Bonferroni
The problem of multiple tests of significance. In many kinds of research, particularly surveys and studies using batteries of tests, and with certain statistical methods (such as multiple regression and analysis of variance), there are frequently many (more than, say, 10) statistical hypotheses being tested (perhaps on different dependent and independent variable combinations). In such studies, we find that setting 
= 0.05 does not provide sufficient protection against the Type I error. What happens is that as the number of separate hypothesis tests increase within a single study, the true
-level for the entire study (regardless of where you think you have set it) will be inflated. An approximate formula for how much
increases as a function of the number of hypotheses you test is
actual -level = 1 - (1- you think you are setting)j
where j is the number of significance tests being done or contemplated. Thus, if we think we are setting
at 0.05 in our study, but we are contemplating testing 20 statistical hypotheses (say 20 t-tests), our actual chance of claiming a significant result when there should not be one is 1 - (1 - 0.05)20 = 0.64. That is, we really have a 64% chance of making a Type I error. This is quite a substantial chance!! Moreover, you do not know which ones are errors and which ones arenÕt.
There are a couple of ways of coping with this problem in multivariate studies. The best way is to use special multivariate statistical procedures that are designed to help protect against this
inflation problem either by conducting a single test on all dependent variables simultaneously or by providing ways to statistically reduce the number of variables being dealt with.
Another approach is attributed to a gentleman named Bonferroni. He devised a simple way to help control for the
-inflation problem. To have a study with an overall
-level equal to some pre-specified level (say 0.05), merely divide this up equally among the j tests of significance being contemplated. Thus, for each significance test, we use
new -level = 
In the previous example, with 20 significance tests, we would consider using a new
-level of 0.05/20 = 0.0025 for each separate significance test. In this way, we gain better protection against
-level inflation due to doing multiple tests of significance. We can check this out by computing the inflated
formula using the new
-level: 1 - (1 - 0.0025)20 = 0.0488 which is very close to an overall
of 0.05. WeÕll refer to this adjustment of the
-level as the Bonferroni adjustment procedure. Of course, we are now in danger of making a Type II error ÐÐ by finding no significant differences when in fact they should be there!. In the present case, a p-value less than 0.0025 might be hard to obtain and hence we might overlook an important finding.
|
