Journal of Modern Applied Statistical Methods Volume 10|Issue 1 Article 10 5-1-2011 Sample Size Considerations for Multiple Comparison Procedures in ANOVA Gordon P. Brooks Ohio University, [email protected] George A. Johanson Ohio University, [email protected] Follow this and additional works at:http://digitalcommons.wayne.edu/jmasm Part of theApplied Statistics Commons,Social and Behavioral Sciences Commons, and the Statistical Theory Commons Recommended Citation Brooks, Gordon P. and Johanson, George A. (2011) "Sample Size Considerations for Multiple Comparison Procedures in ANOVA," Journal of Modern Applied Statistical Methods: Vol. 10 : Iss. 1 , Article 10. DOI: 10.22237/jmasm/1304222940 Available at:http://digitalcommons.wayne.edu/jmasm/vol10/iss1/10 This Regular Article is brought to you for free and open access by the Open Access Journals at DigitalCommons@WayneState. It has been accepted for inclusion in Journal of Modern Applied Statistical Methods by an authorized editor of DigitalCommons@WayneState. Journal of Modern Applied Statistical Methods Copyright © 2011 JMASM, Inc. May 2011, Vol. 10, No. 1, 97-109 1538 – 9472/11/$95.00 Sample Size Considerations for Multiple Comparison Procedures in ANOVA Gordon P. Brooks George A. Johanson Ohio University, Athens, Ohio USA Adequate sample sizes for omnibus ANOVA tests do not necessarily provide sufficient statistical power for post hoc multiple comparisons typically performed following a significant omnibus F test. Results reported support a comparison-of-most-interest approach for sample size determination in ANOVA based on effect sizes for multiple comparisons. Key words: Sample size, multiple comparison procedures, Tukey, ANOVA. Introduction particular statistical analysis that will be used to The determination of an appropriate sample size analyze data. For example, when a t test is used, is an often difficult, but critically important, the researcher commonly estimates an expected, element in the research design process. One of standardized group mean difference effect size the chief functions of experimental design is to (such as Cohen’s d) in order to determine an ensure that a study has adequate statistical power appropriate sample size. Sample sizes in analysis to detect meaningful differences, if indeed they of variance (ANOVA) are often based on an exist (e.g., Hopkins & Hopkins, 1979). There is effect size that represents an overall a very good reason why researchers should standardized difference in the means (such as worry about statistical power a priori: If Cohen’s f), but these recommended sample sizes researchers are going to invest time and money provide statistical power only for the omnibus in carrying out a study, then they would want to null hypothesis (overall ANOVA) that no group have a reasonable chance, perhaps 70% or 80%, means differ. Adequate sample size for the to find a statistically significant difference omnibus test does not necessarily provide between groups if it does exist in the population. sufficient statistical power for the post hoc Thus, a priori power, the probability of rejecting multiple comparisons typically performed a null hypothesis that is indeed false, will inform following a statistically significant (exploratory) researchers about how many subjects per group omnibus test and in many cases the multiple will be needed for adequate power (Light, comparisons are of most interest to a researcher. Singer & Willett, 1990). The purpose of this study was to Among the most important matters determine whether the knowledge that multiple impacting the choice of sample size is the comparison procedures will be used following a statistically significant omnibus ANOVA can be helpful in choosing a sample size for a given study. In particular, results using the Tukey HSD Gordon P. Brooks is an Associate Professor of post hoc multiple comparison procedure (MCP) Educational Research and Evaluation. His were examined to determine whether specific research interests include statistics education, recommendations can be made about sample power and sample size analysis and Monte Carlo sizes when the Tukey MCP is used and three programming. Email him at: [email protected]. groups are compared. This evidence was used to George A. Johanson is a Professor Emeritus of reach conclusions about whether such an Educational Research and Evaluation. His approach to sample size selection has merit. research interests include survey research Note that this is a presentation of a new methods and differential item and person approach to sample size selection – specifically, functioning. Email him at: [email protected]. a new way to think about effect sizes – for 97 MCP SAMPLE SIZES exploratory ANOVA where post hoc method (and other similar methods) concentrates comparisons are relevant. Other approaches are on the statistical power of the omnibus test in both more appropriate and more powerful when ANOVA. Others, Hinkle, Wiersma and Jurs planned comparisons are made in a confirmatory (2003) and Levin (1975), for example, have analysis. recommended approaches based on how large the sample must be to detect a predetermined Theoretical Framework mean difference effect size between any two Several factors play a role in sample size groups, or two extreme groups. Although determination, including that after the statistical Levin’s approach is designed for use with the method and the directionality of the statistical Scheffé multiple comparison procedure, Hinkle, alternative hypotheses have been decided, et al. base their method on Cohen’s d effect size sample size, level of significance, effect size and for comparison between the two groups with the statistical power are all functionally related. largest (most extreme) mean differences, and Other issues also impact statistical power, such therefore do not consider the adjustments to as the reliability of measurements, unequal alpha for multiple comparison procedures. Pan group sizes and unequal group variances. and Dayton (2005) provided sample size However, little consideration has been given to requirements for patterns of ordered means, but the role of post hoc multiple comparison tests in focused on an information criteria approach to choosing adequate sample sizes. pair-wise comparison procedures. In order to maintain reasonable experiment-wise Type I error rates when group Comparison-of-Most-Interest means are compared, researchers often use When determining sample sizes for a ANOVA followed by an appropriate MCP. The factorial ANOVA, researchers may choose the overall ANOVA is tested using an omnibus test sample size that provides sufficient statistical at a predetermined level of significance (e.g., power for all sources of variation (e.g., main 0.05). The post hoc tests that follow a effects and interactions). Alternatively, statistically significant omnibus test are then researchers may determine which effect is most often performed at an adjusted level of important to them and select a sample size based significance, based on the number of on the expected effect size for that particular comparisons to be made. source of variation. For example, researchers For example, when comparing four may have most interest in the interaction effect groups, six pairwise group mean comparisons or a particular main effect. Depending on the possible. If the researcher wishes to perform all structure of the cell means, these effect sizes can six pairwise comparisons, the per comparison vary and therefore result in different required (i.e., per test) level of significance would be sample sizes for the various main effects and adjusted so that the entire set of follow-up tests interaction effects. does not exceed the experiment-wise alpha (e.g., The approach presented in this study is if experiment-wise alpha is 0.05, the adjusted based loosely on this effect-of-most-interest per comparison alpha might be 0.05/6 = 0.0083, approach from factorial ANOVA as applied to using a Bonferroni approach). Each MCP one-way ANOVA: That is, beyond determining performs this adjustment differently, resulting in the sample size required for an omnibus test in different performance for each in terms of Type one-way ANOVA, the new approach also I error and statistical power (e.g., Carmer & determines the sample sizes required for the Swanson, 1973; Einot & Gabriel, 1975; follow-up tests from a given set of population Toothaker, 1991). means. Several methods exist for determining For example, in a 3-group study the sample size for ANOVA. Most common are researcher may be able to estimate that a large statistical power approaches based on Cohen’s effect exists between a control group and two (1988) f effect size, which represents the types of treatment, but may expect a much standardized variability of the group means smaller difference between two types of about the grand mean (Stevens, 2007). This treatment. The comparison-of-most-interest may 98 BROOKS & JOHANSON be the difference between the treatments and the of 100,000 samples resulted in statistically control; however, the much smaller difference significant omnibus F statistics for the ANOVA between the two treatments may be the most among the three groups. However, the number interesting. The researcher would use this of correct statistically significant Tukey HSD information to determine an appropriate sample comparisons between groups 1 and 3 and size for the study by selecting a sample size between groups 2 and 3 (with a sample size of large enough for the smaller effect size between 24 in each group), was approximately 64.7%. At the types of treatment. This differs from an a the adjusted alpha used by the Tukey HSD priori set of planned comparisons in that the procedure, approximately 1.9% of the researcher may have a special interest in comparisons between groups 1 and 2 were particular comparisons, but not have specific statistically significant (and therefore Type I alternative research hypotheses to predict the errors because both group 1 and 2 had the same direction of the mean differences. The procedure mean). studied here is an adaptation of the Hinkle, et al. These illustrative power analysis results (2003) approach that looks at meaningful effect imply that a number of samples from among the sizes between any groups rather than the Hinkle, 100,000 had statistically significant omnibus F et al. difference between only the two most statistics while, at most, one of the non-null extreme groups. Tukey post hoc comparisons was statistically Even in an exploratory ANOVA, it is significant. The MC4G program reported that rarely satisfactory knowing only that a approximately 78.9% of samples had at least one difference exists in the means (as given by the significant Tukey comparison following a omnibus test); researchers typically also want to significant omnibus test. However, because only know between which groups the differences 64.7% of each non-null comparison were exist. Without consideration of the multiple statistically significant, and because the group 1 comparison procedures during the sample size versus group 2 comparison was significant as a analysis, it is possible to find a statistically Type I error in about 1.9% of the samples, this significant omnibus test with no pairwise group implies that - in many of those samples - only differences determined to be statistically one of the two large, non-null comparisons was significant in post hoc tests. Although other statistically significant. potential reasons for such a result exist, it may From another perspective, in order to sometimes be an issue of statistical power. reach statistical power of 0.80 for the two non- null Tukey comparisons (i.e., group 1 vs. group An Example of the Problem 3 and group 2 vs. group 3), 32 cases are needed Suppose a researcher is analyzing the per group, for a total sample size of 96 mean differences for three groups, where the (compared to 24 per group based solely on the means for groups 1 and 2 are both 0.0, but the omnibus test). With a total sample size of 96 the third group mean is 0.8. This represents a omnibus F test, however, had a power rate of relatively large pairwise difference between approximately 0.91. group 3 and both groups 1 and 2. Using the Cohen (1988) effect size, f, for ANOVA, this Methodology might be characterized as a relatively large An existing Monte Carlo program was modified effect: Cohen’s large effect size is f = 0.40 and so that it can ascertain appropriate sample sizes in this example f = 0.38. Cohen’s sample size for pairwise comparisons calculated using the analysis, as implemented by the SPSS Tukey multiple comparison procedure. The SamplePower program, indicates that 24 cases MC4G: Monte Carlo Analyses for up to 4 per group are required to achieve statistical Groups program was originally developed by power of 0.80 for the omnibus test in such a one of the authors to perform Monte Carlo situation. analyses for t tests and ANOVA in a Windows When performing a Monte Carlo environment (Brooks, 2008). The current analysis for this condition using the MC4G version of the program (MC4G version v2008) program (Brooks, 2008), approximately 80.8% 99 MCP SAMPLE SIZES was upgraded to include the sample size For example, whether the three group analyses required for this study. means were set at 0.2, 0.4 and 0.6 or at 0.3, 0.5 The MC4G program was compiled in and 0.7, the pattern for both resulting Delphi 2007. The program uses the L’Ecuyer standardized mean difference effect sizes (all (1988) uniform pseudorandom number standard deviations were 1.0) would be 0.2, 0.2 generator. Specifically, the FORTRAN code of and 0.4, respectively. The mean differences - as Press, et al. (1992), was translated into Delphi effect sizes - are the key to the sample size Pascal. The L’Ecuyer generator was chosen due analyses, not the absolute sizes of the means. to its large period and because combined Therefore, each pattern of mean differences was generators are recommended for use with the only included once. The result was 16 non- Box-Muller method for generating random redundant comparison patterns that fit the mean normal deviates (Park & Miller, 1988), as is the difference conditions described (see Table 1). case in MC4G. The computer algorithm for the Box-Muller method used in MC4G was adapted Results for Delphi Pascal from the standard Pascal code Three primary findings of interest were observed provided by Press, et al. (1989). Simulated from this study. First, when the pattern of means samples were chosen randomly to test program resulted in a pattern where two of the three function by comparison with results provided by means are equal – and different from the third – SPSS. there was a consistent pattern of sample sizes required for the comparison relative to the Monte Carlo Design sample size required for the omnibus test. In all simulations, normally distributed Second, when the pattern of means resulted in standardized data were generated to fit the given two of the three mean differences being equal – conditions for each simulation; that is, all and different from the third – there was a variances were set to 1.0, while group means consistent pattern of sample sizes required for varied between 0.0 and 0.8, depending on the the comparison relative to the sample size given effect size. A minimum of 10,000 required for the omnibus test. Third, no matter replications were performed for the final sample what the pattern of means, a given absolute size analysis in each condition. Specifically, a standardized mean difference effect size default value of 20,000 was used with the consistently required the same sample size to MC4G sample size analysis, which guaranteed achieve the power desired. that the final results would be based on at least 10,000 iterations (i.e., simulated samples). Two Equal Means Samples sizes for all three groups were restricted In situations where two groups had the to be equal. Some of the Monte Carlo same mean and a third group mean differed, the simulations were run multiple times with non-null multiple comparisons required larger different seeds to verify that the results were not sample sizes than the omnibus ANOVA. For an artifact of a poor seed choice. example, the condition where the pattern of Conditions included varying standardized means was 0.0, 0.0 and 0.5 standardized mean differences among groups for (therefore a pattern of mean differences of 0.0, a three-group ANOVA. In particular, groups 0.5 and 0.5) resulted in per group sample sizes varied such that all possible non-redundant of roughly 81 cases to achieve power of 0.80 for patterns of pairwise mean differences were the two multiple comparisons with a varied across groups from 0.0 to 0.8. The standardized mean difference of 0.5 (see Table minimum non-null standardized mean difference 2). This was compared to the 60 cases per group between groups of 0.2 was chosen because of needed to achieve statistical power of 0.80 for the very large sample sizes required for smaller the omnibus test. effects; the maximum of 0.8 was chosen because All patterns with two similar means, of the very small sample sizes required when the regardless of the magnitude of the mean mean differences are larger. differences, resulted in a relative efficiency of sample sizes (omnibus per group sample size 100 BROOKS & JOHANSON Table 1: Patterns of Means Studied Group 1 Group 2 Group 3 Comparison Cohen f Cohen Cohen A nalysis Mean Mean Mean Patterna Effect Size Total N N Per Group 1 0.0 0.0 0.2 0.0, 0.2, 0.2 0.0943 1089 363 2 0.0 0.0 0.3 0.0, 0.3, 0.3 0.1414 486 162 3 0.0 0.0 0.4 0.0, 0.4, 0.4 0.1886 276 92 4 0.0 0.0 0.5 0.0, 0.5, 0.5 0.2357 177 59 5 0.0 0.0 0.6 0.0, 0.6, 0.6 0.2828 126 42 6 0.0 0.0 0.7 0.0, 0.7, 0.7 0.3300 93 31 7 0.0 0.0 0.8 0.0, 0.8, 0.8 0.3771 72 24 8 0.0 0.2 0.4 0.2, 0.2, 0.4 0.1633 366 122 9 0.0 0.2 0.5 0.2, 0.3, 0.5 0.2055 234 78 10 0.0 0.2 0.6 0.2, 0.4, 0.6 0.2494 159 53 11 0.0 0.2 0.7 0.2, 0.5, 0.7 0.2944 117 39 12 0.0 0.2 0.8 0.2, 0.6, 0.8 0.3399 87 29 13 0.0 0.3 0.6 0.3, 0.3, 0.6 0.2449 165 55 14 0.0 0.3 0.7 0.3, 0.4, 0.7 0.2867 123 41 15 0.0 0.3 0.8 0.3, 0.5, 0.8 0.3300 93 31 16 0.0 0.4 0.8 0.4, 0.4, 0.8 0.3266 96 32 aC omparison pattern indicates the standardized mean differ ence between Group 1 vs. Group 2, Group 2 vs. G roup 3, and Group 1 vs. Group 3, respectively divided by multiple comparison per group sample size as the omnibus test. For example, in sample size) of approximately 0.70. Stated the case where the pattern of means was 0.0, 0.3 another way, in all cases where two groups had and 0.6 (therefore a pattern of mean differences the same mean while a third group differed, the of 0.3, 0.3 and 0.6, respectively), the smaller multiple comparisons required approximately mean comparisons required approximately 228 1.4 times more cases than the omnibus test did cases per group, while the third mean in order to achieve power of 0.80. For example, comparison required 57 cases per group. These in the condition where the pattern of means was values were compared to the omnibus test 0.0, 0.0 and 0.5, the multiple comparisons sample size of 55 cases per group for a power required 1.35 times more cases than did the rate of 0.80. overall test. For 0.0, 0.0 and 0.8, the multiple Like the two similar means pattern comparisons resulted in 1.38 times more cases. described above, the relative efficiencies of the Complete relative efficiency results from the two similar mean differences pattern were studied conditions can be reviewed in Table 2. consistent across results. In all cases where two mean differences were the same, the multiple Two Equal Mean Differences comparison tests required approximately 4.2 In conditions where two of the three times more cases than the omnibus test. For the mean differences were the same and the third third, different comparison, approximately 1.1 mean difference was twice as large, the two times more cases were needed. For example, in smaller mean comparisons required a much the 0.0, 0.4, 0.8 condition, the two equal larger sample size than the overall test, while the multiple comparison tests (i.e., group 1 vs. group third comparison required roughly the same 2 and group 2 vs. group 3) required 101 MCP SAMPLE SIZES Table 2: Sample Size Results for the Tukey H SD Multiple Comparison Procedure for the Primary Monte Carlo Design at Statistical Power of 0.80 Group 1 Group 2 Group 3 Comparison Total Sample Sample Size per Relative Mean Mean Mean Tested Size Group Efficiencya Omnibus 1080 360 G1 v G2 * * 0 0 0.2 G2 v G3 1521 507 1.41 G3 v G1 1524 508 1.41 Omnibus 483 161 G1 v G2 * * 0 0 0.3 G2 v G3 678 226 1.40 G3 v G1 681 227 1.41 Omnibus 276 92 G1 v G2 * * 0 0 0.4 G2 v G3 375 125 1.36 G3 v G1 381 127 1.38 Omnibus 180 60 G1 v G2 * * 0 0 0.5 G2 v G3 243 81 1.35 G3 v G1 246 82 1.37 Omnibus 123 41 G1 v G2 * * 0 0 0.6 G2 v G3 171 57 1.39 G3 v G1 174 58 1.41 Omnibus 93 31 G1 v G2 * * 0 0 0.7 G2 v G3 126 42 1.35 G3 v G1 126 42 1.35 Omnibus 72 24 G1 v G2 * * 0 0 0.8 G2 v G3 99 33 1.38 G3 v G1 99 33 1.38 Notes: * indicates that the Null Hypothesis was true for th e given comparison, thus no sample size analysis was performed; aRelative efficiency is calculated as the total sample size for the particular comparison divided by the total sample size for the omni bus test for the condition 102 BROOKS & JOHANSON Table 2 (continued): Sample Size Results for the Tukey HSD Multiple Comparison Procedure for the Primary Monte Carlo Design at Statistical Power of 0.80 Group 1 Group 2 Group 3 Comparison Total Sample Sample Size per Relative Mean Mean Mean Tested Size Group Efficiencya Omnibus 366 122 G1 v G2 1524 508 4.16 0 0.2 0.4 G2 v G3 1527 509 4.17 G3 v G1 378 126 1.03 Omnibus 231 77 G1 v G2 1524 508 6.60 0 0.2 0.5 G2 v G3 690 230 2.99 G3 v G1 246 82 1.06 Omnibus 156 52 G1 v G2 1527 509 9.79 0 0.2 0.6 G2 v G3 384 128 2.46 G3 v G1 171 57 1.10 Omnibus 114 38 G1 v G2 1515 505 13.29 0 0.2 0.7 G2 v G3 246 82 2.16 G3 v G1 126 42 1.11 Omnibus 87 29 G1 v G2 1527 509 17.55 0 0.2 0.8 G2 v G3 171 57 1.97 G3 v G1 99 33 1.14 Omnibus 165 55 G1 v G2 684 228 4.15 0 0.3 0.6 G2 v G3 684 228 4.15 G3 v G1 171 57 1.04 Omnibus 120 40 G1 v G2 675 225 5.63 0 0.3 0.7 G2 v G3 384 128 3.20 G3 v G1 126 42 1.05 Notes: * indicates that the Null Hypothesis was true for the given comparison, thus no sample size analysis was performed; aRelative efficiency is calculated as the total sample size for the particular comparison divided by the total sample size for the omnibus test for the condition 103 MCP SAMPLE SIZES Table 2 (continued): Sample Size Results for the Tukey HSD Multiple Comparison Procedure for the Primary Monte Carlo Design at Statistical Power of 0.80 Group 1 Group 2 Group 3 Comparison Total Sample Sample Size per Relative Mean Mean Mean Tested Size Group Efficiencya Omnibus 93 31 G1 v G2 678 226 7.29 0 0.3 0.8 G2 v G3 246 82 2.65 G3 v G1 99 33 1.06 Omnibus 93 31 G1 v G2 381 127 4.10 0 0.4 0.8 G2 v G3 378 126 4.06 G3 v G1 99 33 1.06 Notes: * indicates that the Null Hypothesis was true for th e given comparison, thus no sample size analysis was performed; aRelative efficiency is calculated as the total sample size for the particular comparison divided by the total sample size for the omni bus test for the condition approximately 4.10 times more cases than the expected to be approximately 0.3, regardless of omnibus test (i.e., 127 vs. 31), while the third the expected effect sizes for the other possible different mean comparison (i.e., group 1 vs. comparisons, they would choose a total sample group 3) required just 33 cases, for a relative size of approximately 681 cases. Alternatively, efficiency of 1.06. Very much the same results if there are multiple comparisons-of-interest, occurred for the (0.0, 0.2, and 0.4) and (0.0, 0.3, then researchers in this example would choose and 0.6) conditions of two similar mean 0.3 as the smallest among the set of most differences (see Table 2). interesting comparisons and therefore choose sample sizes based on that smallest comparison- Absolute Mean Difference Effect Sizes of-interest. There were also consistent required sample sizes for absolute standardized group Conclusion mean difference effect sizes regardless of the Perhaps even more important than the sample pattern of means, that is, regardless of the size tables produced for this study is the notion pattern of means across the three groups, the that when a researcher is considering sample same sample size was required for any given size, it may not be sufficient to set sample size absolute mean difference (see Table 3). For for the omnibus test being performed. Clearly, example, when examining the specific results for researchers should consider post hoc multiple a comparison-of-most-interest absolute comparisons in the same way they consider standardized mean difference of 0.3, no matter different sources of effects in factorial ANOVA: whether the pattern of means was (0.0, 0.0, 0.3) that is, the most important effects under study or (0.0, 0.3, 0.6) or (0.0, 0.3, 0.8), results must be considered a priori so that adequate indicated that a total sample size of sample sizes may be obtained for the tests of approximately 681 cases (227 per group) was those effects. With group comparison required to achieve a statistical power rate of procedures such as ANOVA, these comparisons- 0.80 for the comparison with a standardized of-most-interest are very frequently performed mean difference effect size of 0.3. Thus, when using post hoc comparison procedures. researchers have a comparison-of-most-interest 104 BROOKS & JOHANSON Table 3: Sample Sizes Required for Statistical P ower of .80 for the Tukey HSD Multiple Comparison Procedure Given Specific Abso lute Standardized Mean Differences (regardless of the pattern of group means) Standardized Mean Total Sample Size Per Group Sample Size Difference Effect Size 0.2 1521 507 0.3 681 227 0.4 381 127 0.5 246 82 0.6 171 57 0.7 126 42 0.8 99 33 These results clearly show that adequate and MANOVA, where the pattern of statistical power for the omnibus ANOVA F test correlations has an important impact on the does not guarantee adequate statistical power for power of the analyses, and therefore also sample given pairwise MCPs performed post hoc. This size determination. Additionally, it is clear that condition may result in overall statistical the absolute size of the given comparison is also significance for the omnibus F test, but no important. Both of these findings could be useful pairwise comparisons showing statistical to researchers as they plan studies that will use significance. Although this will occur at times ANOVA. because the omnibus test is reflecting that a non- pairwise comparison is significant (e.g., one Sample Size Recommendations group compared to an average of two other Based on the results generated, certain groups in an experimental study where one specific recommendations can be made control group is compared to an average of two concerning sample sizes that researchers should experimental treatment groups), it will happen use with ANOVA with three groups. It should sometimes because there is not enough power be remembered that these results were limited to for the adjusted-alpha MCP being performed by Tukey HSD comparisons performed using the researcher. In the end, researchers must statistical power of 0.80. In particular, these determine whether they wish to have sufficient recommendations follow from the three cases power for the overall test or for the often-more- identified in the results. informative post hoc pairwise comparisons. The comparison-of-most-interest approach to sample Case 1: Two Equal Means size selection may be useful for the latter A researcher may be using two control situation. groups and a single treatment group; Results of this study suggest that it may alternatively, the researcher might expect two be inappropriate to select a sample size for treatment groups each to be equally different ANOVA based only on the omnibus test. from the single control group. In such cases, the Clearly the expected pattern among the means researcher should determine the sample size has an impact on the usually important post hoc required for the omnibus ANOVA test and then pairwise multiple comparisons. This may be multiply that sample size by 1.4 to obtain the analogous to situations involving other statistical sample size required for the Tukey comparisons methods, such as principal components analysis between the differing groups. For example, in a 105