Multilevel Modeling

17. Power and Sample Size

As we have emphasized, multilevel models are not about modeling each neighborhood separately; rather, the sample of neighborhoods is seen as one realization from a population of neighborhoods.

When designing a powerful multilevel study it is vital, therefore, to consider the importance of two things:

  1. Determining sample sizes at the various levels of analysis; and
  2. Ensuring the property of exchangeability.

We first discuss the issue of sampling in multilevel analysis.

It is vital that the study design has ‘adequate’ number of units at all the levels of analysis. Specifically, by increasing sample sizes at all levels, estimates and their standard errors become more accurate. The analysis of binomial data in particular requires larger samples than the analysis of normally distributed data (Hox, 2002). Determination of sample sizes at level-1 and level-2 units for efficiency, unbiasedness and consistency of parameter estimates is not entirely straight-forward and this is especially the case if we are interested in the random slopes component.

In a two-level random intercepts model, the sample design question is analogous to computing the effective sample size in two-stage cluster sampling, as given by (Kish, 1965). Effective sample size of a two-stage cluster sampling design, neff, is computed by: neff=n/[1+(nclus –1)ρ], where n is the total number of individuals in the study, that is, the actual sample size; nclus is the number of individuals per neighborhood; and ρ is the intra-class correlation. However, the analogy is not straightforward for random slopes models, because the ICC for these models is a function of the independent variable.

Consensus has yet to be developed on the precise power calculations within multilevel models. Some argue for a sample of at least 30 groups with at least 30 individuals in each group (Kreft, 1996). This advice is considered sound provided the interest is largely in the fixed parameters. Modification to this ‘rule’ is advised if interest is in estimating cross-level interactions and/or variance and covariance components (Hox, 2002). For the former, a 50/20 ‘rule’ is recommended (about 50 neighborhoods with at least 20 individuals per neighborhood) and for a variance-covariance components model about 100 neighborhoods with about 10 individuals per neighborhood is suggested.

Indeed, if this is the case then one has to be cautious about making neighborhood-specific predictions. These ‘rules’ take into account that there are costs attached to data collection, such that if the number of neighborhoods is increased, the number of individuals per neighborhood decreases (Snijders and Bosker, 1993; Snijders, 2001).

Hox, J. (2002) Multilevel analysis: Techniques and applications. Mahwah, NJ: Lawrence Erlbaum Associates.
Kish, L. (1965) Survey sampling. New York: John Wiley.
Kreft, I. G. G. (1996) Are multilevel techniques necessary? An overview including simulation studies. Los Angeles: Calfornia State University.
Snijders, T. A. B. (2001) Sampling. In: A. H. Leyland and H. Goldstein (Eds.). Multilevel modelling of health statistics. Chichester: John Wiley. 159-174.
Snijders, T. A. B., Bosker, R. J. (1993) Standard errors and sample sizes for two-level research. Journal of Educational Statistics 18: 237-259.
Snijders, T. A. B. (2001) Sampling. In: A. H. Leyland and H. Goldstein (Eds.). Multilevel modelling of health statistics. Chichester: John Wiley. 159-174.