About us    Campaigns    Research    Support us    Publications    Media Room    Join Us    Contact us

Improving Learning in Primary Schools of Developing Countries: A Meta-Analysis of Randomized Experiments

Learning Achievements, Teacher education and training

 Patrick J. McEwan

Wellesley College, William and Flora Hewlett Foundation

August 2013

There is a vast non-experimental literature on school effectiveness in developing countries (for reviews, see Velez, Schiefelbein, & Valenzuela, 1993; Fuller & Clarke, 1994; Hanushek, 1995; Kremer, 1995; Glewwe, 2002). It uses regression analysis to identify the putative determinants of student learning, but it is hampered in two ways. First, it does not credibly distinguish between the causal effects of schools, and the confounding effects of the children and families that happen to attend those schools. Second, many datasets contain only simple proxies of school quality, such as teacher credentials and pupil-teacher ratios, that do not encompass the complex menu of investment choices available to policy-makers. Experimental research addresses both issues. The use of random assignment of students or schools to school-based treatments improves the internal validity of causal inferences (Glewwe & Kremer, 2006; Duflo, Glennerster, & Kremer, 2008). Moreover, researchers have evaluated a rich variety of policy-relevant treatments that encompass (1) instructional interventions that combine teacher training, textbooks and other materials, class size reduction, and computer assisted instruction; (2) school-based health and nutrition treatments, such as de-worming and micronutrient supplementation; and (3) interventions that modify stakeholder incentives to improve learning, such as school report cards, performance pay, and school-based management. It is a propitious moment to survey this literature, and assess whether there are lessons for policy-makers and researchers. I conducted a literature search in economics, education, and health, identifying 76 published and unpublished experiments that evaluate 110 treatments (see the Appendix). In each case, researchers randomly assigned children, schools, or entire villages to receive a school-based treatment, versus “business-as-usual” in the same setting. I coded effect sizes and their standard errors for outcome variables in language and mathematics. I further coded study attributes that describe the category of treatment, details on the country and experimental sample, outcome measures, and the study quality. Two categories of interventions—monetary grants and school-based deworming—have mean effects that are close to zero and statistically insignificant (based on a random effects model). School-based nutritional treatments, treatments that provide information to parents or students, and treatments that improve school management and supervision tend to have small mean effect sizes—from 0.04 to 0.06 standard deviations—that are not always robust to controls for study moderators. The largest average effect sizes are observed for treatments that incorporate instructional materials (0.08); computers or instructional technology (0.15); teacher training (0.12); smaller classes, smaller learning groups within classes, or ability grouping (0.12); student and teacher performance incentives (0.10); and contract or volunteer teachers (0.10). The categories are not mutually exclusive, however, and meta-regressions that control for treatment heterogeneity and other moderators suggest that the effects of materials and contract teachers, in particular, are partly accounted for by overlapping treatments. For example, instructional materials have few effects on learning in the absence of teacher training (Glewwe et al., 2004, 2009), and contract and volunteer teacher interventions usually overlap with class size reduction and/or instructional treatments (e.g., Banerjee et al., 2007; Bold et al., 2012).

A challenge to the interpretation of learning effects is that some treatments, particularly deworming and school feeding programs, affect enrollment and attainment, despite weaker effects on learning (Miguel & Kremer, 2004; Baird et al., 2012; Petrosino et al., 2013). Even when attainment increases, it is plausible that student time is not being used productively in classrooms. This suggests that interventions with a primary focus on access should be combined—in future research and in practice—with interventions explicitly designed to increase learning. Most papers contain minimal data cost on costs, complicating an assessment of whether specific treatments in the meta-analytic sample—or categories of treatments—are relatively more cost-effective despite smaller effect sizes (or less so despite larger ones). As an alternative, I combine effect sizes with auxiliary cost estimates for 15 treatment arms that are analyzed in Kremer et al. (2013). The results suggest that some interventions are relatively less cost-effective than others, such as computer assisted instruction in India and class size reduction in Kenya. However, the conclusions are tempered by the small samples, the inability to statistically distinguish between ranked cost-effectiveness ratios, and the evidence—cited above—that many treatments affect additional outcomes such as attainment. The review also suggests methodological lessons for the conduct of future experiments. The overwhelming majority of instructional and incentive-based experiments use cluster-randomized assignment of schools (or groups of schools) to a treatment. The statistical power of these experiments is primarily determined by the number of clusters and the intraclass correlation of the outcome variable. The intraclass correlations for test scores are higher in developing-country settings than typical U.S. standards (Zopluoglu, 2012). Further, this paper shows that “typical” effect sizes of some categories of treatments are smaller than commonly assumed. Viewed together, the evidence suggests many smaller cluster-randomized experiments—particularly those with fewer than 50 schools per treatment arm—are under-powered. I further argue that experiments can enhance the potential external validity of their results by (1) adding treatment arms that manipulate key features of the treatment (such as the implementing agency); (2) experimenting within representative samples, and conducting subgroup analysis within policy-relevant and pre-specified subgroups of the full sample; (3) measuring a wider range of learning outcomes and not “cherry-picking” effects across those outcomes; (4) collecting process data that can be used to conduct (non-experimental) tests of the potential causal mechanisms of “black-box” experimental effects; and (5) complementing their findings with high-quality quasi-experimental research that evaluates scaled-up treatments using representative samples of schools. I further suggest that experimental reports, especially in economics, could benefit from a common reporting standard, along the lines of the CONSORT standards widely used in medicine. The next section describes the conceptual framework that organizes the review, including a typology of school-based treaments used in the coding of studies. I then describe data and methods, including the literature search, the criteria for study inclusion and exclusion, the coding of experiments, and the statistical methods used to analyze effect sizes. The results section describes mean effect sizes by categories of treatment, as well as meta-regression models that analyze the correlates of those effects. The final section reviews lessons for policy and future experimental research.

Click here to read more. http://academics.wellesley.edu/Economics/mcewan/PDF/meta.pdf


Leave a Reply

You must be logged in to post a comment.

  Disclaimer: The copyright of the contents of this blog remains with the original author / publisher.