New meta-analysis of after-school programs sheds light on moderators that affect reported youth outcomes

Christensen, K. M., Kremer, K. P., Poon, C. Y., & Rhodes, J. E. (2023). A meta‐analysis of the effects of after‐school programmes among youth with marginalized identities. Journal of Community & Applied Social Psychology.

Summarized by Ellen Parry Luff

Notes of Interest

• There is a growing consensus that after-school programs (ASPs) are a positive resource for youth with marginalized identities.
  • ASPs provide structured time for youth to develop relationships and strengthen skills, such as improving academic performance.
• However, despite evidence that demonstrates the benefits of ASPs, research on ASPs and how they impact underrepresented youth is very limited.
  • Many papers on this topic have mixed results about program effectiveness and don’t critically examine how program quality and differential resources affect outcomes.
• The current meta-analysis study looked at articles from 2014 onward to better understand the general impact ASPs have on underrepresented youths’ social functioning, mental health, behavioral, self-perception/identity, and school-related outcomes.
• Results found a small, statistically significant effect of ASPs, meaning there was evidence that they had a small but positive effect on youth.
• However, the findings also indicated that ASPs might not be effective in improving targeted youth outcomes but have more promise for overall general improvement.
  • In other words, ASPs can not capture all the nuances of counterfactual conditions.
• Youth characteristics, youth outcomes, program characteristics, implementation fidelity, and training characteristics were not significant moderators.
• However, outcome sources and outcome measures were statistically significant moderators.
  • This finding demonstrates self-report bias, the discrepancy between academic metrics (i.e., GPA and report cards) & youths’ perceived performance, and how researchers tend to develop tests that report bigger effect sizes than standardized tests.
• Overall, there is a need for studies to have greater transparency about information surrounding ASPs if they want to have a more comprehensive understanding of the role of moderators.
• Additionally, ASPs and mentoring programs have similar effects on youth, arguing that ASPs may make for a very scalable way to improve youths’ lives with the help of increased funding.

Introduction (Reprinted from Abstract)

After-school programmes (APSs) often provide youth with a safer alternative to unstructured time while providing a context for building skills and forging positive relationships with programme staff and peers. ASPs may be particularly effective for youth with marginalized identities, including youth of color and youth from low-income backgrounds. Despite this promise, few rigorous evaluations of APSs have been conducted and even fewer meta analyses have investigated the effects of ASPs among youth with marginalized identities. Using a multi-level meta-analysis  of 615 effect sizes across 56 studies (overall n = 128,538), the current study examined the overall effects of ASPs on internalizing, externalizing, school-related, social functioning, and self-perception/identity outcomes among kindergarten through 12th grade youth with marginalized identities. Results indicated ASPs to have a small, yet significant positive overall effect of youth outcomes (g = 0.2049, p = 0.001, 95% CI = 0.08-0.33). Moderator  analyses revealed significant differences in effects based on outcome source and outcome measure type. Given  the ubiquity of ASPs and the challenges that youth experiencing marginalization face, this study uniquely adds to the existing literature and outlines important implications and recommendations for research policy, and practice.

Implications (Reprinted from Discussion)

The goal of the current study was to investigate and synthesize the extant literature on the effects of ASPs on outcomes among youth with marginalized identities. This meta-analysis improves upon the limitations of prior meta- analyses by updating the existing literature and empirically assessing a more comprehensive range of outcomes and moderators with rigorous statistical methods among a specific population of youth.

Multi-level meta-analyses revealed an overall effect size of g = 0.20 across 56 studies, 615 effect sizes, and 128,538 youth participants. This finding represents a significant but small effect according to Cohen’s (1998) stan- dards and is consistent with previous studies of the effects of ASPs for youth of all identities (e.g., Ciocanel et al., 2017; Durlak et al., 2010; Lauer et al., 2006). Simply put, these results suggest that ASPs do have the potential to bring about some positive change across a range of developmental domains including social–emotional/interpersonal skills, mental and behavioural health, school success, and identity development for youth who engage in these types of programmes. Of note, specific youth outcome category did not significantly moderate overall effects. This finding could indicate that ASPs may not be particularly effective at improving any one targeted youth outcome, but have some promise for youths’ overall general improvement. Alternatively, it is possible that primary evaluations of ASPs are unable to accurately capture the full nuance of counterfactual conditions (i.e., youth who are not engaged in ASPs or other community programmes). Often, the ASP under evaluation is assessed in a vacuum, with outside activities and supports of youth in both the intervention and control groups unmeasured. It is possible that youth in both the treatment and control conditions are also engaging with external community programmes and supports that are unaccounted for. Thus, a representative counterfactual is needed to accurately capture the full nuance of conditions where ASPs are not implemented in order to understand their effectiveness in the opposite direction (i.e., not just how many positive outcomes were or were not achieved, but how many negative outcomes were avoided). Related to this, it may also be useful for future research to further assess and compare ASPs operating at different levels of the prevention continuum. For example, future studies may seek to investigate whether ASPs functioning as universal prevention programmes are effective in delaying or reducing negative trends in youth functioning and outcomes. Alternatively, if the goal of ASPs is to improve specific outcomes for youth, it will be important for future practice and research to become more comprehensive in their assessment of youth experiences and characteristics and more targeted in their evaluation of programme practices.

To access the article, click here