Like compulsive gold miners: Reflecting on five (and counting) new meta-analyses

/0 Comments/in , /by

By Jean Rhodes

My colleagues and I began conducting meta-analyses of youth mentoring programs around five years ago and we can’t seem to stop. Like compulsive gold miners, we find ourselves returning to this method, again and again, hoping to uncover yet another helpful nugget. A meta-analysis combines the results of multiple evaluations to determine the overall effectiveness of an intervention strategy  (e.g., mentoring programs) and to estimate and compare effects across different approaches or populations. Meta-analysis is not a perfect barometer–effects are often calculated from very different types of programs and the ability to make comparisons depends on the information provided in the original evaluations. Nonetheless, this approach can shed important light on practices and approaches that might be most helpful to the field. In fact, some methodologists place meta-analysis at the top of the evidence pyramid. Given their potential contribution, my colleagues and I have been on a bit of a meta-analysis kick these past few years.

Overall effects

It all began with a comprehensive meta-analysis, led by Professor Liz Raposa. This study encompassed 70 intergenerational, one-on-one mentoring program evaluations that were conducted from 1975 through 2017, and represented more than 25,000 youth. We found that mentoring programs were associated with a range of significant, but, overall, small effects. In fact, the overall effect (g = 0.21) continued a modest trend line of meta-analyses in this field that has not markedly improved in nearly 20 years.  Clearly, there was more work to be done.

As a follow-up, we drew on the Raposa et al. data set to explore programs that took a non-specific, friendship-based approach versus a targeted, problem-specific approach to mentoring. When the type of program was examined, targeted and more problem-specific programs had an average effect size of 0.25, which was more than double the average effect size of non-specific, friendship-based programs (g = 0.11) and for some outcomes triple the average effect size.  These findings are in line with recent calls from mentoring researchers for stronger alignment with theoretical and evidentiary standards of prevention science (e.g., Cavell & Elledge, 2015).

More recently, our meta-analysis team has focused on specific subsets of youth and approaches. This includes, most recently, a study of cross-age peer mentoring. Analyses included six studies and revealed a medium-sized overall effect of cross-age peer mentoring programs (g = 0.45). Several program characteristics moderated effect sizes, most notably, the level of adult oversight and supervision.

 

Although these findings should be interpreted with caution given the limited number of programs, the results do highlight the potential benefits of well-supervised cross-age peer mentoring for youth and indicate a need for further research in this area. In fact, the effect size for cross-age peer mentoring in our study is more than double that observed in these past meta-analyses. This difference is notable, given the far greater emphasis on intergenerational mentoring programs compared to cross-age peer mentoring programs in practice and in the research literature.

Next, we conducted a meta-analysis of youth in the foster care system, estimating the effect size of nine formal mentoring programs in the United States serving youth involved with the foster care system (total n = 55,561).  Our results revealed a small-to-medium overall effect of mentoring programs for youth involved with the foster care system (g = 0.342) with no differences in mentoring impact across different types of youth outcomes. These are stronger overall effects than those revealed in meta-analytic studies of general youth mentoring programs (i.e., Raposa et al., 2019; DuBois et al., 2002). The study results also indicated that near-peer mentors are more effective than intergenerational mentors in formal mentoring programs for youth involved with the foster care system. Moderation analyses of program characteristics revealed that near-peer mentors were more than twice as effective as intergenerational mentors (b = 0.359, p = 0.024). Near-peer mentors refer to mentors who are close in age with mentees; these mentors may also have shared experiences as youth currently or formerly involved in the foster care system. It seems that near-peer mentors may act as credible messengers since their guidance is often grounded in shared experiences.

Finally, we conducted a meta-analysis of youth-initiated mentoring (YIM), a hybrid approach in which youth and their families are helped to identify and recruit caring adult mentors from within their existing social networks and to maintain such relationships, which is a promising strategy for addressing these problems and expanding the reach of youth mentoring. The study revealed that, across a range of outcomes, overall effects were slightly higher (g = 0.30) than those achieved in formal mentoring.

Conclusions

Overall, our studies have produced a range of effects–with traditional, intergenerational, nonspecific mentoring programs leaving the most room for improvement.

After completing the first major meta-analysis of youth mentoring,  DuBois et al. observed that the reality of mentoring programs was “not necessarily consistent with the manner in which results of the large-scale evaluation frequently have been cited by the media as demonstrating a large impact for mentoring relationships.” Nearly 20 years later, the findings from the 2019 meta-analysis led Raposa et al. to call for “more rigorous adherence to evidence-based practices that target specific mechanisms underlying particular youth difficulties, rather than relying on a relatively low-intensity, nonspecific approach with uneven adherence to practices that are research-informed.”

Together the subsequent meta-analyses have reinforced these recommendations, suggesting that stronger mentoring effects may result from programs that:

  • include well-supervised, cross-age peer mentoring
  • address the needs of specific populations(e.g., youth in foster care)
  • provide targeted approaches to mentoring programs.

We are continuing our quest! (expect a meta-analysis on group mentoring from us soon!). Over time, each a new set of findings fits more logically into the broader mentoring landscape and suggests new solutions to the field.