Reflections on a major new evaluation of community-based mentoring

Mentoring experts Carla Herrera, David DuBois, Janet Heubach, and Jean Grossman have just published a major new randomized controlled trial of the effects Big Brothers Big Sisters of America (BBBSA) Community-Based Mentoring (CBM) Program on the social-emotional, behavioral, and academic outcomes of youth. The study included over 700 9- to 14-year-olds who were enrolled in two West coast BBBSA agencies and followed over a 13-month period. The study serves as an important next step following the landmark evaluation of BBBSA CBM, which was conducted in the mid-1990s and helped to launch today’s mentoring movement in the U.S. (Tierney et al., 1995). As in the earlier study, each participating youth was assigned to be immediately eligible for a Big or put on a waitlist (in this case 13 months). Similar to the original evaluation, the study assessed a wide range of youth-reported outcomes. In addition, it included several parent-reported outcomes and created composite indices which reflected the average of youth- and parent-reported outcome measures.

Results: The evaluation found “significant differences between treatment and control groups on only one of the eight youth-reported outcome measures: depressive symptoms [though a small effect (d = .146)]. For the remaining youth-reported outcome measures, estimated effects were in directions favoring the treatment group for all but two measures: misconduct and skipping school.” For parent-reported outcomes, analyses revealed statistically significant differences favoring the treatment group on four of the six tested measures: emotional symptoms, peer problems, conduct problems, and the SDQ Total Difficulties Score. The study also found significant differences favoring the treatment group on the parent-report and combined youth- and parent-report composite indices. Overall, effect sizes were small, ranging from .138 to .253.

As the authors note that “the current study found few significant impacts in the outcome domains it has in common with the earlier P/PV study of BBBSA CBM (Tierney et al., 1995).” They note several factors that might account for these differences.

1. Shorter matches: By the 13-month follow-up, “only about half of the treatment youth had received at least 12 months of mentoring, and the average match length was less than 10 months, whereas the follow-up period for the original P/PV study (Tierney et al., 1995) was 18 months, at which point, matched youth had met with their mentors an average of 12 months (Grossman & Rhodes, 2002).
2. Smaller sample: The P/PV trial was based on a larger sample and thus was in a better position to detect small effects on program outcomes.
3. Lower dosage: While youth in the treatment group reported fairly strong relationships with their mentors, only 15 percent met with their mentor 3 or more times in most of the months they were matched. In contrast, over 70 percent of youth in the original P/PV study met with their mentor at least three times a month, and 45% met at least once a week (Tierney et al., 1995).
4. Implementation issues:

1. 1. Training: Program practices varied across the matches involved in this study. This was especially true for initial “pre-match” mentor training (beyond basic program orientation), which was received by only about one in three mentors.
   2. Monthly support: Implementation of monthly support contacts was also inconsistent, with support contacts not taking place in at least one-third of the months that matches were active for mentors, youth, or parents.

A few reflections:

This was an extremely impressive study. RCT’s are the gold standard, the instruments were well validated, and the discussion section offers a master class in nuanced interpretation. See article. 

One interesting idea that they raise is that small average benefits “may reflect the non-targeted nature of the BBBSA CBM program, which focuses broadly on relationship development, rather than on any particular outcome. A recent meta-analysis by Christensen et al. (2020), in fact, found that targeted mentoring programs yielded substantially larger effect sizes on the outcomes they targeted than did non-targeted programs (typically tested on a broader set of outcomes). Perhaps one of the strengths of non-targeted programs like BBBSA CBM is their ability to yield “small” benefits, but across a wide range of outcomes, whereas targeted programs may yield larger benefits, but only in the specific areas targeted by the programs.”

I would argue, however, that small effects across a broad range of positive outcomes can be scaled and achieved more economically through after-school and other PYD programs (4-H, Boys & Girls Clubs). For example, a recent meta-analysis of the effects of after-school programs among youth with marginalized identities (Christensen et al., 2022), found larger youth-reported effects overall than CBM (g = 0.205  vs. d = .115, respectively). Since only about 5% of youth will ever be assigned a formal mentor, and many youth face long waitlists, an argument could be made for expanding access to/concentration of adults in PYD and after school programs and deploying formal one-on-one mentors to help youth more in need of targeted care and support. After school programs receive more federal funding and most mentored youth are already involved in PYD programs (Jarjoura et al., 2018). Thus, if the goal is to create more youth-adult bonds and engage in PYD activities that modestly effect a range of outcomes, the most efficient structure would involve practice and policy changes that lead to more caring adults in ASPs to improve youth-staff ratios, staff burnout and turnover, and ideally, support longer tenure of staff through better pay.

Another interesting finding that the authors note was that there were many more significant parent-reported (vs. child-reported) impacts. The authors note that this pattern of findings also emerged in a Friends of the Children evaluation (which found only parent effects) and highlight the value of collecting additional sources of data when assessing program effects. Whatever the reason, it is interesting and encouraging that, across two rigorous studies, parents are seeing value in their children’s enrollment in mentoring programs.