A response to the critique of the Brookings Institute report on mentoring

Screen Shot 2014-06-27 at 6.56.50 PM

by Phillip Levine, Ph.D.

I am grateful for the attention that my analysis has received and I believe there is more common ground here than one might otherwise gauge based on this critique (see below). Much of the critique is focused on the scope and nature of the evidence. My analysis was clear in restricting its attention on the poverty-reducing impact that mentoring programs may generate based on evidence from the experimental evaluations that are best capable of identifying causal relationships. This is in contrast from past meta-analyses that make no such restrictions. No experimental studies of which I am aware have lasted long enough to follow mentored youth far enough into adulthood to directly observe subsequent income. Instead, I rely on educational outcomes like grades and educational attainment that can be more easily translated to subsequent wage gains. If the study did not include these outcomes, I did not consider it. In no way did I seek to imply that these are the only potential benefits of mentoring programs. Even within this narrow window, I found evidence to strongly support the expansion of mentoring programs as a means of improving the economic well-being of disadvantaged youth. I would imagine that this is a conclusion with which Professor Rhodes would agree.


Perhaps the more illuminating discussion focuses on the role of public funding to accomplish this goal. The issue is whether society reaps sufficient benefits to expend public funds in this endeavor. Past interventions that have passed this hurdle include early childhood education, which has become quite popular in policy discussions these days. It does so largely by its estimated impact on crime, which is very expensive to society; reducing it has large benefits. My reading of the evidence on mentoring programs provides potential glimmers of hope that they may reduce crime (through, say, reduced delinquency), but no strong link exists. Professor Rhodes raises the interesting point that the broader society may benefit if mentoring programs reduce depression among disadvantaged youth. Generating estimates of those social benefits is an avenue that I did not explore and I believe that this may be a fruitful approach to pursue. Perhaps there is sufficient research enabling us to make that link more strongly. Regardless, my original analysis also made clear that public policy may undertake interventions that largely benefit disadvantaged populations without generating a broader, social return. Advocating that route was not within the scope of my analysis, but it certainly is something we may want to consider.

 

by Jean Rhodes

The Brookings Institution’s Hamilton Project recently published a report by economist Phillip B. Levine which takes stock of the mentoring literature and makes recommendations regarding public and private investments in this intervention strategy. Levine limited his analysis to four experimental studies of  mentoring programs focusing on the results of the evaluations of: Big Brothers Big Sisters community (CBM) and school-based (SBM) models, the Department of Education’s Student Mentoring Program, the Study of Mentoring in the Learning Environment, and the Quantum Opportunities Program. He concludes that since the latter three studies showed “no observable impact” and the BBBSA SBM evaluation had no effect on GPA, the most (and perhaps only) promising model is BBBSA’s CMB approach.

Professor Levine then draws from the CBM evaluation as well as educational research and evaluations of job-placement programs to suggest that programs make same-race and –gender matches. Since, as he notes, we “do not have the luxury of additional experimental evidence to provide strong recommendations regarding the specific content that should be include in model programs,” he extrapolates from the CBM evaluation to make recommendations.

Levine concludes that, although public spending on mentoring programs is not justified (based on the social return on the investment), NGO’s and private sectors that are interested in “helping the poor improve their outcomes from a purely altruistic perspective” might consider making the investment in mentoring (which he estimates will cost in the billions).

Critique

Given the potentially far-reaching influence that this report may have, it is important to step back and examine the basis on which these conclusions were reached. Although Levine is to be credited for tackling a large body of literature, it should be noted that his report on the topic is far from the final word on the topic.

First the report rests on the conclusion that the only evaluation to date that shows any promise is the 20-year old study of BBBS community-based mentoring.  This seems short-sighted, particularly since the one finding that he points to as justifying this claim (i.e., better grade point averages in the treatment group) was modest (i.e., p > .05, amounting to a GPA difference of .08). ). More importantly, his analysis did not include the results of rigorous evaluations of youth mentoring programs that have been released more recently, both of which offer considerable promise to the field.  First, the Centre for Addiction and Mental Health and Big Brothers Big Sisters Canada (BBBSC) recently shared results from a large, ambitious, ongoing study in which children with mentors showed several positive effects relative to those on a wait list.  For example, compared to the girls on the wait list, mentored girls exhibited fewer conduct problems and stronger social skills. Likewise, mentored boys experienced fewer emotional problems, including depression and social anxiety. Also quite promising is the recently released evaluation of the Mentoring At-Risk Youth project. This research includes the first multi-agency randomized controlled evaluation of the Big Brothers Big Sisters of America community-based mentoring program since CBM study. Researchers Carla Herrera, David DuBois, and Jean Grossman followed 1,310 youth who ranged in age from 8 to 15 and who were deemed “higher-risk” by virtue of either individual or environmental risk (or both). In the largest two programs, half the youth were assigned to a waitlist (controls) while the other half was matched right away. In the random assignment portion, mentored youth showed fewer depressive symptoms and were doing better on an aggregate measure of positive change. Among other things, this study highlighted the importance of pre-match and ongoing training.

Likewise, Levine’s analysis omits a series of meta-analyses, summarizing the results of mentoring across multiple studies  (e.g., DuBois et al, 2002; 2011; Eby et al., 2013), including a recent analysis by Patrick Tolan and colleagues (2013) who took stock of its effects on what he deems as taxpayer-justifying outcomes: delinquency (i.e., arrests and convictions) and indicators associated with delinquency (i.e., academic outcomes, aggression, drug use).  Mentoring for high-risk youth had a modest positive effect for delinquency and academic functioning, with trends suggesting similar benefits for aggression and drug use. These findings align with previous meta-analyses  which have helped to advance the field by converging on a core set of practices in the field of youth mentoring that, when rigorously applied, can yield dramatically larger effects (e.g., matching on the basis of shared interest, DuBois et al., 2011).

Since Professor Levine was willing to extrapolate from the P/PV CBM report to make recommendations regarding best practice, he should be willing to consider some of the peer-reviewed empirical work that emerged from this data set. This includes research that suggests that matches that are same-race offer no particular benefits over those that are cross-race and that closeness and consistency are far more important considerations. Likewise, his  recommendation regarding gender is countered by a recently published study (drawing on the BBBS SBM data set). Many other secondary studies have been conducted with these experimental data, which have led to suggestions of best practice that are grounded in peer-reviewed studies.

Finally, his conclusions—that youth mentoring programs are not worth taxpayer investment  because they are unlikely to increase tax revenue, seems short sighted given the findings regarding delinquency cited above and the societal costs of other difficulties (e.g., depression) that mentoring has been shown to affect.