by David DuBois
One of the most common frustrations I have heard voiced by folks in practice and advocacy roles within our field is that the measures used in evaluations of programs do not seem adequate to the task of capturing the benefits that high-quality mentoring can offer to young people. It is tempting for us as researchers to dismiss such sentiments as simply reflecting a lack of full understanding of the ways in which rigorous evaluation methods (e.g., measuring benefits as not simply changes observed for intervention participants, but rather the extent to which these surpass those that are evident for non-participants) can often reveal interventions to have considerably less impact than what anecdote or experience might suggest. To do so, however, in my estimation would be a mistake. A number of considerations lead me to hold this view. No evaluation, for example, can be claimed with confidence to take stock of the full range of outcomes that may be strengthened by a youth’s involvement in a mentoring program. Nor can it be assumed that the outcomes receiving attention were measured with sufficient precision or at the most critical points in time necessary to accurately gauge program benefits. The list could go on. But, for present purposes, I will focus on just one additional consideration: the potential for mentoring to be of benefit to individual youth in different (and possibly also multiple) ways. This is an axiom that few experienced practitioners would debate. Indeed, it is their observations heard informally over the years that have provided the impetus for me to look for ways to go beyond traditional methods of evaluation to achieve greater sensitivity to the varied and multifaceted benefits that may accrue for any given youth in response to mentoring. In conventional approaches to evaluation the focus is on the average changes that entire samples (or subgroups) of youth exhibit on outcomes and each outcome tends to be considered separately from the others. If mentored youth tend to benefit in at least one area, for example, but this area differs quite a bit from youth to youth, program effects may not be revealed with such traditional methods. Nor can it be clarified to what extent the same youth are exhibiting gains in multiple areas.
My first foray into alternative methodologies for assessing program effects was in a meta-analytic review of youth mentoring program evaluations conducted with Jean Rhodes and colleagues (DuBois, Portillo, Rhodes, Silverthorn, & Valentine, 2011). As is typical in such reviews, we reported the average effects (across all evaluations) that programs had on outcomes in each of several domains (e.g., academics, mental health, problem behavior involvement). However, we also looked at whether the youth participating in any given program showed evidence of favorable change in multiple outcome domains (e.g., improved grades and reduced involvement in delinquent behavior). Such a pattern was indeed evident for program youth in approximately half (52%) of the evaluation samples. It is a reality that mentoring programs may offer benefits in specific areas that are not as pronounced as those provided by programs with more exclusive targeting of those areas (e.g., tutoring for academic achievement; substance abuse prevention for problem behavior reduction). Yet, if the goal is to strengthen outcomes more holistically across multiple domains of youths’ development and adjustment, our findings suggest that mentoring still may be a preferred mode of intervention
What the above findings do not address is whether particular youth tend to experience gains in multiple outcome area in association with mentoring. Nor do they help to understand whether mentored youth may be more likely to show gains in at least one area than non-mentored youth, albeit with differences across youth in what that area might be. These issues were able to be addressed in a recent evaluation of the effects of mentoring programs on higher-risk youth that I collaborated on with Carla Herrera and Jean Grossman (Herrera, DuBois, & Grossman, 2013). In this research, we created a measure that was simply a tally of the number of outcomes (out of 10 possible) on which a youth showed positive change. Findings indicated that significantly greater proportions of mentored youth in both the random assignment and quasi-experimental portions of the evaluation (26 and 32 percent, respectively) to show change on at least one outcome measure than non-mentored youth (20 percent). Mentored youth also were significantly more likely to show improvement on multiple outcomes. These types of multi-faceted gains were evident, however, for only a relatively small minority of mentored youth (3 and 7 percent for two mentored groups, respectively, and 1 percent for the non-mentored youth). Of further note, program participation did not appear to reduce the number of outcomes for which youth exhibited deterioration (negative change, such as increased symptoms of depression or declining grades).
Clearly, all of the findings that I have summarized must be regarded as preliminary. My hope, in fact, is that these initial efforts will help to spark interest in the use of similarly purposed evaluation methods and thereby add to our knowledge of the ways in which youths’ lives and futures can be shaped by mentoring. Keeping up with the priorities and insights that emerge from practice is an ongoing challenge, but when embraced can help to bring out the best of us as researchers. Yet, as the present example hopefully illustrates, in some cases even a bit of simple counting can be a big help!
DuBois, D. L., Portillo, N., Rhodes, J. E., Silverthorn, N., & Valentine, J. C. (2011). How effective are mentoring programs for youth? A systematic assessment of the evidence. Psychological Science in the Public Interest, 12, 57-91. Available at http://www.psychologicalscience.org/index.php/publications/journals/pspi/mentoring.html
Herrera, C., DuBois, D. L., & Grossman, J. B. (2013). The role of risk: Mentoring experiences and outcomes for youth with varying risk profiles. New York, NY: A Public/Private Ventures project published by MDRC. Retrieved from http://www.mdrc.org/role-risk