Why evidence for natural mentors can be a “very tricky thing”

Screen Shot 2015-02-12 at 7.08.07 PMby Jean Rhodes

“Circumstantial evidence is a very tricky thing. It may seem to point very straight to one thing, but if you shift your own point of view a little, you may find it pointing in an equally uncompromising manner to something entirely different.”A.C. Doyle

The most recent issue of the American Journal of Community Psychology includes an article that suggests a causal relationship between natural mentors and long-term economic gains for certain, vulnerable youth (see this issue). It’s an impressive study and the findings, which draw from a large, longitudinal survey of adolescents in the U.S., contribute to a growing body of research suggesting the benefits of caring adults. Natural mentors can, no doubt, make a difference in the lives of vulnerable youth. At the same time, readers should consider the potential “self-section” biases inherent in such studies.

In this context, self-selection bias would be at play if youth who found and maintained relationships with natural mentors were systematically different from youth who never did so. If so, these differences may, in part, account for the  better outcomes. For example, youth who sort into natural mentoring relationships may have more enriched networks, special talents, more agreeable personalities, mothers who arrange extracurricular activities, etc.. Those factors may contribute to both meeting a mentor and doing well.  In other words, having a natural mentor could simply be a proxy indicator rather than the source of the youth’s better outcomes. The researchers made some statistical corrections for potential self-selection, and briefly note the “potential bias” of just looking at the correlation between income and having a natural mentor, but the possibility of selection bias may be difficult to resolve. The only way to truly determine if the natural mentor causes (as opposed to simply being correlated with) better outcomes would be to randomly assign natural mentors to youth–a difficult study to imagine.  As my colleague Dr. Jean Grossman recently noted, ” The more experiments I do, the more I realize that most of what we see is due to people sorting themselves into things that best suit them or that reflect their true ability…which is often hard to see or measure a priori.”

Or, as Professor Adar Ben-Eliyahu concluded in an earlier Chronicle post” Not every significant association implies that one variable causes the other. The fact that there might be other explanations [such as self-sorting] for why certain variables are associated, does not necessarily weaken the finding, it just means that we have to be cautious about conclusions that we draw.”