A prevalence of “policy-based evidence-making”

By Grover J. “Russ” Whitehurst, for EducationNext

Does the current drive to incorporate social and emotional learning, or SEL, into the K–12 curriculum represent a positive reform that will lead schools to educate the “whole student” and ultimately boost young people’s academic success? Or is it a distracting fad that comes with high opportunity costs?

Common sense and considerable evidence tell us that many of the abilities that fall under the rubric of social and emotional learning—including individual effort, task-related social skills that enhance group productivity, and self-management abilities such as anger control—contribute to personal effectiveness, whether in school or elsewhere. But should schools try to teach this kind of competency, or stick to the academic domain? Can they even succeed at teaching social and emotional skills?

I don’t think we have the evidence to answer these questions yet, but there are danger signs that the SEL bandwagon is on the wrong road. Two indications stand out: a misfocus on changing student traits and dispositions rather than teaching specific skills, and the prevalence of “policy-based evidence making,” that is, the tendency to cherry-pick studies and disregard methodological quality in order to support a policy that one already favors.


Programs that attempt to teach social and emotional skills tend to focus mistakenly on personality constructs such as conscientiousness and broad dispositions such as grit. As the thinking goes, there is strong evidence that conscientiousness, for instance, is strongly linked to success in school and life. Thus, schools should put as much emphasis on teaching conscientiousness as they do on teaching core academic content. If schools do this effectively, their students will in theory have much better academic and employment outcomes. Therefore, the education system should hold schools accountable for improving conscientiousness and other SEL traits. Accountability, in turn, would require districts to assess students’ social and emotional abilities and provide supports for schools and teachers that aren’t getting the job done.

The principal problem with this line of thinking is that there is little evidence that individual differences in broad personality traits and dispositions can be meaningfully affected through school-based programs. In fact, there is strong evidence to the contrary that comes from research by psychologists on personality going back almost 100 years.

Many personality psychologists today endorse the “five-trait” theory, which centers on the so-called Big Five personality traits: openness to experience, conscientiousness, extraversion, agreeableness, and neuroticism. These traits, which are typically measured by self-report questionnaires, delineate an individual’s stable predispositions to respond in similar ways across a broad range of circumstances.

The soft skills that are often targeted in SEL curricula overlap substantially with aspects of the Big Five personality traits. For example, when the University of Chicago Consortium on School Research says that its model of social and emotional factors in education comprises “such interpersonal qualities as cooperation, assertion, responsibility, and empathy,” it is describing components of the Big Five personality traits.

The problem with SEL models that focus on traits and dispositions is that the influence of genetics looms large relative to that of any particular cultural institution, including schools. The Big Five personality traits are highly heritable. For conscientiousness, the estimate of heritability from the four most recent studies is 49 percent. In other words, the similarity of two children in their degree of conscientiousness is predicted strongly by the extent of their genetic similarity—identical twins will be much more similar than same-sex fraternal twins. Even more important in the present context, similarity in personality traits is not at all predicted by the children’s “shared environment,” that is, whether or not they are reared in the same family or attend the same school.

The expression of a trait such as conscientiousness is surely affected by the environment, but the influences upon it derive from idiosyncratic experiences that are often subject to selection based on an individual’s genetic makeup. Thus, two students, one high in conscientiousness and the other in extraversion, attending the same classes in the same school, will tend to seek out and be selected for environments that fit and strengthen their different propensities. The conscientious student may become editor of the class yearbook while the extraverted student becomes class president. These divergent paths expose the two students to different environments, which, in turn, impart distinctive sets of specific skills that can reinforce preexisting differences in personality traits. It is difficult to imagine how schools could stop this process from happening. And even if they did, the student who was more conscientious at the outset would still be more conscientious in the end.

SEL programs could accomplish much more by shifting their focus from abstract traits and dispositions to specific skills that are observable, close to the classroom, teachable, and linked in straightforward ways to the mission of schools. Such skills include giving effective forms of feedback to others; staying on task in the classroom; monitoring one’s own behavior as to whether it is having the intended effect; engaging in timely and expected social routines; and anticipating and deflecting the occurrence of automatic behaviors and biased beliefs that lead to trouble.

In this regard, there are strong lessons for the SEL movement from research on cognitive development. Cognitive abilities, like social and emotional traits, have a strong genetic component. But schools gave up long ago on the hopeless task of teaching children to be smart or intelligent, focusing instead on teaching specific skills such as reading fluency and mathematical reasoning. The SEL curriculum needs a similar focus on specifics. Conscientiousness, grit, empathy, and the like should be to social and emotional instruction as intelligence and cognitive ability are to academic instruction—reflections of enduring individual differences that provide a context for learning, not what the school tries to teach students directly.

Policy-Based Evidence Making

Advocates for school-based SEL programs promote evidence that they characterize as demonstrating that SEL works. The evidence gathering typically involves relaxing generally accepted standards of research quality (rigor and relevance) and turning a blind eye to discordant findings and reviews. This leads to what has been characterized as policy-based evidence making (as distinguished from evidence-based policymaking).

Policy-based evidence making typically entails the mash-up of large numbers of studies into an analysis that generates summary conclusions about what works. For example, when I read in the SEL literature of a meta-analysis that combined the results of 82 separate studies of the impact of SEL programs and found “impressive” positive impacts at follow-up, I knew without necessarily having to scrutinize the study that the conclusions were not credible.

Why? The likelihood that there are 82 methodologically sound and policy-relevant studies of the impact of school-based SEL interventions is exceedingly small. The What Works Clearinghouse of the U.S. Department of Education’s Institute of Education Sciences has for 15 years been reviewing individual studies of the effectiveness of education programs and practices across multiple domains. It has to date reviewed more than 10,500 studies and found only 383 that report at least one positive effect and meet at least the lower tier of acceptable methodological quality laid out in the clearinghouse’s standards. Only a couple of these 383 studies focused on SEL-like interventions. Thus, the only way to place faith in the conclusion of the analysis of 82 studies is to disregard the low quality of the studies on which it was based. The term of art for a meta-analysis of low-quality studies that generates strong positive conclusions is GIGO (garbage in, garbage out).

Another hallmark of policy-based evidence making is the tendency to ignore contrary research. A salient example is the way SEL advocates have treated a groundbreaking research project carried out by the Institute of Education Sciences. It was a large-scale, multi-site study of schoolwide social- and character-education programs. Schools were randomly assigned to implement an SEL program or to continue with business as usual, and results were collected for students as they moved from 3rd through 5th grade. Findings are reported for the seven SEL programs as a whole and for each individually. The seven programs include some that SEL advocates often hold up as model programs with strong evidence of effectiveness, including PATHS (promoting alternative thinking strategies) and 4Rs (reading, writing, respect, and resolution).

The study found that for the seven SEL interventions analyzed collectively, only 2 of 60 estimated impacts were statistically significant. Some critics of the study have suggested this is because the study was under-powered; that is, it didn’t have a sufficient number of schools to detect modest effects as statistically significant. But a supplementary analysis of the collective findings using a substantially lower statistical bar for identifying effects found about as many detrimental results (seven) as beneficial (nine).

Tellingly, this report has been cited in the scholarly literature only 14 times since its release in 2010. It is not mentioned in the research section (or elsewhere) on the website of the most research-oriented of the SEL advocacy organizations, the Collaborative for Academic, Social, and Emotional Learning. Nor is it included in the previously described meta-analysis of 82 studies.

Achieving evidence-based policy and practice in SEL will require an even-handed consideration of all the evidence that is both methodologically sound and relevant to consequential decisions. This is not the current state of affairs. Educators and policymakers who want to learn what research says about the effectiveness of SEL programs are most likely to turn to those who have skin in the game as developers of SEL programs and advocates of SEL investments. For example, the Collaborative for Academic, Social, and Emotional Learning says about itself, “we are turning this momentum [for SEL] into a movement.” Families looking to install solar panels on their roof shouldn’t turn to the Solar Energy Industries Association for an objective analysis of the costs and benefits of doing so. Likewise, policymakers and practitioners making decisions about investments in social and emotional learning for schools need independent, objective analysis of what works, not advocacy.

Summing Up

Social and emotional learning is important to student success. But try as they might, schools are not going to succeed in making shy students extraverted, careless students meticulous, or contentious students agreeable (or, for that matter, slow students smart). Schools can, to be sure, teach students specific social and emotional skills that they can deploy for advantage in particular situations. For example, the shy student can learn to make eye contact on introductions, the careless student to run a spell check before submitting a class paper, and the contentious student to suppress criticism when it is likely to be counterproductive.

A number of questions need addressing before we can expect SEL to catalyze students’ successful adaptation to the demands of school and life: When and how do we teach social and emotional skills? Which skills should we select? And to which categories of students do we attempt to teach the various skills? Making progress on that bundle of questions will require more than enthusiasm for SEL. It will necessitate specific and well-grounded hypotheses about those questions, and it will require valid measures of success. The slope of the learning curve from this enterprise will depend on the quantity and quality of evidence that is brought to bear on what works and why. That is hard and incremental work, but I know of no other way for social and emotional learning to achieve a permanent and productive place in the mission of schools.

This is part of a forum, “Should Schools Embrace Social and Emotional Learning?.” For an alternate take, see “An Integrated Approach Fosters Student Success,” by Robert Balfanz.

For the original post, please click here.