Inaugural Ephraim Lecture Explores the Lessons of Randomized Controlled Trials
Looking at Fifty Years of Research in Criminal Justice, Professor Megan T. Stevenson Shares a Surprising Insight
What policies and programs can society implement to make a tangible, lasting impact on people’s lives?
Social scientists have been trying to answer this question for decades by conducting randomized controlled trials (RCTs), experiments that aim to identify “what works” to engineer improvements for society, from guaranteed income programs to job training for previously incarcerated individuals. RCTs are considered the gold standard for determining whether a particular policy or intervention is effective.
On October 15 at the inaugural Ephraim Lecture, Megan T. Stevenson, a law professor and professor of economics at the University of Virginia, presented her research on what fifty years of RCT studies in criminal justice teach us about the structure of the social world. Stevenson was the first recipient of the Ephraim Prize for Law and Economics, named for alumnus Donald M. Ephraim, ’55, who established the award in 2023. The prize recognizes an early career scholar in law and economics whose work has advanced the state of knowledge in the field.
Ephraim attended the event and introduced Stevenson, after brief introductory remarks from Dean Thomas J. Miles.
“I am very happy to say that law and economics began here at the University of Chicago,” said Miles. “And more than that, it has been the intellectual powerhouse that propelled these ideas throughout the legal academy and into the world. I’m thrilled that this intellectual leadership continues today through the Donald M. Ephraim Lecture.”
Stevenson’s Central Thesis: The Elusive Impact of RCTs
Most of Stevenson’s research in criminal justice focuses on narrow questions, like how incarceration affects wealth. The paper she presented at the Ephraim Lecture, however, was much bigger in scope, “a turning point” for her as a scholar, she said.
“This paper is built around a key empirical claim that I’ve developed after ten years of reading and conducting empirical research in criminal justice,” she said.
Stevenson’s thesis is this: A broad look at RCTs in criminal justice since the 1970s reveals that most interventions and reforms, as studied by RCTs, were not successful—meaning, they had little lasting effect—and the very few that did have success mostly did not replicate in other times or places.
“This may sound depressing, but it teaches us something really kind of profound and important about the structure of the social world,” she said.
Stevenson argues that, when it comes to the type of limited scope interventions evaluated by RCTs, the social world is full of stabilizing forces that resist change. So, to whatever extent an intervention may have altered an individual’s path for a short period of time, the forces of life and the way the world is structured push a person back onto the path they would have been on absent the intervention.
Cascades—small interventions that lead to large and lasting changes—are rare. Stevenson even ventures to say that they are mostly a myth because of a lack of evidence to support them.
Stevenson took a three-pronged approach in her examination of the impact of RCTs. She looked at a fifty-year study of RCTs with crime as an outcome, she looked at famous instances of success in RCTs, and she examined various hot topics in criminal justice, like intensive probation, drug courts, police body cameras, and job training programs.
Her findings showed that most of the famous successes were never replicated, and the very few attempts at replication found that the intervention was not as successful in other times or places.
“Does this apply in other areas of the social world, outside of the criminal legal space?” posed Stevenson. “I think, yes. Many areas of social science also find that interventions rarely have large or lasting impact.”
Charting a Path Forward
Stevenson concluded her talk by exploring the implications of her research. She described what she called “the engineer’s view,” a viewpoint that is based on the idea that the social world is amenable to manipulation and control—and that we can reliably learn to predict the impacts of reforms using RCTs and other methods.
“I think the engineers view is a myth,” she said. “I don’t think we live in a world that works in the way that it would need to work in order for us to be able to engineer change on the scale we want—at least not with the confidence that comes with having pilot-tested something to prove success first.”
Stevenson said that she didn’t have any definitive answers on what we could do to create real, lasting change for the betterment of society, but she did have three suggestions. First, one could focus on direct harm alleviation—if we are worried about hungry people, for example, feed them. Second, one could pursue systemic reform—and accept the uncertainty that comes with it, because people who are drawn to this option tend to be guided by theory and intuition. Third, one could continue with incrementalism, i.e., taking small steps and hope that this moves the world in the right direction.
“I want to reiterate that I don’t see this as depressing,” said Stevenson. “I truly see this as a testament to people’s will to be able to create things on their own. And to the extent that there remain limits to their ability to create the world that they want on their own, these limits come from deep structural forces that are simply not easy to change.”