Large study presents evidence for behavioral sciences in policymaking

In April 2020, a group of researchers published a highly influential paper with 19 policy recommendations around COVID-19 based on insights from the behavioral sciences. The paper was a large collaboration of over 40 experts, led by Jay Van Bavel of New York University and Robb Willer of Stanford, and was cited thousands of times by governments, researchers, and public figures. Its recommendations covered topics such as official messaging on social distancing, how to get a vaccine once they were available, and the need to work within communities to create real impact. Now, Ruggeri et al.’s new paper in Nature evaluates evidence since the first paper’s publication supports its claims and their applicability for policymaking.

“Governments around the world formulated pandemic policy strategies explicitly on the basis of the behavioral concepts highlighted in the 2020 paper by Jay J. Van Bavel et al.,” says Ruggeri, a professor health policy and management at Columbia University’s Mailman School of Public Health. “Given concerns over a lack of public trust in science, particularly in the context of COVID-19, we believed it was important to evaluate the evidence for public policy recommendations, in a way that promotes transparency and builds trust.”

Two independent teams of 72 experts — including both the 2020 paper’s authors, as well as an independent team of evaluators — reviewed 747 pandemic-related research articles to assess the extent to which claims in the original paper provided valid policy guidance. They treated studies conducted (and replicated) in real-world settings across large populations in multiple settings as the highest level, and flagged arguments that were not backed by empirical evidence.

Alex Haslam, PhD, professor of psychology from the University of Queensland in Australia and study co-author, says, “In recent years, there has been a lot of discussion about the limitations of psychological and behavioral science, especially in the face of the so-called ‘replication crisis.’ As a counterpoint to this, what this research showed is that there is a core of good theory in these fields that provides a strong basis for both scientific prediction and public policy. This theory may not always be flashy, but it is the bedrock of good social science, and this study confirms that it is something we can rely on for guidance when we need it.”

The study finds evidence for 18 of 19 claims in the 2020 paper, including those related to sense of identity and community connectedness, leadership and trust, public health messaging, social cohesion, and misinformation. Of the 18, the 2020 paper correctly identified 16 relevant behavioral concepts during the pandemic as well as likely barriers to mitigating spread of the disease and social challenges that would be faced by policymakers. The researchers found no effect for two proposed policies related to effective public messaging (that messages should emphasize benefits to the recipient, and that they should focus on protecting others). Notably, the team found no evidence to review for one high-profile recommendation in the 2020 paper, which suggested the phrasing “physical distancing” is preferable to “social distancing.”

The most strongly supported claims were the importance of interventions to combat misinformation and polarization, which proved to be vital for ensuring adherence to public health guidelines. Research also underlined the point that, to be effective, messaging needs to emanate from trusted leaders and to emphasize positive social norms.

Public health interventions that received the most attention were not necessarily the ones best supported by the most evidence. For example, handwashing was widely promoted as a strategy for stopping the spread of COVID, yet study effects were small to null, particularly compared to masking, isolation, distancing, and vaccines.

Regarding masking, early guidelines in some countries suggested the practice would not minimize COVID-19, but subsequent evidence pointed to the effectiveness of masking. Likewise, research also undermined guidance on the impacts of school closures and disinfecting surfaces. “While there are understandable pressures to issue guidelines quickly during a crisis, making policy decisions without adequate evidence can be costly in many ways,” says study co-author Katherine Baicker, PhD, Provost of the University of Chicago. “As new scientific evidence comes in over time, some people may view evolving policy guidance as a sign of incompetence — or even conspiracy — undermining trust in expertise. Policymakers must balance the need for expedience with the need for robust evidence and credibility.”

The new study also identifies several domains missing from the 2020 paper. These included threat and risk perception, the role of inequality and racism, skepticism toward science, incentivizing behaviors beyond simply describing benefits (e.g., by providing financial rewards for vaccination) and the absence of clear leadership.

Finally, the research team provides recommendations to help researchers and policymakers respond to future pandemics and disasters. These include the need to study global populations, to do more field testing, and to be more specific in formulating testable questions. “The value of field testing what really works to change health behaviors can’t be overstated, and the strongest conclusions we’ve been able to draw in this article were often thanks to partnerships researchers forged with local governments and healthcare providers to carefully evaluate what actually adds value in the middle of a crisis,” says study co-author Katy Milkman, PhD, professor at the Wharton School of the University of Pennsylvania. The researchers also encourage scientists to forge more alliances with policymakers and decisionmakers — in local government, hospitals, schools, the media, and beyond.

“This work has the potential to increase transparency and build trust in science and public health, and to directly inform the development of tools and knowledge for the next pandemic or other crisis. Researchers can be a viable source of policy advice in the context of a crisis, and our recommendations point to ways to further improve this role of social and behavioral science,” says study co-senior author Robb Willer, PhD, professor of sociology at Stanford University.

“This new paper rigorously evaluated policy recommendations from our original team to see if they were accurate, using large amounts of evidence and a new team of independent reviewers from around the globe. In addition to confirming the vast majority of our original claims, it sets a new gold standard for evaluating evidence when policy decisions, particularly urgent ones, must be made,” says Jay Van Bavel, PhD, professor of psychology, New York University, lead author of the landmark 2020 article, and co-senior author of the new paper.