Regulatory Science and Policy: A Case Study of the National Ambient Air Quality Standards

EPA
by Susan E. Dudley, Director
September 09, 2015

Download the working paper

Background

When it comes to environmental regulation, no one is immune to the temptation to put a spin on science to advance a policy goal. While the media will decry the politicization of science – when political decision-makers attempt to distort what scientific studies conclude, problems also arise when scientists and others attempt to exert influence on policy decisions by selectively presenting, or even distorting, scientific findings (scientization of policy).

This paper focuses on the scientization of policy, and defines and examines two contributors: the “positive-normative fallacy” (not acknowledging that science alone is insufficient to resolve normative policy questions) and “hidden policy judgments” (not acknowledging the policy judgments inherent in assessments of risk). It examines the process by which the Environmental Protection Agency sets National Ambient Air Quality Standards (NAAQS) under the Clean Air Act to illustrate some of the perverse incentives involved in developing regulations, and offers possible mechanisms to improve those incentives and resulting policy.

An Adversarial Process

The NAAQS process exemplifies the incentives at work that compel every party to the regulation to engage in what has been called the “science charade.”  First, Congress directs EPA to set the standards to achieve noble goals, but restricts the agency from considering key factors, falling prey to the positive-normative fallacy by asserting instead the pretense that science alone is sufficient to develop policy.  Combined with tight deadlines, the statutory language permits Congress to take credit for laudable public goals, while blaming the executive branch’s execution for any undesirable outcomes. The courts have reinforced a limited interpretation of the Act, as well as tight deadlines for issuing revised standards.  Executive branch career and policy officials respond by developing scientific-sounding explanations to justify one standard over another, and public interveners vigorously defend alternative standards based on their own interpretation of the science. 

Scientists argue for the primacy of their data, analysts have an incentive to downplay rather than reveal uncertainties regarding their predictions or the implications of key risk assessment policy choices, and decision makers point to science as either requiring a new standard or as determining that existing standards are adequate. 

This has evolved into an adversarial process, characterized by harsh rhetoric in which each party claims the science supports its preferred policy outcome and questions opponents’ credibility and motives, rather than a constructive discussion regarding appropriate assumptions and data.  The real reasons for selecting a particular standard may not even be discussed. 

Furthermore, EPA is setting standards for fine particulate matter and ozone that are unattainable for the foreseeable future for many parts of the country, and the actual public health and welfare benefits of the standards, particularly when one considers the opportunities forgone, are in considerable doubt.

Recommendations

In thinking about reforms to improve how science is used in developing regulations, clarifying which aspects of the decision are matters of science and which are matters of policy is essential to avoid both the politicization of science and the scientization of policy.  Greater transparency in the underlying studies and factors considered in making decisions is also important for more accountable policies. The paper offers recommendations designed to alter the incentives of the parties to the rulemaking process to encourage scientific advancement and better public policies.

  1. Legislators must be more forthright in recognizing that “science” is a positive discipline that can inform, but not decide, appropriate policy.  When delegating regulatory authority to executive agencies, statutes should require a balancing of benefits and costs (as have the last five presidents).
  2. Legislators and policymakers must clarify the appropriate role for scientific advisors. The engagement of scientific advisory panels can provide a valuable source of information and peer review for agency science, but greater efforts should be made to restrict their advice to matters of science, and not ask them to recommend specific regulatory policies.
  3. The executive branch must recognize that risk assessment necessarily involves assumptions and judgments as well as pure scientific inputs, and establish procedures and incentives to make more transparent the effect different credible risk assessment inputs and assumptions have on the range of plausible outcomes.
  4. The executive branch should increase the robustness of regulatory science by institutionalizing reforms that encourage greater feedback and challenge. Successful reforms might involve pre-rulemaking disclosure of risk assessment information to engage broad public comment on the proper choice of studies, models, assumptions, etc. long before any policy decisions are framed, and “positions” established. 
  5. Scientific advisory panels should be required to represent a diversity of perspectives, disciplines, expertise, and experience.  As a 2012 Keystone Group report recommended, since “all potential panelists will have conscious and unconscious biases,” agencies “should strive to engage a wide range of perspectives of qualified scientific experts.
  6. The legislative and executive branches should institutionalize feedback through retrospective review of regulatory outcomes. Institutionalizing a requirement to evaluate whether the predicted effects of the regulation were realized would provide a powerful incentive to improve the use of science for predicting the benefits of interventions. The five-year NAAQS reviews, for example, could be required to apply quasi-experimental (QE) techniques to gather and analyze epidemiology data and health outcome trends in different regions of the country and compare them against predictions.
  7. Regulations should be designed to facilitate natural experimentation and learning. Designing regulations from the outset in ways that allow variation in compliance is essential if we are to go beyond observing mere associations and gather data necessary to test hypotheses of the relationship between regulatory actions, hazards, and risks.
  8. The scientific studies used to support regulation should be subject to peer review and their results reproducible. A greater focus on rigorous peer review, including sharing of data and experimental methods such that others can attempt to replicate studies’ results, would go a long way toward enabling the scientific method that depends on falsifiable hypotheses and challenge.
  9. Legislation should recognize that states have a core interest in environmental quality, and that experimentation and competition among states can be a powerful force for improving environmental outcomes and our practical knowledge of what works. Perhaps a better division of responsibility would be for the federal government to conduct basic risk assessment research and share information on environmental damages, but to defer to states on decisions regarding the risk management policies appropriate for their situations.
  10. Agencies should engage in collaborative tools to generate knowledge. To harness the wisdom of dispersed knowledge, agencies or outside parties might experiment with a collaborative “wiki” approach to public comment, where, rather than each individual or group filing comments in parallel and the agency responding to those comments individually, it could provide a forum for diverse individuals to build on each other’s information, adding, editing, updating, and correcting to engage the wisdom of dispersed knowledge on issues where no one person has complete information.

Download the working paper

Download highlights and recommendations


See also: 

Commentary: Does Reducing Ozone Really Improve Human Health?, by Louis Anthony (Tony) Cox, Jr.

Public Comment: National Ambient Air Quality Standards for Ozone, by Louis Anthony (Tony) Cox, Jr.

Book Chapter: Susan E. Dudley & George M. Gray,  “Improving the Use of Science to Inform Environmental Regulation,” in Institutions and Incentives in Regulatory Science, Lexington Books, Jason Johnston ed. (2012)

Improving Weight of Evidence Approaches to Chemical Evaluations, by By Randall Lutter, Linda Abbott, Rick Becker, Chris Borgert, Ann Bradley, Gail Charnley, Susan Dudley, Alan Felsot, Nancy Golden, George Gray, Daland Juberg, Mary Mitchell, Nancy Rachman, Lorenz Rhomberg, Keith Solomon, Stephen Sundlof, and Kate Willett in Risk Analysis

Comment on Löfstedt’s ‘The substitution principle in chemical regulation: a constructive critique’, by Susan E. Dudley

Rethink chemical risk assessments, by George M. Gray & Joshua T. Cohen in Nature

Improving Causal Inferences in Risk Analysis, by Tony Cox

Testimony before the Subcommittee on Energy and the Environment Committee on Science, Space, and Technology: Fostering Quality Science at EPA: Perspectives on Common Sense Reforms, by Susan E. Dudley