The United States and other developed countries rely on regulatory tools to manage potential risks from exposure to targeted chemicals. While it is standard practice to analyze how proposed regulations might affect regulated entities and consumers before they are issued, regulators have devoted much less analysis to evaluating the impacts of their regulations once they are in effect. This ex-ante regulatory impact assessment has a long tradition in many countries, with established analytical steps and oversight as well as opportunities for public engagement to hold governments accountable for conducting analysis before regulations are issued. But ex-ante analyses necessarily depend on unverifiable assumptions and models of how the world would look absent the regulation, and how responses to regulatory requirements will alter those conditions. In essence, especially in the area of chemical risk management, ex-ante analyses are hypotheses of the effects of regulatory actions.
Better regulatory evaluation is important but challenging
Better regulatory evaluation would allow agencies and others to test those hypotheses against actual outcomes. This would not only inform decisions related to the cost-effectiveness of existing policy, but would provide feedback that would improve future ex-ante analyses and future policies. However, ex-post analysis also poses challenges. Once a regulation is in place, it is not always obvious what the world would have looked like without it. For example, would air emissions have increased directly with economic and population growth, or would technological change and citizen preferences have driven emissions lower? Measuring opportunity costs (what activities or innovations were foregone to achieve regulatory goals?) is not easy, and measuring regulatory benefits is often harder. Furthermore, once a regulation is in place, neither regulators nor regulated entities have strong incentives for examining its actual impact.
A systematic approach to evaluation would improve future regulations
Program evaluation has a long tradition in other areas. Not only do successful businesses routinely measure results and assess outcomes, but government agencies tend to subject programs financed through the fiscal budget to much more rigorous retrospective review than their regulatory programs.
Better evaluation and feedback regarding the efficacy of chemical risk management regulations would contribute both to learning and accountability. It would not only be valuable for understanding the effect of past actions aimed at reducing chemical risk, but for improving future decisions. Achieving better retrospective review demands both better practices and stronger incentives.
Improving retrospective review practices
Conducting reliable retrospective evaluation of regulations’ impacts is challenging, but there are several steps agencies can take to measure the relationship between risk management interventions and desired outcomes, and learn from experience. Agencies should plan for retrospective review at the outset, establishing a framework for empirical testing of assumptions and hypothesized outcomes. They should design regulations to allow for experimental learning – by initiating small scale trials or by randomizing treatments in different areas. Regulations should be based on a theory of change or logic model, so that evaluators can measure moderating or confounding factors that may have affected outcomes. And finally, they should apply advanced statistical tools to establish whether observations are causally related.
Providing incentives for retrospective review
Institutional changes could improve incentives to support a culture of regulatory look back. Just as many jurisdictions will not approve a new regulation unless it is accompanied by ex-ante analysis of expected impacts, they could also not approve a new rule unless it is accompanied by a framework and plan for conducting retrospective review in the future.
Furthermore, retrospective review shouldn’t be left exclusively to regulatory agencies, which have little incentive to find fault with their own regulations. Instead, to introduce objectivity and accountability, regulations should be subject to third-party evaluation. Some countries use mechanisms such as sunset provisions or offsets, both of which could provide incentives for evaluation of regulations’ effects. Finally, shifting resources from ex-ante analysis to ex-post review would not only help with evaluation, but would provide valuable information that could improve ex-ante hypotheses of regulatory effects.
This commentary is based on remarks Susan Dudley presented at an OECD Workshop on Socioeconomic Impact Assessment of Chemicals Management in Helsinki, Finland, July 6-8, 2016.