Learning from Experience: Retrospective Review of Regulations in 2014
Through a series of Executive Orders, President Obama has encouraged federal regulatory agencies to review existing regulations “that may be outmoded, ineffective, insufficient, or excessively burdensome, and to modify, streamline, expand, or repeal them in accordance with what has been learned.” Evaluating whether the intended outcomes of regulations are met ex post can be challenging, so multiple government guidelines instruct agencies to incorporate retrospective review plans into their proposals during the rulemaking process. To support this effort, the George Washington University Regulatory Studies Center examined significant regulations proposed in 2014 to assess whether they included plans for retrospective review, and provided recommendations for how best to do so. This paper finds that, despite these guidelines, agencies are not planning prospectively for ex post analysis of their rules and provides agencies with three recommendations to facilitate transparency, public accountability, and measurement of their rules’ success.
In 2014, the George Washington University Regulatory Studies Center launched a yearlong effort to evaluate high priority proposed rules to determine whether it was designed in a manner that would make its outcomes measurable ex post. As a part of this Retrospective Review Comment Project, the Regulatory Studies Center examined significant proposed regulations to assess whether agencies included a discussion of retrospective review, and submitted comments on the rulemaking record providing suggestions on how best to incorporate plans for retrospective review at the time of the rule’s issuance.
While agencies commonly use prospective evaluation to estimate what the effects of their regulations will be (typically in the form of a benefit-cost analysis), they do not typically use this analysis to measure the effects of their rules after implementation, or to design their rules to aid retrospective review.
As discussed in more detail below, to facilitate meaningful retrospective review after the promulgation of a final rule, multiple government guidelines instruct agencies to incorporate retrospective review plans into their proposals during the rulemaking process. However, based on our review of the rules proposed in 2014, agencies are not designing their rules to facilitate ex post measurement, and are not prospectively planning for retrospective review at the outset of rulemaking.
Retrospective review is a form of program evaluation that reviews the efficacy of a program or policy after implementation. The purpose of retrospective review is to evaluate whether a policy—in this case, a regulation—has had its intended effect, and whether it should be continued or expanded. By examining the effects of existing rules, these reviews can inform policymakers on how best to allocate scarce societal resources to accomplish broad social goals, like improved air quality or wellbeing, through regulation. Retrospective review can provide valuable feedback and learning that will improve the design of future regulations.
In a World Bank report on impact (program) evaluation, Gertler et al. illustrate the importance of applying evaluation to policies:
In a context in which policy makers and civil society are demanding results and accountability from public programs, impact evaluation can provide robust and credible evidence on performance and, crucially, on whether a particular program achieved its desired outcomes.
This argument makes especial sense in the case of regulation. While policymakers have the opportunity to revisit on-budget programs each time federal funds are being appropriated, regulatory programs often exist in perpetuity without a statutory requirement to revisit implementation.
Regulations often receive critical analysis before promulgation, usually in the form of benefit-cost analysis. This prospective analysis details the anticipated results of a proposed rule, including costs, benefits, and unquantifiable effects. While agencies often provide a wealth of information on the anticipated effects of their rules, they seldom return to a rule to evaluate whether the benefits and costs they anticipated actually materialized. In his report to the Administrative Conference of the United States (ACUS), Joseph Aldy writes that federal regulatory agencies have a mixed record on ex post review, despite their “long track record of prospective analysis of proposed regulations that can address these questions.”
Recently, retrospective review has found a proponent in President Barack Obama, who issued three executive orders during his first term directing agencies to conduct retrospective analysis of existing regulations.
 Sofie Miller is a Senior Policy Analyst at the George Washington University Regulatory Studies Center, 805 21st St. NW, Suite 609, Washington, DC. She can be reached at firstname.lastname@example.org or (202) 994-2974.
The author thanks The George Washington University Regulatory Studies Center staff for their dedication to undertaking the ambitious Retrospective Review Project in 2014, on which this paper is based. Specifically, the author thanks Susan E. Dudley, Cassidy B. West, Estelle Raimondo, Julia Morriss, Connor Roth, Mark Bigley, Tracy Mehan, Blake Taylor, Lindsay M. Scherber, Ian Smith, and Philip J. Austin, all of whom contributed to the public comments filed as part of this yearlong project. The author also thanks James Broughel for his constructive comments when a draft of this paper was presented at the Society for Benefit Cost Analysis conference in 2015.
 This comment reflects the views of the author, and does not represent an official position of the GW Regulatory Studies Center or the George Washington University. The Center’s policy on research integrity is available at http://regulatorystudies.columbian.gwu.edu/policy-research-integrity.
 Paul J. Gertler, Sebastian Martinez, Patrick Premand, Laura B. Rawlings, and Christel M. J. Vermeersch. “Impact Evaluation in Practice.” The World Bank. 2011. http://siteresources.worldbank.org/EXTHDOFFICE/Resources/5485726-1295455628620/Impact_Evaluation_in_Practice.pdf
 Joseph Aldy. “Learning from Experience: An Assessment of the Retrospective Reviews of Agency Rules and the Evidence for Improving the Design and Implementation of Regulatory Policy.” A report for the Administrative Conference of the United States. 2014. https://www.acus.gov/report/retrospective-review-report
Commentary: Evaluating Retorospective Review of Regulations in 2014, by Sofie E. Miller