For decades, a central question in rulemaking has been the extent to which public comments on proposed rules affect the substance of agency regulations. On the one hand, the notice and comment process has been likened to Kabuki theater. On the other hand, researchers have discovered that, under certain circumstances, comments can have substantial effects on final rules.
A variety of approaches have been utilized in the quest to uncover the impact of comments. Some analysts have focused on quantifiable changes across proposed and final rules, such as dollar amounts of government allocations, investigating the association between comments the magnitude of these changes. Others have manually coded the content of comments and changes across proposed and final rules. Still others have relied on evidence from regulators themselves, who often point to instances in which comments have fundamentally shaped their agencies’ regulations.
In an intriguing working paper, Andrei Kirilenko, Shawn Mankad, and George Michailidis propose a new approach, dubbed RegRank, to analyzing the impacts of comments. RegRank uses topic modeling to uncover the structure and sentiment of proposed rules, comments, and final rules. Topic modeling is an automated method, developed in the context of natural language processing, that draws off of the meaning and tone of words contained in documents. This method produces scores that quantify documents in terms of their sentiment, for example, the extent to which they are positive or negative in content.
Kirilenko, Mankad, and Michailidis generate and analyze topic modeling scores for proposed rules, comments, and final rules. Their aim is to determine the extent to which changes across proposed and final rules are consistent with what is articulated in comments. The specific research context is approximately 60,000 comments submitted in response to 104 proposed rules that led to 67 final rules issued by the Commodity Futures Trading Commission (CFTC) between January 2010 and September 2013.
The authors’ main finding is, simply put, that comments mattered. The CFTC routinely adjusted proposed rules in the direction of the tone and substance of comments. More specifically, Kirilenko, Mankad, and Michailidis find that comments from finance professionals (e.g., bankers, traders, asset managers) exerted great influence, while comments emanating from other occupations had no discernible impact on final rules. These results do not imply that non-industry comments had no importance at all in CFTC rulemaking. Comments representing the public interest, broadly defined, affected the likelihood that the proposed rules were ultimately finalized.
The approach developed by Kirilenko, Mankad, and Michailidis represents a promising pathway to more fully understanding the place of commenting in the rulemaking process. That said, the authors’ working paper is only a first step in this regard, and the work that has been done thus far is not without its uncertainties.
For starters, the period between publication of a proposed rule and promulgation of a final rule is often rather complex in terms of the analysis that is done and the communications that occur. Such complexities make it difficult to specifically ascribe changes in the substance of regulations during this period to the comments that were submitted. This inferential difficulty is, of course, prevalent not just for RegRank, but in all research seeking to illuminate the impact of comments.
At this point, the work of Kirilenko, Mankad, and Michailidis has not yet been integrated into the larger body of research on commenting that has been produced over the decades by legal scholars and social scientists. Such integration is important because it can help place RegRank in its intellectual context, thereby making it more straightforward to ascribe broader meaning to the results of the analysis. For example, divorced from theoretical considerations, it is not entirely clear just why it is important to find that public interest comments increase the likelihood of proposed rules being finalized.
As with any body of rulemakings, the proposed rules analyzed by Kirilenko, Mankad, and Michailidis vary substantially in the number of comments that were generated. Three rulemakings in particular can be characterized as having generated especially high volumes of comments. It is often the case that the results of rulemaking research vary significantly across high-volume rules and rules that are not high in salience and complexity. Going forward, it will be essential to examine the extent to which the authors’ results hold across rules of varying salience, complexity, and comment volume.
This last point suggests a logical next step of applying RegRank to agencies other than the CFTC. Such an application is especially important given the peculiarities of CFTC rulemaking. For example, both proposed and final rules require votes on the part of CFTC commissioners. Do the results of authors’ analysis hold for agencies that do not share this institutional feature of commissioner approval? Exploring this and other questions that have been raised in this commentary represent good places to start in exploring the general utility of RegRank, which exhibits great promise in its initial application.