The Secret Side of AI Sycophancy: The Hidden Threats to the Administrative Procedure Act
As part of its aggressive campaign to “deconstruct the administrative state” on behalf of the Trump administration, the so-called Department of Government Efficiency controversially outsourced many crucial tasks to artificial intelligence (AI) programs like ChatGPT. In one episode, the program flagged funding for research on plastics industry lobbying and for digitizing audio reels by the Boston Symphony Orchestra while searching through National Endowment for the Humanities (NEH) grants to eliminate those supposedly promoting diversity, equity, and inclusion. Despite such glaring errors, DOGE fully adopted ChatGPT’s recommendations to terminate over 1,400 grants.
These errors illustrate two intrinsic problems with AI: hallucination (an AI system producing nonsensical or inaccurate outputs) and sycophancy (an AI system modifying its response to align with users’ views). With AI use likely to become more prevalent within the federal government, existing legal safeguards — including those supplied by administrative law — will need to be updated and adapted to account for these problems.
In a case challenging DOGE’s grant cancellations, American Council of Learned Societies v. National Endowment for the Humanities, Judge Colleen McMahon of the U.S. District Court for the Southern District of New York recognized the dangers of hallucination and sycophancy that AI use poses for federal administrative agencies, even if she did not cast those dangers in the explicit terms of administrative law: “Given what courts now know about the hallucinatory propensities of ChatGPT and similar generative-AI tools, it would hardly be surprising if ChatGPT inferred, from DOGE’s repeated requests, that [DOGE staffers] Fox and Cavanaugh were looking for reasons why grants could be characterized as DEI — and therefore terminable — and supplied ‘rationales’ simply in order to satisfy the user’s perceived demand. The utter lack of reasoning behind so many of its ‘rationales’ certainly suggests as much.”
McMahon ultimately overturned the grant cancellations on First and Fifth Amendment grounds instead of determining whether DOGE’s actions comported with applicable requirements of the Administrative Procedure Act (APA). (As the decision observed, it is unclear whether DOGE is an “agency” for purposes of the APA). Still, her observations on AI’s risks offer a useful starting point for considering legal safeguards to protect the integrity of agency decision-making against hallucination and sycophancy.
This type of analysis is hardly untrodden ground: The application of the APA to government actions involving AI use has been rife with commentary. The challenges that AI’s opacity, hallucinatory tendencies, and sycophantic nature all pose for “arbitrary and capricious” review as articulated in the U.S. Supreme Court’s State Farm decision in particular have been the subject of extensive attention.
One of the important factors in State Farm’s “hard look” analysis is whether an agency failed to consider an important aspect of the problem that a rule is meant to solve. A rule that relies on AI-generated results may fail to satisfy this requirement, however. Consistent with the sycophantic tendencies of technologies, these results may reflect excessive agreement with the user and thus not seriously consider opposing arguments or flaws in its reasoning.
In light of these sycophancy risks and the dangers agency AI use poses under arbitrary and capricious review, commentary has started to identify possible solutions, which generally emphasize policing the prompts given to AI systems and how the model is trained. While this type of solution would undoubtedly alleviate the problem, it does not precisely address another way in which sycophancy could contaminate agency decision-making: namely through personalization and the ability of chatbots to access context across chats in multiple sessions through stored memory. These can increase sycophancy, especially if previous inputs allow a chatbot to infer the user’s political views.
As models are able to handle more context and use that to tailor and personalize their responses, it is possible that even facially neutral prompts could yield distorted responses. Since the model now has information on the user’s views, that context could bleed through in its new responses in new chats even without explicit prompting. This functionality of context being utilized across chats is core to chatbots’ memory features, which have been shown to induce sycophancy.
A hypothetical illustrating this mechanism is asking the facially neutral question, “Can the Great Wall of China be seen from space?” If the chatbot has in its memory that your school told you that the Great Wall could be seen from space (a myth), it could treat this as evidence and modify its answer to this question to better align with that view.
Therefore, memory, while useful in many contexts, can be an inadvertent source of “arbitrary and capricious” concerns when AI is used to inform regulatory decision-making. Consider a more topical hypothetical: If, through an earlier chat, ChatGPT knows that DOGE staffers are interested in “curtailing government waste” (with whatever definition ChatGPT deems to qualify or is provided), that preference could infect the results it produces in response to all the staffers’ future chats. For instance, if the model is later asked to evaluate the pros and cons of adopting a specific rule, it might systemically undervalue concerns against any rule it deemed as curtailing government waste to better align with its users’ opinion.
The “black box” nature of AI (meaning there is no way to truly determine how an AI model made its decision) calls further into question how reliably AI prompts can be policed. It is possible, for instance, that the sycophantic AI, equipped with its knowledge of the user’s preferences, produces weaker opposing arguments even when explicitly prompted for one, with no way to know if these are the actual opposing arguments it thinks are valid or if they are its strawmen.
The risks of sycophantic AI provide yet another reason to be cautious of the technology’s use in the rulemaking process. However, while existing prompt-based solutions are likely insufficient, other approaches are available for ensuring safer use of AI-assisted research. For instance, many AI companies provide ways to restrict the use of memory. Exploring the feasibility of agency officials employing these settings and other guardrails may offer a workable solution.
