The debate over how to regulate artificial intelligence has produced a stark new warning from researchers: partial or weak regulation may be more dangerous than doing nothing at all. A study published Monday in the Proceedings of the National Academy of Sciences argues that poorly designed AI safety rules can create incentives that make AI products riskier, while also giving the public a false sense of security.
Using theoretical economics and game theory, a team of researchers from Cornell University and Carnegie Mellon University built a model to explore how AI regulation can most effectively ensure safety. Their central finding is that regulation must be strict and must target the entire supply chain, especially the companies that develop general-purpose AI models such as OpenAI, Google, and Anthropic. Focusing only on downstream companies that apply AI in real-world settings, like medical diagnostic systems or e-commerce customer service chatbots, can backfire and reduce overall safety.
What the researchers found
The researchers discovered a classic free-rider problem. When the government regulates downstream companies but leaves AI model developers largely unregulated, the developers have little incentive to invest heavily in safety measures such as third-party audits, red-teaming, or robust alignment research. Instead, they offload safety responsibilities onto the downstream specialists, who may lack the technical expertise or leverage to fully assess and mitigate risks embedded in a general-purpose model.
Benjamin Laufer, the study's principal author, described the behavior as "free-riding." He said, "The regulation acts as a tool for the general provider to offload the safety burden onto the downstream specialist." In other words, a developer could point to downstream regulation and argue that safety is someone else's job, even though the developer controls the model's architecture, training data, and core capabilities.
The result is a product that may pass regulatory checkpoints but is actually less safe than an AI product created without any regulation. Under a no-regulation scenario, developers know they bear full responsibility for their model's safety and may act more cautiously. Under weak regulation, they can shift blame and cut corners in ways that are hard for downstream companies or regulators to detect.
The problem with regulating only downstream applications
At first glance, focusing on specific AI use cases seems logical. Governments often regulate products based on how they affect consumers. For example, an AI-powered medical diagnosis tool should meet certain accuracy standards because a mistake could harm a patient. Similarly, an AI chatbot used in customer service should not be allowed to dish out harmful or deceptive information.
But the study argues that this approach overlooks how AI is actually built. General-purpose AI models are not static products; they are platforms that can be fine-tuned and adapted for many different uses. The same model that powers a medical assistant could also be used for legal research, financial analysis, or creative writing. If regulators fixate on the final use, they miss the enormous influence that model developers exert upstream. A developer can choose to reduce safety investments, knowing that downstream regulation will hold the end user accountable rather than the creator.
The authors describe this as a failure of regulatory design. When rules target the last mile of the AI supply chain, they create a moral hazard. The party best positioned to improve safety is the one with the least incentive to do so. Meanwhile, the parties with the strongest legal obligation to ensure safety have the least control over the model's fundamental properties.
Free-riding: the core failure
The study frames the problem as a coordination failure. General-purpose AI developers and downstream specialists both make safety decisions, but their incentives are not aligned. In the absence of strong regulation, each side may assume the other will invest in safety. The developer assumes the downstream company will test, filter, and guard against harmful outputs. The downstream company assumes the developer has already made the model safe.
This mutual assumption leads to underinvestment on both sides. The researchers call this a "race to the bottom" in safety, where each player tries to minimize their own costs while relying on the other to carry the safety burden. In a market where speed and cost efficiency are rewarded, this dynamic is especially dangerous. Cutting corners on safety can bring products to market faster and cheaper, giving irresponsible companies a competitive advantage.
The prisoner's dilemma is at the heart of this dynamic. In game theory, the prisoner's dilemma describes a situation where two rational actors, unable to trust each other, choose to betray rather than cooperate, even though cooperation would produce the best collective outcome. For AI companies, the choice is between cooperating on safety and defecting to save money. Without binding regulation, defecting is often the rational choice for an individual company, even though it leads to worse outcomes for everyone.
A prisoner's dilemma for AI companies
The researchers explain that AI safety is a classic prisoner's dilemma. If both the general-purpose developer and the downstream specialist invest heavily in safety, the end product is robust and both players benefit from a good reputation and lower liability. But if one side invests while the other doesn't, the investor bears the cost while the free-rider enjoys the benefit. As a result, companies may choose to defect—reducing safety investments—to avoid being exploited.
Strict regulation changes the game. When regulators set clear safety standards for both developers and downstream companies, cooperation becomes the rational choice. Companies no longer fear that their competitors will undercut them on safety, because the rules apply to everyone. This creates a "sweet spot" where regulation improves safety and economic utility for all players. The researchers define utility as revenue share minus investment cost, and they argue that stronger, well-placed regulation can mutually benefit everyone in the supply chain.
The broader regulatory debate
The study arrives at a moment of intense disagreement over AI governance. In the United States, two main camps have formed. One camp opposes strict regulation, arguing that AI innovation should not be hindered by unnecessary guardrails. This group often claims that the United States can only win the global AI race against China by allowing companies to move quickly and freely. They sometimes dismiss supporters of stricter regulation as "doomers" or accuse them of attempting regulatory capture.
The other camp insists that under-regulated AI development poses serious risks, from AI "psychosis" and biased decisions to the health consequences of data centers and a feared unemployment crisis. These advocates argue that the AI industry, driven by profit margins, underestimates or undersells the dangers of moving too fast.
But the authors of the new study say this is a false choice. Safety and revenue do not have to be an either-or proposition. Their model shows that well-designed regulation can improve both safety and economic returns by fostering trust and cooperation across the supply chain. The key is to set meaningful safety standards and enforce them uniformly.
What this means for policy
The findings have practical implications for regulators in the United States and around the world. Policymakers should avoid narrow rules that focus only on consumer-facing applications. Instead, they should consider the entire AI supply chain, including the developers of foundational models, the companies that fine-tune models for specific tasks, and the businesses that deploy them in real-world settings.
Regulation should include clear safety expectations for model developers, such as mandatory risk assessments, third-party audits, and robust documentation of training data and evaluation methods. Downstream companies should also be required to conduct their own testing and monitoring, but they should not be treated as a substitute for upstream accountability. The goal is to create a system where safety is a shared responsibility, not a disposable cost.
Laufer emphasized the need for a holistic view. "People think of AI as a single object, but actually AI involves a very complicated set of stakeholders and actors that each have their own contributions to the technology," he said. "To regulate in a thoughtful way, we need to consider the whole supply chain, not just a single provider or entity."
Source: Gizmodo News