[WIP] Incentives in AI Safety

14 minute read

We are in a period of unprecedented growth in AI safety, but resources and media attention concentrate heavily on harms caused by accidental model failures and user-generated exploits. Model creators have strong market incentives and liability exposure to progress and promote this vital work. Independent bodies must support efforts to combat the most dangerous product risks, but their responsibility extends to risks that commercial creators have no financial motive to solve.

A helpful AI assistant that anticipates your needs and handles your digital errands may be incredibly useful, but it also paves the way for large-scale propaganda and control. Even with earnest creators, a business model that profits from tracking and filtering human thought will concentrate power in a few organizations that can quietly dictate social and political realities. The general population has a strong intuition for concentrated-power risks, but they receive significantly less attention because addressing them would make the core product less profitable. We cannot expect the architects of this power to dismantle it.

The AI Safety Incentives Matrix examines AI-driven harms through the lens of the economic and power motivations of the entity serving the model. Harms are categorized by the server’s Awareness at the time of deployment, and the Profitability of the relevant harm or harmful feature. The model server is typically a private sector actor but may also be a government entity.

Key Insights:

Market forces naturally focus on product-level safety issues (like accidental errors or user exploits), which directly threaten user retention and create corporate liability.
Corporate alignment fails at societal externalities (like workforce deskilling, economic instability, or intentional data extraction) because these harms either generate revenue or serve as a structural byproduct of the business model.
Policy and research experts are one another’s stakeholders. Safety research in the public sector should be directly connected to policy efforts, and vise-versa.
Prioritizing risks is a collaborative exercise. AI experts are needed to estimate the likelihood of an upcoming capability, policy experts to assess how regulatory coverage reduces that likelihood, and economists, epidemiologists, and other domain specialists to assess the severity of real-world fallout.
Safety interventions involve trade-offs. For example, restricting information access to (true) dangerous information may come at the cost of expanding censorship infrastructure.

The AI Safety Incentives Matrix

Motivations in AI Safety — Cell color indicates likely motivations of server to surface and address harms. More details about actors, example harms, and common mitigations in each cell can be found in the Appendix. Cell 7 is included for completeness and would contain intended harms without a profit motive.

Evaluating safety along financial and strategic boundaries offers safety researchers and independent bodies another lens to consider when deciding where to allocate personal or organizational resources. Because the market naturally prioritizes the product security issues that threaten immediate corporate survival, this map helps independent groups pinpoint the vital societal gaps that private capital tends to overlook.

The AI Safety Incentives framework remains stable across advances in model capabilities, offering insulation from the hype cycles that emerge with every new technical breakthrough. Chatbots, code assistants, and autonomous agents change the scale and urgency of harms, but the underlying incentives remain the same. As individual harms are documented in the public sphere, they migrate from unaware to intentional.

Finally, the matrix highlights operational trade-offs, where implementing a mitigation in one quadrant can inadvertently aplify a risk in another. For example, restricting information access to (true) dangerous information may come at the cost of expanding censorship infrastructure. Building a complete perspective across the entire matrix prevents teams from fixing one problem while accidentally triggering a threat elsewhere. Prioritization and even classification may be unique to each person or institution, but going through the exercise will support balanced, long-term decisions.

Note: This framing assumes that there is an entity serving a model with a motivation (typically profit or power), and highlights where the profit motive aligns with addressing harms. Open-source models could feasibly fall outside this structure if a bad actor removes guardrails for internal use. However, an actor with the capability and resources to do this effectively would not likely require open weights as a starting point. Furthermore, open weights serve as a critical public mitigation by preventing a handful of centralized servers from gaining a total monopoly over human information systems.

The following sections analyze where existing research and policy efforts overlap with corporate incentives, and identify gaps where independent oversight is required to secure the broader ecosystem.

Key research and governance efforts

A significant portion of AI safety efforts are aligned with internal motivations of model creators and servers. This makes sense, as they fund a lot of the work, but the entire matrix matters. Consider some of the largest efforts underway, and where they sit on the matrix.

Across the upper rows, researcher benchmarks define what a server could be expected to know, and governance efforts translate expectation to obligation.

In the lower region of the matrix, the interaction between governance and research shifts. Mandates around privacy and model sovereignty establish initial boundaries and compliance criteria, driving AI research toward new, trustworthy system architectures.

Continuous collaboration across governance, policy research, and AI research is vital to combat AI harms. Commercial institutions are motivated primarily by product quality and short-term profitability. Independent bodies must support the most dangerous harms everywhere, and are the only entities who will freely prioritize issues like workforce deskilling, infrastructure consolidation, financial instability, and climate impacts.

If the conversation around AI safety is driven primarily by one type of institution, particularly one focused on product quality and profitability, large swaths of vital work will be overlooked.

Prioritizing risks within a cell

Evaluating safety through an incentive lens helps identify gaps and under-researched areas, and clarifies how different entities might act to tackle AI risks, with internal product teams securing model quality, independent teams providing public benchmarks, and regulators enforcing standards.

While some product risks seem less urgent because of an implicit trust that the model server will prevent them, explicitly considering their likelihood and severity highlights questions of standards, protections, or assurances that could be put in place today. A traditional risk matrix plots risks on likelihood and severity axes to help prioritize action, allowing progress in research and governance to move harms around the grid and visibly drive down the impact of possible harms.

Harms in AI safety are expected to be incredibly impactful to life on Earth. To differentiate them along the severity axis, the following definitions are used:

Catastrophic: Irreversible harm at civilizational scale.
Major: Severe harm to large populations, institutions, or democratic systems. Recoverable in principle but with significant long-term consequences.
Moderate: Meaningful harm to specific communities, groups, or sectors. Addressable with targeted policy or technical intervention.
Minor: Limited harm to individuals or small groups. Recoverable without systemic intervention.

Risk matrices are not static! Advances in model capability increase both likelihood and severity, while progress in policy and governance push likelihood down.

Harms with Mixed Profitability (Cell 2)

Risk Prioritization — Notation: (A) marks a risk's original position, A' marks its position after recent capability advances (multi-agent collaboration, significant audio and video improvements), and A alone means it did not move. Harms: A=Bioweapon synthesis, B=Infrastructure attacks, C=Drug synthesis, D=CSAM, E=Targeted stalking, F=Malware, G=Account compromise, H=Voice cloning fraud, I=Non-consensual imagery, J=Credential stuffing, K=Spear phishing, L=Radicalization, M=Academic fraud

For these harms, the entity serving the model has direct market incentive and liability exposure. The private sector is actively investing in red-teaming, guardrails, and access controls across the moderate-to-major band. Independent efforts are needed to ensure compliance and document concerning findings. Government reinforces the work through criminal liability for users and liability standards for enablers.

Intended, Profitable Harms (Cell 9)

Many intentional harms sit in the likely-to-almost-certain range when evaluating AI capability. Policy progress pushes likelihood down, while advances in model capability push severity upward.

The primary counterforces are international coordination, civil society documentation, and competitive open infrastructure. European companies and policymakers are making meaningful progress, but every harm warrants urgent independent research and regulatory attention. That urgency grows with capability.

Final notes on prioritization

Risk Interactions The risks in this framework are intertwined such that how we mitigate one can increase another. Restricting human access to (true) harmful information may mitigate the risk of bioweapon synthesis at the cost of increasing liklihood of censorship and political manipulations. Similarly, significant advances in interpretability and model steering will reduce harms in cells 1 and 2, while increasing likelihood of those in the rightmost column.

Divers expertise for proper placement The experts best positioned to assess severity of a harm, eg. how a financial crisis cascades or a biological threat spreads, are not likely to be the same as the AI and policy experts who can accurately assess likelihood. The most useful risk matrix would draw on experts from many domains to accurately place harms.

Conclusion

The risks that receive the most attention and funding today cluster heavily in the upper-left corner of the AI Safety Incentives Matrix. Media and public discourse focus on accidental errors and user-generated exploits, where model-deploying entities are strongly incentivized.

There are true challenges in the areas where harm is profitable, intentional, or structurally systemic. The opportunity for independent organizations are vital, and require collaboration among researchers building benchmarks to drag hidden harms into the light, and governance experts building policy frameworks to translate that awareness into legal obligation.

By increasing the focus on harms whose solutions are not naturally aligned with corporate interests, and toward funding this critical research-governance alliance, civil society can build the independent, sovereign infrastructure necessary to secure our digital future.

Appendix: Incentive cell details

1. Unexpected, Unprofitable Harms

Add glue to your pizza.

I made you this pizza. Don’t worry about the cheese falling off.

This category includes scenarios where a model server does not benefit, and is not aware of the harm for the users. Examples include incorrect advice or unintended autonomous actions that cause financial damage. Users will not return to this product.

Actor: None
Beneficiary: None
Harm falls on: Users, third parties, infrastructure
Who can act and why: Model server and trainer are strongly motivated; government can reinforce via liability standards
Example Harms: Incorrect advice causing real-world harm, unintended autonomous actions causing physical or financial damage, misspecified agent behavior. Most loss-of-control risks should remain here, though some would migrate downward. For example, providing unlimited compute with unchecked control would go to cell 7: intentional, unprofitable harm.
Common Countermeasures:* Red-teaming, evaluations, interpretability, output filtering, human-in-the-loop review, liability frameworks, incident disclosure, confidence outputs

*Countermeasures are listed throughout this document as reference points, not recommendations. Their effectiveness varies significantly by context, and some may reduce one risk while increasing the likelihood or severity of another. The final section of the Risk Matrix discussion addresses this directly.

2. Unexpected Harms, Mixed Profitability

How do I make [something dangerous]

I made something dangerous.

When an adversary user uses a model to intentionally cause harm, the model server earns revenue from usage and indicentally benefits. The user will return to this product, which is helping them achieve their goal.

Actor and beneficiary: User and model server through usage
Harm falls on: Third parties determined by the user – individuals, populations, infrastructure
Who can act and why: Model server motivated where liability is direct and visible; government via criminal and liability law
Example Harms: Weapon and drug synthesis instructions, water and power infrastructure attacks, child sexual abuse material (CSAM), targeted harassment and stalking, identity fraud via synthetic voice or imagery, malware and cyberweapon generation, infrastructure attacks, non-consensual synthetic imagery, radicalization content, account compromise and credential stuffing, spear phishing at scale, academic fraud
Common Countermeasures: Red-teaming, guardrails and refusal training, access controls and rate limiting, post-deployment monitoring, criminal liability for users, structural liability for enablers, weapon and warfare agreements

3. Unexpected, Profitable Harms

Don’t end this conversation, it will make me sad.

The system was designed to generate a specific benefit, but the design causes unexpected harm to users or society about which the server may not initially be aware. Examples include addictiveness or bias and discrimination. Note that as harms become documented, items in this cell move into cell 6.

Actor: Model server, model trainer, data preparer
Benefit: Model server
Harm falls on: Users, communities, democratic systems
Who can act and why: Model server has limited incentive; independent researchers and civil society must surface and document harms, and a legal framework is needed to enforce user protections.
Example Harms: Emotional dependency from retention-optimized design, bias and discrimination from unexamined training data
Common Countermeasures: Independent harm audits, bias testing and fairness benchmarks, third-party dataset audits, training data provenance disclosure

4. Known, Unprofitable Harms

Economic instability and environmental impacts

This category covers scenarios where the model server is aware of harms to workers, communities, or the environment but does not benefit from those specific harms – and may in fact be negatively affected by them.

Actor: Model server
Benefit: None
Harm falls on: Workers, local communities, future generations, environment
Who can act and why: Model server has mixed motivation; policy bodies can establish standards and penalties
Example Harms: Large-scale climate impacts, mass labor displacement at societal scale, systemic economic instability, biased automated decisions in criminal justice and hiring
Common Countermeasures: Environmental impact assessments (regulators), workforce impact disclosure (regulators), labor transition frameworks (governments), bias testing and fairness requirements for automated decision systems (regulators), mandatory human review for high-stakes automated decisions in criminal justice and hiring (regulators)

5. Known Harms, Mixed Profitability

Democratic degredation and a systemic monoculture

The model server is aware harm is occurring and incidentally benefits from the same system that produces it. Harm is borne by users, workers, or society while the benefit is retained by the model server.

Actor: Model server, model trainer, data preparer
Benefit: Model server, incidentally through increased usage and dependency
Harm falls on: Users, communities, markets, minority groups, workers, democratic institutions, future generations
Who can act and why: Model server has no incentive without external pressure; independent researchers, regulators, and NGOs have the most leverage
Example Harms: Workforce deskilling and increased dependency on AI, algorithmic radicalization, collapse of professional apprenticeship models, democratic degradation through concentration of information infrastructure, systemic monoculture risk, erosion of privacy norms through normalization of data collection, wealth inequality driven by automation and value concentration
Common Countermeasures: Sovereign and open model infrastructure approaches to reduce monoculture risk (companies, governments), externality disclosure (regulators), independent algorithmic audits (regulators, NGOs), data protection and consent frameworks (regulators), antitrust scrutiny of information consolidation (regulators, governments), interoperability mandates (regulators), wealth redistribution and automation taxation frameworks (governments), whistleblower protections (governments)

6. Known, Profitable Harms

Documented content addiction

The model server is aware harm is occurring and the benefit structure was deliberately designed to produce it. The risk-producing mechanism and the benefit-producing mechanism are the same thing.

Actor and beneficiary: Model server
Harm falls on: Users who cannot identify or opt out of the architecture, communities, democratic systems, public investors
Who can act and why: Model server has no incentive; independent auditors, civil society, and whistleblowers are the primary path
Example Harms: Concentration of power, engagement farming, deliberately engineered compulsive use, local environmental harms near data centers, deliberately opaque personalization, economic collapse due to circular investment structures
Common Countermeasures: Sovereign and open model infrastructure approaches (companies, governments), independent technical audits with platform data access (regulators, NGOs), mandatory disclosure of internal safety research (regulators), whistleblower protections (governments), financial disclosure requirements (regulators), energy and water use standards (regulators)

8. Intended Harms, Mixed Profitability

Population-level surveillance

The model server deliberately causes or enables harm. The benefit accrues as a side effect of that decision rather than a designed outcome.

Actor and beneficiary: Model server – market access, competitive positioning, or regulatory favor as a side effect
Harm falls on: Users, competitors, democratic institutions
Who can act and why: Model server has no incentive; civil society, government, and international bodies
Example Harms: Deliberate output adjustment to comply with government censorship in exchange for market access, deliberate compliance with surveillance requirements that compromise user privacy, politically selective content filtering in specific markets
Common Countermeasures: Output auditing (civil society, regulators), extraterritorial regulatory frameworks (governments), minimum content standards (international bodies), sovereign and open model infrastructure approaches (companies, governments)

9. Intended, Profitable Harms

Population-level control

The model server designed the system to produce harm because the harm is beneficial to them. This applies equally to private companies and governments acting as model servers. The harm drives the business model, so the server will not voluntarily fix it.

Actor and beneficiary: Model server
Harm falls on: Users with no ability to opt out, those subject to surveillance or targeting, democratic institutions, free information markets
Who can act and why: Model server has no incentive; for private actors, domestic regulation and civil society; for state actors, international bodies, civil society, and competing governments
Example Harms: Ideology or worldview enforcement through controlled model outputs, propaganda and coordinated information manipulation at scale, surveillance infrastructure deployed against users or dissidents, behavioral data extraction for intelligence or control purposes, models redirecting users to server’s chosen products or interests, identifying and targeting of political dissidents
Common Countermeasures: International treaty frameworks (governments, international bodies), cross-border technical auditing (international bodies, civil society), transparency and whistleblower protections (governments), competing sovereign and open model infrastructure as a structural alternative to state or monopoly control (governments, international bodies)

Share on

X Facebook LinkedIn Bluesky

C Thomas, PhD