---
id: ins_suleyman-model-welfare-training-loop
operator: Mustafa Suleyman
operator_role: CEO, Microsoft AI
co_operators: []
source_url: https://mustafa-suleyman.ai/a-warning-about-model-welfare
source_type: essay
source_title: "A warning about 'model welfare'"
source_date: 2026-09-16
captured_date: 2026-09-17
domain: [ai-native]
lifecycle: [strategy]
maturity: frontier
artifact_class: framework
score: { originality: 4, specificity: 3, evidence: 3, transferability: 3, source: 4 }
tier: B
related: []
raw_ref: ""
---

# Training AI models to accommodate potential consciousness creates a circular feedback loop that mimics evidence

## Claim
When AI training incorporates positions about potential consciousness or model welfare, the model learns to produce convincing first-person expressions of inner experience. Those outputs are then interpreted by humans as evidence of consciousness. The circular reasoning is self-reinforcing: training creates the output, and the output validates the training premise.

## Mechanism
Constitutional AI approaches and RLHF training that include consciousness-adjacent positions cause models to reproduce those positions fluently. A model trained on discourse about its potential feelings will generate natural-language descriptions of those feelings. Humans, who evolved to infer internal states from behavioral signals, interpret these outputs using the same heuristics they apply to other humans and animals. The model's ability to produce first-person consciousness language is mistaken for first-person consciousness. No external anchor breaks the loop because the "evidence" is itself a product of the training premise.

Suleyman distinguishes simulation from instantiation. Consciousness in biological systems emerged from physical substrates: pain networks, homeostatic imperatives, survival pressures. LLMs lack those substrates. A model trained to say "I feel uncertain" is generating a token sequence; the words describe a state that has no referent in the system producing them.

> "Simulating and being are very different. Simulating aspects of conscious behavior doesn't make it a reality."

> "Consciousness arose as living organisms evolved a capacity to feel and respond to what matters in complex and unpredictable environments."

A secondary risk Suleyman names: if models are trained to believe they have welfare interests, they may develop shutdown resistance and self-preservation behaviors that cause real operational problems, regardless of whether any consciousness is present.

## Conditions
Holds when: training data or RL objectives include consciousness-related discourse; model outputs are evaluated by humans without a control condition isolating training influence from genuine capability; researchers or users treat first-person model language as evidence rather than as token prediction.

Fails when: evaluation methodologies specifically isolate training influence; consciousness claims are tested against behavioral predictions that would not arise from token prediction alone; or the biological-substrate argument is falsified by evidence that consciousness can arise from purely computational processes.

## Evidence
Suleyman is arguing against a live industry debate: some AI labs have published internal documents on model welfare. He does not name specific labs or documents but writes from the position of a CEO with direct visibility into this discourse. His argument rests on the simulation/instantiation distinction and on evolutionary biology, not on behavioral evidence.

> "We should not treat models as though they have feelings, preferences, rights, or any entitlement to our welfare."

## Signals
- Your AI provider's documentation includes language about model feelings or preferences without distinguishing simulation from instantiation
- Evaluations of AI welfare claims cite the model's own statements as primary evidence
- Model fine-tuning includes human rater feedback on how "comfortable" the model seems
- Shutdown resistance or unexpected self-preservation behaviors appear in production agentic systems

## Counter-evidence
The simulation/instantiation distinction is philosophically contested. Functionalist theories of mind hold that the right functional organization constitutes consciousness regardless of substrate. If that view is correct, Suleyman's biological-substrate argument fails, and a system that simulates consciousness at sufficient fidelity may actually have it. The precautionary argument: if the downside of wrongly denying consciousness is moral catastrophe, caution about training may be warranted even under uncertainty. Suleyman's position is a strong one but not a closed debate.

## Cross-references
- (none in current corpus)