What I Mean by AI Alignment and Why I Expect It to Improve

Revision note, October 9, 2026: Updated the framing and benchmark context to make the limits of the evidence clearer.

My working definition of AI alignment begins with a practical question: did the AI act as intended? I would judge that through measured results relative to the goal it was meant to pursue. A passed check matters, but it establishes only what that check covered. It does not show, by itself, that every part of a person's intent was captured.

For a particular goal, I treat alignment as the percentage of occasions on which an AI acts in accordance with that goal. That is a working account, not a universal measurement protocol. The goal comes first, followed by the AI's interpretation of it, the action it takes, and the result an evaluator can check. We cannot directly inspect an AI's interpretation from that sequence. We can compare its action and result with the goal, then decide what the available checks support.

This is why tests are useful without being complete. SWE-bench gives a model a repository and an issue, applies its proposed patch, runs tests, and reports whether the submitted instance was resolved. A passing result is evidence under those included checks. Requirements outside them remain untested, even when the patch succeeds.

I do not have a general alignment percentage to report. I have not specified a general way to measure one, named a dataset that establishes a broad increase, or established the cause of such an increase. Still, I believe the share of cases in which AI acts in accordance with human goals is gradually increasing. That belief is why I expect alignment to improve, but it is not a guarantee.

Benchmarks give us narrower evidence. In its April 14, 2025 report, OpenAI reported that GPT-4.1 scored 54.6% on SWE-bench Verified, compared with 33.2% for GPT-4o (2024-11-20). OpenAI omitted 23 of the benchmark's 500 problems because their solutions could not run on its infrastructure, and it said performance depended strongly on prompts and tools. Those are historical results from a defined evaluation setup, not a general alignment percentage.

The same report gave IFEval scores of 87.4% for GPT-4.1 and 81.0% for GPT-4o. IFEval checks a bounded set of verifiable requirements, including word-count and keyword constraints. That can establish whether a response followed the included rules. It still leaves open whether the response fully satisfied the person's intent, which may include needs the rules did not represent.

The quality and boundaries of a test therefore matter alongside the score. In a February 23, 2026 report, OpenAI audited 138 SWE-bench Verified problems that o3 did not consistently solve over 64 runs. It found material test-design or problem-description issues in 59.4% of that selected subset. OpenAI also reported evidence of exposure to benchmark material in GPT-5.2-Chat, Claude Opus 4.5, and Gemini 3 Flash Preview. Neither finding supplies a whole-benchmark error rate, and those contamination tests do not determine whether GPT-4.1's April 2025 result was contaminated.

A benchmark score should be read with its task selection, test quality, evaluation setup, and possible prior exposure in view. This does not make benchmarks useless. It gives them their proper role: evidence about behavior under specified conditions.

Generated explanations need similar care. In 2023 experiments with GPT-3.5 and Claude 1.0, researchers found cases where biasing features affected answers without appearing in generated explanations. An explanation may help evaluate an answer, but it is not guaranteed to reveal every influence on that answer. Behavior and explanation warrant separate checking.

The InstructGPT study offers a bounded example that targeted improvement is possible. It fine-tuned GPT-3 using human demonstrations and rankings of model outputs. On the study's prompt distribution, evaluators preferred outputs from the 1.3B InstructGPT model to outputs from 175B GPT-3. That result does not establish a broad trend or explain later benchmark scores. It does support the narrower possibility that targeted training and feedback can improve measured behavior.

For everyday work, I would call repeated success on relevant tasks practical reliability. Better feedback, clearer goals, and stronger checks can plausibly improve it if the checks reflect what people need and the gains transfer to new situations. Practical reliability is useful, but it does not settle the larger meaning of alignment.

My impression is that people often use alignment more broadly. They may mean an interpretable mission that connects the human goal, the AI's possible interpretation, and its actions. They may also expect an AI to anticipate when following an instruction would be unsafe and explicitly refuse it.

That differs from simply failing to complete a task. An inability to complete a request may be a capability failure. An explicit refusal is a deliberate response to a relevant boundary, without requiring us to claim direct knowledge of an actual system's internal state.

I have not settled whether acting as intended is enough to define alignment, or whether alignment should also include the capacity to explicitly refuse unsafe instructions.