The Limit May Not Be in the Model

Revised October 9, 2026.

Four Questions, Again

I asked an unnamed console-style AI product for a longer, iterative interview. It came back with four clarifying questions. Fair enough. I answered them and asked it to continue.

Then it gave me four questions again.

That was the part I could not stop thinking about. Four is not obviously a bad number, and I have no access to the product's internal implementation. But my request for a longer exchange did not seem to change the shape of the interaction.

Maybe the model could have handled more. Maybe it could not. The evidence available to me does not establish why the product returned to four questions. It only establishes the repeated behavior I saw.

Users experience model capability through a product. A limit in an interface can feel like a limit in the model, even when the product is the layer applying the boundary.

I did not come away thinking that four questions was wrong. I came away wondering who owned the number. Was it selected deliberately? Is there a current evaluation behind it? What evidence would cause a review? Or has it survived model changes because nobody reopened the decision?

That is the engineering question. An inherited default can constrain how capability reaches a user without telling the user, or perhaps even the team, where that constraint came from.

The Model and the Product Move on Different Clocks

Model capability can change meaningfully on a particular task. As one example, Anthropic reported that Claude Sonnet 4.5 scored 61.4% on OSWorld, a computer-use benchmark, compared with 42.2% for Sonnet 4 four months earlier.

That is a vendor-reported result from one named evaluation. It does not show that every newer model is better at every task, and it does not establish a universal curve for model progress. It does show why a default deserves reconsideration when the capability assumptions behind it may have changed.

Product behavior has its own clock. A team can swap in a newer model, confirm that the existing flow still operates, and ship. That compatibility work is necessary. It does not answer whether the inherited flow remains the best way to expose the model's capability to users.

A model upgrade can be real while the experience remains tied to assumptions made for an earlier release. The gap becomes a governance problem when no one owns the decision to inspect it.

More interaction is not automatically better. It can add latency and cost, complicate compatibility, produce different quality outcomes, or create safety concerns. Those tradeoffs belong in the review once, where they can shape a decision, instead of appearing as a repeated defense of leaving every inherited default alone.

Where Four Can Become Embedded

A question count sounds like a single setting. In a product, it can become part of everything around the model.

The interface may be designed for a predictable number of prompts. State handling may assume a short exchange. Evaluation coverage may stop at the existing behavior, leaving longer exchanges untested. Latency and cost budgets may reflect the current path. Client integrations may rely on its shape. Release confidence may depend on a flow that teams already understand.

Any of those could help preserve a four-question default. None of them is an established cause of the interaction I saw.

Once a default appears in interface copy, orchestration, tests, budgets, and rollout plans, it is no longer a small implementation detail. It functions as part of the product contract: the set of product-level choices that determines how model capability is presented, measured, and changed.

When that contract is invisible, inheritance can be mistaken for intent.

Review the Question-Count Contract

The useful alternative is to make the default reviewable.

For the interview flow, start by treating the question-count rule as versioned configuration. Every served interaction should be traceable to the contract version that selected its behavior. Google's monitoring guidance recommends observing binary versions, feature-related flags, and dynamic-configuration versions so production behavior can be connected to intended changes. Applied here, that means telemetry can distinguish what happened under one contract version from what happened under another.

The contract also needs one accountable owner. That could be the product or engineering owner responsible for the interview flow, but it cannot remain implicit across several teams. The owner does not need authority to raise a number by instinct. They need responsibility for bringing the decision, evidence, and next review together.

Consider a hypothetical alternative to the fixed four-question flow: continue the interview until the information required for its next step has been collected, while letting the user stop and keeping an explicit interaction budget. This is a candidate to evaluate, not a claim about what the unnamed product can do or a recommendation to remove every limit.

OpenAI's evaluation guidance distinguishes industry benchmarks from application-specific tests and recommends scoped evaluations that reflect real-world distributions. Here, both contracts would face the same representative interview tasks. The owner would assess whether each collected the necessary information and whether the next step was useful, alongside completion, abandonment, latency, cost, compatibility, and safety.

Suppose the adaptive candidate merely asks more questions without improving the information available for the next step. The owner would retain four because extra interaction would have no demonstrated benefit. A shorter fixed flow could also be worth testing if it preserved the needed information. If the adaptive candidate instead improved task outcomes in the evaluation without breaching the product's agreed constraints, that would justify a limited canary, not a conclusion that it should replace the default everywhere.

No such comparison has been supplied for the interaction I saw. These conditional decisions show what the contract owner would need to learn. A higher completion rate alone could hide poorer information; better answers among finishers alone could hide unacceptable abandonment. The decision has to account for both the result and what it took to obtain it.

If a candidate behavior merits production testing, release it through a canary against a control. Google's canarying guidance describes limited exposure, comparison with a control, and using attributable evidence to decide whether to proceed, pause, or roll back. The important connection is attribution: compare outcomes for the populations served by each contract version instead of treating an aggregate shift as proof that the default caused it.

The number can remain simple. The decision around it should be inspectable.

Close the Release Loop

A review that occurs only when someone remembers it is not much of a contract. The next review needs a trigger.

Under this proposed operating model, a material model release triggers task-specific evaluation and owner review. It does not automatically trigger a larger limit. The owner evaluates candidate behaviors, decides whether a canary is justified, and reads version-linked telemetry against a control. The evidence informs the current decision and records what should be revisited later.

Observed product behavior can trigger the same loop. A repeated interaction pattern, an unusual abandonment change, a quality regression, or a latency shift may indicate that the current contract deserves inspection before another model release arrives.

OpenAI's evaluation guidance describes evaluation as continuous work as an application changes. That supports the broader discipline. The specific release and behavior triggers here are a product recommendation: define them in the contract so capability changes do not rely on memory or informal concern to become visible.

The trigger starts a review. It does not decide its outcome.

An Upgrade Is More Than a Model Swap

Under this proposed operating model, shipping a stronger model without reviewing the product contract that exposes its capability is an incomplete upgrade.

Before calling the next model upgrade complete, select one consequential default. Identify its accountable owner. Define the task-specific evaluation that could change it. Name the evidence the owner must weigh. Then name the material release or observed behavior that schedules the next review.

The result may be to retain the default. It may be to lower it, raise it, or adapt it. What changes is that the limit becomes a current, testable, owned decision instead of a number quietly inherited from an earlier product.