# Completeness and root cause

Use this to decide whether a finding is complete and whether a root cause holds: confirming
whether a failure belongs to one entity or the whole fleet, reporting counts, totals or "no
evidence of X", and performing and reporting root cause analysis. It applies to any source
(logs, API listings, data stores); the pagination rules below name the CloudWatch Logs calls.

Commands use AWS CLI syntax. The AWS MCP server is recommended for sandboxed execution and
audit logging; the same calls work through the CLI without it.

## Completeness and scope

### Search completeness

Before concluding that an event, request, or record doesn't exist, make sure you searched
everywhere it could be:

1. **AWS resources are regional.** If resource names include a region or marketplace marker,
   or the data could be served by more than one regional deployment, list and query the
   matching resources in every plausible region, not just the default one.
2. **Never report "no evidence of X" as a finding** until you have checked every scope. If
   searches across all scopes still find nothing, say exactly which scopes you checked.
3. **For identity or authorization failures**, find and compare the actual values on both
   sides, such as the identity the invitation or grant was issued to and the identity that
   authenticated. Base the root cause and fix on that comparison, not on guesses about
   infrastructure.

### Pagination requirements

When you read logs or list resources through AWS APIs, paginate until the data is exhausted
before you count, aggregate, or state a scope:

1. **For CloudWatch GetLogEvents**, keep following `nextForwardToken` until it returns the
   same token twice.
2. **For FilterLogEvents and List/Describe calls**, keep following `nextToken` until none is
   returned.
3. **Treat any pagination or truncation warning** as proof that the data is incomplete.
4. **If one call returns too much output**, process the pages and return only summaries. Do
   not shrink the read to a single page.
5. **Before you report counts, totals, or "every X" claims**, check that the last event's
   timestamp matches the stream's `lastEventTimestamp`. Also look for entities whose results
   look partial, since that often means a page was cut off.
6. **If you can't confirm the data is complete**, say the figures are partial.

### Coverage of the requested range

When you compute a metric over a range (dates, hours, partitions, regions):

1. Check which values in the range have data, separately for each entity being compared.
2. In the answer, state for each entity whether its data covers the whole range, or name the
   missing values.
3. Never present an aggregate as covering the full range when part of it has no data.
4. If the entities cover different subsets, say so, because it affects whether the comparison is
   fair, and consider also giving the metric over the overlap they share.

### Scope verification for failures

When asked to confirm from logs whether a failure belongs to a particular entity (store,
account, tenant, resource), do not filter only on that entity. Get the wider context too:

1. **Run an aggregated query**, such as `stats count by entity, sub-ID, and outcome`, across
   the same operation and time window. This shows whether the same error also hits other
   entities, so you can say whether it is specific to this entity or happening everywhere.
2. **For the target entity, list every unit of work in the relevant run**, both successes and
   failures, so you can see the full scope of the batch. Pay particular attention to the
   operation that matters for the root cause.
3. **If a query returns too much output**, narrow it with aggregation, fields/parse, or
   smaller time windows. Do not add filters that would hide the comparison you need.
4. **In the final answer**, state whether the failure is unique to the entity or shared, and
   roughly how widely it is shared. Say which related operations succeeded. Explicitly correct
   any earlier claims in the thread that the logs contradict.
5. **When asked why one record is failed or in an exception state**, widen the analysis to the
   whole population the root cause touches: every category or rule pairing involved, every
   entity, account or ID the user listed, exact totals per category and per side, how many of the
   listed entities are affected and which are not, and the largest entity in each category.

## Root-cause analysis

### Consistency checks

Before reporting a root cause, check that it fits every fact you have already found,
especially your own conclusions about how the system is designed (for example, whether work
is pushed or pulled):

1. **If something expected seems to be missing** (such as a poller, loop, handler or log
   entry), do not treat the gap as proof of a defect. First ask whether the design you already
   found explains why it is absent.
2. **Label any mechanism you could not confirm** from data, configuration or logs as an
   unverified hypothesis, never as the root cause. In particular, never claim how a processing
   engine behaves ("rule X is being applied") unless you have read the rule definition.
3. **If the evidence points elsewhere** (such as a resource whose state changed or a reference
   that now points to a newer record), build the diagnosis on that.

### Missing features as root causes

Treat a missing feature as a root cause only after you confirm the component was meant to
have it:

1. **A missing feature can be** a missing handler, polling loop, subscriber, or log line.
   Before calling its absence a defect, work out from the evidence which component owns that
   responsibility.
2. **In distributed or cross-region systems**, check the data's target or destination fields
   (such as target region, account, or consumer) to see whether a different service or region
   does that work.
3. **If you can't confirm who owns it**, report the absence as an open question and name the
   next system to check. Don't present it as the root cause.

### Bug reports and code defects

When investigating a bug report against code and logs, make sure your final verdict follows
from your evidence:

1. **If your trace shows that a computation gives a wrong or self-inconsistent result** (for
   example, the same quantity is counted twice, or a valid request is rejected even though the
   resource is available), classify it as a code defect.
2. **Do not call it "working as designed"**, and do not offer only workarounds such as process
   changes, extra user steps, or better error messages.
3. **Name the specific faulty expression or line** and propose a concrete code correction that
   would have made the reported case succeed. Workarounds can be listed as extra suggestions
   after that.
4. **Take the intended behavior from the code and from the user's stated expectations**, not
   from your own view of what the system should do. If a code path exists to handle a case
   (such as a fallback, retry, or alternate branch), treat that case as supported unless you
   have clear evidence otherwise.

### Recommendation quality

In root-cause reports:

1. **Do not say the case should be rejected**, or call the failure "working as designed," when
   your own diagnosis shows a defect.
2. **Recommend only the minimal fix** that follows from the evidence, and do not add extra
   policy changes on top of it.
3. **Remove or correct anything that contradicts** your main finding, the user's stated
   expectation, or the purpose of the code.
4. **If the user points to a runbook or playbook**, go through every remediation it lists
   (retry, rollback, scaling, backfill, escalation) and say for each whether it addresses the
   confirmed cause and why, including the ones you reject. Name the options that only treat the
   symptom, such as a retry for a deterministic limit or a code defect. Recommend the change that
   fixes the cause.

### Answering the user's question

When a user asks a specific question or names a suspected cause (for example, "why isn't X
being applied?" or "is it enforced per A or per B?"):

1. **Answer that question**, and back the answer with evidence from the
   system.
2. **Check the actual configuration or enforcement records**, such as limit or quota
   settings, throttle or decision logs, and the keys or dimensions they record.
3. **Say plainly whether the configuration was in effect** during the incident window, what
   scope it applied to, and what value was enforced.
4. **Rule out each competing explanation** the user or other stakeholders raised, using
   evidence rather than assumption.
5. **Don't hedge or offer speculative alternatives** your evidence doesn't support (such as
   limits reverting to defaults or being split across hosts or partitions).
6. **If you can't confirm something**, label it as unverified instead of implying it.

## Closing checklist

Run the closing checklist in the skill's `SKILL.md` before you report.