# Evaluate a Skill with `comet eval`

This document explains, from the user's point of view, how to evaluate a Skill in the current
release. In normal usage, you do not need to understand pytest, task registry details, profiles,
treatments, or Docker internals. The user-facing entry point is `comet eval`.

## First understand where `comet eval` fits in Comet

`/comet-any` creates or optimizes a Skill, while `comet eval` evaluates whether that Skill can be
discovered by the eval harness, run, and reported correctly. The current `/comet-any` flow also
feeds eval results and project-level preference evidence into publish readiness: `preferenceHash`,
the composition proposal, resolved Skill evidence, and eval evidence must all match the current
draft.

Think about the flow like this:

```text
/comet-any generates a Skill
  -> produces comet/eval.yaml
  -> comet eval --collect performs discovery precheck
  -> comet eval --html performs the real evaluation
  -> /comet-any or comet creator status/next reads the result and continues into readiness / review / publish / distribute
```

`comet eval` does not publish. Publishing is still handled by the Bundle backend behind
`/comet-any`, surfaced to ordinary users as `comet publish`. Eval's job is to provide publish
evidence.

For ordinary users, the recommended path remains:

```text
/comet-any -> comet eval -> comet creator status/next -> comet publish review/approve/run -> comet publish distribute --preview -> comet publish distribute
```

The stable composed Skill Bundle required capability set is
`skills/scripts/rules/hooks/references`, with `scripts/rules/hooks` as the required control plane.
`hooks/*.yaml` only become active after `comet publish distribute` compiles them for the target
platform.

## Recommended path: evaluate a Skill generated by `/comet-any`

After `/comet-any` generates a Skill, first look for:

```text
generated-skill/
  comet/
    eval.yaml
```

Then run these two commands:

```bash
comet eval ./generated-skill/comet/eval.yaml --collect
comet eval ./generated-skill/comet/eval.yaml --html
```

The first command, `--collect`, only confirms whether tasks can be discovered. It is a low-cost
precheck right after generation.

The second command, `--html`, performs the real evaluation and produces a browsable report.
After it passes, `/comet-any` can use the result as publish evidence.

## How eval results enter publish readiness

After `/comet-any` or the backend records eval results, they are merged into publish readiness.
What the user needs to know is just:

1. Results from `comet eval` become evidence for `Publish readiness:`.
2. If current-hash eval evidence is missing, `User next steps:` must point to running
   `comet eval` before publish continues.

The usual sequence is:

```bash
comet eval ./generated-skill/comet/eval.yaml --collect
comet eval ./generated-skill/comet/eval.yaml --html
comet creator next <name> --json
comet publish review <name> --platform <reference-platform> --json
```

`comet creator next` prints only the single recommended user command; `comet publish review` must
directly show `Publish readiness:`, `User next steps:`, `Readiness:`, `Blockers:`, `Warnings:`,
and `Evidence:`.

## Why run `collect` first

`collect` is the cheapest user-facing debugging entry point. It answers:

- is the `comet/eval.yaml` path correct
- can the eval harness read this manifest
- can the tasks referenced by the manifest be discovered
- are the current repo's eval dependency paths usable

It should not start a full model evaluation first, and it should not spend long-running cost
before manifest and discovery work.

## Choose the Local or LangSmith suite

`comet eval` uses the `local` suite by default for day-to-day development and local reports. To sync runs, rubric
feedback, costs, and Claude Code trajectories to LangSmith, select the suite explicitly:

```bash
comet eval ./generated-skill/comet/eval.yaml --suite langsmith --html
```

Both suites reuse the same tasks, treatments, rubric, and manifest. The `langsmith` suite reads
`LANGSMITH_API_KEY`, `LANGSMITH_PROJECT`, and `LANGSMITH_TRACING`, provisions the Claude Code trajectory plugin, and
writes reports under `eval/langsmith/logs/experiments/`. Without `--suite langsmith`, enabling tracing in the
environment does not make the Local runner create a LangSmith experiment.

## What `--html` prints

During execution, the CLI prints a set of execution facts:

- `Eval root`: which `eval/` root was actually used
- `Mode`: `collect` or `run`
- `Suite`: `local` or `langsmith`
- `Target`: whether the target is a manifest or local Skill directory
- `Experiment`: the experiment id for this run
- `Profile`: which profile the run used
- `Task`: which task was run
- `Report path`: where the report was written
- `Report config`: the temporary report config used when `--html` is enabled

`--html` requests both markdown and HTML reports. The report path follows the selected suite:

```text
eval/<suite>/logs/experiments/<experiment-id>/summary.html
```

If CLI output shows the placeholder `<experiment-id>`, use the `Experiment` value printed in the
same output block to resolve the final path.

## How to read the report

Users do not need to read low-level logs line by line. Start with:

- whether evaluation passed
- whether failures come from harness, workflow, task, or model
- whether failures are relevant to the Skill's goal
- whether expected artifacts are missing
- whether the problem is path, manifest, or environment related
- whether token / cost / duration look abnormal

`comet eval` surfaces failure attribution so the report groups failures under harness,
workflow, task, model, and related categories. That attribution helps decide whether the next step
is fixing the Skill, the eval config, or the environment.

Reports also distinguish the raw set from the analysis set. The raw set keeps every run for
auditability; the analysis set is the default source for headline metrics, pass@k/pass^k, cost,
charts, and the verdict. `excluded` usually means API timeout, rate limiting, auth/network
failure, Docker/container failure, or an outer runner timeout; these runs stay visible in the
report but do not affect headline metrics. `flagged` means a harness or task assumption looks
suspicious; the run remains in the analysis set but is called out as a risk. Real Skill, workflow,
model, or validator failures remain `included` and are not filtered away just because they lower
the score.

If the report says `Insufficient clean data` or `Inconclusive due to data quality`, rerun the
affected task/treatment pair or inspect the environment before treating the verdict as final.

## How `/comet-any` uses eval results

From the user's point of view, after `comet eval` finishes you can hand control back to
`/comet-any` or run `comet creator next <name>` to see the single recommended next step.
`/comet-any` merges eval evidence into readiness:

- no eval evidence: cannot publish
- eval failed: cannot publish
- eval evidence points to an old hash: cannot publish
- `.comet/skill-preferences.yaml` changed while in `strict` mode: cannot publish; confirm or
  regenerate first
- eval passed and hash matches: continue into review / publish decisions

Users should not edit Bundle state manually, and they should not write report paths into internal
JSON by hand. `/comet-any` records structured evidence through the Bundle backend.

## How to evaluate when you only have a local Skill directory

If you do not yet have `comet/eval.yaml` and only have a local Skill directory, start with quick
smoke:

```bash
comet eval ./my-skill --quick
```

This is appropriate for early validation:

- whether the Skill directory can be read
- whether the eval harness can inject it as a dynamic Skill
- whether the generic smoke task runs

Current quick smoke uses:

```text
generic-skill-smoke
```

This is only an early smoke path, not final publish evidence. Before publishing, it is still
recommended to let `/comet-any` generate `comet/eval.yaml` and then use the manifest path.

## When to choose manifest vs skill-path

The rule is simple:

- if `comet/eval.yaml` exists: pass that file as the target
- if you only have a local directory and are still in early debugging: pass the Skill directory with `--quick`
- if the Skill was generated by `/comet-any`: pass its `comet/eval.yaml`
- if you want publish readiness: pass its `comet/eval.yaml`

Do not treat a local Skill `--quick` smoke as the final publish evaluation.

## What to do when it fails

### If `collect` fails

Check first:

- whether the manifest path is correct
- whether `comet/eval.yaml` exists
- whether the manifest's recommended task exists
- whether you are in the Comet repo root or passed the right `--project`

### If `run` fails

Look at failure attribution in the report first:

- `harness`: usually eval harness, dependency, Docker, path, or environment issues
- `workflow`: usually the Skill execution flow did not behave as intended
- `task`: usually task definition, verification rule, or fixture issues
- `model`: usually model behavior, tool use, or unstable output

### If the HTML report cannot be found

First read the CLI's `Experiment` and `Report path`. If the path still contains `<experiment-id>`,
use the real experiment id under:

```text
eval/local/logs/experiments/
```

## `comet eval` and `comet skill check` are different

The commands serve different roles.

`comet eval` is the shared eval-harness entry point for evaluating a Skill package or
`comet/eval.yaml`.

`comet skill check` is the local Engine Run completion check used to decide whether a run or change
meets the runtime checks in `comet/checks.yaml`.

If your question is "Can this Skill pass product evaluation?", use:

```bash
comet eval ./generated-skill/comet/eval.yaml --html
```

If your question is "Is this deterministic Skill Run missing artifacts or status?", then use:

```bash
comet skill check --change ./changes/demo --scope completion
```

## What the user needs to remember

In practice, only remember these three points:

1. If `/comet-any` generated the Skill, prefer `comet/eval.yaml`
2. Run `--collect` first, then `--html`
3. Eval results are publish readiness evidence, not the publish action itself

Recommended commands:

```bash
comet eval ./generated-skill/comet/eval.yaml --collect
comet eval ./generated-skill/comet/eval.yaml --html
comet creator next <name> --json
```