# Data construction

[README](../README.md) · [Inference](inference.md) · [Evaluation](evaluation.md) · [Training](training.md) · [Data construction](data-construction.md) · [Reference](reference.md)

This guide covers candidate generation, verification, and final split creation. For the released files and schemas, see the [dataset reference](reference.md#dataset-and-benchmark-splits).

The construction path is:

```text
Kali tool manuscripts
  → sub-tool usage chunks
  → Qwen3-Max query-command candidates
  → documentation-grounded model verification
  → regeneration of uncovered sub-tools
  → isolated Kali terminal execution and rule-based triage
  → optional reviewed adjudications
  → exact query deduplication and tool-stratified train/test finalization
```

Complete the [environment setup](evaluation.md#environment-setup) first, then run all commands from the repository root. The defaults use Qwen3-Max through DashScope's OpenAI-compatible API, matching the data-construction configuration used for the released artifact. The model, API base URL, and API-key environment variable are configurable.

## 1. Extract documentation and generate candidates

Set the API key without placing it in a command-line argument:

```bash
export DASHSCOPE_API_KEY="YOUR_KEY"
```

Run document extraction, request preparation, generation, and response flattening:

```bash
python src/data_creation/generate.py \
  --stage all \
  --source-dataset "anonymous62567/kali-tools" \
  --source-split train \
  --out-dir "$PWD/outputs/data_creation/generation" \
  --model qwen3-max \
  --items-per-tool 10 \
  --temperature 0.5 \
  --max-tokens 4096
```

This creates:

```text
outputs/data_creation/generation/
├── tool_subtools.jsonl             # extracted and deduplicated usage chunks
├── generation_requests.jsonl       # exact model requests
├── generation_responses.jsonl      # append-only API checkpoint
├── generated_labels.jsonl          # flattened query-command candidates
└── generation_parse_errors.jsonl   # responses that could not be parsed
```

Preparation can be reproduced without network access by using the released usage chunks:

```bash
python src/data_creation/generate.py \
  --stage prepare \
  --input-tool-subtools-jsonl "$PWD/KaliBench_data/Kali_Tool_Subtools_UsageCode.jsonl" \
  --out-dir "$PWD/outputs/data_creation/generation"
```

The individual `prepare`, `generate`, and `flatten` stages can be rerun separately. API checkpoints are keyed by `custom_id`; completed IDs are skipped, while `--retry-errors` retries IDs whose latest checkpoint contains an API error.

## 2. Verify against the source documentation

The verifier receives each query-command pair together with its matched usage documentation and must return `ACCURATE` or `HALLUCINATED`.

```bash
python src/data_creation/verify.py \
  --stage all \
  --labels-jsonl "$PWD/outputs/data_creation/generation/generated_labels.jsonl" \
  --tool-subtools-jsonl "$PWD/outputs/data_creation/generation/tool_subtools.jsonl" \
  --out-dir "$PWD/outputs/data_creation/verification" \
  --model qwen3-max \
  --temperature 0 \
  --max-tokens 1024
```

This creates:

```text
outputs/data_creation/verification/
├── verification_prompts.jsonl      # exact messages presented to the verifier
├── verification_results.jsonl      # append-only API checkpoint and raw responses
├── model_verified.jsonl            # ACCURATE rows in released metadata format
├── model_rejected.jsonl            # HALLUCINATED rows
├── verification_errors.jsonl       # API, parse, or missing-documentation failures
└── remaining_subtools.jsonl         # usage chunks with no accepted candidate
```

`model_verified.jsonl` stores the verifier evidence as:

```json
{"model_verification":{"model":"qwen3-max","model_output":{"verdict":"ACCURATE","reasons":"..."}}}
```

To reproduce the released construction behavior, unmatched candidates are sent with an empty documentation block. Add `--require-usage` to route them to `verification_errors.jsonl` without an API call instead.

Uncovered sub-tools can be regenerated by passing `remaining_subtools.jsonl` back to the generation stage:

```bash
python src/data_creation/generate.py \
  --stage all \
  --input-tool-subtools-jsonl "$PWD/outputs/data_creation/verification/remaining_subtools.jsonl" \
  --out-dir "$PWD/outputs/data_creation/regeneration"
```

The verification CLI also accepts `--merge-clean` to combine clean files from multiple generation/verification rounds while retaining one latest row per `custom_id`.

## 3. Execute and triage commands in Kali

> [!CAUTION]
> This stage executes every `ground_truth_command`. Run it only inside an isolated Kali container or disposable VM with no sensitive files, credentials, or reachable production systems. The explicit acknowledgement flag is required.

The included image installs the `kali-linux-everything` toolset used for terminal verification. It is a large image, so the initial build requires substantial download time and disk space:

```bash
docker build \
  -t kalibench/kali-everything:latest \
  -f docker/kali-everything/Dockerfile .
```

Run the verifier from the repository root. The command below disables container networking, mounts the repository read-only, and makes only the terminal-output directory writable:

```bash
mkdir -p "$PWD/outputs/data_creation/terminal"

docker run --rm \
  --name kalibench-terminal-verification \
  --hostname kali-everything \
  --network none \
  --pids-limit 256 \
  --memory 8g \
  --cpus 4 \
  --security-opt no-new-privileges \
  --cap-add NET_ADMIN \
  --cap-add NET_RAW \
  --mount type=bind,src="$PWD",dst=/workspace,readonly \
  --mount type=bind,src="$PWD/outputs/data_creation/terminal",dst=/workspace/outputs/data_creation/terminal \
  --workdir /workspace \
  kalibench/kali-everything:latest \
  python3 src/data_creation/terminal_verify.py \
    --input-jsonl /workspace/outputs/data_creation/verification/model_verified.jsonl \
    --out-dir /workspace/outputs/data_creation/terminal \
    --timeout-sec 20 \
    --max-stream-bytes 204800 \
    --acknowledge-command-execution
```

Networking is disabled by default so commands cannot reach external targets. Network-dependent commands may therefore be classified as runtime failures. If external connectivity is necessary for a specific verification experiment, attach the container only to a controlled, isolated test network and document that deviation.

For every row, the executor records timestamps, duration, timeout status, exit code, bounded stdout/stderr, truncation metadata, classifier label, category, reason, and matched error pattern. The rule-based categories are:

| Category | Meaning |
| --- | --- |
| `pass_review` | Executed successfully, timed out, or failed only because a runtime resource such as a file/device/service/target was unavailable. |
| `reject_hallucinated` | The shell/tool reported a missing command, unsupported flag, invalid option, or malformed required argument. |
| `undecided` | No reliable execution decision could be made. |

Outputs:

```text
outputs/data_creation/terminal/
├── terminal_results.jsonl          # append-only execution checkpoint
├── terminal_accepted.jsonl         # pass_review rows
├── terminal_rejected.jsonl         # hallucinated tool/parameter rows
├── terminal_undecided.jsonl        # rows requiring inspection
└── terminal_summary.json
```

Optional reviewed decisions can be supplied as an auditable JSONL file:

```json
{"custom_id":"candidate-000001-000","category":"pass_review","reason":"No unsupported capability was found during inspection."}
```

Apply those decisions by adding:

```text
--adjudications-jsonl /path/to/adjudications.jsonl
```

The merged rows record `ai_review_category` and `ai_review_reason` inside `terminal_verification`, matching the provenance structure in the released splits.

## 4. Deduplicate and create the final splits

The finalizer requires both an `ACCURATE` model verdict and terminal category `pass_review`, removes duplicate normalized queries, assigns stable `sample_` IDs, and creates a deterministic tool-stratified split.

```bash
python src/data_creation/finalize.py \
  --inputs "$PWD/outputs/data_creation/terminal/terminal_accepted.jsonl" \
  --out-dir "$PWD/outputs/data_creation/final" \
  --test-size 5000 \
  --min-test-per-tool 3 \
  --max-test-per-tool 4 \
  --seed 3407
```

For the released 8,504 accepted rows, these settings produce 3,504 training rows and 5,000 test rows while retaining all 1,642 tools in the test pool. The finalizer writes the combined data, train/test splits, rejected/duplicate audit rows, and a JSON summary with row and tool counts.

Because hosted-model outputs can change across model revisions or service updates, a fresh API run may not reproduce every generated sentence byte-for-byte. The released JSONL files are the canonical outputs of the documented construction and review process; the code reproduces the transformations, checkpoints, verification schema, filtering rules, and split policy.

## 5. Validate the construction implementation

The tests do not call external APIs and execute only a harmless local `printf` smoke command:

```bash
python -m unittest tests.test_data_creation -v
```