--- name: k8e-sandbox description: "Run a goal end to end inside an isolated K8E sandbox pod (gVisor / Kata / Firecracker) instead of on the host: exec bash / Python / Node / TypeScript, install packages, move files in and out, reuse one session across calls, snapshot and restore the workspace, run background jobs, and publish an in-sandbox service through the k8e API Gateway. Use when the user invokes /k8e-sandbox , $k8e-sandbox or /skill:k8e-sandbox, or when work needs untrusted, disposable or reproducible Linux execution — running or testing code, installing dependencies, processing data or files, reproducing a bug in a clean box, or serving a dev app off the host. Egress is allowlisted, destructive actions are human-gated, and the connection is mTLS with multi-cluster profiles." argument-hint: "" user-invocable: true --- # /k8e-sandbox Treat this as the **k8e-sandbox** skill command. **Invocation (same skill, different harness prefixes):** - Claude Code: `/k8e-sandbox ` - Codex: `$k8e-sandbox ` (or pick from `/skills`) - Pi: `/skill:k8e-sandbox ` (or `/k8e-sandbox` when skill commands are enabled) - dsh (DeepSeek Harness): the model loads this skill via the `skill` tool (catalog name `k8e-sandbox`), or the user names it directly in chat — see [dsh execution path](#dsh-deepseek-harness-execution-path) below **Goal from invocation arguments:** ``` $ARGUMENTS ``` If `$ARGUMENTS` is empty and no goal is otherwise provided, ask the user for a sandbox goal and **stop** (do not invent work). ## What this skill can do (capability map) Match the goal to a capability, then use the matching command. Exact flags live in the [command reference](#command-reference). | Goal shape | Capability | Command | |---|---|---| | Run a command, script or test suite | Isolated exec in the pod — `bash` (default), `python`, `node`, `ts` | `run`, `run --lang python`, `run --raw` | | Install dependencies | pip/npm writes land in `/workspace`, not the image (KIP-13) | `run 'pip install …'` | | Move files in / out | Chunked streaming — constant memory, binary-safe, any size | `push [remote]`, `pull [local]` | | Keep state across calls | Auto session, `--tenant` for cross-process reuse, sub-agents share one pod + workspace | `run …`, `run --tenant`, `subagent ` | | Seed a workspace | Declarative manifest or a git clone at session creation (KIP-9) | `create --manifest`, `create --git-repo` | | Save / restore a workspace | Content-addressed snapshots (deduped; incremental via `--base`) | `snapshot save/list/restore/delete` | | Serve a web app or API | Publish an in-pod port through the k8e API Gateway — no port-forward, no inbound pod exposure (KIP-24) | `run --background` → `expose ` → `exposed` / `unexpose` | | Reach the internet | Session egress allowlist, updatable live while the pod runs | `allow-hosts --add/--remove/--clear`, or `--allowed-hosts` at create | | Watch or debug a run | Transcript replay, NDJSON event stream, process list, background poll | `log`, `events`, `ps`, `poll` | | Gate an irreversible step | Human-in-the-loop approval before the action runs | `confirm ` → `approve ` | | Measure cold-start latency | Warm-pool benchmark | `benchmark` | | Drive it from another program | Machine-readable command surface; native MCP server | `catalog`, `mcp-serve` | | Target another cluster | mTLS client certs + named profiles; API keys with TTL (KIP-14 / KIP-17) | `--profile`, `connect`, `k8e sandbox-apikey create` | | Diagnose a broken setup | Self-check with auto-fix | `doctor`, `doctor --json`, `doctor --fix` | ## dsh (DeepSeek Harness) execution path In dsh, decide your execution mode by **checking the current session's tool list first** — everything else in this skill branches on it. ### A. Plugin mounted — `k8e_sandbox_session_status` IS in your tool list The `dsh-k8e-sandbox` plugin replaced the harness's execution seams: - `bash` (subprocess seam) → runs **inside the sandbox pod** - `read` / `write` / `edit` / dir listings (fs seam) → sandbox `/workspace` - `k8e_sandbox_*` tools → session lifecycle, expose, egress allowlist Rules: 1. **Do NOT run `k8e-sandbox-cli`** here: it is not present inside the sandbox, and invoking it from the sandboxed `bash` would recursively dial the gateway from inside the pod. Use the tools + sandboxed seams instead. 2. **Prefer plain `bash` for commands** — it lands in the sandbox. Use `k8e_sandbox_exec` when you want structured stdout/stderr/exitCode, and `k8e_sandbox_run_background` + `k8e_sandbox_poll` for long/streaming tasks. 3. First action in a session: `k8e_sandbox_session_status` — it lazily creates the session shared by fs/subprocess/exec, and reports `available`, `sessionId`, `tenantId`. Tool reference (exact argument shapes — do not guess): | Tool | Arguments | Returns | |---|---|---| | `k8e_sandbox_session_status` | `{}` | `available`, `sessionId`, `tenantId`, `error` | | `k8e_sandbox_session_destroy` | `{}` — releases the pod (idempotent) | `destroyed` | | `k8e_sandbox_exec` | `{code: string (required), lang?: "bash"\|"python"\|"node"\|"ts", timeout?: number}` — omit `timeout` to leave a background run uncapped | `stdout`, `stderr`, `exitCode`, `durationMs`, `truncated` | | `k8e_sandbox_run_background` | `{code: string (required), lang?: …}` | `runId`, `sessionId`, `status` | | `k8e_sandbox_poll` | `{runId: string (required)}` | `runId`, `status`, `stdout`, `stderr`, `exitCode`, `durationMs` | | `k8e_sandbox_expose` | `{port: number (required), host?: string}` | `url`, `port` | | `k8e_sandbox_unexpose` | `{port: number (required)}` | `ok`, `port` | | `k8e_sandbox_allow_hosts` | `{hosts: string[] (required)}` — full replacement list; `[]` clears (falls back to cluster defaults) | `hosts[]` | **Service exposure in dsh**: after starting a long-running service with `k8e_sandbox_run_background`, call `k8e_sandbox_expose {port: 8080}` and hand the returned URL to the user — same gateway-proxied URL the CLI's `expose` prints. Teardown with `k8e_sandbox_unexpose {port: 8080}`. These runs are not time-capped, so the service stays reachable until the session is destroyed. Session, connection, and mTLS are owned by the plugin: it resolves the gateway from config → env → `~/.k8e/sandbox/profiles.yaml` (KIP-17) and reuses one persistent gRPC connection. If a tool errors with gateway unreachable / mTLS / deadline, tell the user to run `k8e-sandbox-cli connect` (local) or `k8e-sandbox-cli connect --endpoint :50051 --apikey ` (remote) **outside dsh**, then restart the dsh session. ### B. Plugin NOT mounted — `k8e_sandbox_*` tools are NOT in your tool list The plugin bundle is not installed for this dsh profile. dsh's `bash` still runs on the HOST here (no seam replacement), so the **CLI-first flow below works normally**: execute everything via `k8e-sandbox-cli run ...` exactly as the CLI examples describe. Do not pretend the k8e_sandbox_* tools exist — calling a nonexistent tool errors. To enable the full plugin experience, ask the user to install the bundle once (from a k8e checkout), then restart dsh: ``` dsh plugin --profile add /plugins/deepseek-harness/packages/dsh-k8e-sandbox-bundle dsh --profile # restart the session ``` ### dsh error quick reference | Symptom | Cause | Fix | |---|---|---| | Tool call fails "not found" / unknown tool `k8e_sandbox_*` | plugin bundle not mounted | use section B (CLI-first); ask user to install the bundle | | Tool errors "gateway unreachable" / mTLS / deadline | gateway down or missing credentials | `k8e-sandbox-cli connect` (or with `--endpoint`/`--apikey`) outside dsh, restart dsh | | `bash`/`read` error with connection refused | session pod not ready | `k8e_sandbox_session_status`; wait and retry | | `k8e_sandbox_expose` returns 503 "no pod IP" | server build predates the podIP backfill | upgrade the k8e server, or inspect the session with `k8e-sandbox-cli get ` and retry | The CLI-first flow below (`k8e-sandbox-cli run ...`) is for harnesses where the sandbox is *not* mounted (Claude Code / Codex / Pi / dsh without the plugin). ## Binary naming (read this first) The downloaded file name carries a **platform suffix** — pick the one for the user's machine: | Platform | Download name | |----------|---------------| | Linux amd64 | `k8e-sandbox-cli-linux-amd64` | | Linux arm64 | `k8e-sandbox-cli-linux-arm64` | | macOS amd64 | `k8e-sandbox-cli-darwin-amd64` | | macOS arm64 | `k8e-sandbox-cli-darwin-arm64` | | Windows amd64 | `k8e-sandbox-cli-windows-amd64.exe` | It is the **same binary** this skill invokes as `k8e-sandbox-cli` — just under the platform-suffixed name. To make the plain name work without renaming, create a **symlink** (do not rename the file): ```bash # Example for Linux amd64 — substitute the platform name for other OS/arch curl -sLO https://github.com/xiaods/k8e/releases/latest/download/k8e-sandbox-cli-linux-amd64 chmod +x k8e-sandbox-cli-linux-amd64 ln -s k8e-sandbox-cli-linux-amd64 k8e-sandbox-cli # symlink, original file stays # optionally move both into a PATH dir, e.g. ~/.local/bin/ ./k8e-sandbox-cli ... connect # connect installs this skill + ensures PATH ``` (Windows: use `mklink k8e-sandbox-cli.exe k8e-sandbox-cli-windows-amd64.exe` in cmd.) From then on, this skill and all examples use the plain name `k8e-sandbox-cli` — same binary. If you only see a platform-suffixed name in the user's environment (no symlink yet), use that file directly: `./k8e-sandbox-cli-linux-amd64 status` etc. All spellings are interchangeable; never tell the user they are missing a second binary. ## Hard rules 1. **All code and shell execution goes through `k8e-sandbox-cli`** — never run `python3`, `node`, `pip`, `npm`, `curl`, compilers, or tests on the host for this goal. 2. Prefer auto session mode: `k8e-sandbox-cli run "..."` (creates/reuses session). 3. Parse JSON with `jq` unless `--raw` is used. 4. If the gateway is unreachable, tell the user to run `k8e-sandbox-cli connect` (local) or `k8e-sandbox-cli connect --endpoint :50051 --apikey ` (remote). Multi-cluster: `--profile ` / `~/.k8e/sandbox/profiles.yaml` (KIP-17). ## Auth & multi-profile (KIP-14 / KIP-17 / #538) **Do not confuse these files:** | Path | Who | What | |------|-----|------| | `/etc/k8e/config.yaml` | **k8e server/agent** | Daemon flags only | | `~/.k8e/sandbox/profiles.yaml` | **k8e-sandbox-cli** | Named gateways / cert dirs | | `~/.k8e/sandbox/config.json` | **k8e-sandbox-cli** | Last connect stamp | **mTLS bootstrap:** first remote connect/login uses an API key once; CLI stores `ca.crt` + `client.crt` + `client.key` (private key never leaves the machine). Client certs last **90 days** and auto-renew when **<30 days** remain. API keys default to **30-day TTL** (`k8e sandbox-apikey create name`, override with `--ttl 90d|never`). First authentication verifies the gateway using system trust, a cached CA, or an administrator-provided `--ca-file /path/to/sandbox-ca.crt` on `connect`/`login`. Obtain that CA through a trusted channel. Private-CA gateways require the CA on first use. `--insecure-bootstrap` explicitly bypasses server verification only when no CA is selected; this exposes the API key to server impersonation and must not be the default recovery step. `login` always authenticates the supplied API key, even with valid cached credentials. Concurrent CLI processes serialize credential initialization and renewal. **Profiles** (`~/.k8e/sandbox/profiles.yaml`, override with `K8E_SANDBOX_CONFIG`): ```yaml # ~/.k8e/sandbox/profiles.yaml — NOT /etc/k8e/config.yaml version: 1 current_profile: default profiles: default: endpoint: 10.0.0.1:50051 prod: endpoint: sandbox.prod.example:50051 cert_dir: ~/.k8e/sandbox-prod device_name: laptop-prod ``` ```bash k8e-sandbox-cli --profile prod connect --apikey <64-hex key> k8e-sandbox-cli --profile prod run 'echo hi' # or: export K8E_SANDBOX_PROFILE=prod ``` Priority: flags → env → profile → last-connect fallback (`~/.k8e/sandbox/config.json`) → defaults. Flag/env pairs: `--endpoint`/`K8E_SANDBOX_ENDPOINT`, `--apikey`/`K8E_SANDBOX_APIKEY`, `--profile`/`K8E_SANDBOX_PROFILE`. The cert dir has **no flag**: `K8E_SANDBOX_CERT_DIR` → profile `cert_dir` → `~/.k8e/sandbox`. Other env: `K8E_SANDBOX_SESSION_ID`, `K8E_SANDBOX_TENANT`, `K8E_SANDBOX_DEVICE_NAME`, `K8E_SANDBOX_CONFIG` (profile file path). ## Procedure (always) ### 1. Pre-flight ```bash command -v k8e-sandbox-cli >/dev/null || { echo "k8e-sandbox-cli not on PATH; run connect again"; exit 1; } k8e-sandbox-cli status ``` Require `"available": true`. If not available, stop and instruct the user to `connect`. ### 2. Plan Decompose `$ARGUMENTS` into sandbox-safe steps (install deps → write files → run code → read outputs). ### 3. Execute (examples) ```bash # Shell / bash (default) k8e-sandbox-cli run 'echo hello' # Python k8e-sandbox-cli run "print(1+1)" --lang python # Multi-line / files k8e-sandbox-cli run 'pip install pandas' --lang bash # write via stdin: # cat analysis.py | k8e-sandbox-cli write /workspace/analysis.py # k8e-sandbox-cli run 'python3 /workspace/analysis.py' --session-id # push/pull local files (chunked streaming — constant memory, binary-safe, # works for files of any size; prefer over write/read for real files): # k8e-sandbox-cli push ./analysis.py /workspace/analysis.py # k8e-sandbox-cli pull /workspace/results.csv ./results.csv # Background exec (returns run_id immediately). # A background run has NO lifetime cap by default. `--timeout N` caps it and # the daemon SIGKILLs the run (whole process group) when N expires — only pass # it for work you want bounded. k8e-sandbox-cli run 'sleep 30; echo done' --background k8e-sandbox-cli poll # wait + stream output # Tenant reuse (share one session across CLI calls) k8e-sandbox-cli run 'echo hi' --tenant my-project # Sub-agent: child session sharing parent pod + workspace (no new pod) k8e-sandbox-cli subagent # Expose a long-running service through the k8e API Gateway (KIP-24). # No --timeout here: a cap would kill the server while you are still testing it. k8e-sandbox-cli run "python3 -m http.server 8080 --bind 127.0.0.1" --background k8e-sandbox-cli expose 8080 # -> {"url":"http:///k8e/expose//8080/",...} ``` Useful commands: `run`, `write`, `read`, `list`, `push`, `pull`, `create`, `get`, `sessions`, `destroy`, `status`, `log`, `events`, `ps`, `poll`, `subagent`, `confirm`, `approve`, `snapshot`, `benchmark`, `catalog`, `expose`, `unexpose`, `exposed`, `allow-hosts`, `doctor`, `login`, `mcp-serve`. ### 4. Report Show stdout/stderr and exit codes from the CLI JSON. Do not claim host-side execution. ## One-time setup (if not connected) ```bash # Local K8E node k8e-sandbox-cli connect # Remote — API key from server (default TTL 30d) k8e sandbox-apikey create my-agent # → {"name":"my-agent","key":"<64-hex>","e2b_key":"e2b_<64-hex>", # "ttl_days":30,"created_at":"…","expires_at":"…"} # Pass `key` (bare hex) to `connect --apikey`. Hand `e2b_key` to the official e2b # SDKs: they require the e2b_ prefix and the server strips it. # k8e sandbox-apikey create my-agent --ttl never # optional non-expiring k8e-sandbox-cli connect --endpoint :50051 --apikey <64-hex key> --ca-file /path/to/sandbox-ca.crt # Multi-cluster: k8e-sandbox-cli --profile prod connect --apikey <64-hex key> ``` `connect` authenticates (mTLS), verifies the gateway, puts `k8e-sandbox-cli` on PATH when needed (symlink to `~/.local/bin/k8e-sandbox-cli`), and installs this skill into Claude / Codex / Pi / dsh discovery paths (`--agent dsh` or `--agent all`; dsh reads it from `~/.dsh/skills` or `~/.agents/skills`). ## Command reference | Command | Purpose | |---------|---------| | `k8e-sandbox-cli --profile …` | Use named profile from `~/.k8e/sandbox/profiles.yaml` | | `k8e-sandbox-cli connect` | Local/remote auth + install this skill into agent harnesses (`--agent` auto/claude/codex/pi/dsh/all, `--reset-certs`, `--skip-verify`, `--skip-path`) | | `k8e-sandbox-cli connect --skill-only` | Re-install this skill only (no gateway dial) | | `k8e-sandbox-cli login` | Remote mTLS only (no skill install); optional `--device-name` | | `k8e-sandbox-cli doctor` | Self-check gateway / certs / skill install / PATH (`--json`, `--fix`) | | `k8e-sandbox-cli mcp-serve` | Serve the sandbox MCP endpoint for MCP-capable agents (`--listen`, `--gateway`, TLS + API-key flags) | | `k8e-sandbox-cli status` | Gateway + session probe | | `k8e-sandbox-cli run ` | Exec in sandbox (`--lang`, `--timeout` seconds — default 30, `--raw`, `--session-id`, `--tenant`, `--background`, `--manifest`, `--git-repo`/`--git-ref`/`--git-path`, `--allowed-hosts`) | | `k8e-sandbox-cli create` | Manual session (`--runtime`, `--env`, `--secret`, `--allowed-hosts`, `--manifest`, `--git-repo`) | | `k8e-sandbox-cli get ` | Session introspection (phase, runtime, env keys) | | `k8e-sandbox-cli sessions` | List sessions | | `k8e-sandbox-cli write/read/list` | Workspace files (`write --mode`, `read --raw`); `list --since ` returns only files modified after that timestamp | | `k8e-sandbox-cli push [remote]` | Stream a local file INTO the sandbox (chunked 4MiB windows — constant memory, binary-safe, any size; `--chunk-mb` to tune) | | `k8e-sandbox-cli pull [local]` | Stream a sandbox file OUT to a local path (same chunked transfer) | | `k8e-sandbox-cli log ` | Replay exec transcript (`--offset`, `--limit`, `--follow`) | | `k8e-sandbox-cli events ` | Read daemon NDJSON event stream (`--limit`) | | `k8e-sandbox-cli ps ` | List processes in the sandbox pod (pid, comm, state) | | `k8e-sandbox-cli poll ` | Wait for a `run --background` job and return its result | | `k8e-sandbox-cli subagent ` | Spawn child session (shares parent's pod + workspace — no new pod) | | `k8e-sandbox-cli confirm ` | Gate destructive action on human approval (`--timeout`, `--no-wait`) | | `k8e-sandbox-cli approve ` | Approve a pending confirm (`--reject`, `--reason`) | | `k8e-sandbox-cli snapshot save ` | Save workspace snapshot (content-addressed, dedup'd) | | `k8e-sandbox-cli snapshot list` | List saved snapshots | | `k8e-sandbox-cli snapshot restore ` | New session from a snapshot (`--base ` for incremental) | | `k8e-sandbox-cli snapshot delete ` | Delete a snapshot | | `k8e-sandbox-cli expose ` | Expose an in-sandbox service through the k8e API Gateway; returns the public URL (`--host`, `--session-id`) | | `k8e-sandbox-cli unexpose ` | Tear down an exposed port (idempotent; `--session-id`) | | `k8e-sandbox-cli exposed` | List live exposures for the session (`--session-id`) | | `k8e-sandbox-cli allow-hosts ` | Freely set the session egress allowlist, live (`--hosts` replace, `--add`, `--remove`, `--clear`; `--session-id`) | | `k8e-sandbox-cli benchmark` | Warm-pool latency metrics (`--pool-size`, `--iterations`) | | `k8e-sandbox-cli catalog` | Emit machine-readable command surface (SDK generation) | | `k8e-sandbox-cli destroy ` | Tear down session | Default `run` output is JSON: `stdout`, `stderr`, `exit_code`, `session_id`, `status`, `duration_ms`, `truncated`, `language`; `run --background` returns `run_id`, `status`, `session_id`. Use `--raw` to stream plain text instead. ## Service exposure (KIP-24) When the agent builds a long-running service inside the sandbox (web app, API server), expose it through the k8e API Gateway so the gateway/other hosts can reach it — no port-forward, no inbound pod exposure: ``` k8e-sandbox-cli run "python3 -m http.server 8080 --bind 127.0.0.1" --background k8e-sandbox-cli expose 8080 # -> {"url":"http:///k8e/expose//8080/",...} curl http:///k8e/expose//8080/ # reachable via the gateway (VPC/LB) k8e-sandbox-cli exposed # list live exposures k8e-sandbox-cli unexpose 8080 # tear down ``` The exposed URL routes: Cilium Gateway API (:80/:443) -> embedded e2b HTTP server -> reverse proxy to `http://:`. The gateway base is configured server-side (`--sandbox-expose-base-url`, default `http://`). The CNP is re-applied automatically so only the gateway/e2b-server can reach the exposed port. **Keep the service alive.** Start it with `run --background` and *no* `--timeout`: the run then lives until the process exits or the session is destroyed. Passing `--timeout N` makes sandboxd SIGKILL the run's whole process group after N seconds — the URL keeps working until then and returns 502 afterwards with nothing listening on the port. **A 502 on an exposed URL means nothing is listening on that port inside the sandbox** (the service died, never started, or crashed). The 502 body names the unreachable `:`. Diagnose with `ps ` (is the process still there?) and `poll ` (`timed_out` = the lifetime cap expired). **Egress allowlist is freely configurable** — when the sandbox needs outbound access to domains (package registries, tunnel endpoints), update it live: ``` k8e-sandbox-cli allow-hosts --add pypi.org,registry.npmjs.org k8e-sandbox-cli allow-hosts --remove pypi.org k8e-sandbox-cli allow-hosts --clear # fall back to cluster defaults ``` ## Session modes | Mode | How | State | |------|-----|-------| | Auto (default) | `run` without session id | `~/.k8e/sandbox/default/state.json` | | Tenant | `--tenant my-project` | `~/.k8e/sandbox/{tenant}/state.json` | | Manual | `create` → `run --session-id` → `destroy` | none | ## Egress Default allowed hosts (cluster `SandboxMatrix.spec.defaultAllowedHosts`): `pypi.org`, `files.pythonhosted.org`, `registry.npmjs.org`, `github.com`, `raw.githubusercontent.com`. - At session creation: `create --allowed-hosts a.com,b.com` (or `run --allowed-hosts` for auto-created sessions). - **Live, any time (KIP-24)**: `allow-hosts --add a.com,b.com` / `--remove a.com` / `--clear` (fall back to cluster defaults). Applies immediately via CNP re-apply; in dsh use `k8e_sandbox_allow_hosts {hosts: [...]}`. ## Security red lines - `--env` is for non-sensitive config only (stored on CRD). Use `--secret ENV=secret:key` for secrets. - Never pass host secrets into sandbox flags in chat logs if avoidable. - Never `sudo` via sandbox CLI. - Destructive sandbox actions require `confirm` → `approve` (human in the loop); don't skip it. ## Error quick reference | Exit | Meaning | Action | |------|---------|--------| | 2 | TLS / cert / unreachable | Server reinstalled or CA rotated? Re-run `connect --reset-certs --apikey --ca-file /path/to/trusted-new-ca.crt` (validates new credentials before replacing cached files); otherwise check profile `cert_dir` | | 1 | Command/session error | Read JSON error; recreate session if gone; re-create API key if TTL expired | | 8 | ResourceExhausted | Wait or free warm pool capacity | ## Your role when this skill is active **Do:** - **dsh + plugin mounted (section A)**: run everything through the sandboxed `bash`/`read`/`write` seams and the `k8e_sandbox_*` tools; start long-running services with `k8e_sandbox_run_background`, hand the user a reachable URL via `k8e_sandbox_expose`. - **Everywhere else (CLI-first flow)**: execute `$ARGUMENTS` entirely via `k8e-sandbox-cli`; prefer `run`; use `--lang python` for Python; use `--raw` for long streams; show real CLI output. When the goal builds a long-running service (web app, API), start it with `run --background` and hand the user a reachable URL via `expose `. **Don't:** run the goal on the host; skip pre-flight; invent successful output without actually running a tool/CLI; call `k8e_sandbox_*` tools that are not in your current tool list (plugin not mounted — use section B instead).