# hermes-grok-bench

**Public dogfood benchmark of every xAI Grok model under Hermes-style agentic workloads.**

One weekly run. One public matrix. Real API calls, real cost in USD, real reasoning tokens.

Built for [Hermes Agent](https://github.com/NousResearch/hermes-agent) operators who want a non-marketing answer to: *which Grok model should I route this task to?*

---

## Why this exists

xAI ships fast. The Grok lineup as of v0.1.0:

```
grok-3              grok-3-mini           grok-4-0709
grok-4-fast-{r,nr}  grok-4-1-fast-{r,nr}
grok-4.20-0309-{r,nr}                     grok-4.20-multi-agent-0309
grok-code-fast-1
```

Eleven SKUs, three context-window tiers, divergent reasoning support, two transport surfaces (`/v1/chat/completions` vs `/v1/responses`). Cherry-picking the wrong one wastes tokens and breaks tool-use.

`hermes-grok-bench` runs a fixed micro-suite against the live xAI API and publishes the matrix below. No simulation, no static pricing — pricing is fetched from `/v1/language-models` at run time, cost is computed from the actual `usage` payload returned by each call.

## What it measures

Three intentionally narrow tasks, each graded deterministically:

1. **`math_multiplication`** — exact arithmetic (`31847 * 49123`). Tests reasoning under a single-shot constraint.
2. **`tool_use_weather`** — emit a valid `tool_calls[]` for `get_weather(city, unit)`. Tests function-calling compatibility and JSON arg correctness.
3. **`code_is_prime`** — write a Python `is_prime(n)` that passes 8 reference cases under `exec`. Tests code generation.

Each (model × task) cell records: pass/fail, latency, input/output/cached/reasoning tokens, USD cost.

## Live matrix

<!-- BENCH:START -->
| Model | code_is_prime | math_multiplication | tool_use_weather | Pass rate |
|---|---|---|---|---|
| `grok-3` | ✓ 1205ms $0.0010 | ✗ | ✓ 1069ms $0.0014 | **2/3** |
| `grok-3-mini` | ✓ 7719ms $0.0000 | ✓ 19584ms $0.0000 | ✓ 5127ms $0.0001 | **3/3** |
| `grok-4-0709` | ✓ 20680ms $0.0020 | ✓ 27650ms $0.0007 | ✓ 5538ms $0.0018 | **3/3** |
| `grok-4-1-fast-non-reasoning` | ✓ 978ms $0.0001 | ✗ | ✓ 2002ms $0.0001 | **2/3** |
| `grok-4-1-fast-reasoning` | ✓ 6627ms $0.0001 | ✓ 23127ms $0.0000 | ✓ 2674ms $0.0001 | **3/3** |
| `grok-4-fast-non-reasoning` | ✓ 1095ms $0.0001 | ✗ | ✓ 1764ms $0.0001 | **2/3** |
| `grok-4-fast-reasoning` | ✓ 3244ms $0.0001 | ✓ 13788ms $0.0000 | ✓ 2044ms $0.0001 | **3/3** |
| `grok-4.20-0309-non-reasoning` | ✓ 1203ms $0.0004 | ✗ | ✓ 1361ms $0.0003 | **2/3** |
| `grok-4.20-0309-reasoning` | ✓ 9266ms $0.0003 | ✓ 22737ms $0.0007 | ✓ 2031ms $0.0003 | **3/3** |
| `grok-4.3` | ✓ 5517ms $0.0003 | ✓ 11862ms $0.0001 | ✓ 2453ms $0.0003 | **3/3** |
| `grok-code-fast-1` | ✓ 20860ms $0.0001 | ✓ 22412ms $0.0000 | ✓ 13643ms $0.0001 | **3/3** |
| **Total** | **11/11** | **7/11** | **11/11** | |

_Run: `2026-05-04T081941Z` · 2026-05-04T08:19:41.006432+00:00_
_11 models × 3 tasks = 33 calls · total cost: **$0.0109** · reasoning tokens: 13,637_

Excluded from v0.1 matrix (different API surface): `grok-4.20-multi-agent-0309`.
<!-- BENCH:END -->

## Install

```bash
pip install hermes-grok-bench
# or, from this repo:
uv pip install -e .
```

Requires Python 3.11+.

## Run it yourself

```bash
export XAI_API_KEY=xai-...
hermes-grok-bench run
```

A full run against all 10 chat-completion models hits ~30 calls, takes ~2 minutes, and costs **less than $0.05** (figure refreshed in the matrix above on every run).

Results land in `results/<run_id>.json` and `results/latest.json`. The README matrix is regenerated in place.

## Design notes

- **No SDK, just `httpx`.** xAI's REST is small enough that a Python SDK would obscure more than it abstracts.
- **Pricing is API-truth, not docs-truth.** Docs lag; `/v1/language-models` does not.
- **`grok-4.20-multi-agent-0309` is excluded from the v0.1 chat matrix** — it's `/v1/responses`-only and refuses `/v1/chat/completions`. A separate `responses` suite is on the roadmap.
- **Reasoning tokens are surfaced as a first-class column** because they dominate cost on Grok 4 reasoning models and are invisible in the `content` field.
- **Determinism over coverage.** This is a compatibility/cost matrix, not a capability ranking. For broad capability evals use lm-evaluation-harness.

## Roadmap

- [ ] `responses-suite` for `grok-4.20-multi-agent-0309` (reasoning effort `low/medium/high/xhigh`, cost ladder)
- [ ] `prompt-cache` task — measure `cached_tokens` recovery via repeated calls with `x-grok-conv-id`
- [ ] `vision` task — image input compatibility per model
- [ ] GitHub Action: scheduled weekly run, auto-commit refreshed matrix

## License

MIT — see `LICENSE`.

## Author

Built by [Julien Talbot](https://github.com/Julientalbot) — Hermes Agent contributor (xAI integration). Issues and PRs welcome.