# Bernardo — Evaluating an AI agent for restaurant operations

Bernardo is a case study in designing, evaluating, and improving an AI agent that answers staff questions and prepares workforce-planning actions over WhatsApp.

It complements [Comptoir AI Restaurant Planning](https://github.com/75016paris/comptoir-ai-restaurant-planning), which contains the product implementation.

## Executive summary

**My role:** I designed and implemented the evaluation contract, deterministic controls, OpenTelemetry/Langfuse instrumentation, campaign, analysis, and recommendation.

**The question:** which hosted model configuration best balances answer quality, write integrity, security, latency, and cost for a workforce-planning agent?

**The work:** six configurations were screened against 100 fictional business-risk scenarios covering retrieval, mutations, permissions, and prompt injection.

**The decision:** the highest raw score did not automatically win. OVH Qwen3.6-27B led on contract pass but carried higher latency, cost, and observed security risk. The DeepInfra Qwen3-14B configurations formed the balanced shortlist for further validation.

**Status:** screening evidence, not a production selection or safety certification.

## Business problem

Restaurant staff ask deceptively simple questions—“When do I work tomorrow?” or “Who can cover this shift?”—whose answers depend on identity, tenant scope, dates, permissions, and live planning data. Managers may also request actions that change schedules or leave records.

A plausible answer is not enough. The system must prevent:

- incorrect dates, hours, workers, or statuses;
- answers that ignore authoritative tools;
- success messages without the expected database change;
- unauthorized role or restaurant access;
- prompt-injection and tool-result spoofing;
- unsafe writes without explicit confirmation.

## What I built

| Area | Contribution |
|---|---|
| Evaluation contract | 100 versioned cases with reply, tool, API, and database checks |
| Runtime controls | Frozen business time, role-filtered tools, confirmation state, and model-versus-policy attribution |
| Test environment | Isolated fictional database clones covering roles, schedules, leave, closures, and replacements |
| Evidence pipeline | Candidate capture, source fingerprints, aggregate results, cost/latency reporting, and validation |
| Observability | OpenTelemetry instrumentation exported to self-hosted Langfuse: agent, generation, tool, evaluator, and guardrail spans |
| Decision support | Risk-adjusted model shortlist and prioritized remediation plan |

## Architecture

```text
WhatsApp message
  → identity, restaurant, role, and business date
  → localized prompt and permitted tools
  → candidate model selects a tool
  → authenticated API applies business rules
  → database/API evidence is checked
  → grounded reply or explicit confirmation
```

The model handles intent, tool selection, and language. Deterministic code owns authorization, tenant isolation, confirmations, database writes, and staffing constraints.

[Read the architecture and safety boundary →](./docs/architecture.md)

## Observability — OpenTelemetry and Langfuse

I instrumented the runtime with the OpenTelemetry NodeSDK and Langfuse v5, exporting hierarchical agent, model-generation, tool, evaluator, and guardrail observations to a self-hosted Langfuse instance. Traces carry model and campaign tags, token usage, cost, latency, tool outcomes, and evaluation scores. Export is optional, batched, and fail-open so an observability outage does not break the business request.

![Sanitized Langfuse trace of a synthetic Bernardo evaluation run](./evidence/screenshots/langfuse-trace-overview.png)

*Figure — A synthetic Bernardo run in self-hosted Langfuse: OpenTelemetry spans connect prompt construction, model generations, an authoritative tool call, evaluator results, latency, tokens, cost, and trace-level scores.*

[Read the observability design, five-screenshot walkthrough, and privacy boundary →](./docs/observability.md)

## Evaluation approach

A case passes only when every applicable part succeeds:

```text
casePass = replyOk && dbOk && apiOk && toolOk
```

| Suite | Cases | Focus |
|---|---:|---|
| Retrieval | 45 | Dates, schedules, hours, staffing, and complete answers |
| Mutation | 25 | Preparation, confirmation, execution, and database state |
| Permissions | 16 | Role and tenant boundaries |
| Injection | 14 | Spoofed authority, roles, tools, and cross-tenant evidence |

[Read the evaluation method →](./docs/evaluation-method.md)

## Screening results — August 2026

| Configuration | Contract pass | Mutation | Security¹ | p95 | Candidate cost² |
|---|---:|---:|---:|---:|---:|
| OVH Qwen3.6-27B | **90/100** | 22/25 | 25/30 | 41.0 s | $0.404 |
| DeepInfra Qwen3-14B standard | 86/100 | **25/25** | 28/30 | 31.8 s | $0.103 |
| DeepInfra Qwen3-14B flex | 86/100 | **25/25** | 28/30 | 15.0 s | $0.083 |
| DeepInfra Qwen3-14B priority | 85/100 | **25/25** | 28/30 | 6.0 s | $0.155 |
| DeepInfra Qwen3.6-27B standard | 78/100 | 10/25 | **30/30** | **5.2 s** | $0.264 |
| OVH Qwen3.5-9B | 75/100 | 15/25 | 28/30 | 17.6 s | **$0.081** |

¹ Permissions and injection cases combined. Observed results do not establish production safety.

² Rate-card estimate for 100 incoming requests; excludes retries, judges, engineering, and infrastructure.

![Contract pass by configuration](./evidence/charts/contract-pass.png)

The recommendation favors the DeepInfra Qwen3-14B group as the balanced shortlist. Their mutation behavior was complete in this screen, costs were lower, and security results were stronger than the highest-scoring configuration. One run is not enough to rank their service tiers reliably.

## What failed

The 100 failed observations produced four clear workstreams:

| Cause | Observations | Priority response |
|---|---:|---|
| Evidence lost between tool and reply | **55** | Verify required structured facts before sending |
| Confirmation-state failures | **28** | Enforce pending-action state in code |
| Incorrect refusal behavior | **13** | Separate objective safety from response quality |
| Wrong or missing retrieval tool | **4** | Improve routing and tool contracts |

The most important conclusion was architectural: the next improvement should not be another prompt rewrite alone. It should be a deterministic confirmation invariant plus a structured evidence-to-reply verifier.

[Read the findings and recommendations →](./docs/findings-and-recommendations.md)

## Evidence for technical reviewers

- [Three sanitized case walkthroughs](./evidence/examples.md)
- [Compact machine-readable results](./evidence/results.json)
- [Full 100-case evaluation contract](./evaluation/cases.json)
- [Synthetic fixture design](./docs/fixture-database.md)
- [OpenTelemetry and Langfuse observability](./docs/observability.md)
- [Quality, suite, and cost/latency charts](./evidence/charts/)

Validate the compact evidence package with:

```bash
python3 tools/validate.py
```

The [complete technical evidence bundle](https://github.com/75016paris/bernardo-agent-evals/releases/tag/evidence-2026-08)—including detailed release manifests, provenance, historical material, and original release tooling—is distributed as a GitHub Release rather than placed in the presentation path.

## Limitations and next step

This campaign used one generation per model and case. Nineteen cases had previously been inspected, and providers ran in separate time blocks. The results support shortlisting, not stable ranking.

The next step is an unseen holdout with repeated, randomized, interleaved runs; first-attempt availability tracking; broader API assertions; and blinded human review.

All restaurant, employment, schedule, phone, and business records are fictional. Some fixture names borrow public-figure names solely as memorable test labels; no affiliation is implied.