{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "welcome",
      "metadata": {},
      "source": [
        "# A reproducible model-evaluation record\n",
        "\n",
        "Evaluate a deliberately simple deterministic classifier, keep the examples visible, and produce a provenance record. The goal is measurement discipline—not a performance claim."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "imports",
      "metadata": {},
      "outputs": [],
      "source": [
        "import hashlib\n",
        "import json\n",
        "import math\n",
        "import platform\n",
        "from datetime import datetime, timezone"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "examples-note",
      "metadata": {},
      "source": [
        "## Freeze the examples and prediction rule\n",
        "\n",
        "All labels and inputs are synthetic. The keyword rule is intentionally inspectable and deterministic, so the same notebook produces the same result without model downloads or hidden services."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "examples",
      "metadata": {},
      "outputs": [],
      "source": [
        "examples = [\n",
        "    {\"text\": \"the release checks passed\", \"label\": 1},\n",
        "    {\"text\": \"verification failed safely\", \"label\": 0},\n",
        "    {\"text\": \"tests passed after review\", \"label\": 1},\n",
        "    {\"text\": \"the scanner failed closed\", \"label\": 0},\n",
        "    {\"text\": \"review remains pending\", \"label\": 0},\n",
        "    {\"text\": \"all required checks passed\", \"label\": 1},\n",
        "]\n",
        "\n",
        "def predict(text):\n",
        "    return int(\"passed\" in text.lower())\n",
        "\n",
        "predictions = [predict(row[\"text\"]) for row in examples]\n",
        "assert predictions == [1, 0, 1, 0, 0, 1]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "metrics",
      "metadata": {},
      "outputs": [],
      "source": [
        "tp = sum(pred == 1 and row[\"label\"] == 1 for pred, row in zip(predictions, examples))\n",
        "tn = sum(pred == 0 and row[\"label\"] == 0 for pred, row in zip(predictions, examples))\n",
        "fp = sum(pred == 1 and row[\"label\"] == 0 for pred, row in zip(predictions, examples))\n",
        "fn = sum(pred == 0 and row[\"label\"] == 1 for pred, row in zip(predictions, examples))\n",
        "accuracy = (tp + tn) / len(examples)\n",
        "precision = tp / (tp + fp) if tp + fp else 0.0\n",
        "recall = tp / (tp + fn) if tp + fn else 0.0\n",
        "f1 = 2 * precision * recall / (precision + recall) if precision + recall else 0.0\n",
        "metrics = {\n",
        "    \"sample_count\": len(examples),\n",
        "    \"accuracy\": accuracy,\n",
        "    \"precision\": precision,\n",
        "    \"recall\": recall,\n",
        "    \"f1\": f1,\n",
        "    \"confusion_matrix\": {\"tp\": tp, \"tn\": tn, \"fp\": fp, \"fn\": fn},\n",
        "}\n",
        "print(json.dumps(metrics, indent=2))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "uncertainty-note",
      "metadata": {},
      "source": [
        "## Report uncertainty and boundaries\n",
        "\n",
        "Six synthetic examples cannot support a general model-quality claim. A Wilson interval makes the sampling uncertainty visible, but it does not correct dataset bias, leakage, label errors, or distribution shift."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "interval",
      "metadata": {},
      "outputs": [],
      "source": [
        "z = 1.96\n",
        "n = len(examples)\n",
        "center = (accuracy + z*z/(2*n)) / (1 + z*z/n)\n",
        "margin = z * math.sqrt(accuracy*(1-accuracy)/n + z*z/(4*n*n)) / (1 + z*z/n)\n",
        "metrics[\"accuracy_wilson_95\"] = [max(0.0, center - margin), min(1.0, center + margin)]\n",
        "print(json.dumps(metrics, indent=2))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "provenance",
      "metadata": {},
      "outputs": [],
      "source": [
        "canonical_examples = json.dumps(examples, sort_keys=True, separators=(\",\", \":\"))\n",
        "record = {\n",
        "    \"schema_version\": 1,\n",
        "    \"created_at\": datetime.now(timezone.utc).isoformat(),\n",
        "    \"dataset_sha256\": hashlib.sha256(canonical_examples.encode()).hexdigest(),\n",
        "    \"prediction_rule\": \"case-insensitive substring: passed\",\n",
        "    \"python\": platform.python_version(),\n",
        "    \"metrics\": metrics,\n",
        "    \"claim_boundary\": \"Synthetic tutorial smoke test; not evidence of general model quality.\",\n",
        "}\n",
        "print(json.dumps(record, indent=2))"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.12"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}