{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "welcome",
      "metadata": {},
      "source": [
        "# Create and verify a small dataset\n",
        "\n",
        "Build a deterministic CSV dataset, calculate basic statistics, write a Data Card, and produce a SHA-256 manifest before publishing. This local tutorial uses only Python's standard library."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "imports",
      "metadata": {},
      "outputs": [],
      "source": [
        "import csv\n",
        "import hashlib\n",
        "import json\n",
        "from pathlib import Path\n",
        "from statistics import mean\n",
        "\n",
        "OUTPUT = Path(\"superii-dataset-tutorial\")\n",
        "OUTPUT.mkdir(exist_ok=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "data",
      "metadata": {},
      "source": [
        "## Define transparent source records\n",
        "\n",
        "These rows are synthetic tutorial data. A real Data Card must document collection, consent, licensing, exclusions, known bias, and intended use."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "write-csv",
      "metadata": {},
      "outputs": [],
      "source": [
        "rows = [\n",
        "    {\"sample_id\": \"sii-001\", \"text\": \"clear documentation\", \"quality\": 5},\n",
        "    {\"sample_id\": \"sii-002\", \"text\": \"reproducible example\", \"quality\": 4},\n",
        "    {\"sample_id\": \"sii-003\", \"text\": \"bounded preview\", \"quality\": 5},\n",
        "]\n",
        "csv_path = OUTPUT / \"train.csv\"\n",
        "with csv_path.open(\"w\", newline=\"\", encoding=\"utf-8\") as handle:\n",
        "    writer = csv.DictWriter(handle, fieldnames=list(rows[0]))\n",
        "    writer.writeheader()\n",
        "    writer.writerows(rows)\n",
        "print(csv_path.read_text(encoding=\"utf-8\"))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "statistics",
      "metadata": {},
      "outputs": [],
      "source": [
        "stats = {\n",
        "    \"rows\": len(rows),\n",
        "    \"columns\": list(rows[0]),\n",
        "    \"unique_sample_ids\": len({row['sample_id'] for row in rows}),\n",
        "    \"mean_quality\": mean(row[\"quality\"] for row in rows),\n",
        "    \"missing_values\": sum(value in (None, \"\") for row in rows for value in row.values()),\n",
        "}\n",
        "assert stats[\"unique_sample_ids\"] == stats[\"rows\"]\n",
        "assert stats[\"missing_values\"] == 0\n",
        "print(json.dumps(stats, indent=2))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "card",
      "metadata": {},
      "source": [
        "## Write the Data Card and immutable manifest\n",
        "\n",
        "The manifest binds the published file path, byte size, media type, and checksum. Recalculate it after any content change."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "manifest",
      "metadata": {},
      "outputs": [],
      "source": [
        "data_card = \"\"\"# Tutorial quality phrases\n",
        "\n",
        "## Summary\n",
        "Three synthetic English phrases created solely for this Super ii tutorial.\n",
        "\n",
        "## License\n",
        "CC0-1.0 for these synthetic rows.\n",
        "\n",
        "## Intended use\n",
        "Demonstrating local dataset validation. Not suitable for training or evaluation claims.\n",
        "\n",
        "## Provenance and limitations\n",
        "Created manually for this notebook; no people, private data, or external sources are represented.\n",
        "\"\"\"\n",
        "(OUTPUT / \"README.md\").write_text(data_card, encoding=\"utf-8\")\n",
        "\n",
        "def file_record(path):\n",
        "    content = path.read_bytes()\n",
        "    return {\n",
        "        \"path\": path.name,\n",
        "        \"size_bytes\": len(content),\n",
        "        \"sha256\": hashlib.sha256(content).hexdigest(),\n",
        "    }\n",
        "\n",
        "manifest = [file_record(OUTPUT / name) for name in (\"README.md\", \"train.csv\")]\n",
        "(OUTPUT / \"manifest.json\").write_text(json.dumps(manifest, indent=2), encoding=\"utf-8\")\n",
        "print(json.dumps(manifest, indent=2))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "verify",
      "metadata": {},
      "outputs": [],
      "source": [
        "for record in manifest:\n",
        "    path = OUTPUT / record[\"path\"]\n",
        "    assert path.stat().st_size == record[\"size_bytes\"]\n",
        "    assert hashlib.sha256(path.read_bytes()).hexdigest() == record[\"sha256\"]\n",
        "print(\"Verified every manifest entry. Review the files before uploading them.\")"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.12"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}