{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cell-00",
   "metadata": {},
   "source": [
    "# Train a language model\n",
    "\n",
    "In this notebook we train a small decoder-only transformer on the complete works of Shakespeare, one byte at a time, and then let it write. The model has the parts current open models use: RMSNorm, rotary position embeddings, grouped-query attention and a gated MLP. Dew builds it from the registry, `LMObjective` holds the next-token loss, and the same `Trainer` as the diffusion notebooks runs the training.\n",
    "\n",
    "The notebook expects one NVIDIA GPU. It takes about four minutes on a Colab L4. The corpus is Tiny Shakespeare, 1.1 MB of text."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-01",
   "metadata": {},
   "source": [
    "## Install"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "cell-02",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-09-22T20:18:25.832806Z",
     "iopub.status.busy": "2026-09-22T20:18:25.832594Z",
     "iopub.status.idle": "2026-09-22T20:18:48.483848Z",
     "shell.execute_reply": "2026-09-22T20:18:48.483135Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
      "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[?25l   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/532.1 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m532.1/532.1 kB\u001b[0m \u001b[31m36.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h\u001b[?25l   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/216.9 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m216.9/216.9 kB\u001b[0m \u001b[31m25.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Building wheel for dew-ml (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n"
     ]
    }
   ],
   "source": [
    "%pip install -q \"dew-ml @ git+https://github.com/AshishKumar4/dew\" \"jax[cuda12]\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-03",
   "metadata": {},
   "source": [
    "## Settings\n",
    "\n",
    "`SEQUENCE_LENGTH` is how many bytes the model sees at once. The model is 6 layers of width 384, about 14M parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "cell-04",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-09-22T20:18:48.486714Z",
     "iopub.status.busy": "2026-09-22T20:18:48.486491Z",
     "iopub.status.idle": "2026-09-22T20:18:48.490391Z",
     "shell.execute_reply": "2026-09-22T20:18:48.489665Z"
    }
   },
   "outputs": [],
   "source": [
    "SEQUENCE_LENGTH = 256\n",
    "BATCH_SIZE = 64\n",
    "STEPS = 1500\n",
    "LEARNING_RATE = 1e-3\n",
    "EMB_FEATURES = 384\n",
    "NUM_LAYERS = 6\n",
    "NUM_HEADS = 6\n",
    "MAX_NEW_TOKENS = 400\n",
    "PROMPT = \"ROMEO:\"\n",
    "DATA_DIR = \"data/05-shakespeare\"\n",
    "RUN_DIR = \"runs/05-lm\"\n",
    "SEED = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "cell-05",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-09-22T20:18:48.492832Z",
     "iopub.status.busy": "2026-09-22T20:18:48.492626Z",
     "iopub.status.idle": "2026-09-22T20:18:51.717531Z",
     "shell.execute_reply": "2026-09-22T20:18:51.716954Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CudaDevice(id=0)]\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "import urllib.request\n",
    "from pathlib import Path\n",
    "\n",
    "import jax\n",
    "import jax.numpy as jnp\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "print(jax.devices())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-06",
   "metadata": {},
   "source": [
    "## From text to tokens\n",
    "\n",
    "A language model predicts the next token from the ones before it. The simplest tokenizer is the byte tokenizer: every byte of UTF-8 text is one token, so the vocabulary has 256 entries and needs no download. It makes sequences long, since every character is a token, but it is easy to read and fine for 1 MB of English.\n",
    "\n",
    "Dew's token loader reads a folder with three files: `train.bin` and `val.bin`, the token ids as one flat array each, and `meta.json`, which says how to read them. We download the text, encode it, and keep the first 5% as validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "cell-07",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-09-22T20:18:51.720483Z",
     "iopub.status.busy": "2026-09-22T20:18:51.719984Z",
     "iopub.status.idle": "2026-09-22T20:18:53.686498Z",
     "shell.execute_reply": "2026-09-22T20:18:53.685744Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "First Citizen:\n",
      "Before we proceed any further, hear me speak.\n",
      "\n",
      "All:\n",
      "Speak, speak.\n",
      "\n",
      "First Citizen:\n",
      "You are all resolved rather to die than to famish?\n",
      "\n",
      "All:\n",
      "Resolved. resolved.\n",
      "\n",
      "First Citizen:\n",
      "First, you know Caius Marcius is chief enemy to the people.\n",
      "\n",
      "All:\n",
      "We know't, we know't.\n",
      "\n",
      "First Citizen:\n",
      "Let us\n"
     ]
    }
   ],
   "source": [
    "from dew.data import ByteTokenizer\n",
    "\n",
    "data_dir = Path(DATA_DIR)\n",
    "data_dir.mkdir(parents=True, exist_ok=True)\n",
    "text_path = data_dir / \"input.txt\"\n",
    "urllib.request.urlretrieve(\n",
    "    \"https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt\", text_path)\n",
    "text = text_path.read_text(encoding=\"utf-8\")\n",
    "print(text[:300])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "cell-08",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-09-22T20:18:53.689677Z",
     "iopub.status.busy": "2026-09-22T20:18:53.689288Z",
     "iopub.status.idle": "2026-09-22T20:18:53.744261Z",
     "shell.execute_reply": "2026-09-22T20:18:53.743629Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'tokenizer': 'byte', 'vocab_size': 256, 'dtype': 'uint8', 'train_tokens': 1059625, 'val_tokens': 55769, 'eos_id': None}\n"
     ]
    }
   ],
   "source": [
    "tokenizer = ByteTokenizer()\n",
    "ids = np.asarray(tokenizer.encode(text), np.uint8)\n",
    "val_len = len(ids) // 20\n",
    "ids[:val_len].tofile(data_dir / \"val.bin\")\n",
    "ids[val_len:].tofile(data_dir / \"train.bin\")\n",
    "meta = {\"tokenizer\": \"byte\", \"vocab_size\": tokenizer.vocab_size, \"dtype\": \"uint8\",\n",
    "        \"train_tokens\": len(ids) - val_len, \"val_tokens\": val_len, \"eos_id\": None}\n",
    "(data_dir / \"meta.json\").write_text(json.dumps(meta, indent=2))\n",
    "print(meta)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-09",
   "metadata": {},
   "source": [
    "`TokenWindows` cuts the training stream into windows of `SEQUENCE_LENGTH + 1` tokens. The first 256 ids of a window are the input and the same ids shifted by one are the targets, so every position learns to predict the byte after it. `val_batches=8` scores up to eight batches of validation windows at each evaluation; our validation split fills three."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "cell-10",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-09-22T20:18:53.747279Z",
     "iopub.status.busy": "2026-09-22T20:18:53.746709Z",
     "iopub.status.idle": "2026-09-22T20:18:53.765270Z",
     "shell.execute_reply": "2026-09-22T20:18:53.764685Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training windows: 4139 | steps per epoch: 64\n",
      "(64, 257)\n",
      "'u Kate,\\nAnd bring you from a wild Kate to a Kate\\nConformable as other household Kates.\\nHere comes your father: never mak'\n"
     ]
    }
   ],
   "source": [
    "from dew.data import Loading, TokenWindows\n",
    "\n",
    "data = TokenWindows(\n",
    "    path=DATA_DIR, seq_len=SEQUENCE_LENGTH, val_batches=8,\n",
    "    loading=Loading(workers=0, threads=1, read_buffer=2),\n",
    ").load(batch=BATCH_SIZE)\n",
    "print(\"training windows:\", data.records, \"| steps per epoch:\", data.steps_per_epoch)\n",
    "\n",
    "batch = next(iter(data.train()))\n",
    "print(batch[\"text\"].shape)\n",
    "print(repr(tokenizer.decode(batch[\"text\"][0][:120])))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-11",
   "metadata": {},
   "source": [
    "## The model\n",
    "\n",
    "`models.build(\"causal_transformer\", ...)` builds the decoder. `max_seq_len` sets the size of the KV cache used for generation, so it has to cover the prompt plus every token we generate at the end, not only the training length. We compute in bfloat16 and keep the weights in float32. `dropout_rate=0.2` randomly zeroes a fifth of the activations during training. It slows memorisation, which matters here: 1 MB of text cuts into only about 4,000 training windows, and a model this size can learn them by heart."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "cell-12",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-09-22T20:18:53.768097Z",
     "iopub.status.busy": "2026-09-22T20:18:53.767609Z",
     "iopub.status.idle": "2026-09-22T20:19:01.622107Z",
     "shell.execute_reply": "2026-09-22T20:19:01.621491Z"
    }
   },
   "outputs": [],
   "source": [
    "from dew import models\n",
    "\n",
    "model = models.build(\n",
    "    \"causal_transformer\",\n",
    "    vocab_size=meta[\"vocab_size\"],\n",
    "    emb_features=EMB_FEATURES,\n",
    "    num_layers=NUM_LAYERS,\n",
    "    num_heads=NUM_HEADS,\n",
    "    dropout_rate=0.2,\n",
    "    max_seq_len=len(PROMPT) + MAX_NEW_TOKENS,\n",
    "    dtype=\"bfloat16\",\n",
    "    attention_impl=\"auto\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-13",
   "metadata": {},
   "source": [
    "## The objective and the trainer\n",
    "\n",
    "`LMObjective` is next-token cross entropy. It shifts the batch into inputs and targets itself and computes the loss in float32. Like the diffusion objective, it keeps an EMA of the weights.\n",
    "\n",
    "The learning rate warms up over the first 10% of steps and then decays along a cosine. `eval_every` runs the validation pass from the EMA weights, and `metrics.perplexity()` turns it into one number, which the next section explains."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "cell-14",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-09-22T20:19:01.625214Z",
     "iopub.status.busy": "2026-09-22T20:19:01.624774Z",
     "iopub.status.idle": "2026-09-22T20:19:02.636039Z",
     "shell.execute_reply": "2026-09-22T20:19:02.635478Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "14.3M parameters\n"
     ]
    }
   ],
   "source": [
    "import optax\n",
    "from dew import Checkpoints, LocalTracker, Trainer, metrics\n",
    "from dew.objectives.lm import LMObjective\n",
    "\n",
    "objective = LMObjective(model, SEQUENCE_LENGTH, ema_decay=0.999)\n",
    "schedule = optax.warmup_cosine_decay_schedule(\n",
    "    init_value=0.0, peak_value=LEARNING_RATE,\n",
    "    warmup_steps=STEPS // 10, decay_steps=STEPS, end_value=LEARNING_RATE / 10)\n",
    "trainer = Trainer(\n",
    "    objective, optax.adamw(schedule),\n",
    "    key=jax.random.key(SEED),\n",
    "    checkpoints=Checkpoints(RUN_DIR),\n",
    "    tracker=LocalTracker(f\"{RUN_DIR}/tracking\"),\n",
    ")\n",
    "\n",
    "variables = jax.eval_shape(objective.init, jax.random.key(0))\n",
    "n_params = sum(x.size for x in jax.tree_util.tree_leaves(variables[\"params\"]))\n",
    "print(f\"{n_params / 1e6:.1f}M parameters\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-15",
   "metadata": {},
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "cell-16",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-09-22T20:19:02.638884Z",
     "iopub.status.busy": "2026-09-22T20:19:02.638425Z",
     "iopub.status.idle": "2026-09-22T20:21:42.494787Z",
     "shell.execute_reply": "2026-09-22T20:21:42.494120Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training from step 0 to 1500 on {'data': 1, 'expert': 1, 'fsdp': 1, 'tensor': 1, 'sequence': 1, 'stage': 1} (1 process(es))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step 150: loss 2.0032\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step 300: loss 1.5412\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step 450: loss 1.4044\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluation val at step 500: 3 coordinated batches, 192 records, uneven_shards=False, event_key=(1753219770, 2831660841): {'val/perplexity': 11.98130281857004}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step 600: loss 1.3257\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step 750: loss 1.2502\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step 900: loss 1.2015\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluation val at step 1000: 3 coordinated batches, 192 records, uneven_shards=False, event_key=(586219947, 2283116771): {'val/perplexity': 5.102528755990834}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step 1050: loss 1.1233\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step 1200: loss 1.1310\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step 1350: loss 1.1093\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step 1500: loss 1.0713\n",
      "Evaluation val at step 1500: 3 coordinated batches, 192 records, uneven_shards=False, event_key=(352661883, 2305041505): {'val/perplexity': 4.1767878417399285}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Goodput: first step after 40.58 s, 67.6% of the wall time in steps\n"
     ]
    }
   ],
   "source": [
    "state = trainer.fit(data, steps=STEPS, log_every=150, eval_every=500, checkpoint_every=STEPS,\n",
    "                    metrics=(metrics.perplexity(),))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-17",
   "metadata": {},
   "source": [
    "## Perplexity\n",
    "\n",
    "Perplexity is $e^{\\text{cross entropy}}$: roughly, how many tokens the model is choosing between at each position. An untrained byte model scores about 256, one guess out of every possible byte. The plot shows the training loss and the validation perplexity from the tracker's journal. Validation perplexity falls from about 12 at step 500 to about 4 at the end: the model has narrowed each next byte down to a handful of choices."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "cell-18",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-09-22T20:21:42.499898Z",
     "iopub.status.busy": "2026-09-22T20:21:42.499072Z",
     "iopub.status.idle": "2026-09-22T20:21:42.758530Z",
     "shell.execute_reply": "2026-09-22T20:21:42.757959Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAzoAAAE8CAYAAAAId8SAAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlHJYcgAAAAlwSFlzAAAPYQAAD2EBqD+naQAAZN9JREFUeJzt3XlYVGX/BvD7zLBvg+ygCLgvoKK44pp7ZqmvS2YqWqapqS1m1i/Nyijr7c1yyazUctfUSs0k911QUXFXQFBBRGTfZ57fH8jkCCj7mRnuz3XNpZw5Z849B3gevnOe8xxJCCFARERERERkRBRyByAiIiIiIqpsLHSIiIiIiMjosNAhIiIiIiKjw0KHiIiIiIiMDgsdIiIiIiIyOix0iIiIiIjI6LDQISIiIiIio8NCh4iIiIiIjA4LHSIiIiIiMjosdKhG8/b2RlBQULm27d69O7p3716peUqrIrmJiAhYuXIlJElCdHS0dllp2/X9+/dDkiTs37+/UjNJkoSPPvqoUl/TkBX3Paps7E+NGwsd0mtHjx7FRx99hOTkZLmjEBERVdjOnTtZzOixixcv4qOPPqrS4oqqj4ncAYie5OjRo5g3bx6CgoJgb29f6a9/5coVKBTlq/d3795dyWmIiEhO1dGu79y5E4sXLy622MnKyoKJCf80q06P/x1w8eJFzJs3D927d4e3t7d8wahS8LeJjIZGo0Fubi4sLCxKvY25uXm592dmZlbubYmISP/I3a6Xpf8yNEIIZGdnw9LSUu4oOirydwDpPw5dI7310UcfYebMmQAAHx8fSJKkM1ZXkiRMnToVa9asQfPmzWFubo5du3YBAL766it06tQJjo6OsLS0RJs2bbB58+Yi+3h8bG7heOAjR47grbfegrOzM6ytrTF48GDcu3dPZ9vHx3IXjtneuHEj5s+fjzp16sDCwgI9e/bE9evXi+x78eLFqFevHiwtLdGuXTscOnSoQtf9REZGYtiwYXBwcICVlRU6dOiAHTt2FFnvu+++Q/PmzWFlZYVatWohICAAa9eu1T6flpaGGTNmwNvbG+bm5nBxcUHv3r1x+vTpcuUiIqqozZs3Q5IkHDhwoMhzy5YtgyRJiIiIAACcO3cOQUFBqFevHiwsLODm5obx48fj/v37T91PcW3wrVu3MGjQIFhbW8PFxQVvvvkmcnJyimx76NAhDBs2DHXr1oW5uTk8PT3x5ptvIisrS7tOUFAQFi9eDADaPk2SJO3zxV2jc+bMGfTv3x92dnawsbFBz549cfz4cZ11ytJ3FScoKAg2NjaIjIxE3759YW1tDQ8PD3z88ccQQuisq9Fo8M0336B58+awsLCAq6srJk6ciAcPHuis5+3tjeeeew5///03AgICYGlpiWXLlmnfZ2H/3bhxY1hYWKBNmzY4ePDgU7MCwF9//YUuXbrA2toatra2GDBgAC5cuKB9fu/evVAoFJgzZ47OdmvXroUkSVi6dKlOzsK/A1auXIlhw4YBAHr06KH9/uzfvx9jx46Fk5MT8vLyiuTp06cPGjduXKrsVL14Rof01pAhQ3D16lWsW7cO//vf/+Dk5AQAcHZ21q6zd+9ebNy4EVOnToWTk5P2NPPChQvx/PPPY9SoUcjNzcX69esxbNgwbN++HQMGDHjqvt944w3UqlULc+fORXR0NL755htMnToVGzZseOq2n3/+ORQKBd555x2kpKRgwYIFGDVqFE6cOKFdZ+nSpZg6dSq6dOmCN998E9HR0Rg0aBBq1aqFOnXqlPFIAXfv3kWnTp2QmZmJadOmwdHREatWrcLzzz+PzZs3Y/DgwQCA5cuXY9q0aRg6dCimT5+O7OxsnDt3DidOnMBLL70EAJg0aRI2b96MqVOnolmzZrh//z4OHz6MS5cuoXXr1mXORkRUUQMGDICNjQ02btyIbt266Ty3YcMGNG/eHL6+vgCAkJAQREZGYty4cXBzc8OFCxfwww8/4MKFCzh+/LhOYfE0WVlZ6NmzJ2JiYjBt2jR4eHjg119/xd69e4usu2nTJmRmZuL111+Ho6MjTp48ie+++w63bt3Cpk2bAAATJ07EnTt3EBISgl9//fWp+79w4QK6dOkCOzs7vPvuuzA1NcWyZcvQvXt3HDhwAO3bt9dZvyJ9l1qtRr9+/dChQwcsWLAAu3btwty5c5Gfn4+PP/5Yu97EiROxcuVKjBs3DtOmTUNUVBQWLVqEM2fO4MiRIzA1NdWue+XKFYwcORITJ07EhAkTdIqBAwcOYMOGDZg2bRrMzc2xZMkS9OvXDydPntR+L4vz66+/YuzYsejbty+++OILZGZmYunSpejcuTPOnDkDb29vPPPMM5g8eTKCg4MxaNAgtG7dGnFxcXjjjTfQq1cvTJo0qdjX7tq1K6ZNm4Zvv/0W77//Ppo2bQoAaNq0KUaPHo1ffvkFf//9N5577jntNvHx8di7dy/mzp371GNMMhBEeuzLL78UAERUVFSR5wAIhUIhLly4UOS5zMxMna9zc3OFr6+veOaZZ3SWe3l5ibFjx2q/XrFihQAgevXqJTQajXb5m2++KZRKpUhOTtYu69atm+jWrZv263379gkAomnTpiInJ0e7fOHChQKAOH/+vBBCiJycHOHo6Cjatm0r8vLytOutXLlSANB5zZI8nnvGjBkCgDh06JB2WVpamvDx8RHe3t5CrVYLIYR44YUXRPPmzZ/42iqVSkyZMuWpGYiIqtPIkSOFi4uLyM/P1y6Li4sTCoVCfPzxx9plj7f/Qgixbt06AUAcPHhQu6ywvX+0f3m8Xf/mm28EALFx40btsoyMDNGgQQMBQOzbt++J+w0ODhaSJImbN29ql02ZMkWU9OcXADF37lzt14MGDRJmZmbixo0b2mV37twRtra2omvXrkXeS2n6ruKMHTtWABBvvPGGdplGoxEDBgwQZmZm4t69e0IIIQ4dOiQAiDVr1uhsv2vXriLLvby8BACxa9euYt8nABEWFqZddvPmTWFhYSEGDx5c5H0Vfo/S0tKEvb29mDBhgs7rxcfHC5VKpbO88PvUvHlzkZ2dLQYMGCDs7Ox0vheFOR/tTzdt2lTkeyuEEGq1WtSpU0eMGDFCZ/nXX38tJEkSkZGRRd4nyY9D18igdevWDc2aNSuy/NExwA8ePEBKSgq6dOlS6uFXr732ms6nfl26dIFarcbNmzefuu24ceN0xnl36dIFQMHQMgAICwvD/fv3MWHCBJ2LTkeNGoVatWqVKt/jdu7ciXbt2qFz587aZTY2NnjttdcQHR2NixcvAgDs7e1x69YthIaGlvha9vb2OHHiBO7cuVOuLEREVWHEiBFISEjQmdJ58+bN0Gg0GDFihHbZo+1/dnY2EhMT0aFDBwAo8xDcnTt3wt3dHUOHDtUus7KywmuvvVZk3Uf3m5GRgcTERHTq1AlCCJw5c6ZM+wUKzrDs3r0bgwYNQr169bTL3d3d8dJLL+Hw4cNITU3V2aYifRcATJ06Vfv/wuFlubm5+OeffwAUnLVSqVTo3bs3EhMTtY82bdrAxsYG+/bt03k9Hx8f9O3bt9h9dezYEW3atNF+XbduXbzwwgv4+++/oVari90mJCQEycnJGDlypM7+lUol2rdvr7N/KysrrFy5EpcuXULXrl2xY8cO/O9//0PdunVLdSwep1AoMGrUKPzxxx9IS0vTLl+zZg06deoEHx+fcr0uVS0WOmTQSmpYtm/fjg4dOsDCwgIODg5wdnbG0qVLkZKSUqrXfbwhLCxAHh+DXJ5tCzucBg0a6KxnYmJS7hlebt68Wez44MLT7oX7nDVrFmxsbNCuXTs0bNgQU6ZMwZEjR3S2WbBgASIiIuDp6Yl27drho48+0hZpRERy6devH1Qqlc4wrA0bNqBVq1Zo1KiRdllSUhKmT58OV1dXWFpawtnZWdtXlLYPKHTz5k00aNCgyHC34trbmJgYBAUFwcHBATY2NnB2dtYOsyvrfgHg3r17yMzMLLFt12g0iI2N1Vlekb5LoVDoFFQAtMe18NrYa9euISUlBS4uLnB2dtZ5pKenIyEhQWf7J/3x37BhwyLLGjVqhMzMzBKvK7p27RoA4Jlnnimy/927dxfZf2BgIF5//XWcPHkSffv2xfjx4598EJ5izJgxyMrKwtatWwEUDM07deoURo8eXaHXparDa3TIoBU3e8uhQ4fw/PPPo2vXrliyZAnc3d1hamqKFStW6Fx0/yRKpbLY5eKxizIre9uq1rRpU1y5cgXbt2/Hrl278Ntvv2HJkiWYM2cO5s2bBwAYPnw4unTpgq1bt2L37t348ssv8cUXX2DLli3o37+/zO+AiGoqc3NzDBo0CFu3bsWSJUtw9+5dHDlyBJ999pnOesOHD8fRo0cxc+ZMtGrVCjY2NtBoNOjXrx80Gk2VZFOr1ejduzeSkpIwa9YsNGnSBNbW1rh9+zaCgoKqbL+Pq+r+R6PRwMXFBWvWrCn2+UevoQWK76Mrun+g4DodNze3Is8/PjV3Tk6O9gzgjRs3kJmZCSsrq3Lvv1mzZmjTpg1Wr16NMWPGYPXq1TAzM8Pw4cPL/ZpUtVjokF4ry0WjhX777TdYWFjg77//1pk2csWKFZUZrdy8vLwAANevX0ePHj20y/Pz8xEdHY0WLVqU6zWvXLlSZPnly5d19gkA1tbWGDFiBEaMGIHc3FwMGTIE8+fPx+zZs7VTm7q7u2Py5MmYPHkyEhIS0Lp1a8yfP5+FDhHJasSIEVi1ahX27NmDS5cuQQihM2ztwYMH2LNnD+bNm6cz41bhmYCy8vLyQkREBIQQOv3R4+3t+fPncfXqVaxatQpjxozRLg8JCSnymqXt15ydnWFlZVVi265QKODp6Vnat/JUGo0GkZGROmfHrl69CgDa0Qb169fHP//8g8DAwAoXMcV9T65evQorK6siBVOh+vXrAwBcXFzQq1evp+5j7ty5uHTpEr766ivMmjUL7733Hr799tsnbvO078+YMWPw1ltvIS4uDmvXrsWAAQPKPeycqh6HrpFes7a2BgAkJyeXehulUglJknTG+EZHR2Pbtm2VnK58AgIC4OjoiOXLlyM/P1+7fM2aNaUaXlCcZ599FidPnsSxY8e0yzIyMvDDDz/A29tbex3T49OrmpmZoVmzZhBCIC8vD2q1usgQCxcXF3h4eBQ7nSoRUXXq1asXHBwcsGHDBmzYsAHt2rXTGR5VeEbj8TMY33zzTbn29+yzz+LOnTs6tyfIzMzEDz/8oLNecfsVQmDhwoVFXrO0/ZpSqUSfPn3w+++/a4eOAQWzbK5duxadO3eGnZ1dWd/SEy1atEj7fyEEFi1aBFNTU/Ts2RNAwdkytVqNTz75pMi2+fn5Zeqrjx07pnPNVGxsLH7//Xf06dOnxDNTffv2hZ2dHT777LNip3l+dMjbiRMn8NVXX2HGjBl4++23MXPmTCxatKjYKcof9bTvz8iRIyFJEqZPn47IyEi8/PLLT3urJCOe0SG9Vnih4gcffIAXX3wRpqamGDhwoLYhKs6AAQPw9ddfo1+/fnjppZeQkJCAxYsXo0GDBjh37lx1RS+RmZkZPvroI7zxxht45plnMHz4cERHR2PlypWoX79+uc5ivffee1i3bh369++PadOmwcHBAatWrUJUVBR+++037V2f+/TpAzc3NwQGBsLV1RWXLl3CokWLMGDAANja2iI5ORl16tTB0KFD0bJlS9jY2OCff/5BaGgo/vvf/1b2oSAiKhNTU1MMGTIE69evR0ZGBr766iud5+3s7NC1a1csWLAAeXl5qF27Nnbv3o2oqKhy7W/ChAlYtGgRxowZg1OnTsHd3R2//vprkeFPTZo0Qf369fHOO+/g9u3bsLOzw2+//Vbsh1eF/dq0adPQt29fKJVKvPjii8Xu/9NPP0VISAg6d+6MyZMnw8TEBMuWLUNOTg4WLFhQrvdUEgsLC+zatQtjx45F+/bt8ddff2HHjh14//33tWdYunXrhokTJyI4OBjh4eHo06cPTE1Nce3aNWzatAkLFy7UmbjhSXx9fdG3b1+d6aUBaIdRF8fOzg5Lly7F6NGj0bp1a7z44otwdnZGTEwMduzYgcDAQCxatAjZ2dkYO3YsGjZsiPnz52tf988//8S4ceNw/vz5Ev+OaNWqFZRKJb744gukpKTA3NwczzzzDFxcXAAUnGnr168fNm3aBHt7+1LdsoJkJNNsb0Sl9sknn4jatWsLhUKhM80kgBKnQf7pp59Ew4YNhbm5uWjSpIlYsWKFmDt3bpEpPUuaXjo0NFRnvcKpox+dbrKk6aU3bdqks21UVJQAIFasWKGz/NtvvxVeXl7C3NxctGvXThw5ckS0adNG9OvX76nH5PHcQghx48YNMXToUGFvby8sLCxEu3btxPbt23XWWbZsmejatatwdHQU5ubmon79+mLmzJkiJSVFCFEw9fXMmTNFy5Ytha2trbC2thYtW7YUS5YseWomIqLqEBISIgAISZJEbGxskedv3bolBg8eLOzt7YVKpRLDhg0Td+7cKTJ1c2mmlxaiYNrj559/XlhZWQknJycxffp07XTKj/YJFy9eFL169RI2NjbCyclJTJgwQZw9e7ZI+5+fny/eeOMN4ezsLCRJ0umXHs8ohBCnT58Wffv2FTY2NsLKykr06NFDHD16VGedsvRdxRk7dqywtrYWN27cEH369BFWVlbC1dVVzJ07V3t7gkf98MMPok2bNsLS0lLY2toKPz8/8e6774o7d+5o1/Hy8hIDBgwodn+F/ffq1au1fbW/v3+RnMV9jwrfV9++fYVKpRIWFhaifv36IigoSDtddeG02idOnNDZLiwsTJiYmIjXX39dJ+fj/eny5ctFvXr1hFKpLPb4bdy4UQAQr732WrHvj/SHJIQeXCFNRNBoNHB2dsaQIUOwfPlyueMQEVENERQUhM2bNyM9Pb1a9idJEqZMmaIzVM6Q/P777xg0aBAOHjyovYUE6Sdeo0Mkg+zs7CJjyH/55RckJSWhe/fu8oQiIiKip1q+fDnq1aunc+860k+8RodIBsePH8ebb76JYcOGwdHREadPn8ZPP/0EX19fDBs2TO54RERE9Jj169fj3Llz2LFjBxYuXFiua2qperHQIZKBt7c3PD098e233yIpKQkODg4YM2YMPv/8c5iZmckdj4iIiB4zcuRI2NjY4JVXXsHkyZPljkOlwGt0iIiIiIjI6PAaHSIiIiIiMjosdIiIiIiIyOgYxDU6Go0Gd+7cga2tLS/8IiKqRkIIpKWlwcPDQ3vjWSrAvomISB6l7ZsMotC5c+cOPD095Y5BRFRjxcbGok6dOnLH0Cvsm4iI5PW0vskgCh1bW1sABW/Gzs5O5jRERDVHamoqPD09te0w/Yt9ExGRPErbNxlEoVM4JMDOzo6dCRGRDDg0qyj2TURE8npa38QB10REREREZHRY6BARERERkdExiKFrRERExkStETgZlYSEtGy42FqgnY8DlAoODyQiqkw8o0NEREbn4MGDGDhwIDw8PCBJErZt26Z9Li8vD7NmzYKfnx+sra3h4eGBMWPG4M6dO9WSbVdEHDp/sRcjlx/H9PXhGLn8ODp/sRe7IuKqZf9ERDVFmQqd4OBgtG3bFra2tnBxccGgQYNw5cqVp263adMmNGnSBBYWFvDz88POnTvLHbg84lKycPRGIuJSsqp1v0REJI+MjAy0bNkSixcvLvJcZmYmTp8+jQ8//BCnT5/Gli1bcOXKFTz//PNVnmtXRBxeX30acSnZOsvjU7Lx+urTLHaIiCpRmYauHThwAFOmTEHbtm2Rn5+P999/H3369MHFixdhbW1d7DZHjx7FyJEjERwcjOeeew5r167FoEGDcPr0afj6+lbKm3iSDaExmL3lPDQCUEhA8BA/jGhbt8r3S0RE8unfvz/69+9f7HMqlQohISE6yxYtWoR27dohJiYGdetWTR+h1gjM+/MiRDHPCQASgHl/XkTvZm4cxkZEVAnKVOjs2rVL5+uVK1fCxcUFp06dQteuXYvdZuHChejXrx9mzpwJAPjkk08QEhKCRYsW4fvvvy9n7NKJS8nCe1vOQzzsVTQCeH9LBLo2coa7yrJK901ERIYjJSUFkiTB3t6+xHVycnKQk5Oj/To1NbVM+zgZlVTkTM6jBIC4lGycjEpCx/qOZXptIiIqqkLX6KSkpAAAHBwcSlzn2LFj6NWrl86yvn374tixYyVuk5OTg9TUVJ1HeUQlZmiLnEJqIRCdmFmu1yMiIuOTnZ2NWbNmYeTIkU+8H05wcDBUKpX24enpWab9JKSVXOSUZz0iInqychc6Go0GM2bMQGBg4BOHoMXHx8PV1VVnmaurK+Lj40vcpqKdSSEfJ2s8fvZfKUnwdrIq1+sREZFxycvLw/DhwyGEwNKlS5+47uzZs5GSkqJ9xMbGlmlfLrYWlboeERE9WbkLnSlTpiAiIgLr16+vzDwAKt6ZFHJXWSJ4iB8erXU+G+LLYWtERKQtcm7evImQkJAnns0BAHNzc9jZ2ek8yqKdjwPcVRYo6eobCYC7qmCqaSIiqrhyFTpTp07F9u3bsW/fPtSpU+eJ67q5ueHu3bs6y+7evQs3N7cSt6loZ/KoEW3rYvPrHQEUdCI9mriU+7WIiMg4FBY5165dwz///ANHx6q/JkapkDB3YDMAKLbYEQBm92/CiQiIiCpJmQodIQSmTp2KrVu3Yu/evfDx8XnqNh07dsSePXt0loWEhKBjx45lS1oBbbwc0NLTHgLAX+dLHjJHRETGIT09HeHh4QgPDwcAREVFITw8HDExMcjLy8PQoUMRFhaGNWvWQK1WIz4+HvHx8cjNza3SXP183bH05dZwU+kOTyssbXZfvAuNprh52YiIqKzKNOvalClTsHbtWvz++++wtbXVXmejUqlgaVkwHGzMmDGoXbs2goODAQDTp09Ht27d8N///hcDBgzA+vXrERYWhh9++KGS38qTDWzhjrOxydh+7g7GdvKu1n0TEVH1CgsLQ48ePbRfv/XWWwCAsWPH4qOPPsIff/wBAGjVqpXOdvv27UP37t2rNFs/X3f0buaGk1FJSEjLhoutBdQaDcatDMX2c3HwdLDCrH5NqjQDEVFNUKZCp/BCzcc7gRUrViAoKAgAEBMTA4Xi3xNFnTp1wtq1a/F///d/eP/999GwYUNs27atWu6h86gBLdzx6Y5LCI1+gDvJWfCw53U6RETGqnv37hCPT7v5iCc9Vx2UCqnIFNKfD2mBtzedxdL9N1DXwQoj2/Geb0REFVGmQqc0HcP+/fuLLBs2bBiGDRtWll1VOneVJdp5O+BkdBJ2no/Dq13qyZqHiIjoUf9pUwcxSZlYuOca/m9bBDzsLdGtkbPcsYiIDFaF7qNjaJ5r6Q4A+PPsHZmTEBERFTWjV0MM8a8NtUZgyprTuBRXvvvIERFRDSt0+vu6QyEBZ2+lIOY+bxpKRET6RZIkfP6fFuhQzwHpOfkYvzIUd1N5A1EiovKoUYWOs625dkz0n+d4VoeIiPSPmYkCy14OQH1na8SlZGPcilCk5+TLHYuIyODUqEIHAAa28AAAbD8XJ3MSIiKi4qmsTLFyXDs42ZjhYlwq3lh7GvlqjdyxiIgMSo0rdPr5usFEIeFSXCquJ6TLHYeIiKhYng5W+HFsW1iYKrDvyj189OcF2WeLIyIyJDWu0LG3MkOXhk4AgO0cvkZERHqslac9vhnhD0kCVh+PwY+HouSORERkMGpcoQMAzz0cvvbn2Tv8dIyIiPRaP183fPBsUwDAZ39dwl/nOfSaiKg0amSh07u5K8xMFLhxLwOX49PkjkNERPREr3T2wZiOXhACmLEhHKdjHsgdiYhI79XIQsfOwhTdH96EjffUISIifSdJEuY81wzPNHFBTr4GE1aF8TYJRERPUSMLHQAY2PLf2dc4fI2IiPSdiVKB70b6o7mHHe5n5CJo5UkkZ+bKHYuISG/V2EKnZ1MXWJoqEZOUiXO3UuSOQ0RE9FTW5ib4OagtPFQWiLyXgYm/nkJOvlruWEREeqnGFjpWZibo2dQFAGdfIyIiw+FqZ4Gfx7WFjbkJTkQl4b3fznNkAhFRMWpsoQP8O/va9nNx0GjYSRARkWFo4maHJaNaQ6mQsPXMbfzvn2tyRyIi0js1utDp3tgZNuYmiEvJ5gw2RERkULo2csb8Qb4AgG/3XMPmU7dkTkREpF9qdKFjYapEn2auADj7GhERGZ4X29XF5O71AQDv/XYOR68nypyIiEh/1OhCB/h39rUd5+Oh5vA1IiIyMO/0aYyBLT2QrxGYuPoUrt3l/eGIiAAWOghs4ASVpSkS03NwIvK+3HGIiIjKRKGQ8OXQFgjwqoW07HwErQhFQlq23LGIiGRX4wsdMxMF+vu6AQD+PBcncxoiIqKyszBV4ocxAfBxssbt5Cy8uioMmbn5csciIpJVjS90gH9nX/srIg55ao3MaYiIiMrOwdoMK4LaopaVKc7dSsH09eEckk1ENRoLHQAd6jnAycYMyZl5OMILOYmIyEB5O1lj+ZgAmJkoEHLxLubvuCR3JCIi2bDQAWCiVKC/rzsA4M+zHL5GRESGK8DbAV8PbwkA+PlIFFYeiZI5ERGRPFjoPFQ4+9ruC/HIyVfLnIaIiKj8nmvhgVn9mgAAPt5+ESEX78qciIio+rHQeSjAqxbc7CyQlpOPA1fuyR2HiIioQiZ1q4eR7TyhEcC0dWdw/laK3JGIiKoVC52HFAoJA1oUDF/bztnXiIjIwEmShI9f8EWXhk7IylNj/KpQ3HqQKXcsIqJqw0LnEc89LHT+uXQXWbkcvkZERIbNVKnAklGt0cTNFvfScjB+ZShSs/PkjkVEVC1Y6Dyilac96tSyRGauGnsvJ8gdh4iIqMJsLUzxc1BbuNqZ4+rddLy++hRy83krBSIyfix0HiFJkvaeOn+evSNzGiIiosrhYW+Jn8a2hZWZEkeu38cHW89DCN5jh4iMGwudxwxsWTB8bd+VBKTx9D4RERkJ39oqLH6pNRQSsOnULSzed13uSEREVYqFzmOauduhnpM1cvI1+OcSp+MkIiLj0aOJC+Y93xwA8NXuq/g9/LbMiYiIqg4LncdIkoTnHt5TZztvHkpEREZmdEdvTOjiAwCYuekcTkYlyZyIiKhqsNApxsCHs68dvHYPKZkcvkZERMZldv+m6O/rhly1Bq/9GoYb99LljkREVOlY6BSjoastmrjZIk8t8PeFeLnjEBERVSqFQsL/RrRCK097JGfmYdyKUNxPz5E7FhFRpWKhU4KBD4ev/XmOs68REZHxsTBV4sexAfB0sERMUiYm/BKG7DzeQ46IjAcLnRIU3jz06I37SOSnXEREZIScbMyxIqgdVJamOB2TjLc2hkOj4bTTRGQcylzoHDx4EAMHDoSHhwckScK2bdueus2aNWvQsmVLWFlZwd3dHePHj8f9+/fLk7faeDlao0UdFdQagb8iOHyNiIiMUwMXGywb3QamSgk7z8fji78vyx2JiKhSlLnQycjIQMuWLbF48eJSrX/kyBGMGTMGr7zyCi5cuIBNmzbh5MmTmDBhQpnDVrfCszrbefNQIiIyYh3qOWLB0BYAgGUHIrHmxE2ZExERVVyZC53+/fvj008/xeDBg0u1/rFjx+Dt7Y1p06bBx8cHnTt3xsSJE3Hy5Mkyh61uA1oUXKdzMjoJd1OzZU5DRERUdQb718FbvRsBAOb8fgH7riTInIiIqGKq/Bqdjh07IjY2Fjt37oQQAnfv3sXmzZvx7LPPlrhNTk4OUlNTdR5yqG1viTZetSAEsOMc76lDRETG7Y1nGmBomzpQawSmrjmNi3fk6X+JiCpDlRc6gYGBWLNmDUaMGAEzMzO4ublBpVI9cehbcHAwVCqV9uHp6VnVMUtUeE8dzr5GRETGTpIkfDbYD53qOyIjV43xK0MRl5IldywionKp8kLn4sWLmD59OubMmYNTp05h165diI6OxqRJk0rcZvbs2UhJSdE+YmNjqzpmiZ71c4ckAWdikhGblClbDiIioupgZqLA0pfboKGLDeJTszF+ZRjSc/LljkVEVGZVXugEBwcjMDAQM2fORIsWLdC3b18sWbIEP//8M+Liih8OZm5uDjs7O52HXFzsLNDBxxEAsOM8h68REZHxU1ma4uegtnCyMceluFRMWXMa+WqN3LGIiMqkygudzMxMKBS6u1EqlQAAIQxjrv7nWj6cfY3D14iIqIbwdLDCT2MDYGGqwIGr9zDnjwsG028TEQHlKHTS09MRHh6O8PBwAEBUVBTCw8MRExMDoGDY2ZgxY7TrDxw4EFu2bMHSpUsRGRmJI0eOYNq0aWjXrh08PDwq511Usf6+7lAqJETcTkVUYobccYiI6Cmeds83IQTmzJkDd3d3WFpaolevXrh27Zo8YfVYS097fPuiPyQJWHsiBj8cjJQ7EhFRqZW50AkLC4O/vz/8/f0BAG+99Rb8/f0xZ84cAEBcXJy26AGAoKAgfP3111i0aBF8fX0xbNgwNG7cGFu2bKmkt1D1HKzNENjACQDvqUNEZAieds+3BQsW4Ntvv8X333+PEydOwNraGn379kV2Nm8l8Lg+zd3w4YBmAIDgvy5zFlIiMhiSMIDz0KmpqVCpVEhJSZHtep1NYbGYufkcGrnaYPeb3WTJQERU3fSh/a0oSZKwdetWDBo0CEDB2RwPDw+8/fbbeOeddwAAKSkpcHV1xcqVK/Hiiy+W6nWN4diUxUd/XMDKo9EwM1Fg3YT2aOPlIHckIqqhStv+Vvk1OsaiT3M3mCkVuHo3HVfi0+SOQ0RE5RQVFYX4+Hj06tVLu0ylUqF9+/Y4duxYidvpyz3e5PLhc83Qq6kLcvM1mPDLKdy8z6HcRKTfWOiUksrSFF0bOQPgpARERIYsPj4eAODq6qqz3NXVVftccfTpHm9yUCokfDvSH361VUjKyMW4FaF4kJErdywiohKx0CmDgdrZ1+I48wwRUQ2jT/d4k4uVmQl+GhuA2vaWiEzMwMRfTyEnXy13LCKiYrHQKYNeTV1hYapAVGIGLtypWUMWiIiMhZubGwDg7t27Osvv3r2rfa44+nSPNzm52Flgxbi2sDU3wcnoJLy7+Rw//CMivcRCpwyszU3wTBMXAMCfHL5GRGSQfHx84Obmhj179miXpaam4sSJE+jYsaOMyQxHI1dbLH25DUwUEn4Pv4OvQ67KHYmIqAgWOmU0sEXBvX+2n+XwNSIiffWke75JkoQZM2bg008/xR9//IHz589jzJgx8PDw0M7MRk/XuaETPhviBwD4bu91bAyteUP5iEi/mcgdwND0aOICazMlbidn4UxsMlrXrSV3JCIiekxYWBh69Oih/fqtt94CAIwdOxYrV67Eu+++i4yMDLz22mtITk5G586dsWvXLlhYWMgV2SAND/BEbFImvtt7He9vPQ8Pe0t0bugkdywiIgC8j065zFh/BtvC72BcoDfmDmwudxwioiqjb+2vPuGxKSCEwIwN4fg9/A5szU2w+fVOaOxmK3csIjJivI9OFXru4fC1HefioNbofZ1IRERUZSRJwoKhLdDO2wFpOfkYvzIUCanZcsciImKhUx5dGjnBzsIECWk5CI1OkjsOERGRrMxNlFg2ug3qOVnjdnIWXlkVhszcfLljEVENx0KnHMxNlOjbvGAKUt48lIiICKhlbYYV49rCwdoM52+nYNq6Mxz1QESyYqFTTgNbFgxf++t8PPLVGpnTEBERyc/L0RrLxwTAzESBfy4l4JPtF+WOREQ1GAudcupU3xEO1ma4n5GLY5H35Y5DRESkF9p41cL/hrcCAKw8Go2fD0fJG4iIaiwWOuVkolSgv2/B8LU/z3L4GhERUaEBLdwxu38TAMAnOy7i7wvxMiciopqIhU4FFM6+tisiHrn5HL5GRERU6LWu9fBS+7oQApi+/gzOxibLHYmIahgWOhXQzscBLrbmSM3Ox6Fr9+SOQ0REpDckScLHzzdHt0bOyM7T4JVVoYhNypQ7FhHVICx0KkCpkPCsnzsAYPu5OJnTEBER6RcTpQKLR7VGU3c7JKbnYtzKUKRk5ckdi4hqCBY6FVQ4+9ruC/HIzlPLnIaIiEi/2Jib4OegALjZWeB6Qjom/XqKw72JqFqw0Kmg1nXtUdveEhm5auy/kiB3HCIiIr3jrrLEz0FtYW2mxLHI+5i95TyE4D12iKhqsdCpIEmS8FyLguFrf57l8DUiIqLiNPOww6JRraFUSPjt9C18t/e63JGIyMix0KkEhbOv7bl8Fxk5+TKnISIi0k89Grvg4xeaAwC+DrmKrWduyZyIiIwZC51K4FvbDt6OVsjO0+CfS3fljkNERKS3RrX3wsRu9QAA724+h+O86TYRVREWOpWgYPhawVkdzr5GRET0ZLP6NsEAP3fkqQVe+yUM1xPS5Y5EREaIhU4lKZx97cCVe5w6k4iI6AkUCgn/Hd4SrevaIzU7H+NWnkRieo7csYjIyLDQqSSN3WzR0MUGuWoNQi5y+BoREdGTWJgqsXxMAOo6WCE2KQuvrgrjbRqIqFKx0KlEhWd1/jx7R+YkRERE+s/RxhwrxrWFytIU4bHJmLE+HBoNp50mosrBQqcSFU4zfeR6IpIycmVOQ0REpP/qO9vgh9FtYKZUYNeFeAT/dUnuSERkJFjoVKJ6zjZo7mGHfI3Aroh4ueMQEREZhPb1HPHlsBYAgOWHovDrsWh5AxGRUWChU8n+nX2Nw9eIiIhK64VWtfFOn0YAgLl/XMDey7zelYgqhoVOJSscvnY88j4S0rJlTkNERGQ4pvRogOEBdaARwNS1ZxBxO0XuSERkwFjoVDJPByu08rSHRgB/nefwNSIiotKSJAnzB/uhcwMnZOaqMX5lKO4kZ8kdi4gMFAudKsDZ14iIiMrHVKnAkpdbo5GrDRLScjB+ZSjSsnl/OiIqOxY6VWCAnzskCQi7+YCfRBEREZWRnYUpfg5qC2dbc1yOT8PkNaeRp9bIHYuIDAwLnSrgprJAW28HAMCOc3EypyEiIjI8dWpZ4eexbWFpqsSha4n4cFsEhOA9doio9Mpc6Bw8eBADBw6Eh4cHJEnCtm3bnrpNTk4OPvjgA3h5ecHc3Bze3t74+eefy5PXYAx8OCkBZ18jIiIqH786Knw30h8KCVgfGoulB27IHYmIDEiZC52MjAy0bNkSixcvLvU2w4cPx549e/DTTz/hypUrWLduHRo3blzWXRuU/n7uUEjA2VspuHk/Q+44REREBqlXM1fMea4ZAGDBriu8/pWISs2krBv0798f/fv3L/X6u3btwoEDBxAZGQkHh4LhXN7e3mXdrcFxsjFHp/pOOHw9EdvPxWFKjwZyRyIiIjJIQYE+iEnKws9HovD2prNwV1kg4OEQcSKiklT5NTp//PEHAgICsGDBAtSuXRuNGjXCO++8g6yski/Sz8nJQWpqqs7DEA1sWTB8jZ8+ERERVcwHA5qidzNX5OZrMOGXMEQncrQEET1ZlRc6kZGROHz4MCIiIrB161Z888032Lx5MyZPnlziNsHBwVCpVNqHp6dnVcesEn2bu8FEIeFyfBquJ6TJHYeIiMhgKRUSFr7YCi3qqPAgMw9BK04iKSNX7lhEpMeqvNDRaDSQJAlr1qxBu3bt8Oyzz+Lrr7/GqlWrSjyrM3v2bKSkpGgfsbGxVR2zSthbmaFrI2cAwJ9nOfsaERFRRViZmeDHsQGobW+J6PuZeO2XMGTnqeWORUR6qsoLHXd3d9SuXRsqlUq7rGnTphBC4NatW8VuY25uDjs7O52HoXrukdnXOC0mERFRxbjYWmDluLawtTBB2M0HmLn5HDQa9q9EVFSVFzqBgYG4c+cO0tPTtcuuXr0KhUKBOnXqVPXuZde7mSvMTBS4cS8Dl+I4fI2IiKiiGrraYtnLbWCikPDn2Tv4avcVuSMRkR4qc6GTnp6O8PBwhIeHAwCioqIQHh6OmJgYAAXDzsaMGaNd/6WXXoKjoyPGjRuHixcv4uDBg5g5cybGjx8PS0vLynkXeszWwhQ9GhcMX+M9dYiIiCpHpwZO+Pw/LQAAS/bfwPqTMTInIiJ9U+ZCJywsDP7+/vD39wcAvPXWW/D398ecOXMAAHFxcdqiBwBsbGwQEhKC5ORkBAQEYNSoURg4cCC+/fbbSnoL+m9gSw8AwJ8cvkZERFRphrapg2k9GwIAPtgWgYNX78mciIj0iSQM4C/v1NRUqFQqpKSkGOT1Opm5+WjzyT/IylPj9ymBaOlpL3ckIqJSMfT2tyrx2OgHIQTe2ngWW8/cho25CTa/3hFN3Pj9IDJmpW1/q/waHSqYJaZXM1cAvKcOERFRZZIkCZ//xw/tfRyQnpOPcStCcTc1W+5YRKQHWOhUk8LZ13acj+PsMERERJXI3ESJH0YHoJ6zNeJSsjF+ZSgycvLljkVEMmOhU026NXKGrbkJ4lKycSrmgdxxiIiIjIrKyhQrg9rB0doMF+6k4o11Z5Cv1sgdi4hkxEKnmliYKtG7ecHwteUHIxGXUvzNUomIiKh86jpa4cexATA3UWDv5QTM+/MiJwEiqsFY6FQjlaUpAGD3xbsI/HwvNoRyKkwiIqLK5F+3Fr4Z0QqSBPx6/CZ+OhwldyQikgkLnWoSl5KFVUejtV9rBPD+lgie2SEikoFarcaHH34IHx8fWFpaon79+vjkk0/46b+R6O/njvf7NwUAzN95Cbsi4mRORERyYKFTTaISM/D4HARqIRCdmClPICKiGuyLL77A0qVLsWjRIly6dAlffPEFFixYgO+++07uaFRJXu3ig9EdvCAEMH19OM7w+liiGoeFTjXxcbKGQiq6/EwsG14ioup29OhRvPDCCxgwYAC8vb0xdOhQ9OnTBydPnpQ7GlUSSZIwd2Az9GjsjJx8DV5dFYaY+/xwkagmYaFTTdxVlgge4gelVFDtFNY8C3ZdQfBflzjlNBFRNerUqRP27NmDq1evAgDOnj2Lw4cPo3///iVuk5OTg9TUVJ0H6TcTpQKLXmqNZu52uJ+Ri3ErTyIlM0/uWERUTUzkDlCTjGhbF10bOSM6MRNejpbYfOo2vg65imUHInErKQv/Hd4SFqZKuWMSERm99957D6mpqWjSpAmUSiXUajXmz5+PUaNGlbhNcHAw5s2bV40pqTJYm5vg56C2GLzkCG7cy8DE1WH4ZXx7mJnws14iY8ff8mrmrrJEx/qO8LC3wrSeDfG/ES1hqpSw43wcXlp+HPfTc+SOSERk9DZu3Ig1a9Zg7dq1OH36NFatWoWvvvoKq1atKnGb2bNnIyUlRfuIjY2txsRUEW4qC/wc1BY25iY4HpmE9347x4kniGoASRjAb3pqaipUKhVSUlJgZ2cnd5xKdzzyPl77JQyp2fnwcrTCiqC2qOdsI3csIiKjbX89PT3x3nvvYcqUKdpln376KVavXo3Lly+X6jWM9dgYswNX72H8ylCoNQLTezbEm70byR2JiMqhtO0vz+jogQ71HLFlcifUqWWJm/czMWTpUYRGJ8kdi4jIaGVmZkKh0O0ClUolNBqNTImoOnRr5IxPB/kCABbuuYbNp27JnIiIqhILHT3RwMUWWycHoqWnPZIz8zBq+Qn8cfaO3LGIiIzSwIEDMX/+fOzYsQPR0dHYunUrvv76awwePFjuaFTFRrari9e71wcAzN5yDkdvJMqciIiqCgsdPeJsa471Ezqgb3NX5Ko1mLbuDJbsv85xxEREley7777D0KFDMXnyZDRt2hTvvPMOJk6ciE8++UTuaFQNZvZpjOdauCNPLTDx11O4djdN7khEVAV4jY4eUmsEPtt5CT8djgIAvNjWE58M8oWpknUpEVWvmtb+lgWPjWHLzlNj1I8ncOrmA9SpZYmtkwPhbGsudywiKgVeo2PAlAoJHz7XDPOebw6FBKwPjcX4laFIy+bc/0RERJXBwlSJ5WMC4O1ohVsPsvDqqlBk5arljkVElYiFjh4b28kbP4wOgKWpEoeuJWLY98dwJzlL7lhERERGwcHaDCvGtUMtK1OcvZWC6evPQM0beBMZDRY6eq5XM1dsnNgRzrbmuByfhsFLjuDCnRS5YxERERkFHydr/DAmAGZKBXZfvIvPdl6SOxIRVRIWOgbAr44KWyd3QiNXG9xNzcHw749h35UEuWMREREZhbbeDvhqeEsAwE+Ho7DqaLS8gYioUrDQMRB1allh06ROCGzgiIxcNV5dFYbVx2/KHYuIiMgoPN/SAzP7NgYAzPvzAv65eFfmRERUUSx0DIjK0hQrgtphaJs6UGsE/m9bBIJ3XoKG44mJiIgqbHL3+nixrSc0Anhj3Rmcv8Wh4kSGjIWOgTEzUeDLoS3wVu9GAIBlByPxxrozyM7jTDFEREQVIUkSPhnkiy4NnZCVp8b4VaG4zUmAiAwWCx0DJEkSpvVsiP+NaAlTpYQd5+Pw0vLjuJ+eI3c0IiIig2aqVGDxqNZo7GqLe2k5GLfiJFJ5ewcig8RCx4AN9q+DX19pDzsLE5yOScaQpUcReS9d7lhEREQGzc7CFD+PawsXW3NcvZuOyatPI0+tkTsWEZURCx0D16GeI7ZM7gRPB0vcvJ+JIUuPIjQ6Se5YREREBq22vSV+DmoLKzMlDl9PxAdbz0MIXhNLZEhY6BiBBi622Do5EC097ZGcmYdRy0/gj7N35I5FRERk0Hxrq/DdSH8oJGBj2C0s2X9D7khEVAYsdIyEk4051k/ogL7NXZGr1mDaujNYvO86P30iIiKqgJ5NXfHR880BAF/+fQW/h9+WORERlRYLHSNiaabEklFt8EpnHwAFDfLsLec5rpiIiKgCxnT0xqsP+9aZm87hZBSHiBMZAhY6RkapkPDhc80w7/nmUEjA+tBYjF8ZijTOGENERFRu7z/bVDtq4rVfwzj5D5EBYKFjpMZ28sYPowNgaarEoWuJGPb9MdzhvQCIiIjKRaGQ8M0If+31sONWhvK2DkR6joWOEevVzBUbJ3aEs605LsenYfCSI7hwh3d5JiIiKg9LMyV+HBOAOrUKZjqd8EsYb9hNpMdY6Bg5vzoqbJ3cCY1cbXA3NQfDvz+GfZcT5I5FRERkkJxtzbFyXFvtPeze3ngWGg0n/iHSRyx0aoA6taywaVInBDZwREauGq+sCsXq4zfljkVERGSQGrjYYtnoAJgqJew4H4cFf1+ROxIRFaPMhc7BgwcxcOBAeHh4QJIkbNu2rdTbHjlyBCYmJmjVqlVZd0sVpLI0xYqgdhjapg40Avi/bREI3nkJtx9k4uiNRMSl8PodIiKi0upY3xFf/KcFAOD7Azew9kSMzImI6HFlLnQyMjLQsmVLLF68uEzbJScnY8yYMejZs2dZd0mVxMxEgS+HtsDbvRsBAJYdjETgF/vw0vITCPx8LzaEspEmIiIqrSGt62BGr4YAgA9/j8D+KxwaTqRPylzo9O/fH59++ikGDx5cpu0mTZqEl156CR07dizrLqkSSZKEN3o2xNyBTXWWawTw/pbzPLNDRERUBtN7NsSQ1rWh1ghMWXMaF++kyh2JiB6qlmt0VqxYgcjISMydO7dU6+fk5CA1NVXnQZWrsZtdkWVqAfx59g6E4EWVREREpSFJEj4f0gId6xVcBzt+ZSjiU7LljkVEqIZC59q1a3jvvfewevVqmJiYlGqb4OBgqFQq7cPT07OKU9Y8Pk7WUEhFl3+28zL+s/QoDl69x4KHiIioFMxMFPj+5Tao72yN+NRsjFsZivScfLljEdV4VVroqNVqvPTSS5g3bx4aNWpU6u1mz56NlJQU7SM2NrYKU9ZM7ipLBA/xg1IqqHYUEtC5gRPMTRQ4HZOMMT+fxJClR3GABQ8REdFTqaxMsXJcOzjZmOFSXCqmrj2NfLVG7lhENZokKvBXrCRJ2Lp1KwYNGlTs88nJyahVqxaUSqV2mUajgRACSqUSu3fvxjPPPPPU/aSmpkKlUiElJQV2dkWHXFH5xaVkIToxE95OVnBXWSIhNRvLDkZi9fGbyMkvaKD969pjes+G6NbIGZJUzGkgIjJabH9LxmNDxQmPTcaLPxxDdp4Go9rXxaeDfNl3ElWy0ra/VXpGx87ODufPn0d4eLj2MWnSJDRu3Bjh4eFo3759Ve6eSsFdZYmO9R3hrrIEALjYWeDD55rh0KweeLWzDyxMFTgTk4ygFaEYvOQo9l9J4BkeIiKiErTytMc3I/whScCaEzFYfihS7khENVaZC5309HRt0QIAUVFRCA8PR0xMwdTEs2fPxpgxYwpeXKGAr6+vzsPFxQUWFhbw9fWFtbV15b0TqlQuthb4v+ea4eC7/xY84bEFBc+gJUexjwUPERFRsfr5uuGDZwtmN/1s52XsPB8ncyKimqnMhU5YWBj8/f3h7+8PAHjrrbfg7++POXPmAADi4uK0RQ8ZvsKC59C7z2BCl4KC52xsMsatCMWgxUew7zILHiIiose90tkHYzt6AQDe3BCOUzcfyJyIqOap0DU61YXjoPXHvbQcLD8UiV+ORSM7r+AanpZ1VJjeqyF6NHbhOGQiI8P2t2Q8NvQ0ao3Aa7+EYc/lBDhYm2Hr5E7wcuRoFqKK0otrdMj4ONua4/1nm+LQu8/gta71YGmqxNlbKRi/MgwvLD6CvZfv8gwPERERAKVCwrcj/eFb2w5JGbkYtzIUyZm5csciqjFY6FC5aAueWT0w8WHBc+6RgmfPJRY8RERE1uYm+HlsW3ioLBB5LwOv/XoKOflquWMR1QgsdKhCnGzMMbuw4On2b8HzyqowPL/oCP65yIKHiIhqNhc7C/w8ri1szU1wMioJ724+x76RqBqw0KFK4WRjjtn9m+Lww4LHykyJ87dT8OovYRi46DBCWPAQEVEN1sTNDktebg0ThYTfw+/gfyFX5Y5EZPRY6FClcnxY8Bx6twcmdasPKzMlIm6nYsIvYXjuOxY8RERUc3Vp6Iz5g30BAN/uvY6NYbEyJyIybix0qEo42pjjvf5NcHjWM3i9e0HBc+HOvwXP7gvxLHiIiKjGGdG2Lqb0qA8AeH/LeRy5nihzIiLjxUKHqpSDtRlm9SsoeCZ3rw/rhwXPa7+ewoBvD+NvFjxERFTDvN27MZ5v6YF8jcCkX0/h6t00uSMRGSUWOlQtHKzN8G6/Jjj0SMFzMS4VEx8WPLsi4qHRCMSlZOHojUTEpWTJHZmIiKhKKBQSvhzWAm29ayEtJx/jVoQiIS1b7lhERoc3DCVZPMjIxY+HI7HySDQycgum2XRXWSA+JRsCgEICgof4YUTbuvIGJarh2P6WjMeGKupBRi6GLD2KqMQM+NVWYcPEDrAyM5E7FpHe4w1DSa/VsjbDzL4FQ9qm9mgAS1MF4h4WOQCgEcDsLed5ZoeIiIxWLWszrAhqi1pWpjh/OwXT1oVDrdH7z5+JDAYLHZJVLWszvNO3MRa+6F/kOY0Afj4cxRurERGR0fJ2ssaPYwNgZqLAP5fu4pPtF+WORGQ0WOiQXvCro4JCKrp8+aEodF2wDz8eikRmbn71ByMiIqpibbwc8PXwlgCAlUej8fPhKJkTERkHFjqkF9xVlgge4gelVFDtKCTguRbucLUzx93UHHy64xICP9+L7/ZcQ0pWnsxpiYiIKtdzLTwwq18TAMAnOy5i94V4mRMRGT5ORkB6JS4lC9GJmfB2soK7yhI5+WpsOX0bS/ffQExSJgDAxtwEozt6YXygD5xtzWVOTGTc2P6WjMeGKpsQAu9vjcC6kzGwMFVgw2sd0dLTXu5YRHqHkxGQQXJXWaJjfUe4qywBAOYmSoxsVxd73+6GhS+2QmNXW6Tn5GPp/hvo/MVezP09AreTOWEBEZXd7du38fLLL8PR0RGWlpbw8/NDWFiY3LGoBpMkCZ+80BxdGzkjO0+DV1aFIfbhh3xEVHYsdMggmCgVeKFVbfw1vQuWjwlAS0975ORrsOrYTXRbsA8zN53FjXvpcsckIgPx4MEDBAYGwtTUFH/99RcuXryI//73v6hVq5bc0aiGM1EqsPglfzRxs0Vieg7GrwzlkG2icuLQNTJIQggcvXEfi/ddx9Eb9wEAkgQ86+uOyT3qo7mHSuaERMbBWNvf9957D0eOHMGhQ4fK/RrGemxIP8SlZGHQ4iO4m5qDTvUdsXJcO5iZ8PNpIoBD18jISZKEwAZOWDuhA7ZM7oReTV0gBLDjfBwGfHsY41acxKmbSXLHJCI99ccffyAgIADDhg2Di4sL/P39sXz58iduk5OTg9TUVJ0HUVVxV1ni56C2sDZT4uiN+3h/63kYwGfTRHqFhQ4ZvNZ1a+HHsW3x1/QueL6lBxQSsO/KPfxn6TGMWHYMB6/eY+dARDoiIyOxdOlSNGzYEH///Tdef/11TJs2DatWrSpxm+DgYKhUKu3D09OzGhNTTdTcQ4VFL7WGQgI2n7qF7/ZelzsSkUHh0DUyOlGJGVh24AZ+O30LeeqCH+8WdVSY3L0B+jRzhaK4G/YQUbGMtf01MzNDQEAAjh49ql02bdo0hIaG4tixY8Vuk5OTg5ycHO3Xqamp8PT0NLpjQ/pn9fGb+L9tEQCAb0a0wiD/2jInIpIXh65RjeXjZI3P/9MCB9/tgXGB3rAwVeDcrRRMWn0Kfb45iC2nbyFfrZE7JhHJyN3dHc2aNdNZ1rRpU8TExJS4jbm5Oezs7HQeRNXh5Q5eeK1rPQDAu5vP4XjkfZkTERkGFjpktNxVlpg7sDmOzHoGU3s0gK25Ca4npOOtjWfR47/7sfr4TWTnqeWOSUQyCAwMxJUrV3SWXb16FV5eXjIlInqy9/o1QX9fN+SqNZj46ynONEpUCix0yOg52pjjnb6NcWT2M5jZtzEcrc0Qm5SF/9sWga4L9mH5wUhk5OTLHZOIqtGbb76J48eP47PPPsP169exdu1a/PDDD5gyZYrc0YiKpVBI+N+IVvCva4+UrDyMWxGKxPScp29IVIPxGh2qcbJy1dgQGoNlByMRl5INALC3MkVQJ28EdfKGvZWZzAmJ9Icxt7/bt2/H7Nmzce3aNfj4+OCtt97ChAkTSr29MR8b0l+J6TkYvOQIYpOy4F/XHusmdICFqVLuWETVqrTtLwsdqrFy8zXYduY2lh64gajEDACAtZkSL3fwwitdfKDWCEQlZsDHyRruKkuZ0xLJg+1vyXhsSC7XE9Lxn6VHkZKVh/6+blj8UmtOtEM1CgsdolJSawR2no/D4n3XcTk+DQCgVEhQawp+NRQSEDzEDyPa1pUzJpEs2P6WjMeG5HQ88j5G/3QCeWqB17rWw/vPNpU7ElG14axrRKWkVEgY2NIDf03vgp+DAuDrYactcgBAI4D3fjuPI9cTeT8eIiLSCx3qOeLLoS0BAD8cjMSvx2/KnIhI/5jIHYBIX0iShGeauMLCRImXfjyh85wAMOrHE/BytELvpq7o3cwVbbxqwUTJzwqIiEgeg/xrIyYpE1+HXMXc3yNQx94SPZq4yB2LSG/wrzSix/g4W6O4oc6mSgk372fix8NRGPHDcbSd/w/e3ngWuyLikZnLWduIiKj6vfFMAwxtUwcaAUxZexoRt1PkjkSkN1joED3GXWWJ4CF+UEoF1Y5SkvDFf/wQPqcPlo5qjSH+taGyNMWDzDz8dvoWJq0+Bf+PQ/DKylCsPxmDe2mc7pOIiKqHJEn4bLAfOtV3RGauGq+sCkVcSpbcsYj0AicjICpBXEoWohMz4e1kVWTWtXy1BqHRDxBy8S5CLsUjNunfTkWSAH9Pe/Ru5obezVzRwMWmuqMTVRq2vyXjsSF9kpKVh6FLj+JaQjqauNli06SOsLUwlTsWUZXgrGtE1UQIgSt30xBy4S5CLt3FuVu6wwbqOVmjd3NX9GnmilaetaDkFKBkQNj+lozHhvTNrQeZGLT4KBLTc9C1kTN+GhsAU15LSkaIhQ6RTOJSsvDPpQSEXLyLYzcSkaf+91fMycYMPZsUTGbQuaETb/JGeo/tb8l4bEgfnbuVjBHLjiMrT42R7Tzx2WA/SBI/YCPjwkKHSA+kZefhwNV7CLl4F3svJyAt+99JCyxMFeja0Bm9m7miZ1NXOFibyZiUqHhsf0vGY0P6aveFeExcfQpCALP6NcHr3evLHYmoUlXZfXQOHjyIgQMHwsPDA5IkYdu2bU9cf8uWLejduzecnZ1hZ2eHjh074u+//y7rbokMkq2FKZ5r4YGFL/rj9Ie9sfqV9hjb0QseKgtk52mw++JdzNx8DgGfhmD498ew/GAkohMz5I5NREQGrE9zN8x5rhkA4Itdl7H93B2ZExHJo8yFTkZGBlq2bInFixeXav2DBw+id+/e2LlzJ06dOoUePXpg4MCBOHPmTJnDEhkyU6UCnRs6Yd4Lvjjy3jPY/kZnTO/ZEM3c7aARwMnoJMzfeQndv9qP3l8fwIJdl3Em5gE0D29eGpeShaM3EjmbDhERPdW4QB8EdfIGALy18SzCopPkDUQkgwoNXZMkCVu3bsWgQYPKtF3z5s0xYsQIzJkzp1Trc3gAGbtbDzLxz8WCyQxORCYhX/Pvr6WLrTm8nawRGp0EIQCFBAQP8cOItnVlTEw1BdvfkvHYkL5TawQm/noK/1y6i1pWptg6ORDeTtZyxyKqsCobulZRGo0GaWlpcHBwKHGdnJwcpKam6jyIjFmdWlYICvTBmlc74NSHvbHwxVYY0MIdNuYmSEjLwcmogiIHADQCmL3lPM/sEBHREykVEr4d2Qot6qjwIDMP41aG4kFGLtQagWM37uP38Ns4duM+1Bq9v1ybqFxMqnuHX331FdLT0zF8+PAS1wkODsa8efOqMRWR/lBZmuKFVrXxQqvayMlXY8WRaHz+12WddTQC+H7/Dcx+tilnbiMiohJZmZngx7EBGLz4KKISMzD0+6PIyMlHfOq/N7d2V1lg7sBm6OfrLmNSospXrWd01q5di3nz5mHjxo1wcXEpcb3Zs2cjJSVF+4iNja3GlET6w9xEiRdaeaC4W++sOnYTXRbsw4+HIpGVq67+cEREZBBcbC2wYlxbWJgocONehk6RAwDxKdl4ffVp7IqIkykhUdWotkJn/fr1ePXVV7Fx40b06tXrieuam5vDzs5O50FUU7mrLBE8xA/Kh/dBUEjAYH8P1La3xL20HHy64xK6LNiH5QcjkZmb/5RXIyKimqi+sw2szIsfyFM4cG3enxc5jI2MSrUMXVu3bh3Gjx+P9evXY8CAAdWxSyKjMqJtXXRt5IzoxEx4O1nBXWWJ3HwNtpy+hUX7ruPWgyzM33kJ3x+4gQld62F0By9Yl9ChERFRzXMyKglJGbklPi8AxKVk42RUEjrWd6y+YERVqMx/CaWnp+P69evar6OiohAeHg4HBwfUrVsXs2fPxu3bt/HLL78AKBiuNnbsWCxcuBDt27dHfHw8AMDS0hIqlaqS3gaR8XNXWcJdZan92sxEgRfb1cV/2tTB1jO3sXjfddy8n4nP/7qMZQdu4NUu9TC2kzdsWPAQEdV4CWnZpVpv+vozaOlpj/rONqjvbI36Ljao72QDlZVpFSckqnxlnl56//796NGjR5HlY8eOxcqVKxEUFITo6Gjs378fANC9e3ccOHCgxPVLg1N4Ej1dvlqDbeF3sGjvNUTfzwQA2FuZ4tXOPhjbyRu2FuykqOzY/paMx4YMybEb9zFy+fFyb+9kY456ztY6BVADZxt42FtCWdyFpERVqLTtb4Xuo1Nd2JkQlV6+WoM/z93Bd3uvI/JeBgDAzsIEr3Suh6BAb6gsWfBQ6bH9LRmPDRkStUag8xd7EZ+SjeL+8JMAONua48uhLRB9PxM37qUXPBIyEJ9a8tkgMxMF6jnpFkD1nGxQz9maQ6ipyrDQIarh1BqB7efu4Ns913DjYcFja2GC8YE+GB/ow2EIVCpsf0vGY0OGZldEHF5ffRoAdIqdwvMxS19uXewU0+k5+Yi8l47Iexk6BVBUYgZy1ZoS9+eusihSANV3sYabnQUkiWeBqPxY6BARgIKCZ+f5OHy39xqu3k0HANiamyAo0BuvdPaBvZWZzAlJn7H9LRmPDRmiXRFxmPfnRcSl/HuWprz30VFrBG4/yPq3+HlYAN24l477T5j4wNpMiXoPC6CCfwsKIG9Ha94bjkqFhQ4R6dBoBP6KiMe3e67hyt00AAWdzdhO3ni1Sz04WLPgoaLY/paMx4YMlVojcDIqCQlp2XCxtUA7H4dKv84mOTMXNx47AxSZmI6b9zNLnMJakgDPWla6BdDDs0GO1mY8C0RaLHSIqFgajcDui/FYuOc6LsWlAgCszJQY09EbE7r4wNHGXOaEpE/Y/paMx4ao7HLzNYhJ+vcaoMLhcNcT0pGWXfK94FSWpsUWQHUdrGCqrLbbQpKeYKFDRE8khEDIxbtYuOcaLtwpKHgsTZUY3dELr3WtBycWPAS2v0/CY0NUeYQQSEzPLVIA3biXjlsPslDSX6smCgl1Ha0eFj+cErumYKFDRKUihMCeSwn4du81nLuVAgCwMFXg5fZeeK1bPbjYWsickOTE9rdkPDZE1SM7T42oxAzdyRAeDofLylOXuJ2TjVmRM0D1nWxQuxanxDZ0LHSIqEyEENh/5R6+2XMNZ2OTAQDmJgq81L4uJnWrD1c7/S144lKyEJWYAR8na52bqlLFsf0tGY8Nkbw0GoH41OxiCyBOiW3cWOgQUbkIIXDwWiIW/nMVp2OSARR0Ci+1Kyh43FTyFjwajUBmnhoZOfnIyMnHtjO38d2+6xACUEhA8BA/jGhbV9aMxoTtb8l4bIj0V3pOPqKKKYCi7mcgN59TYhs6FjpEVCFCCBy+noiF/1xD2M0HAAAzpQIj2nriP61rIzNP/dQzKEII5ORrHhYlamTk5iMzNx/pOWpk5uQjPScfmbkFy7XrFLcst+DfzNyC556mqbsdGrrYwMvRCnUdrODlaA1vRys425qzoyojtr8l47EhMjwlTYkdmZiOxHROiW0oWOgQUaUQQuDYjfv4Zs81nIxKKvJ8yzoqONuaP1KQPFrUqEucRrSiFFLB0LqsvJI/mXucpakSdR2sUNfRCl4OVgWFkKM1vBysULuWJWfuKQbb35Lx2BAZl8enxC4cEscpsfUPCx0iqnTbz97B1HVnyrWtpakS1uYmsDZXwsrMBDbaf01gZfb4c48uM4F14f/NTGBlroSNuQnMTRSIT81G4Od78Wj/o5CA+YN8kZqdj5tJmYi5n4mbSRm4/SALT6q5lAoJHvYW8Ha0fngWyAp1Hazh5Vjwfyuzmjlum+1vyXhsiGqGikyJXc/ZusiMcJwSu+JK2/7WzJ6biMrFwab4m4pO6OIDvzr2OgWJtXnB/63MCoqXqpjhxl1lieAhfnh/SwTUQkApSfhsiG+x1+jkqTW4/SAL0fczEJOUiZv3Cx4xSQVfZ+dpEJuUhdikrGL35WRjXlD0PDwj5O1orT0z5PDYp3acHIGIyHiYmSjQwMUGDVxsdJY/OiX24xMi3HqQhZSsPJyJScaZh9e7FuKU2NWHhQ4RlZqPkzUUEnTOjCglCeM7+8j2B/2ItnXRtZEzohMz4e1kVWIOU6UC3k7W8HayLvKcRiNwLz0H0YkZj5wFykTM/YKvkzPzkJieg8T0HJx6eL3So2zMTbRngTJz83HwaiIEODkCEZExkyQJzrbmcLY1R4d6jjrPPW1K7Mh7Bc+F4K7OdpwSu3Jx6BoRlcmG0JhSnUExJilZeYi5n/nI2aCMh2eDMhGXUvIUpoWmdK+PXs1c4VtbZXDDFdj+lozHhojKilNiVw5eo0NEVSYuJeupZ1Bqiuw8NW49KBgGt//KPfx6/GaJ61qaKtHayx5tvR3QzscB/p61YGmm37P1sP0tGY8NEVWm4qbEjryXgcjE0k+J/eiMcMY8JTYLHSKiahaXklVkcgQJQGADJ0TcSUFyZp7O+qZKCX61VWjn44j2Pg5o410Ldhb6NT6b7W/JeGyIqDoUOyX2vQxE3ivdlNg6EyIYyZTYLHSIiGRQ0tA+jUbgWkI6TkYn4WRUEk5G3cfd1BydbSUJaOpmh3Y+BWd82no7wNnWXKZ3UoDtb8l4bIhIbhWZEru4GeEMZUpsFjpERDIpzdA+IQRik7JwIuo+Qh8WP9H3M4usV8/JWlv0tPNxQJ1altXaCbH9LRmPDRHpq/JOiW1nYVIwAUIVTomt1gicjEpCQlo2XGwt0M7HocwTLbDQISIyMAmp2Y+c8UnClbtpeLyF9lBZoO3DMz7tvB3QwMWmSgsftr8l47EhIkPztCmxS6oKKmtK7F0RcZj350WdiXzcVRaYO7AZ+vm6l/p1WOgQERm4lMw8hN0sKHpORCUh4nYK8h8biuBgbYa23rXQ1tsB7X0c0dTdFiaPfOpW0Xv6sP0tGY8NERmT7Dw1ou9n4EZC0QkRMnPVJW5XZErsh/9/fErsXRFxeH31aTxeeBSusfTl1qUudljoEBEZmczcfJyJScaJqCSERiXhdMwD5Dw2E4+1mRJtvB3Q3scBadl5+OFgJDSi/Pf0YftbMh4bIqoJhCiYEru4AuhJt1h4dEpsHycrrD4eg+SsvGLXlQC4qSxweNYzpRrGxkKHiMjI5eZrcP52inZyg7CbD5449lopSTj8Xo8yndlh+1syHhsiqunKOyV2SdZN6ICO9R2ful5p29+ad4chIiIjYWaiQBuvWmjjVQuvd68PtUbgcnwqQqOS8FdEHE5EPdBZXy0EohMza/y9j4iIqHLYmJvAr44KfnVUOssfnxJ7z+UEHLtx/6mvl5D29Jtwl4Vh3aKbiIhKpFRIaO6hQlCgD7550R+Pn/1XShK8nazkCafnPv/8c0iShBkzZsgdhYjI4CkfTl7Qo4kLXu1SD9OeaViq7VxsLSo1BwsdIiIj5K6yRPAQPygfzshWeE8fns0pKjQ0FMuWLUOLFi3kjkJEZJTa+TjAXWWBkq6+kVAw+1o7H4dK3S+HrhERGakRbeuiayPnp97TpyZLT0/HqFGjsHz5cnz66adyxyEiMkpKhYS5A5vh9dWnIQE6M68VFj9zBzYr8/10noZndIiIjJi7yhId6zuyyCnBlClTMGDAAPTq1eup6+bk5CA1NVXnQUREpdPP1x1LX24NN5Xu8DQ3lUWZppYuC57RISKiGmn9+vU4ffo0QkNDS7V+cHAw5s2bV8WpiIiMVz9fd/Ru5oaTUUlISMuGi23BcLXKPpNTiIUOERHVOLGxsZg+fTpCQkJgYVG6i19nz56Nt956S/t1amoqPD09qyoiEZFRUiqkUk0hXRlY6BARUY1z6tQpJCQkoHXr1tplarUaBw8exKJFi5CTkwOlUqmzjbm5OczNzas7KhERlRMLHSIiqnF69uyJ8+fP6ywbN24cmjRpglmzZhUpcoiIyPCw0CEiohrH1tYWvr6+Osusra3h6OhYZDkRERkmgyh0hCiYhI4z3BARVa/CdrewHaZ/sW8iIpJHafsmgyh00tLSAIAXfRIRySQtLQ0qlUruGFVq//79ZVqffRMRkbye1jdJwgA+ptNoNLhz5w5sbW0hSVUz/dyjCmfSiY2NhZ2dXZXvrzIZcnbAsPMzu3wMOb++ZxdCIC0tDR4eHlAoeOu1R1V331TZ9P1nr6rx/fP98/0b7vsvbd9kEGd0FAoF6tSpU+37tbOzM8hvPmDY2QHDzs/s8jHk/Pqc3djP5JSXXH1TZdPnn73qwPfP98/3b5jvvzR9Ez+eIyIiIiIio8NCh4iIiIiIjA4LnWKYm5tj7ty5BnljOEPODhh2fmaXjyHnN+TsZNhq+s8e3z/fP9+/8b9/g5iMgIiIiIiIqCx4RoeIiIiIiIwOCx0iIiIiIjI6LHSIiIiIiMjosNAhIiIiIiKjU2MKneDgYLRt2xa2trZwcXHBoEGDcOXKFZ11srOzMWXKFDg6OsLGxgb/+c9/cPfuXZ11YmJiMGDAAFhZWcHFxQUzZ85Efn5+db4VfP7555AkCTNmzDCI7Ldv38bLL78MR0dHWFpaws/PD2FhYdrnhRCYM2cO3N3dYWlpiV69euHatWs6r5GUlIRRo0bBzs4O9vb2eOWVV5Cenl7l2dVqNT788EP4+PjA0tIS9evXxyeffIJH5/DQl/wHDx7EwIED4eHhAUmSsG3bNp3nKyvnuXPn0KVLF1hYWMDT0xMLFiyo8vx5eXmYNWsW/Pz8YG1tDQ8PD4wZMwZ37tzRi/xPO/aPmjRpEiRJwjfffKMX2cm4GHJ7W1GG1F5XFkNv9yvKkPuNysC+pxREDdG3b1+xYsUKERERIcLDw8Wzzz4r6tatK9LT07XrTJo0SXh6eoo9e/aIsLAw0aFDB9GpUyft8/n5+cLX11f06tVLnDlzRuzcuVM4OTmJ2bNnV9v7OHnypPD29hYtWrQQ06dP1/vsSUlJwsvLSwQFBYkTJ06IyMhI8ffff4vr169r1/n888+FSqUS27ZtE2fPnhXPP/+88PHxEVlZWdp1+vXrJ1q2bCmOHz8uDh06JBo0aCBGjhxZpdmFEGL+/PnC0dFRbN++XURFRYlNmzYJGxsbsXDhQr3Lv3PnTvHBBx+ILVu2CABi69atOs9XRs6UlBTh6uoqRo0aJSIiIsS6deuEpaWlWLZsWZXmT05OFr169RIbNmwQly9fFseOHRPt2rUTbdq00XkNufI/7dgX2rJli2jZsqXw8PAQ//vf//QiOxkPQ29vK8qQ2uvKYujtfkUZcr9RGdj3PF2NKXQel5CQIACIAwcOCCEKfiFMTU3Fpk2btOtcunRJABDHjh0TQhT8QCkUChEfH69dZ+nSpcLOzk7k5ORUeea0tDTRsGFDERISIrp166YtdPQ5+6xZs0Tnzp1LfF6j0Qg3Nzfx5ZdfapclJycLc3NzsW7dOiGEEBcvXhQARGhoqHadv/76S0iSJG7fvl1l2YUQYsCAAWL8+PE6y4YMGSJGjRql1/kfb/AqK+eSJUtErVq1dH5mZs2aJRo3blyl+Ytz8uRJAUDcvHlTr/KXlP3WrVuidu3aIiIiQnh5eel0NvqSnQybobe3FWWo7XVlMfR2v6IMud+oDOx7ildjhq49LiUlBQDg4OAAADh16hTy8vLQq1cv7TpNmjRB3bp1cezYMQDAsWPH4OfnB1dXV+06ffv2RWpqKi5cuFDlmadMmYIBAwboZNT37H/88QcCAgIwbNgwuLi4wN/fH8uXL9c+HxUVhfj4eJ3sKpUK7du318lub2+PgIAA7Tq9evWCQqHAiRMnqiw7AHTq1Al79uzB1atXAQBnz57F4cOH0b9/f4PIX6iych47dgxdu3aFmZmZdp2+ffviypUrePDgQbW8l0IpKSmQJAn29vZ6n1+j0WD06NGYOXMmmjdvXuR5fc5OhsPQ29uKMpb2urIYY7tfUYbUb1QG9j016BqdR2k0GsyYMQOBgYHw9fUFAMTHx8PMzEz7w1/I1dUV8fHx2nUeLRQKny98riqtX78ep0+fRnBwcJHn9Dl7ZGQkli5dioYNG+Lvv//G66+/jmnTpmHVqlU6+y4u26PZXVxcdJ43MTGBg4NDlR/39957Dy+++CKaNGkCU1NT+Pv7Y8aMGRg1apRB5C9UWTnl/B14VHZ2NmbNmoWRI0fCzs5Ou399zf/FF1/AxMQE06ZNK/Z5fc5OhsPQ29uKMpb2urIYW7tfUYbWb1QG9j2AidwB5DBlyhRERETg8OHDckcpldjYWEyfPh0hISGwsLCQO06ZaDQaBAQE4LPPPgMA+Pv7IyIiAt9//z3Gjh0rc7qn27hxI9asWYO1a9eiefPmCA8Px4wZM+Dh4WEQ+Y1RXl4ehg8fDiEEli5dKnecpzp16hQWLlyI06dPQ5IkueOQETP09rai2F5TSQyt36gM7HsK1LgzOlOnTsX27duxb98+1KlTR7vczc0Nubm5SE5O1ln/7t27cHNz067z+ExmhV8XrlMVTp06hYSEBLRu3RomJiYwMTHBgQMH8O2338LExASurq56m93d3R3NmjXTWda0aVPExMTo7Lu4bI9mT0hI0Hk+Pz8fSUlJVZodAGbOnKn9lNDPzw+jR4/Gm2++qT2zpu/5C1VWTrl+jgoVdlY3b95ESEiI9lO5wv3rY/5Dhw4hISEBdevW1f7+3rx5E2+//Ta8vb31OjsZFkNvbyvKWNrrymIs7X5FGWK/URnY9xSoMYWOEAJTp07F1q1bsXfvXvj4+Og836ZNG5iammLPnj3aZVeuXEFMTAw6duwIAOjYsSPOnz+v80NR+EvzeOdSmXr27Inz588jPDxc+wgICMCoUaO0/9fX7IGBgUWm8b569Sq8vLwAAD4+PnBzc9PJnpqaihMnTuhkT05OxqlTp7Tr7N27FxqNBu3bt6+y7ACQmZkJhUL310SpVEKj0RhE/kKVlbNjx444ePAg8vLytOuEhISgcePGqFWrVpW+h8LO6tq1a/jnn3/g6Oio87y+5h89ejTOnTun8/vr4eGBmTNn4u+//9br7GRYDL29rShjaa8rizG0+xVlqP1GZWDf85DMkyFUm9dff12oVCqxf/9+ERcXp31kZmZq15k0aZKoW7eu2Lt3rwgLCxMdO3YUHTt21D5fOEVznz59RHh4uNi1a5dwdnau1umlCz0665o+Zz958qQwMTER8+fPF9euXRNr1qwRVlZWYvXq1dp1Pv/8c2Fvby9+//13ce7cOfHCCy8UO/2lv7+/OHHihDh8+LBo2LBhtUz3OXbsWFG7dm3tdKVbtmwRTk5O4t1339W7/GlpaeLMmTPizJkzAoD4+uuvxZkzZ7Szy1RGzuTkZOHq6ipGjx4tIiIixPr164WVlVWlTDP5pPy5ubni+eefF3Xq1BHh4eE6v8OPzgQjV/6nHfvHPT7zjZzZyXgYentbUYbUXlcWQ2/3K8qQ+43KwL7n6WpMoQOg2MeKFSu062RlZYnJkyeLWrVqCSsrKzF48GARFxen8zrR0dGif//+wtLSUjg5OYm3335b5OXlVfO7KVro6HP2P//8U/j6+gpzc3PRpEkT8cMPP+g8r9FoxIcffihcXV2Fubm56Nmzp7hy5YrOOvfv3xcjR44UNjY2ws7OTowbN06kpaVVefbU1FQxffp0UbduXWFhYSHq1asnPvjgA51GUl/y79u3r9if8bFjx1ZqzrNnz4rOnTsLc3NzUbt2bfH5559Xef6oqKgSf4f37dsne/6nHfvHFdfZyHnsyXgYcntbUYbUXlcWQ2/3K8qQ+43KwL7n6SQhHrllMBERERERkRGoMdfoEBERERFRzcFCh4iIiIiIjA4LHSIiIiIiMjosdIiIiIiIyOiw0CEiIiIiIqPDQoeIiIiIiIwOCx0iIiIiIjI6LHSIiIiIiMjosNAhIiIiIiKjw0KHqJIEBQVh0KBBcscgIiICwH6JiIUOEREREREZHRY6RGW0efNm+Pn5wdLSEo6OjujVqxdmzpyJVatW4ffff4ckSZAkCfv37wcAxMbGYvjw4bC3t4eDgwNeeOEFREdHa1+v8BO3efPmwdnZGXZ2dpg0aRJyc3PleYNERGRQ2C8RFc9E7gBEhiQuLg4jR47EggULMHjwYKSlpeHQoUMYM2YMYmJikJqaihUrVgAAHBwckJeXh759+6Jjx444dOgQTExM8Omnn6Jfv344d+4czMzMAAB79uyBhYUF9u/fj+joaIwbNw6Ojo6YP3++nG+XiIj0HPslopKx0CEqg7i4OOTn52PIkCHw8vICAPj5+QEALC0tkZOTAzc3N+36q1evhkajwY8//ghJkgAAK1asgL29Pfbv348+ffoAAMzMzPDzzz/DysoKzZs3x8cff4yZM2fik08+gULBE69ERFQ89ktEJeNPKlEZtGzZEj179oSfnx+GDRuG5cuX48GDByWuf/bsWVy/fh22trawsbGBjY0NHBwckJ2djRs3bui8rpWVlfbrjh07Ij09HbGxsVX6foiIyLCxXyIqGc/oEJWBUqlESEgIjh49it27d+O7777DBx98gBMnThS7fnp6Otq0aYM1a9YUec7Z2bmq4xIRkZFjv0RUMhY6RGUkSRICAwMRGBiIOXPmwMvLC1u3boWZmRnUarXOuq1bt8aGDRvg4uICOzu7El/z7NmzyMrKgqWlJQDg+PHjsLGxgaenZ5W+FyIiMnzsl4iKx6FrRGVw4sQJfPbZZwgLC0NMTAy2bNmCe/fuoWnTpvD29sa5c+dw5coVJCYmIi8vD6NGjYKTkxNeeOEFHDp0CFFRUdi/fz+mTZuGW7duaV83NzcXr7zyCi5evIidO3di7ty5mDp1KsdBExHRE7FfIioZz+gQlYGdnR0OHjyIb775BqmpqfDy8sJ///tf9O/fHwEBAdi/fz8CAgKQnp6Offv2oXv37jh48CBmzZqFIUOGIC0tDbVr10bPnj11Pknr2bMnGjZsiK5duyInJwcjR47ERx99JN8bJSIig8B+iahkkhBCyB2CqCYLCgpCcnIytm3bJncUIiIi9ktkNHj+kYiIiIiIjA4LHSIiIiIiMjocukZEREREREaHZ3SIiIiIiMjosNAhIiIiIiKjw0KHiIiIiIiMDgsdIiIiIiIyOix0iIiIiIjI6LDQISIiIiIio8NCh4iIiIiIjA4LHSIiIiIiMjr/D+p38ayHzCj5AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 1000x300 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "final validation perplexity: 4.18\n"
     ]
    }
   ],
   "source": [
    "rows = [json.loads(line) for line in open(f\"{RUN_DIR}/tracking/scalars.jsonl\")]\n",
    "train = [(row[\"step\"], row[\"scalars\"][\"train/loss\"]) for row in rows if \"train/loss\" in row[\"scalars\"]]\n",
    "val = [(row[\"step\"], row[\"scalars\"][\"val/perplexity\"]) for row in rows if \"val/perplexity\" in row[\"scalars\"]]\n",
    "\n",
    "figure, (left, right) = plt.subplots(1, 2, figsize=(10, 3))\n",
    "left.plot(*zip(*train), marker=\".\")\n",
    "left.set(xlabel=\"step\", title=\"training loss\")\n",
    "right.plot(*zip(*val), marker=\"o\")\n",
    "right.set(xlabel=\"step\", title=\"validation perplexity\")\n",
    "plt.show()\n",
    "print(\"final validation perplexity:\", round(val[-1][1], 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-19",
   "metadata": {},
   "source": [
    "## Generating text\n",
    "\n",
    "A decoder writes one token at a time, and each new token attends to every token before it. The KV cache stores the keys and values of every earlier position, so the prompt runs through the model once and each later step runs only the newest token.\n",
    "\n",
    "`generate` runs that loop compiled. `Sampling` sets the temperature and top-k: temperature 0 always takes the most likely byte, and higher temperatures pick more freely. We sample from the EMA weights, `state.averaged`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "cell-20",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-09-22T20:21:42.761614Z",
     "iopub.status.busy": "2026-09-22T20:21:42.761075Z",
     "iopub.status.idle": "2026-09-22T20:21:51.778495Z",
     "shell.execute_reply": "2026-09-22T20:21:51.777889Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ROMEO:\n",
      "Tut Tybalt, Gracious fazous to deparal,\n",
      "Your bring shall be find that is in the sign,\n",
      "Why both his hardy nundrest to an all ment.\n",
      "\n",
      "RICHMOND:\n",
      "Let must be, my lord, if you that day:\n",
      "But yea, sir, then time and with himself;\n",
      "Her oft protection to his death to the supper your\n",
      "mother married to men.\n",
      "\n",
      "HERMIONE:\n",
      "And not that his by the duke a heaveness.\n",
      "\n",
      "Third I change it another canst thou beast;\n",
      "When \n"
     ]
    }
   ],
   "source": [
    "from dew.sampling import Sampling, generate\n",
    "\n",
    "prompt = jnp.asarray([tokenizer.encode(PROMPT)], jnp.int32)\n",
    "out = generate(model, state.averaged, prompt, max_new_tokens=MAX_NEW_TOKENS,\n",
    "               key=jax.random.key(1), sampling=Sampling(temperature=0.8, top_k=40))\n",
    "print(tokenizer.decode(out.tokens[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-21",
   "metadata": {},
   "source": [
    "Greedy decoding (temperature 0) always picks the most likely byte. It tends to fall into a loop, as it does below after one line, which is why sampling is usually preferred for text like this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "cell-22",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-09-22T20:21:51.781038Z",
     "iopub.status.busy": "2026-09-22T20:21:51.780781Z",
     "iopub.status.idle": "2026-09-22T20:21:57.430261Z",
     "shell.execute_reply": "2026-09-22T20:21:57.429556Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ROMEO:\n",
      "The sear that shall be the seat of the seas,\n",
      "And the strain the seat of the seat of the seas,\n",
      "And the strain the seat of the seat of the seas,\n",
      "And the straight the seat of the seat of the seas,\n",
      "And t\n"
     ]
    }
   ],
   "source": [
    "greedy = generate(model, state.averaged, prompt, max_new_tokens=200,\n",
    "                  key=jax.random.key(0), sampling=Sampling(temperature=0.0))\n",
    "print(tokenizer.decode(greedy.tokens[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-23",
   "metadata": {},
   "source": [
    "## Reloading the checkpoint\n",
    "\n",
    "`fit` wrote a checkpoint to `RUN_DIR` at the end of training: the weights, the EMA, the optimizer state, the step counters and the position in the data. A new `Trainer` over the same objective restores it with `place()`, and greedy decoding from the restored weights gives the same text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "cell-24",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-09-22T20:21:57.432938Z",
     "iopub.status.busy": "2026-09-22T20:21:57.432692Z",
     "iopub.status.idle": "2026-09-22T20:21:58.547662Z",
     "shell.execute_reply": "2026-09-22T20:21:58.546888Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Resumed from step 1500 in /content/nb/runs/05-lm\n",
      "restored step: 1500\n",
      "same greedy text: True\n"
     ]
    }
   ],
   "source": [
    "restored, _, _ = Trainer(objective, optax.adamw(schedule), key=jax.random.key(SEED),\n",
    "                         checkpoints=Checkpoints(RUN_DIR)).place()\n",
    "again = generate(model, restored.averaged, prompt, max_new_tokens=200,\n",
    "                 key=jax.random.key(0), sampling=Sampling(temperature=0.0))\n",
    "print(\"restored step:\", int(restored.step))\n",
    "print(\"same greedy text:\", bool(np.array_equal(again.tokens, greedy.tokens)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-25",
   "metadata": {},
   "source": [
    "## Where to go next\n",
    "\n",
    "`tools/tokenize_text.py` in the repository writes the same three token files for any text file or folder, with the byte tokenizer or a Hugging Face one. `recipes/lm/train.py` runs this training from the command line. [Notebook 07](07-scaling-on-many-devices.ipynb) trains this kind of model on several devices at once, and [notebook 08](08-load-a-pretrained-decoder.ipynb) starts from a pretrained decoder instead of random weights."
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}