{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Continual learning in context\n", "\n", "A trained language model is frozen: whatever it reads, its weights stay where training left them. Nested Learning ([Behrouz et al., 2025](https://arxiv.org/abs/2512.24695)) asks what happens if some of them keep learning while the model reads. Its continuum memory system gives a model levels of parameters that update at different frequencies: a fast level adapts every few dozen tokens and forgets as quickly, a slow one every few hundred and keeps what it learned for longer. Each level takes a step of gradient descent on the model's own next-token loss over the tokens it just read, and the model reads on with the stepped weights.\n", "\n", "In this notebook we take a small pretrained model, SmolLM2-135M, and turn the feed-forward blocks of its last four layers into such levels, each at its own frequency. We measure what learning in context buys it on held-out text, train the levels' starting points by continued pre-training through their own steps, and then put the model through class-incremental learning: ten tasks of intent classification, learned one after another, with the forgetting of earlier tasks measured after each." ], "id": "cell-00" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Install dependencies\n", "\n", "Dew from GitHub with its cuda13 and streaming extras. Dew streams the text and the intent dataset from the Hub." ], "id": "cell-01" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%pip install -q \"dewml[cuda13,streaming] @ git+https://github.com/AshishKumar4/dew@7715ecf18adf6827aeae6025f95203f78df0b707\" --extra-index-url https://pypi.dewml.dev/simple" ], "id": "cell-02" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Settings" ], "id": "cell-03" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "MODEL = \"HuggingFaceTB/SmolLM2-135M\"\n", "SEQUENCE_LENGTH = 512\n", "BATCH_SIZE = 16\n", "LEVEL_LAYERS = (26, 27, 28, 29)\n", "LEVEL_WINDOWS = (32, 64, 128, 256)\n", "LEVEL_RATES = (1e-3, 5e-4, 2.5e-4, 1.25e-4)\n", "PRETRAIN_STEPS = 600\n", "TASKS = 10\n", "INTENTS_PER_TASK = 15\n", "TASK_STEPS = 60\n", "TEST_PER_INTENT = 4\n", "RUN_DIR = \"runs/13-continual\"\n", "SEED = 0" ], "id": "cell-04" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import jax\n", "import jax.numpy as jnp\n", "import numpy as np\n", "\n", "from dew import figures\n", "from dew.training import prepare_process\n", "\n", "prepare_process() # Dew's setup for a training process, before the first JAX call\n", "print(jax.devices())" ], "id": "cell-05" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## A pretrained model and its levels\n", "\n", "`Pretrained.load` reads SmolLM2-135M from the Hub as a native `CausalTransformer`. Nested Learning's section 7.3 starts a continuum memory system from a pretrained model's own blocks: each level is one of its feed-forward blocks, its trained weights the level's start. So the model itself does not change at all. What changes is the objective, which now steps those blocks in context.\n", "\n", "`ChunkAdaptation(layers=...)` names the layers whose feed-forward blocks become levels, with a window and a learning rate each. The block of layer 26 steps every 32 tokens, layer 29's every 256. Within a row, the model reads the text a chunk of 32 tokens at a time, through its own decode cache, and each chunk reads every level at the weights its window started with. At the end of a level's window, its weights take one step down the gradient of the loss over the window's tokens." ], "id": "cell-06" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from dew.interop import Pretrained\n", "from dew.objectives.lm import ChunkAdaptation, LMObjective\n", "\n", "loaded = Pretrained.load(MODEL, dtype=jnp.bfloat16, attention_impl=\"auto\", max_seq_len=SEQUENCE_LENGTH)\n", "model, pretrained = loaded.model, loaded.variables\n", "adaptation = ChunkAdaptation(window=LEVEL_WINDOWS, rate=LEVEL_RATES, layers=LEVEL_LAYERS)\n", "adapted = LMObjective(model, SEQUENCE_LENGTH, adaptation=adaptation, variables=pretrained)\n", "plain = LMObjective(model, SEQUENCE_LENGTH)\n", "print(f\"{sum(leaf.size for leaf in jax.tree.leaves(pretrained['params'])) / 1e6:.0f}M parameters, \"\n", " f\"{sum(leaf.size for layer in LEVEL_LAYERS for leaf in jax.tree.leaves(pretrained['params'][f'layers_{layer}']['mlp'])) / 1e6:.1f}M of them in levels\")" ], "id": "cell-07" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Learning while reading\n", "\n", "We read FineWeb-Edu text the model never saw, windows of 512 tokens, and score every token twice: with the model as it is, and with its levels stepping in context. The difference is what test-time training on the model's own loss gives a frozen model." ], "id": "cell-08" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from dew.data import Hub, presets\n", "from dew.data.compile import compile as compile_data\n", "from dew.data.dataset import DataPartition\n", "\n", "fineweb = Hub(\"HuggingFaceFW/fineweb-edu\", name=\"sample-10BT\")\n", "crawl = Hub(\"HuggingFaceFW/fineweb-edu\", name=\"CC-MAIN-2025-26\")\n", "windows = {\"seq_len\": SEQUENCE_LENGTH, \"batch\": BATCH_SIZE, \"pack\": None}\n", "data = compile_data(fineweb.shuffle(seed=0, window=64).repeat().pipe(presets.lm, MODEL, **windows),\n", " validation=crawl.pipe(presets.lm, MODEL, **windows, records=4), read=\"streamed\")\n", "held_out = [batch[\"text\"] for batch in data.val(DataPartition())]\n", "\n", "\n", "def perplexity(objective, variables):\n", " losses = [objective.token_scores(variables, jnp.asarray(rows)).losses for rows in held_out]\n", " return float(jnp.exp(jnp.mean(jnp.stack(losses))))\n", "\n", "\n", "before = {\"frozen\": perplexity(plain, pretrained), \"in context\": perplexity(adapted, pretrained)}\n", "print(before)\n", "figures.bars([list(before.values())], labels=list(before), y_max=max(before.values()),\n", " y_label=\"validation perplexity\")" ], "id": "cell-09" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Training the levels' starts\n", "\n", "The levels start from the pretrained blocks, which were trained to be read, not to be stepped. Continued pre-training trains them, and every other weight, through the levels' own steps: the outer loss differentiates through each in-context step back to the starting weights, a second-order gradient. This is the meta-learning step of Nested Learning's outermost level." ], "id": "cell-10" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from dew import Checkpoints, LocalTracker, Trainer\n", "from dew.config import OptimConfig\n", "from dew.optim import Cosine\n", "\n", "schedule = Cosine(peak=3e-5, warmup_steps=PRETRAIN_STEPS // 10, decay_steps=PRETRAIN_STEPS, end=3e-6)\n", "trainer = Trainer(adapted, OptimConfig(schedule=schedule), key=jax.random.key(SEED),\n", " checkpoints=Checkpoints(f\"{RUN_DIR}/pretrain\"), tracker=LocalTracker(f\"{RUN_DIR}/tracking\"))\n", "state = trainer.fit(data, steps=PRETRAIN_STEPS, log_every=PRETRAIN_STEPS // 10, checkpoint_every=PRETRAIN_STEPS)\n", "trained = state.variables\n", "after = {\"frozen\": perplexity(plain, trained), \"in context\": perplexity(adapted, trained)}\n", "print(after)\n", "figures.bars([list(before.values()), list(after.values())], labels=list(before),\n", " captions=[\"pretrained\", \"after continued pre-training\"],\n", " y_max=max(*before.values(), *after.values()), y_label=\"validation perplexity\")" ], "id": "cell-11" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Class-incremental learning\n", "\n", "CLINC150 ([Larson et al., 2019](https://arxiv.org/abs/1909.02027)) has 150 intents of short user queries. We split its intents into ten tasks of fifteen, and fine-tune on the tasks one after another, each on its own training queries alone. After every task we test on every task's held-out queries, choosing among every intent seen so far: the model reads `Query: ...` and we score the log-likelihood it gives each intent's name after `Intent:`, picking the highest (`dew.eval.label_scores`). That is class-incremental learning, the setting Nested Learning's section 9.1 measures.\n", "\n", "`continual_run` collects the accuracies into a matrix: row `i` is the accuracies after task `i`, column `j` task `j`'s. Its last row's average is the final accuracy, and a task's forgetting is the most it lost from its best." ], "id": "cell-12" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from dew.data import HFTokenizer, Memory\n", "\n", "tokenizer = HFTokenizer(MODEL)\n", "clinc = Hub(\"clinc_oos\", name=\"plus\")\n", "names = clinc.builder.info.features[\"intent\"].names\n", "train_rows = [record for record in clinc.records(\"indexed\")]\n", "test_rows = [record for record in clinc.with_split(\"test\").records(\"indexed\")]\n", "rng = np.random.default_rng(SEED)\n", "intents = [index for index, name in enumerate(names) if name != \"oos\"]\n", "rng.shuffle(intents)\n", "tasks = [intents[at * INTENTS_PER_TASK:(at + 1) * INTENTS_PER_TASK] for at in range(TASKS)]\n", "\n", "\n", "def query(text):\n", " return tokenizer.encode(f\"Query: {text}\\nIntent:\", add_special_tokens=False)\n", "\n", "\n", "def label(intent):\n", " return tokenizer.encode(\" \" + names[intent].replace(\"_\", \" \"), add_special_tokens=False)\n", "\n", "\n", "def task_data(task):\n", " kept = [{\"text\": f\"Query: {row['text']}\\nIntent: {names[row['intent']].replace('_', ' ')}\"}\n", " for row in train_rows if row[\"intent\"] in task]\n", " return compile_data(Memory(kept).shuffle(seed=0).repeat().pipe(presets.lm, MODEL, seq_len=64, batch=BATCH_SIZE,\n", " pack=\"first-fit\"))\n", "\n", "\n", "def tests(task):\n", " picked = {}\n", " for row in test_rows:\n", " if row[\"intent\"] in task and len(picked.setdefault(row[\"intent\"], [])) < TEST_PER_INTENT:\n", " picked[row[\"intent\"]].append(row[\"text\"])\n", " return [(query(text), intent) for intent, texts in picked.items() for text in texts]" ], "id": "cell-13" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from dew.eval import continual_run, label_accuracy, label_scores\n", "\n", "\n", "def run(steps_in_context, name):\n", " objective = LMObjective(model, 64, adaptation=adaptation if steps_in_context else None, variables=trained)\n", " learner = Trainer(objective, OptimConfig(schedule=Cosine(peak=1e-4, warmup_steps=5,\n", " decay_steps=TASKS * TASK_STEPS, end=1e-5)),\n", " key=jax.random.key(SEED), checkpoints=Checkpoints(f\"{RUN_DIR}/{name}\"))\n", " held = {}\n", "\n", " def fit(stage, task):\n", " held[\"state\"] = learner.fit(task_data(task), steps=TASK_STEPS * (stage + 1), log_every=TASK_STEPS,\n", " checkpoint_every=TASK_STEPS)\n", "\n", " def accuracy(stage, scored):\n", " if scored > stage:\n", " # A task not yet trained has no intent among those seen: class-incremental scores it 0.\n", " return 0.0\n", " seen = [intent for task in tasks[:stage + 1] for intent in task]\n", " examples = tests(tasks[scored])\n", " variables = held[\"state\"].variables\n", " scores = label_scores(lambda rows: objective.token_scores(variables, rows).losses,\n", " [prompt for prompt, _ in examples], [label(intent) for intent in seen],\n", " pad_id=tokenizer.eos_id or 0)\n", " return label_accuracy(scores, [seen.index(intent) for _, intent in examples])\n", "\n", " return continual_run(tasks, fit, accuracy)\n", "\n", "\n", "matrices = {\"frozen\": run(False, \"frozen\"), \"in context\": run(True, \"in-context\")}\n", "for name, matrix in matrices.items():\n", " print(name, matrix.record())" ], "id": "cell-14" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "figures.heatmap([matrix.values for matrix in matrices.values()], rows=[str(at) for at in range(TASKS)],\n", " columns=[str(at) for at in range(TASKS)], captions=list(matrices), x_label=\"task scored\",\n", " y_label=\"after task\")\n", "figures.bars([[matrix.average(), matrix.average_forgetting()] for matrix in matrices.values()],\n", " labels=[\"final accuracy\", \"forgetting\"], captions=list(matrices))" ], "id": "cell-15" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Where to go next\n", "\n", "The levels here are a pretrained model's own blocks. `CausalTransformer(continuum=...)` gives a model separate levels on its final states instead, a chain of them at their own frequencies, which `ChunkAdaptation(window=..., rate=...)` steps the same way; `ChunkAdaptation(carry=\"run\")` keeps what they learned across documents and training steps, the memory a continual learner carries. `dew.nn.memory` has the in-context memories of Titans, TTT and Miras, which learn from their own reconstruction losses rather than the model's." ], "id": "cell-16" } ], "metadata": { "accelerator": "GPU", "colab": { "gpuType": "A100", "provenance": [] }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 5 }