{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# World models: acting and imagining in one loop\n", "\n", "Suppose you have taught a program to balance a pole on a cart by letting it practise in a simulator, pushing the cart left and right millions of times until it stops dropping the pole. Now suppose the simulator is slow, or the real thing breaks when it falls, and every practice step costs you something. You would want the program to do some of its practising in its head, the way you might rehearse a parking manoeuvre before you try it. That is what a world model is for. It is a small, learned copy of the world, good enough to rehearse in.\n", "\n", "Here is the thought this tutorial is built on. In Dew, an environment is two functions and nothing more. One starts an episode, the other takes a state and an action and answers with the next observation and a reward. A physics simulator fits that shape, and so does a learned model, if we let its state be a bundle of numbers it keeps in its head and its step be a guess at what comes next. Put either one in the loop below and the loop cannot tell the difference.\n", "\n", "![One rollout loop with two kinds of step. What the policy reads goes in, the policy answers with one action here, and a step answers with the next thing to read and a reward. From the simulator the policy reads the cartpole's five numbers; in imagination it reads the latent feature, h and z joined. The step can be a simulator's physics, whose state is the cartpole, or a world model's prior, whose state is a latent, and the same scan runs both.](https://raw.githubusercontent.com/AshishKumar4/dew/main/docs/assets/world-loop.svg)\n", "\n", "We will run that loop on MuJoCo Playground's cartpole: first a random policy, then PPO in the simulator, then DreamerV3 practising inside its model. After that, Dreamer 4 learning from video alone, and racing drones, where the physics stays in JAX and only the camera lives outside it.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Setup\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [ "skip-execution" ] }, "outputs": [], "source": [ "%pip install -q \"dewml[cuda13,playground,hyperlux] @ git+https://github.com/AshishKumar4/dew@ff1a61a94d3b31152bd10d606204633e44534b85\" --extra-index-url https://pypi.dewml.dev/simple" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "execution": { "iopub.execute_input": "2026-10-31T20:01:45.091405Z", "iopub.status.busy": "2026-10-31T20:01:45.091278Z", "iopub.status.idle": "2026-10-31T20:01:48.292318Z", "shell.execute_reply": "2026-10-31T20:01:48.291917Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "cuda:0 observation 5 action 1\n" ] } ], "source": [ "import time\n", "\n", "import jax\n", "import jax.numpy as jnp\n", "import numpy as np\n", "import optax\n", "\n", "from dew import figures\n", "from dew.rl.env import env_rollout, start\n", "from dew.rl.playground import Playground, playground\n", "\n", "TASK = \"CartpoleBalance\"\n", "probe = Playground.load(TASK)\n", "print(jax.devices()[0], \"observation\", probe.env.observation_size, \"action\", probe.action_size)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## A pole, a cart, and a policy that guesses\n", "\n", "The task is `CartpoleBalance`. At every step the policy is shown five numbers (we just printed the sizes above) and answers with one number between -1 and 1, how hard to push the cart and which way. An episode runs for at most 1000 steps, which is how Playground trains these tasks. Keep the pole up the whole time and the return gets close to 1000.\n", "\n", "Before we teach anything, what does guessing get us? A random policy is the smallest policy there is. It ignores the observation completely and draws its push from its random key. It still has the shape every policy here has: weights, observation, key and memory in; action and memory out. Because it has that shape, the rollout loop can already run it, 256 carts at once.\n" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "execution": { "iopub.execute_input": "2026-10-31T20:01:48.310812Z", "iopub.status.busy": "2026-10-31T20:01:48.310656Z", "iopub.status.idle": "2026-10-31T20:02:11.914042Z", "shell.execute_reply": "2026-10-31T20:02:11.913616Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "random policy: mean return 358.9 over 256 episodes\n" ] } ], "source": [ "env = playground(TASK, 256)\n", "\n", "def random_policy(params, observation, key, memory):\n", " action = jax.random.uniform(key, (observation.shape[0], env.action_size), minval=-1, maxval=1)\n", " return action, {}, memory\n", "\n", "rolled = jax.jit(lambda carry: env_rollout(env, random_policy, None, carry, 1000))\n", "_, steps = rolled(start(env, jax.random.key(0)))\n", "returns = np.asarray(steps.following.reward.sum(0))\n", "print(f\"random policy: mean return {returns.mean():.1f} over {len(returns)} episodes\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "358.9, on average, out of about 1000. That number is the sum of rewards over each episode, averaged over the 256 carts. These episodes run to their 1000-step time limit rather than ending when the pole falls, so read it as reward collected under random pushes, not as time spent with the pole up. It gives us a floor to measure against: anything we teach from here has to beat a policy that is not even looking.\n", "\n", "## Learning in the simulator with PPO\n", "\n", "The obvious way to get better is to practise in the simulator and adjust. PPO does exactly that. It keeps two networks, an actor that says how to push and a critic that guesses how much reward is still to come from here. The critic's guesses let it say, for each push it tried, whether things went better or worse than expected, and the actor is nudged towards the pushes that went better. The loss clips the gain from moving far from the policy that collected the batch, which discourages very large changes in one update, though nothing caps how far the weights themselves move. That clip is the one you may have met in [tutorial 10](10-rl-with-verifiable-rewards.ipynb). Here the actor's push is a draw from a squashed bell curve rather than a choice among tokens, but the idea is the same.\n", "\n", "Dew's version follows Brax's PPO, in big parallel batches. Below it practises on 2048 carts at once, collects a stretch of steps, and takes several passes of updates over that stretch before collecting again. At the end we score it greedily: always the centre of its bell curve, no exploring.\n" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "execution": { "iopub.execute_input": "2026-10-31T20:02:11.915311Z", "iopub.status.busy": "2026-10-31T20:02:11.915175Z", "iopub.status.idle": "2026-10-31T20:03:01.765130Z", "shell.execute_reply": "2026-10-31T20:03:01.764684Z" } }, "outputs": [ { "data": { "text/html": [ "
╭─  dew · ActorCritic  ───────────────────────────────────────────────────────────────────────────────────────────╮\n",
       "│ 268,387 parameters  ·  on 1 × NVIDIA RTX PRO 6000 Blackwell Server Edition  ·  batch 1  ·  float32              │\n",
       "│                                                                                                                 │\n",
       "│ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4096/4096  100%  133.9 step/s  0:00:44 │\n",
       "│                                                                                                                 │\n",
       "│ train                                                                                                           │\n",
       "│   loss                     0.2480  █▇██▇▇▄▁     ↓ 1,352×                                                        │\n",
       "│   entropy                 0.08654  ████▇▇▅▁        ↓ 86%                                                        │\n",
       "│   pg_clipfrac              0.1693  ▄▁▅▁█▁▇▇       ↑ 138%                                                        │\n",
       "│   policy                 -0.05414  ▇█▆█▁▇▅▁       ↓ 314%                                                        │\n",
       "│   ppo_kl                  0.08489  ▃▂▄▂▆▁██       ↑ 319%                                                        │\n",
       "│   value                    0.3030  █▇██▇▇▄▁     ↓ 1,107×                                                        │\n",
       "│   data_wait_ms            0.01406  █▇▃▂▂▁▂▁        ↓ 14%                                                        │\n",
       "│   grad_norm                 175.8  █▇▇▆▇▇▃▁        ↓ 15×                                                        │\n",
       "│   peak_memory_gib          0.3771  ▁▇██████       ↑ 0.2%                                                        │\n",
       "│   pg_clipfrac_lower 0   flops 6.532e+10   compiles 1.000   cache_hits 763.0   cache_misses 0                    │\n",
       "│ throughput                                                                                                      │\n",
       "│   step_time_ms              6.110  █▄▄▄▄▄▄▁        ↓ 51%                                                        │\n",
       "│   samples_per_sec         167,592  ▁▅▅▅▅▅▅█       ↑ 103%                                                        │\n",
       "│   mfu                        2.1%  ▁▅▅▅▅▅▅█     ↑ 1.1 pt                                                        │\n",
       "│   hlo_flops_util             2.1%  ▁▅▅▅▅▅▅█     ↑ 1.1 pt                                                        │\n",
       "│   hfu                        2.1%  ▁▅▅▅▅▅▅█     ↑ 1.1 pt                                                        │\n",
       "│   rollout_seconds           3.946  █▄▁▁▁▁▁▁        ↓ 63%                                                        │\n",
       "│ optimizer                                                                                                       │\n",
       "│   accepted 100.0%                                                                                               │\n",
       "│ rollout                                                                                                         │\n",
       "│   episode_return              nan  ▁▂█                                                                          │\n",
       "╰──────────────────────────────────────────────────────────────────────────────────────────────────  ✓ finished  ─╯\n",
       "
\n" ], "text/plain": [ "╭─ dew · ActorCritic ───────────────────────────────────────────────────────────────────────────────────────────╮\n", "│ 268,387 parameters · on 1 × NVIDIA RTX PRO 6000 Blackwell Server Edition · batch 1 · float32 │\n", "│ │\n", "│ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4096/4096 100% 133.9 step/s 0:00:44 │\n", "│ │\n", "│ train │\n", "│ loss 0.2480 █▇██▇▇▄▁ ↓ 1,352× │\n", "│ entropy 0.08654 ████▇▇▅▁ ↓ 86% │\n", "│ pg_clipfrac 0.1693 ▄▁▅▁█▁▇▇ ↑ 138% │\n", "│ policy -0.05414 ▇█▆█▁▇▅▁ ↓ 314% │\n", "│ ppo_kl 0.08489 ▃▂▄▂▆▁██ ↑ 319% │\n", "│ value 0.3030 █▇██▇▇▄▁ ↓ 1,107× │\n", "│ data_wait_ms 0.01406 █▇▃▂▂▁▂▁ ↓ 14% │\n", "│ grad_norm 175.8 █▇▇▆▇▇▃▁ ↓ 15× │\n", "│ peak_memory_gib 0.3771 ▁▇██████ ↑ 0.2% │\n", "│ pg_clipfrac_lower 0 flops 6.532e+10 compiles 1.000 cache_hits 763.0 cache_misses 0 │\n", "│ throughput │\n", "│ step_time_ms 6.110 █▄▄▄▄▄▄▁ ↓ 51% │\n", "│ samples_per_sec 167,592 ▁▅▅▅▅▅▅█ ↑ 103% │\n", "│ mfu 2.1% ▁▅▅▅▅▅▅█ ↑ 1.1 pt │\n", "│ hlo_flops_util 2.1% ▁▅▅▅▅▅▅█ ↑ 1.1 pt │\n", "│ hfu 2.1% ▁▅▅▅▅▅▅█ ↑ 1.1 pt │\n", "│ rollout_seconds 3.946 █▄▁▁▁▁▁▁ ↓ 63% │\n", "│ optimizer │\n", "│ accepted 100.0% │\n", "│ rollout │\n", "│ episode_return nan ▁▂█ │\n", "╰────────────────────────────────────────────────────────────────────────────────────────────────── ✓ finished ─╯\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
╭─  ✓ ActorCritic  ───────────────────────────╮\n",
       "│    trained  4096 steps in 0:00:44           │\n",
       "│ first step  after 9.03 s                    │\n",
       "│       then  115.7 step/s                    │\n",
       "│    goodput  79.7% of the wall time in steps │\n",
       "│ final loss  0.2480                          │\n",
       "╰─────────────────────────────────────────────╯\n",
       "
\n" ], "text/plain": [ "\u001b[38;2;42;167;201m╭─\u001b[0m\u001b[38;2;42;167;201m \u001b[0m\u001b[38;2;63;174;106m✓ \u001b[0m\u001b[1;38;2;42;167;201mActorCritic\u001b[0m\u001b[38;2;42;167;201m \u001b[0m\u001b[38;2;42;167;201m──────────────────────────\u001b[0m\u001b[38;2;42;167;201m─╮\u001b[0m\n", "\u001b[38;2;42;167;201m│\u001b[0m \u001b[2m trained\u001b[0m\u001b[2m \u001b[0m4096 steps in 0:00:44 \u001b[38;2;42;167;201m│\u001b[0m\n", "\u001b[38;2;42;167;201m│\u001b[0m \u001b[2mfirst step\u001b[0m\u001b[2m \u001b[0mafter 9.03 s \u001b[38;2;42;167;201m│\u001b[0m\n", "\u001b[38;2;42;167;201m│\u001b[0m \u001b[2m then\u001b[0m\u001b[2m \u001b[0m115.7 step/s \u001b[38;2;42;167;201m│\u001b[0m\n", "\u001b[38;2;42;167;201m│\u001b[0m \u001b[2m goodput\u001b[0m\u001b[2m \u001b[0m79.7% of the wall time in steps \u001b[38;2;42;167;201m│\u001b[0m\n", "\u001b[38;2;42;167;201m│\u001b[0m \u001b[2mfinal loss\u001b[0m\u001b[2m \u001b[0m\u001b[1m0.2480 \u001b[0m \u001b[38;2;42;167;201m│\u001b[0m\n", "\u001b[38;2;42;167;201m╰─────────────────────────────────────────────╯\u001b[0m\n" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "44 s; greedy return 999.743896484375\n" ] } ], "source": [ "import itertools\n", "\n", "from dew.data.dataset import Loader\n", "from dew.objectives.rl.control import ActorCritic, ControlPPOObjective, ControlRollout, episode_return\n", "from dew.training import Trainer\n", "\n", "objective = ControlPPOObjective(ActorCritic(probe.action_size), int(probe.env.observation_size),\n", " discount=0.995, reward_scaling=10.0, entropy_cost=1e-2)\n", "rollout = ControlRollout(objective, playground(TASK, 2048), unroll_length=30, unrolls=16, epochs=16,\n", " minibatches=32)\n", "idle = Loader(train=lambda partition: itertools.repeat({\"step\": np.zeros((1,), np.float32)}), val=None,\n", " records=1, batch=1)\n", "began = time.perf_counter()\n", "ppo = Trainer(objective, optax.adam(1e-3), key=jax.random.key(1), rollout=rollout).fit(idle, steps=8 * 512,\n", " log_every=512)\n", "print(f\"{time.perf_counter() - began:.0f} s; greedy return\",\n", " episode_return(objective, playground(TASK, 64), ppo.variables, jax.random.key(2), 1000))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "44 seconds, and the greedy return is 999.7. The pole simply stays up. When practising is this cheap, there is not much more to ask for. The rest of this tutorial only makes sense once practice stops being cheap.\n", "\n", "So keep the question in mind. What would we do if each step ran on a real robot, or in a simulator too slow to call millions of times? We would want far more learning from each real step: remember every step taken, learn a model of what the world does, and practise inside the model, where steps are just arithmetic. That is DreamerV3's plan ([Hafner et al., 2023](https://arxiv.org/abs/2301.04104)).\n", "\n", "## A model you can practise in\n", "\n", "DreamerV3's world model never tries to remember the cartpole in full detail. It keeps a summary, a latent state in two parts, and the split between the parts is the whole trick, so let's go slowly.\n", "\n", "The first part, usually written $h$, is 512 numbers carried by a recurrent network, a block GRU. Think of it as the model's running memory of how things have been going. It is updated step by step from the previous $h$, the previous draw $z$ and the action just taken. The update itself makes no new draw: given those three, the next $h$ is worked out rather than sampled. Its history still runs through the earlier draws, since each step's $z$ feeds the next step's $h$. The second part, $z$, is where the guessing lives. It is 32 separate draws, each one picking one of a few classes, 4 in the small model we run here. If $h$ is what the model can work out, $z$ is its way of saying \"and then one of several things could be true\".\n", "\n", "Now the important move. There are two ways to get $z$ at a step. If we have actually seen this step's observation, we can read $z$ off with the help of that observation. That is the posterior, $q(z_t \\mid h_t, o_t)$, the summary of a step we watched. If we have not seen it, because we are rehearsing a future that has not happened, we have to guess $z$ from $h$ alone. That is the prior, $p(z_t \\mid h_t)$. Training pulls the prior towards the posterior, so that the guess made in the dark lands near the summary we would have written in the light. Where the guess can be right, imagining will track watching. Where the world is genuinely up in the air, the draws in $z$ give the imagination its spread.\n", "\n", "![One step of the latent. The deterministic state h, 512 numbers from a block GRU, carries the past. The stochastic part z is 32 categorical draws, from the posterior when the step's observation is there to read and from the prior, h alone, when the model imagines. Training pulls the prior towards the posterior, and the two parts together feed heads that rebuild the observation and predict the reward and whether the episode continues.](https://raw.githubusercontent.com/AshishKumar4/dew/main/docs/assets/world-rssm.svg)\n", "\n", "Off the pair $(h, z)$ hang a few small heads. One rebuilds the observation, which forces the latent to describe the step it sits on. One predicts the reward, and one whether the episode goes on. With those heads the model can play the environment's part: hand it a latent and an action and it advances $h$, draws the next $z$ from the prior, and answers with a predicted reward. In Dew that is `LatentEnv`. The actor and critic learn by rolling out in it, starting from latents of real steps stored in the replay.\n", "\n", "Below, the agent runs 16 carts in the real simulator to fill that replay and reuses what it stored very heavily, which is rather the point. It uses a learning rate of $10^{-4}$ where DreamerV3's own recipe uses $4 \\times 10^{-5}$ over far longer runs. We take the larger rate for this short run and say so.\n" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "execution": { "iopub.execute_input": "2026-10-31T20:03:01.766638Z", "iopub.status.busy": "2026-10-31T20:03:01.766270Z", "iopub.status.idle": "2026-10-31T20:13:43.218545Z", "shell.execute_reply": "2026-10-31T20:13:43.218101Z" } }, "outputs": [ { "data": { "text/html": [ "
╭─  dew · WorldModel  ────────────────────────────────────────────────────────────────────────────────────────────╮\n",
       "│ 637,446 parameters  ·  on 1 × NVIDIA RTX PRO 6000 Blackwell Server Edition  ·  batch 1  ·  float32              │\n",
       "│                                                                                                                 │\n",
       "│ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  20000/20000  100%  32.0 step/s  0:10:37 │\n",
       "│                                                                                                                 │\n",
       "│ train                                                                                                           │\n",
       "│   loss                        3.357  █▂▁▁▁▁▁▁▁▁        ↓ 53%                                                    │\n",
       "│   imagined_return             182.5  ▁▅▆▆▆▇▇▇██       ↑ 663%                                                    │\n",
       "│   loss_continue             0.02044  █▂▁▁▁▁▁▁▁▁       ↓ 0.6%                                                    │\n",
       "│   loss_decoder              0.01217  █▅▆▅▅▄▄▃▁▂        ↓ 80%                                                    │\n",
       "│   loss_dynamics               1.380  ▁▃▅▇█▇▇▇▆▆        ↑ 13%                                                    │\n",
       "│   loss_policy             -0.006085  █▁▁▁▁▁▁▁▁▁       ↓ 101%                                                    │\n",
       "│   loss_replay_value           1.123  █▄▂▂▂▂▂▁▁▁        ↓ 76%                                                    │\n",
       "│   loss_representation         1.380  ▁▃▅▇█▇▇▇▆▆        ↑ 13%                                                    │\n",
       "│   loss_reward                0.5109  █▄▃▂▂▁▂▂▄▂        ↓ 24%                                                    │\n",
       "│   loss_value                 0.9650  █▂▂▂▂▁▂▁▁▁        ↓ 68%                                                    │\n",
       "│   return_scale                106.0  ▁▃▅▆▇▇▇▇██        ↑ 14×                                                    │\n",
       "│   data_wait_ms              0.01646  ▄▁▇▂▇▁▃█▁▅       ↑ 0.7%                                                    │\n",
       "│   grad_norm                   3.477  █▆▅▄▃▂▂▂▁▁        ↓ 59%                                                    │\n",
       "│   flops 3.288e+10   compiles 1.000   cache_hits 1,394   cache_misses 0   peak_memory_gib 0.3771                 │\n",
       "│ throughput                                                                                                      │\n",
       "│   step_time_ms                31.25  █▅▅▅▅▅▁▅▅▅       ↓ 5.5%                                                    │\n",
       "│   samples_per_sec             512.0  ▁▄▄▄▄▄█▄▄▄       ↑ 5.8%                                                    │\n",
       "│   mfu                          0.2%  ▁▄▄▄▄▄█▄▄▄            →                                                    │\n",
       "│   hlo_flops_util               0.2%  ▁▄▄▄▄▄█▄▄▄            →                                                    │\n",
       "│   hfu                          0.2%  ▁▄▄▄▄▄█▄▄▄            →                                                    │\n",
       "│   rollout_seconds             55.34  █▃▃▃▃▃▁▃▃▃        ↓ 18%                                                    │\n",
       "│ optimizer                                                                                                       │\n",
       "│   accepted 100.0%                                                                                               │\n",
       "│ rollout                                                                                                         │\n",
       "│   episode_return                nan                                                                             │\n",
       "╰──────────────────────────────────────────────────────────────────────────────────────────────────  ✓ finished  ─╯\n",
       "
\n" ], "text/plain": [ "╭─ dew · WorldModel ────────────────────────────────────────────────────────────────────────────────────────────╮\n", "│ 637,446 parameters · on 1 × NVIDIA RTX PRO 6000 Blackwell Server Edition · batch 1 · float32 │\n", "│ │\n", "│ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20000/20000 100% 32.0 step/s 0:10:37 │\n", "│ │\n", "│ train │\n", "│ loss 3.357 █▂▁▁▁▁▁▁▁▁ ↓ 53% │\n", "│ imagined_return 182.5 ▁▅▆▆▆▇▇▇██ ↑ 663% │\n", "│ loss_continue 0.02044 █▂▁▁▁▁▁▁▁▁ ↓ 0.6% │\n", "│ loss_decoder 0.01217 █▅▆▅▅▄▄▃▁▂ ↓ 80% │\n", "│ loss_dynamics 1.380 ▁▃▅▇█▇▇▇▆▆ ↑ 13% │\n", "│ loss_policy -0.006085 █▁▁▁▁▁▁▁▁▁ ↓ 101% │\n", "│ loss_replay_value 1.123 █▄▂▂▂▂▂▁▁▁ ↓ 76% │\n", "│ loss_representation 1.380 ▁▃▅▇█▇▇▇▆▆ ↑ 13% │\n", "│ loss_reward 0.5109 █▄▃▂▂▁▂▂▄▂ ↓ 24% │\n", "│ loss_value 0.9650 █▂▂▂▂▁▂▁▁▁ ↓ 68% │\n", "│ return_scale 106.0 ▁▃▅▆▇▇▇▇██ ↑ 14× │\n", "│ data_wait_ms 0.01646 ▄▁▇▂▇▁▃█▁▅ ↑ 0.7% │\n", "│ grad_norm 3.477 █▆▅▄▃▂▂▂▁▁ ↓ 59% │\n", "│ flops 3.288e+10 compiles 1.000 cache_hits 1,394 cache_misses 0 peak_memory_gib 0.3771 │\n", "│ throughput │\n", "│ step_time_ms 31.25 █▅▅▅▅▅▁▅▅▅ ↓ 5.5% │\n", "│ samples_per_sec 512.0 ▁▄▄▄▄▄█▄▄▄ ↑ 5.8% │\n", "│ mfu 0.2% ▁▄▄▄▄▄█▄▄▄ → │\n", "│ hlo_flops_util 0.2% ▁▄▄▄▄▄█▄▄▄ → │\n", "│ hfu 0.2% ▁▄▄▄▄▄█▄▄▄ → │\n", "│ rollout_seconds 55.34 █▃▃▃▃▃▁▃▃▃ ↓ 18% │\n", "│ optimizer │\n", "│ accepted 100.0% │\n", "│ rollout │\n", "│ episode_return nan │\n", "╰────────────────────────────────────────────────────────────────────────────────────────────────── ✓ finished ─╯\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
╭─  ✓ WorldModel  ────────────────────────────╮\n",
       "│    trained  20000 steps in 0:10:38          │\n",
       "│ first step  after 16.30 s                   │\n",
       "│       then  32.2 step/s                     │\n",
       "│    goodput  97.4% of the wall time in steps │\n",
       "│ final loss  3.357                           │\n",
       "╰─────────────────────────────────────────────╯\n",
       "
\n" ], "text/plain": [ "\u001b[38;2;42;167;201m╭─\u001b[0m\u001b[38;2;42;167;201m \u001b[0m\u001b[38;2;63;174;106m✓ \u001b[0m\u001b[1;38;2;42;167;201mWorldModel\u001b[0m\u001b[38;2;42;167;201m \u001b[0m\u001b[38;2;42;167;201m───────────────────────────\u001b[0m\u001b[38;2;42;167;201m─╮\u001b[0m\n", "\u001b[38;2;42;167;201m│\u001b[0m \u001b[2m trained\u001b[0m\u001b[2m \u001b[0m20000 steps in 0:10:38 \u001b[38;2;42;167;201m│\u001b[0m\n", "\u001b[38;2;42;167;201m│\u001b[0m \u001b[2mfirst step\u001b[0m\u001b[2m \u001b[0mafter 16.30 s \u001b[38;2;42;167;201m│\u001b[0m\n", "\u001b[38;2;42;167;201m│\u001b[0m \u001b[2m then\u001b[0m\u001b[2m \u001b[0m32.2 step/s \u001b[38;2;42;167;201m│\u001b[0m\n", "\u001b[38;2;42;167;201m│\u001b[0m \u001b[2m goodput\u001b[0m\u001b[2m \u001b[0m97.4% of the wall time in steps \u001b[38;2;42;167;201m│\u001b[0m\n", "\u001b[38;2;42;167;201m│\u001b[0m \u001b[2mfinal loss\u001b[0m\u001b[2m \u001b[0m\u001b[1m3.357 \u001b[0m \u001b[38;2;42;167;201m│\u001b[0m\n", "\u001b[38;2;42;167;201m╰─────────────────────────────────────────────╯\u001b[0m\n" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "638 s, 21024 environment steps; return 810.866943359375\n" ] } ], "source": [ "from dew.objectives.rl.dreamer import DreamerRollout, DreamerV3Objective, WorldModel, episodes_return\n", "from dew.optim import laprop\n", "\n", "sys_env = playground(TASK, 16)\n", "dreamer = DreamerV3Objective(WorldModel(int(probe.env.observation_size), probe.action_size))\n", "replay = DreamerRollout(dreamer, sys_env, capacity=4000, batch=16, length=64, train_ratio=1024.0, prefill=64)\n", "began = time.perf_counter()\n", "agent = Trainer(dreamer, laprop(learning_rate=1e-4), key=jax.random.key(3), rollout=replay).fit(idle, steps=20_000,\n", " log_every=2000)\n", "print(f\"{time.perf_counter() - began:.0f} s, {replay.environment_steps} environment steps; return\",\n", " episodes_return(dreamer, playground(TASK, 16), agent.variables, jax.random.key(4), 1000))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The return is 810.9, from 21,024 real environment steps in all, and it took 638 seconds. Set against PPO's run, those three numbers are the trade to judge.\n", "\n", "The model is not buying you a better cartpole. PPO reached 999.7 in 44 seconds, because MuJoCo steps are nearly free and PPO spent them freely, thousands of carts at a time. Dreamer spent its 638 seconds mostly on arithmetic, replaying and imagining, and asked the simulator for only about twenty one thousand steps. On this task that trade is a bad one.\n", "\n", "But turn the prices around in your head. Give each real step a real cost: a slow simulator, a robot that wears out, a drone that crashes. Then the same trade starts to look very different. The model's whole value is that it converts a small pile of real steps into a large amount of practice. Whether that conversion is worth its price depends entirely on what a real step costs you, and the last part of this tutorial is about a task where steps are not free at all.\n", "\n", "First, though, we should check the thing itself. We have been told the actor learns from imagined rewards. Do those rewards have anything to do with the real ones?\n", "\n", "## Checking the imagination against the replay\n", "\n", "Here is a direct way to look. Take a stretch from the replay and watch it for 32 steps, every observation fixing the latent through the posterior. Then close your eyes: from that latent, let the prior run on for 32 more steps with the policy acting on what it imagines, and collect the predicted rewards. Beside them, put the rewards that same replayed stretch actually went on to earn. Keep in mind what this comparison can show. The imagined stretch acts with fresh draws from the policy, not with the actions stored in the replay, so the two stretches are different trajectories that share only their starting latent. Even a perfect model would not reproduce the stored rewards step for step, and matching average levels cannot rule out a model that predicts an average reward. This is a distribution-level sanity check: do the imagined rewards sit at about the right level?\n", "\n", "![Imagination branching off a replayed run. The run is watched for 32 steps with every observation fixing the latent through the posterior, then the observations stop and the prior runs on alone for 32 steps under the policy, its continuations fanning out because each step draws its z. The imagined rewards come from the reward head; the replayed ones were read from the run itself.](https://raw.githubusercontent.com/AshishKumar4/dew/main/docs/assets/world-imagine.svg)\n", "\n", "In training the same move is made constantly and more modestly: 15 imagined steps from every replayed state in a batch, and the critic learns its values from those short rehearsals. The cell below makes one 32-step rehearsal so we can compare it with what happened next.\n" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "execution": { "iopub.execute_input": "2026-10-31T20:13:43.219897Z", "iopub.status.busy": "2026-10-31T20:13:43.219785Z", "iopub.status.idle": "2026-10-31T20:13:44.530006Z", "shell.execute_reply": "2026-10-31T20:13:44.529601Z" } }, "outputs": [ { "data": { "text/html": [ "reward against step0.4250.450.4750102030steprewardreplayed 0.40imagined 0.44" ], "text/plain": [ "Figure(460 x 290, 18 shapes)" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from dew.objectives.rl.dreamer import LatentEnv\n", "from dew.rl.distributions import twohot_bins\n", "from dew.rl.env import Carry\n", "\n", "windows = replay.sample(jax.random.key(5))\n", "model = dreamer.model.bind(agent.variables, rngs={\"latent\": jax.random.key(6)})\n", "deter, stoch = model.initial((16,))\n", "first = jnp.asarray(windows[\"first\"]).at[:, 0].set(True)\n", "previous = jnp.concatenate([jnp.zeros_like(windows[\"action\"][:, :1]), windows[\"action\"][:, :-1]], 1)\n", "for t in range(32):\n", " deter, stoch, _ = dreamer.observe(model, deter, stoch, windows[\"observation\"][:, t], previous[:, t], first[:, t])\n", "world = LatentEnv(dreamer.model, agent.variables, twohot_bins())\n", "state, feature = world.at(deter, stoch, jax.random.key(7))\n", "\n", "def imagined_policy(params, feature, key, memory):\n", " mean, std = dreamer.model.apply(params, feature, method=WorldModel.policy)\n", " return jnp.clip(mean + std * jax.random.normal(key, mean.shape), -1, 1), {}, None\n", "\n", "_, dream = env_rollout(world, imagined_policy, agent.variables, Carry(state, feature, jax.random.key(8)), 32,\n", " reset=False)\n", "figures.curve({\"replayed\": np.asarray(windows[\"reward\"][:, 32:]).mean(0),\n", " \"imagined\": np.asarray(dream.following.reward).mean(1)}, x_label=\"step\", y_label=\"reward\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The two lines wander in the same band and end close, the replayed mean at 0.40 and the imagined at 0.44, with the imagined line the smoother of the two in this plot. Nothing guarantees that smoothness: each imagined reward is the reward head's mean at a freshly drawn latent under freshly drawn actions, averaged over the 16 carts, and a mean can still jump when its input does. What the check supports is the level. Over these 32 steps the imagined rewards stay near what the replay went on to earn, which is what a rehearsal needs from them. Push the imagination much further and the fan in the picture is the truth of it. Every prior step draws a fresh $z$, small disagreements compound, and the rehearsal slowly becomes a story the model tells itself. Dreamer keeps its rehearsals at 15 steps for that reason.\n", "\n", "## A world model that learns from video\n", "\n", "DreamerV3's model is built around acting. It needs actions and rewards, and its latent is one step of an agent's life. Dreamer 4 ([Hafner et al., 2025](https://arxiv.org/abs/2509.24527)) asks a different question. What if the world model first learns from video alone, and only later has actions threaded through it? Two pieces change. A tokenizer squeezes each frame, on its own so nothing leaks backwards in time, into a small set of latents. Then a transformer over those latents learns how a sequence of frames continues.\n", "\n", "The transformer is trained as a diffusion model, more or less. Each frame's latents are mixed with noise up to some signal level $\\tau$, different for every frame, and the model has to name the clean latents. Generating then means walking a frame from pure noise at $\\tau = 0$ to clean at $\\tau = 1$, and the natural worry is the one we had with samplers in [tutorial 01](01-diffusion-from-scratch.ipynb): small safe steps take forever, and nobody wants a hundred steps per frame.\n", "\n", "Shortcut forcing is Dreamer 4's answer, and it is a lovely piece of self discipline. The model is told, along with $\\tau$, the size $d$ of the step it is being asked to take. At the smallest size it simply learns to name the clean latents. At any larger size it is asked to agree with itself. Take one step of size $d$, then take two of its own half steps of size $d/2$ over the same stretch, the second starting where the first landed, and average where the two half steps point. The large step is trained to match that average. At larger sizes the target is the model's own half steps, so training encourages the sizes to agree with each other. Agreement by itself says nothing about accuracy, as the run below shows.\n", "\n", "![Shortcut forcing on one stretch between noise and clean. Two half steps of size d over 2 cross from tau to tau plus d, the second starting where the first lands, and training pulls the endpoint of one step of size d to where the two half steps land. The notebook generates each frame in four such large steps.](https://raw.githubusercontent.com/AshishKumar4/dew/main/docs/assets/world-shortcut.svg)\n", "\n", "Our test world is as small as a world gets: two balls bounce around a $32 \\times 32$ box for 12 frames, in 2048 clips of video, with no actions or rewards. The tokenizer squeezes each frame into 16 latents of 8 numbers and the dynamics model learns over those. At the end we show it four frames of an unseen clip and ask for the next eight, four steps a frame, beside what really happened.\n" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "execution": { "iopub.execute_input": "2026-10-31T20:13:44.531422Z", "iopub.status.busy": "2026-10-31T20:13:44.531278Z", "iopub.status.idle": "2026-10-31T20:15:57.671880Z", "shell.execute_reply": "2026-10-31T20:15:57.671523Z" } }, "outputs": [ { "data": { "text/html": [ "
╭─  dew · CausalTokenizer  ───────────────────────────────────────────────────────────────────────────────────────╮\n",
       "│ 1.6M parameters  ·  on 1 × NVIDIA RTX PRO 6000 Blackwell Server Edition  ·  batch 32  ·  float32                │\n",
       "│                                                                                                                 │\n",
       "│ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  2000/2000  100%  42.7 step/s  0:00:51 │\n",
       "│                                                                                                                 │\n",
       "│ train                                                                                                           │\n",
       "│   loss                   0.1465  ▁█       ↑ 0.2%                                                                │\n",
       "│   mse                    0.1465  ▁█       ↑ 0.2%                                                                │\n",
       "│   data_wait_ms          0.01679  ▁█       ↑ 3.3%                                                                │\n",
       "│   grad_norm              0.4110  █▁        ↓ 35%                                                                │\n",
       "│   compiles                1.000  ▁▁            →                                                                │\n",
       "│   cache_hits              1,598  ▁▁            →                                                                │\n",
       "│   cache_misses                0  ▁▁            →                                                                │\n",
       "│   peak_memory_gib         2.516  ▁█            →                                                                │\n",
       "│ throughput                                                                                                      │\n",
       "│   step_time_ms            23.41  █▁       ↓ 6.1%                                                                │\n",
       "│   samples_per_sec         1,367  ▁█       ↑ 6.5%                                                                │\n",
       "│   hlo_flops_util           2.7%  ▁█     ↑ 0.2 pt                                                                │\n",
       "│   hfu                      2.7%  ▁█     ↑ 0.2 pt                                                                │\n",
       "│ optimizer                                                                                                       │\n",
       "│   accepted               100.0%  ▁▁            →                                                                │\n",
       "╰──────────────────────────────────────────────────────────────────────────────────────────────────  ✓ finished  ─╯\n",
       "
\n" ], "text/plain": [ "╭─ dew · CausalTokenizer ───────────────────────────────────────────────────────────────────────────────────────╮\n", "│ 1.6M parameters · on 1 × NVIDIA RTX PRO 6000 Blackwell Server Edition · batch 32 · float32 │\n", "│ │\n", "│ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2000/2000 100% 42.7 step/s 0:00:51 │\n", "│ │\n", "│ train │\n", "│ loss 0.1465 ▁█ ↑ 0.2% │\n", "│ mse 0.1465 ▁█ ↑ 0.2% │\n", "│ data_wait_ms 0.01679 ▁█ ↑ 3.3% │\n", "│ grad_norm 0.4110 █▁ ↓ 35% │\n", "│ compiles 1.000 ▁▁ → │\n", "│ cache_hits 1,598 ▁▁ → │\n", "│ cache_misses 0 ▁▁ → │\n", "│ peak_memory_gib 2.516 ▁█ → │\n", "│ throughput │\n", "│ step_time_ms 23.41 █▁ ↓ 6.1% │\n", "│ samples_per_sec 1,367 ▁█ ↑ 6.5% │\n", "│ hlo_flops_util 2.7% ▁█ ↑ 0.2 pt │\n", "│ hfu 2.7% ▁█ ↑ 0.2 pt │\n", "│ optimizer │\n", "│ accepted 100.0% ▁▁ → │\n", "╰────────────────────────────────────────────────────────────────────────────────────────────────── ✓ finished ─╯\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
╭─  ✓ CausalTokenizer  ───────────────────────╮\n",
       "│    trained  2000 steps in 0:00:51           │\n",
       "│ first step  after 4.49 s                    │\n",
       "│       then  42.7 step/s                     │\n",
       "│    goodput  91.2% of the wall time in steps │\n",
       "│ final loss  0.1465                          │\n",
       "╰─────────────────────────────────────────────╯\n",
       "
\n" ], "text/plain": [ "\u001b[38;2;42;167;201m╭─\u001b[0m\u001b[38;2;42;167;201m \u001b[0m\u001b[38;2;63;174;106m✓ \u001b[0m\u001b[1;38;2;42;167;201mCausalTokenizer\u001b[0m\u001b[38;2;42;167;201m \u001b[0m\u001b[38;2;42;167;201m──────────────────────\u001b[0m\u001b[38;2;42;167;201m─╮\u001b[0m\n", "\u001b[38;2;42;167;201m│\u001b[0m \u001b[2m trained\u001b[0m\u001b[2m \u001b[0m2000 steps in 0:00:51 \u001b[38;2;42;167;201m│\u001b[0m\n", "\u001b[38;2;42;167;201m│\u001b[0m \u001b[2mfirst step\u001b[0m\u001b[2m \u001b[0mafter 4.49 s \u001b[38;2;42;167;201m│\u001b[0m\n", "\u001b[38;2;42;167;201m│\u001b[0m \u001b[2m then\u001b[0m\u001b[2m \u001b[0m42.7 step/s \u001b[38;2;42;167;201m│\u001b[0m\n", "\u001b[38;2;42;167;201m│\u001b[0m \u001b[2m goodput\u001b[0m\u001b[2m \u001b[0m91.2% of the wall time in steps \u001b[38;2;42;167;201m│\u001b[0m\n", "\u001b[38;2;42;167;201m│\u001b[0m \u001b[2mfinal loss\u001b[0m\u001b[2m \u001b[0m\u001b[1m0.1465 \u001b[0m \u001b[38;2;42;167;201m│\u001b[0m\n", "\u001b[38;2;42;167;201m╰─────────────────────────────────────────────╯\u001b[0m\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
╭─  dew · Dynamics  ──────────────────────────────────────────────────────────────────────────────────────────────╮\n",
       "│ 6.3M parameters  ·  on 1 × NVIDIA RTX PRO 6000 Blackwell Server Edition  ·  batch 32  ·  float32                │\n",
       "│                                                                                                                 │\n",
       "│ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  3000/3000  100%  44.1 step/s  0:01:13 │\n",
       "│                                                                                                                 │\n",
       "│ train                                                                                                           │\n",
       "│   loss                  0.01494  █▁▁        ↓ 74%                                                               │\n",
       "│   frame_loss            0.01494  █▁▁        ↓ 74%                                                               │\n",
       "│   data_wait_ms          0.01695  ▁█▄       ↑ 0.8%                                                               │\n",
       "│   grad_norm              0.9320  █▂▁        ↓ 84%                                                               │\n",
       "│   compiles 1.000   cache_hits 1,600   cache_misses 0   peak_memory_gib 2.516                                    │\n",
       "│ throughput                                                                                                      │\n",
       "│   step_time_ms            22.65  █▁▁       ↓ 8.3%                                                               │\n",
       "│   samples_per_sec         1,413  ▁██       ↑ 9.1%                                                               │\n",
       "│   hlo_flops_util           4.7%  ▁██     ↑ 0.4 pt                                                               │\n",
       "│   hfu                      4.7%  ▁██     ↑ 0.4 pt                                                               │\n",
       "│ optimizer                                                                                                       │\n",
       "│   accepted 100.0%                                                                                               │\n",
       "╰──────────────────────────────────────────────────────────────────────────────────────────────────  ✓ finished  ─╯\n",
       "
\n" ], "text/plain": [ "╭─ dew · Dynamics ──────────────────────────────────────────────────────────────────────────────────────────────╮\n", "│ 6.3M parameters · on 1 × NVIDIA RTX PRO 6000 Blackwell Server Edition · batch 32 · float32 │\n", "│ │\n", "│ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3000/3000 100% 44.1 step/s 0:01:13 │\n", "│ │\n", "│ train │\n", "│ loss 0.01494 █▁▁ ↓ 74% │\n", "│ frame_loss 0.01494 █▁▁ ↓ 74% │\n", "│ data_wait_ms 0.01695 ▁█▄ ↑ 0.8% │\n", "│ grad_norm 0.9320 █▂▁ ↓ 84% │\n", "│ compiles 1.000 cache_hits 1,600 cache_misses 0 peak_memory_gib 2.516 │\n", "│ throughput │\n", "│ step_time_ms 22.65 █▁▁ ↓ 8.3% │\n", "│ samples_per_sec 1,413 ▁██ ↑ 9.1% │\n", "│ hlo_flops_util 4.7% ▁██ ↑ 0.4 pt │\n", "│ hfu 4.7% ▁██ ↑ 0.4 pt │\n", "│ optimizer │\n", "│ accepted 100.0% │\n", "╰────────────────────────────────────────────────────────────────────────────────────────────────── ✓ finished ─╯\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
╭─  ✓ Dynamics  ──────────────────────────────╮\n",
       "│    trained  3000 steps in 0:01:14           │\n",
       "│ first step  after 5.77 s                    │\n",
       "│       then  44.1 step/s                     │\n",
       "│    goodput  92.2% of the wall time in steps │\n",
       "│ final loss  0.01494                         │\n",
       "╰─────────────────────────────────────────────╯\n",
       "
\n" ], "text/plain": [ "\u001b[38;2;42;167;201m╭─\u001b[0m\u001b[38;2;42;167;201m \u001b[0m\u001b[38;2;63;174;106m✓ \u001b[0m\u001b[1;38;2;42;167;201mDynamics\u001b[0m\u001b[38;2;42;167;201m \u001b[0m\u001b[38;2;42;167;201m─────────────────────────────\u001b[0m\u001b[38;2;42;167;201m─╮\u001b[0m\n", "\u001b[38;2;42;167;201m│\u001b[0m \u001b[2m trained\u001b[0m\u001b[2m \u001b[0m3000 steps in 0:01:14 \u001b[38;2;42;167;201m│\u001b[0m\n", "\u001b[38;2;42;167;201m│\u001b[0m \u001b[2mfirst step\u001b[0m\u001b[2m \u001b[0mafter 5.77 s \u001b[38;2;42;167;201m│\u001b[0m\n", "\u001b[38;2;42;167;201m│\u001b[0m \u001b[2m then\u001b[0m\u001b[2m \u001b[0m44.1 step/s \u001b[38;2;42;167;201m│\u001b[0m\n", "\u001b[38;2;42;167;201m│\u001b[0m \u001b[2m goodput\u001b[0m\u001b[2m \u001b[0m92.2% of the wall time in steps \u001b[38;2;42;167;201m│\u001b[0m\n", "\u001b[38;2;42;167;201m│\u001b[0m \u001b[2mfinal loss\u001b[0m\u001b[2m \u001b[0m\u001b[1m0.01494 \u001b[0m \u001b[38;2;42;167;201m│\u001b[0m\n", "\u001b[38;2;42;167;201m╰─────────────────────────────────────────────╯\u001b[0m\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "16 imagestruthimagined" ], "text/plain": [ "Figure(527.2 x 120, 3 shapes)" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def bouncing(rows, frames, size, seed):\n", " \"\"\"Videos [rows, frames, size, size, 1] in [-1, 1] of two balls bouncing off the walls.\"\"\"\n", " rng = np.random.default_rng(seed)\n", " grid = np.stack(np.meshgrid(np.arange(size), np.arange(size), indexing=\"ij\"), -1) + 0.5\n", " videos = np.full((rows, frames, size, size, 1), -1.0, np.float32)\n", " for video in videos:\n", " position, velocity = rng.uniform(3, size - 3, (2, 2)), rng.uniform(-1.5, 1.5, (2, 2))\n", " for frame in video:\n", " for ball in range(2):\n", " frame[np.sum(np.square(grid - position[ball]), -1) <= 2.5 ** 2, 0] = 1.0\n", " position = position + velocity\n", " velocity = np.where((position < 2.5) | (position > size - 2.5), -velocity, velocity)\n", " position = np.clip(position, 2.5, size - 2.5)\n", " return videos\n", "\n", "\n", "from dew.objectives.world_model import CausalTokenizer, Dynamics, TokenizerObjective, WorldModelObjective\n", "\n", "videos, held = bouncing(2048, 12, 32, 0), bouncing(8, 12, 32, 1)\n", "clips = Loader(train=lambda partition: ({\"video\": videos[i:i + 32]} for i in itertools.cycle(range(0, 2048, 32))),\n", " val=None, records=2048, batch=32)\n", "tokenizer = CausalTokenizer((32, 32, 1), width=128, depth=3, latents=16, bottleneck=8)\n", "tokens = Trainer(TokenizerObjective(tokenizer), optax.adam(3e-4), key=jax.random.key(9)).fit(clips, steps=2000,\n", " log_every=1000)\n", "dynamics = Dynamics(latents=16, bottleneck=8, spatial=16, width=256, depth=6, frames=12)\n", "world_model = WorldModelObjective(dynamics, tokenizer, tokens.variables)\n", "trained = Trainer(world_model, optax.adam(3e-4), key=jax.random.key(10)).fit(clips, steps=3000, log_every=1000)\n", "latents = world_model.latents(jnp.asarray(held))\n", "predicted = world_model.generate(trained.variables, latents[:, :4], 8, jax.random.key(11), steps=4)\n", "frames = np.asarray(tokenizer.apply(tokens.variables, predicted, method=CausalTokenizer.decode))\n", "def pixels(video):\n", " return np.repeat(np.round((np.clip(video, -1, 1) + 1) * 127.5).astype(np.uint8), 3, -1)\n", "\n", "figures.images([np.concatenate([pixels(held[0, 4:]), pixels(frames[0])])], columns=8, size=52,\n", " rows=[\"truth\", \"imagined\"])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Look at the two rows. The top row is the truth, two pale balls drifting a little across the eight frames. The bottom row is the model's continuation, decoded back into pictures, and there are no balls in it to follow, just a faint wash where they should be. At this size and after these few thousand steps, the visible continuation is not there yet.\n", "\n", "The numbers hint at where it stands. The dynamics loss fell by about three quarters over its 3000 steps, to 0.01494. That loss is the shortcut-forcing objective, whose larger step sizes are scored against the model's own half-step predictions rather than the true next latents, so the fall says the model meets its training objective better. It does not by itself say the next frame is forecast well, and the decoded row above is the forecast test. The tokenizer, after 2000 steps, is still at 0.1465, and whatever the dynamics has learned does not yet survive the trip back through it into pictures you can read. Dreamer 4's own results come from far larger models trained far longer on far richer video, and its dynamics model can also be given the actions taken between frames, which is what would make a model like this one somewhere an agent could act. What this little run gives you is the machinery and the training signal, short of a watchable continuation. The acting half, where the model really does stay in charge of the future, is where the drones come in.\n", "\n", "## Racing drones, where steps cost something\n", "\n", "A first-person racing drone is an awkward case for everything so far. Its physics is ordinary arithmetic: rotors, drag, motor lag, a battery that sags. It fits the two-function shape, and thousands of drones can fly inside one compiled rollout. Its camera is another matter. Drawing what the drone sees means ray tracing gates, pillars, ground and sky, and that work does not live inside JAX. So Dew splits the race in two along that seam.\n", "\n", "![Gate racing split in two. On the left the quadrotor physics is a JAX environment, 4096 drones inside one jitted rollout, and a teacher learns from the privileged state, the next gates' corners and the obstacles. On the right a ray tracer outside JAX draws each drone's camera for the same races, stepped from the host a batch at a time, and the racing policy sees that image alone with the next gate lit.](https://raw.githubusercontent.com/AshishKumar4/dew/main/docs/assets/world-racing.svg)\n", "\n", "On the JAX side sits `FlightEnv`. Every episode gets a course of its own: a loop or a chain of gates with pillars scattered among them. It starts somewhere on that course. Every drone is also drawn a little differently, in its thrust, its rate limits, its command latency and the tilt of its camera, the way real drones differ. On the host side sits `GateRace`, which steps the same races and asks a renderer, here the HyperLux ray tracer over a handful of CC0 textures, to draw each camera. A thing stepped from the host like that is a `Simulator` rather than an `Env`, but it answers the same questions, so rollouts and scoring treat the two alike. Only the drawing lives outside.\n", "\n", "Notice what each side may see, because the next tutorial hangs on it. The flying policy gets the camera image alone, with the next gate lit. The drone's full state, the next gates' corners and the pillars are computed anyway and shown to no policy. They are privileged. A teacher may read them, and a world model may be asked to predict them. In a moment we will do the first of those.\n" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "execution": { "iopub.execute_input": "2026-10-31T20:15:57.673089Z", "iopub.status.busy": "2026-10-31T20:15:57.672986Z", "iopub.status.idle": "2026-10-31T20:16:07.917186Z", "shell.execute_reply": "2026-10-31T20:16:07.916918Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\n", " \"license\": \"CC0 (polyhaven.com/license) textures; geometry Dew's\",\n", " \"ground\": [\n", " \"ground_aerial_grass_rock.glb\",\n", " \"ground_forrest_ground_01.glb\",\n", " \"ground_aerial_asphalt_01.glb\",\n", " \"ground_aerial_beach_01.glb\",\n", " \"ground_rocky_terrain_02.glb\",\n", " \"ground_brown_mud_leaves_01.glb\"\n", " ],\n", " \"sky\": [\n", " \"sky_kloofendal_48d_partly_cloudy_puresky.glb\",\n", " \"sky_lilienstein.glb\",\n", " \"sky_the_sky_is_on_fire.glb\",\n", " \"sky_kloppenheim_02.glb\"\n", " ],\n", " \"gate\": [\n", " \"gate_orange.glb\",\n", " \"gate_blue.glb\",\n", " \"gate_white.glb\",\n", " \"gate_padded.glb\"\n", " ],\n", " \"lit\": [\n", " \"lit_green.glb\",\n", " \"lit_chevrons.glb\",\n", " \"lit_magenta.glb\",\n", " \"lit_cyan.glb\"\n", " ],\n", " \"pillar\": [\n", " [\n", " \"pillar_6_concrete_floor_worn_001.glb\",\n", " \"pillar_6_rust_coarse_01.glb\"\n", " ],\n", " [\n", " \"pillar_12_concrete_floor_worn_001.glb\",\n", " \"pillar_12_rust_coarse_01.glb\"\n", " ],\n", " [\n", " \"pillar_24_concrete_floor_worn_001.glb\",\n", " \"pillar_24_rust_coarse_01.glb\"\n", " ]\n", " ],\n", " \"prop\": [\n", " \"prop_tree.glb\",\n", " \"prop_rock.glb\",\n", " \"prop_banner.glb\"\n", " ],\n", " \"drone\": \"drone.glb\",\n", " \"aspects\": [\n", " 6.0,\n", " 12.0,\n", " 24.0\n", " ],\n", " \"gate_geometry\": {\n", " \"opening\": 1.5,\n", " \"frame\": 0.25,\n", " \"depth\": 0.1\n", " }\n", "}\n" ] }, { "data": { "text/html": [ "16 images at each of 2 pointsstartafter a second of hovering" ], "text/plain": [ "Figure(548 x 170, 4 shapes)" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "!curl -sLO https://raw.githubusercontent.com/AshishKumar4/dew/ff1a61a94d3b31152bd10d606204633e44534b85/tools/fpv_assets.py\n", "!python fpv_assets.py fpv-assets\n", "\n", "from dew.rl.flight import GateRace, Racing\n", "from dew.rl.hyperlux import HyperLuxRenderer\n", "\n", "racing = Racing()\n", "race = GateRace(HyperLuxRenderer(16, \"fpv-assets\"), racing, views=(\"image\", \"passed\"))\n", "shown = [np.asarray(race.reset(0)[\"image\"])]\n", "for _ in range(60):\n", " stepped = race.step(jnp.zeros((16, 4)))\n", "shown.append(np.asarray(stepped.observation[\"image\"]))\n", "figures.images(shown, columns=8, size=64, captions=[\"start\", \"after a second of hovering\"])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Those are 16 drones at their starts and again after 60 steps under a zero action, about a second at this race's 60 steps a second. A zero action asks for hover thrust and no body rates. It is a neutral command, not a command to hold position, and a drone that started already flying keeps its motion under it. Every view is different because every start and every camera is drawn afresh. This, at far lower resolution, is all a racing policy ever gets to see.\n", "\n", "It raises an obvious question. If we want good flying to learn from, why make the first learner peer through a camera? Let one policy read the privileged numbers directly: how the drone moves and turns, the next two gates' corners, the nearest obstacles, even its own airframe's drawn physics, packed into one observation of width `TEACHER`. It flies 4096 drones at once inside JAX, no images anywhere, so its practice is cheap as PPO's was. What it learns could later be shown to a camera policy, or fill a world model's replay; that longer programme lives in `tools/fpv_race.py`. Here we only ask whether the teacher itself can learn the race.\n", "\n", "To keep its first lesson gentle, the course is loosened up. Only loop courses, and both the course difficulty and the drone randomization turned down to a fifth. We score the teacher before and after training on 256 fresh drones of its own arrangement, counting the gates each drone passes in its best episode over the 25 seconds an episode lasts, and how often a drone crashes along the way.\n" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "execution": { "iopub.execute_input": "2026-10-31T20:16:07.918525Z", "iopub.status.busy": "2026-10-31T20:16:07.918396Z", "iopub.status.idle": "2026-10-31T20:19:45.025044Z", "shell.execute_reply": "2026-10-31T20:19:45.024621Z" } }, "outputs": [ { "data": { "text/html": [ "
╭─  dew · ActorCritic  ───────────────────────────────────────────────────────────────────────────────────────────╮\n",
       "│ 370,697 parameters  ·  on 1 × NVIDIA RTX PRO 6000 Blackwell Server Edition  ·  batch 1  ·  float32              │\n",
       "│                                                                                                                 │\n",
       "│ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  25600/25600  100%  131.9 step/s  0:03:25 │\n",
       "│                                                                                                                 │\n",
       "│ train                                                                                                           │\n",
       "│   loss                      1.330  ▁▃▄█▁▅▂▆▂█       ↑ 324%                                                      │\n",
       "│   entropy                  -5.531  █▆▆▅▅▃▃▂▁▁        ↓ 70%                                                      │\n",
       "│   pg_clipfrac             0.09082  ▁▁▄▂▇▆▄▅█▄        ↑ 60%                                                      │\n",
       "│   policy                 0.002198  ▁▄▁▂▃▄▁▃█▃       ↑ 307%                                                      │\n",
       "│   ppo_kl                  0.01384  ▃▁▂▁▆▆▄▇█▄        ↑ 32%                                                      │\n",
       "│   value                     1.327  ▁▃▄█▁▅▂▆▂█       ↑ 322%                                                      │\n",
       "│   data_wait_ms            0.01431  ▄█▃▁▆▃▂▄▁▁       ↓ 4.9%                                                      │\n",
       "│   grad_norm                 15.20  ▁▄▆█▇▆▆▇▇█       ↑ 896%                                                      │\n",
       "│   pg_clipfrac_lower 0   flops 2.104e+10   compiles 1.000   cache_hits 2,752   cache_misses 0   peak_memory_gib  │\n",
       "│ 2.516                                                                                                           │\n",
       "│ throughput                                                                                                      │\n",
       "│   step_time_ms              7.665  █▄▄▃▃▃▃▃▄▁       ↓ 4.5%                                                      │\n",
       "│   samples_per_sec          16,698  ▁▅▅▆▆▆▆▆▅█       ↑ 4.7%                                                      │\n",
       "│   mfu                        0.5%  ▁▅▅▆▆▆▆▆▅█            →                                                      │\n",
       "│   hlo_flops_util             0.5%  ▁▅▅▆▆▆▆▆▅█            →                                                      │\n",
       "│   hfu                        0.5%  ▁▅▅▆▆▆▆▆▅█            →                                                      │\n",
       "│   rollout_seconds           17.23  █▁▁▁▁▁▁▁▁▁        ↓ 23%                                                      │\n",
       "│ optimizer                                                                                                       │\n",
       "│   accepted 100.0%                                                                                               │\n",
       "│ rollout                                                                                                         │\n",
       "│   episode_return            457.2  ▁▁▁▂▄▅▆▆▇█            ↑                                                      │\n",
       "╰──────────────────────────────────────────────────────────────────────────────────────────────────  ✓ finished  ─╯\n",
       "
\n" ], "text/plain": [ "╭─ dew · ActorCritic ───────────────────────────────────────────────────────────────────────────────────────────╮\n", "│ 370,697 parameters · on 1 × NVIDIA RTX PRO 6000 Blackwell Server Edition · batch 1 · float32 │\n", "│ │\n", "│ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 25600/25600 100% 131.9 step/s 0:03:25 │\n", "│ │\n", "│ train │\n", "│ loss 1.330 ▁▃▄█▁▅▂▆▂█ ↑ 324% │\n", "│ entropy -5.531 █▆▆▅▅▃▃▂▁▁ ↓ 70% │\n", "│ pg_clipfrac 0.09082 ▁▁▄▂▇▆▄▅█▄ ↑ 60% │\n", "│ policy 0.002198 ▁▄▁▂▃▄▁▃█▃ ↑ 307% │\n", "│ ppo_kl 0.01384 ▃▁▂▁▆▆▄▇█▄ ↑ 32% │\n", "│ value 1.327 ▁▃▄█▁▅▂▆▂█ ↑ 322% │\n", "│ data_wait_ms 0.01431 ▄█▃▁▆▃▂▄▁▁ ↓ 4.9% │\n", "│ grad_norm 15.20 ▁▄▆█▇▆▆▇▇█ ↑ 896% │\n", "│ pg_clipfrac_lower 0 flops 2.104e+10 compiles 1.000 cache_hits 2,752 cache_misses 0 peak_memory_gib │\n", "│ 2.516 │\n", "│ throughput │\n", "│ step_time_ms 7.665 █▄▄▃▃▃▃▃▄▁ ↓ 4.5% │\n", "│ samples_per_sec 16,698 ▁▅▅▆▆▆▆▆▅█ ↑ 4.7% │\n", "│ mfu 0.5% ▁▅▅▆▆▆▆▆▅█ → │\n", "│ hlo_flops_util 0.5% ▁▅▅▆▆▆▆▆▅█ → │\n", "│ hfu 0.5% ▁▅▅▆▆▆▆▆▅█ → │\n", "│ rollout_seconds 17.23 █▁▁▁▁▁▁▁▁▁ ↓ 23% │\n", "│ optimizer │\n", "│ accepted 100.0% │\n", "│ rollout │\n", "│ episode_return 457.2 ▁▁▁▂▄▅▆▆▇█ ↑ │\n", "╰────────────────────────────────────────────────────────────────────────────────────────────────── ✓ finished ─╯\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
╭─  ✓ ActorCritic  ───────────────────────────╮\n",
       "│    trained  25600 steps in 0:03:25          │\n",
       "│ first step  after 5.62 s                    │\n",
       "│       then  128.4 step/s                    │\n",
       "│    goodput  97.3% of the wall time in steps │\n",
       "│ final loss  1.330                           │\n",
       "╰─────────────────────────────────────────────╯\n",
       "
\n" ], "text/plain": [ "\u001b[38;2;42;167;201m╭─\u001b[0m\u001b[38;2;42;167;201m \u001b[0m\u001b[38;2;63;174;106m✓ \u001b[0m\u001b[1;38;2;42;167;201mActorCritic\u001b[0m\u001b[38;2;42;167;201m \u001b[0m\u001b[38;2;42;167;201m──────────────────────────\u001b[0m\u001b[38;2;42;167;201m─╮\u001b[0m\n", "\u001b[38;2;42;167;201m│\u001b[0m \u001b[2m trained\u001b[0m\u001b[2m \u001b[0m25600 steps in 0:03:25 \u001b[38;2;42;167;201m│\u001b[0m\n", "\u001b[38;2;42;167;201m│\u001b[0m \u001b[2mfirst step\u001b[0m\u001b[2m \u001b[0mafter 5.62 s \u001b[38;2;42;167;201m│\u001b[0m\n", "\u001b[38;2;42;167;201m│\u001b[0m \u001b[2m then\u001b[0m\u001b[2m \u001b[0m128.4 step/s \u001b[38;2;42;167;201m│\u001b[0m\n", "\u001b[38;2;42;167;201m│\u001b[0m \u001b[2m goodput\u001b[0m\u001b[2m \u001b[0m97.3% of the wall time in steps \u001b[38;2;42;167;201m│\u001b[0m\n", "\u001b[38;2;42;167;201m│\u001b[0m \u001b[2mfinal loss\u001b[0m\u001b[2m \u001b[0m\u001b[1m1.330 \u001b[0m \u001b[38;2;42;167;201m│\u001b[0m\n", "\u001b[38;2;42;167;201m╰─────────────────────────────────────────────╯\u001b[0m\n" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "205 s of training\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "fresh teacher over 25 s: 0.0 gates in its best episode, 12.3 crashes a drone\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "trained teacher over 25 s: 34.6 gates in its best episode, 0.2 crashes a drone\n" ] } ], "source": [ "from dew.rl.env import Batched\n", "from dew.rl.flight import TEACHER, FlightEnv\n", "\n", "easy = Racing(families=((\"loop\", 1.0),))\n", "teacher = ControlPPOObjective(ActorCritic(4, (256, 256, 256), (256, 256, 256, 256), spread=0.2), TEACHER,\n", " discount=0.99, clip=0.2, entropy_cost=1e-4, reads=(\"teacher\",))\n", "flights = ControlRollout(teacher, Batched(FlightEnv(easy, (\"teacher\",), difficulty=0.2, ramp=0.2), 4096),\n", " unroll_length=64, epochs=4, minibatches=32)\n", "began = time.perf_counter()\n", "taught = Trainer(teacher, optax.adam(3e-4), key=jax.random.key(12), rollout=flights).fit(idle, steps=25_600,\n", " log_every=2560)\n", "print(f\"{time.perf_counter() - began:.0f} s of training\")\n", "\n", "judge = Batched(FlightEnv(easy, (\"teacher\", \"passed\"), difficulty=0.2, ramp=0.2), 256)\n", "flown = jax.jit(lambda params, carry: env_rollout(judge, teacher.greedy, params, carry, easy.max_steps))\n", "for name, params in ((\"fresh\", teacher.init(jax.random.key(13))), (\"trained\", taught.variables)):\n", " _, steps = flown(params, start(judge, jax.random.key(14)))\n", " passed = np.asarray(steps.following.observation[\"passed\"]).max(0)\n", " crashes = np.asarray(steps.following.terminated).sum(0)\n", " print(f\"{name} teacher over {easy.episode_seconds:.0f} s: {passed.mean():.1f} gates in its best episode, \"\n", " f\"{crashes.mean():.1f} crashes a drone\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Before training, no gates at all and 12.3 crashes a drone, which is what random body rates do to a quadrotor. After 205 seconds of training, 34.6 gates in a best episode and 0.2 crashes a drone. The teacher has plainly learned to race its easy loops, and it learned the way PPO learned the cartpole, by spending cheap steps without counting them.\n", "\n", "What it has not done is learn to see. Everything it knows arrived as numbers no flying drone will ever be handed. A camera pilot has to get the same knowledge out of a small image, and a world model is well shaped for that: train its latent to predict the privileged numbers from the image, so its summary of a flight must contain where the drone and the gate are, and let the policy fly on that summary. [Tutorial 14](14-camera-only-drone-racing.ipynb) does that, from scratch and with no teacher at all, on SkyDreamer's race.\n" ] } ], "metadata": { "accelerator": "GPU", "colab": { "gpuType": "H100", "provenance": [] }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.16" } }, "nbformat": 4, "nbformat_minor": 0 }