{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": "# 6.4210/2 \u2014 ps7 setup\n\nOn a laptop, these Colab-specific cells detect the local runtime and skip themselves. On Colab, run them once per session: the Drive mount, the fetch (a no-op once the code is in your Drive), and `setup()`. See the course README for the optional one-time private-repo setup (Colab secrets `GH_TOKEN` and `PRIVATE_REPO`)." }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": "try:\n import google.colab\n IN_COLAB = True\nexcept ImportError:\n IN_COLAB = False\nif not IN_COLAB:\n print(\"Local runtime: skipping the Colab-only cells below.\")\n" }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": "if IN_COLAB:\n # Mount Google Drive. This is where we will/have clone(d) your course repo to.\n from google.colab import drive\n\n drive.mount(\"/content/drive\")\n" }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": "if IN_COLAB:\n # Fetch the course code into Drive and put it on the import path. With\n # the Colab secrets GH_TOKEN and PRIVATE_REPO set (see the course\n # README), this uses your private homework repo; without them, a plain\n # read-only clone of the public handouts repo.\n import pathlib\n import subprocess\n import sys\n\n from google.colab import userdata\n\n root = pathlib.Path(\"/content/drive/MyDrive/6.4210/repo\")\n public = \"https://github.com/tlpmit/6.4210-2-fall26-handouts.git\"\n try:\n private = userdata.get(\"PRIVATE_REPO\")\n origin = f\"https://github.com/{private}.git\"\n # The token goes in a credential file outside the repo, so it never\n # ends up in .git/config or in what git prints when a fetch fails.\n creds = pathlib.Path(\"/root/.git-credentials\")\n creds.write_text(\n f\"https://x-access-token:{userdata.get('GH_TOKEN')}@github.com\\n\"\n )\n creds.chmod(0o600)\n subprocess.run(\n [\"git\", \"config\", \"--global\",\n \"credential.helper\", f\"store --file={creds}\"],\n check=True,\n )\n print(f\"Colab secrets found: your repo is {private}\")\n except Exception:\n origin = None\n print(\"no Colab secrets configured: \"\n \"using the public repo, read-only\")\n if root.exists():\n print(f\"course code already in Drive at {root}\")\n else:\n print(f\"cloning {'your private repo' if origin else 'the public repo'}: \"\n f\"{origin or public}\")\n root.parent.mkdir(parents=True, exist_ok=True)\n subprocess.run([\"git\", \"clone\", origin or public, str(root)], check=True)\n if origin:\n subprocess.run(\n [\"git\", \"-C\", str(root), \"remote\", \"add\", \"upstream\", public],\n check=True,\n )\n if str(root) not in sys.path:\n sys.path.insert(0, str(root))\n print(\"done: the course code is on the import path\")\n" }, { "cell_type": "markdown", "metadata": {}, "source": "**On Colab, first time only**: stop here, and reopen this notebook from your Drive \u2014 on [drive.google.com](https://drive.google.com) open `My Drive \u2192 6.4210 \u2192 repo \u2192 ps7` and double-click `model_based_push.ipynb`; it opens in Colab. Work in that copy from now on: its edits are saved to Drive and can be committed with git. After opening the Drive copy, start again at the first cell and rerun the mount and fetch cells before continuing." }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": "if IN_COLAB:\n from utils import colab_bootstrap\n colab_bootstrap.setup(\"ps7\")\n import os\n os.chdir(colab_bootstrap.ROOT / \"ps7\")\n" }, { "cell_type": "markdown", "id": "21d4fa14", "metadata": {}, "source": [ "# PS7: Learning what a push does, then planning\n", "\n", "This notebook goes with the PDF problem \u201cLearning what a push does, then planning.\u201d We complete six functions in `model_based_math.py`: `fit_displacement`, `calibrated_length`, `dynamics_loss`, `rollout_cost`, `cost_weights`, and `update_nominal`. The notebook also uses PS6's push simulator and demonstrations from the sibling `ps6/` folder, so keep both folders in the same course checkout.\n", "\n", "**Setup.** Run the setup cell below first. In VS Code, select the course `.venv` Python kernel. If packages are missing, run `%pip install -r requirements.txt` from `ps7`.\n", "\n", "**Running cells.** Run cells one at a time rather than with Run All, since some cells fit a model or run the simulator as soon as they execute. After editing a `.py` file, restart the kernel and run again from the top so the edits are imported.\n", "\n", "**Saved results.** Fitting and controller runs save their outputs in `results/` and reuse them while this problem's code (the `model_*` files in `ps7/` and the PS6 Python files), settings and data are unchanged. A run that stopped with an error, for example on a function that is still a stub, simply runs again. If a completed run is out of date because the code or data changed, the call stops with an error instead of reusing it: rerun that cell with `retrain=True` (for example `fit_models(retrain=True)`), which moves the old files to `results/_previous_runs/`. The supplied files in `data/` are never overwritten." ] }, { "cell_type": "code", "execution_count": null, "id": "93c57a9e", "metadata": {}, "outputs": [], "source": [ "from pathlib import Path\n", "import os, sys\n", "# Works with a kernel started in ps7 or in the repository root.\n", "HERE = next((p if (p / \"model_homework.py\").exists() else p / \"ps7\"\n", " for p in [Path.cwd(), *Path.cwd().parents]\n", " if (p / \"model_homework.py\").exists() or (p / \"ps7/model_homework.py\").exists()), None)\n", "if HERE is None:\n", " raise FileNotFoundError(\"Open this notebook from the course checkout (including ps7/data).\")\n", "sys.path.insert(0, str(HERE))\n", "sys.path.append(str(HERE.parent / \"ps6\"))\n", "os.environ.setdefault(\"OMP_NUM_THREADS\", \"1\")\n", "os.environ.setdefault(\"OPENBLAS_NUM_THREADS\", \"1\")\n", "import numpy as np\n", "import torch\n", "import matplotlib.pyplot as plt\n", "from IPython.display import display\n", "torch.set_num_threads(1)\n", "from model_homework import DATA, RESULTS, MODEL_RESULTS, read\n", "print(\"Code/data:\", HERE, \"\\nYour outputs:\", RESULTS)\n" ] }, { "cell_type": "markdown", "id": "e6b4f427", "metadata": {}, "source": [ "## Demonstrations from PS6\n", "Both models, the affine correction and the neural transition network, learn from the demonstrations PS6 used for behavior cloning, `ps6/data/cautious_push/train.npz` and `validation.npz`. The 64 training episodes supply 886 transitions. For the transition network, the first eight validation episodes (119 transitions) choose the checkpoint, and the last eight (102 transitions, the `diagnostic` split) are held out to measure prediction error. The affine fit uses the 886 training transitions and is checked on all 221 validation transitions.\n", "\n", "`model_reference.py` turns each recorded observation into the eight configuration coordinates the transition network uses: block position, yaw, finger position, block height, roll and pitch. Each duration covers the whole command: approach, push, retract and settling. The physics, reward, [0, 0.125] m command range and 32-command cutoff are the same as in PS6.\n", "\n", "The demonstrations all use commands of 6.4\u20138.4 cm. The planner can try commands outside that range, where low prediction error on the demonstrations does not mean the model is accurate there." ] }, { "cell_type": "code", "execution_count": null, "id": "69048a45", "metadata": {}, "outputs": [], "source": [ "from model_homework import model_data\n", "transitions, splits = model_data()\n", "print(\"Transition-network splits (transitions):\", {name: len(rows) for name, rows in splits.items()})\n", "print(\"Affine-fit transitions:\", {s: sum(r[\"split\"] == s for r in transitions) for s in [\"train\", \"validation\"]})" ] }, { "cell_type": "markdown", "id": "570734a9", "metadata": {}, "source": [ "## Checking the array operations\n", "The function docstrings specify shapes, element types, and units. They explain the least-squares solver, clipping, broadcasting, `keepdims=True`, and the sample and time axes in MPPI (model predictive path integral control). All functions use NumPy except `dynamics_loss`, which uses ordinary arithmetic and `.mean()` on PyTorch arrays. The supplied code handles input checks and fixed targets, so each function only needs its mathematical formula.\n", "\n", "The next cell checks all six functions without a simulator, on both single and batched face sequences.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "28b974eb", "metadata": {}, "outputs": [], "source": [ "from model_based_math import (fit_displacement, calibrated_length, dynamics_loss,\n", " rollout_cost, cost_weights, update_nominal)\n", "x = np.array([.03, .06, .10])\n", "assert np.allclose(fit_displacement(x, .9*x-.02), [.9, -.02])\n", "assert np.isclose(calibrated_length(.07, .9, -.02), .1)\n", "assert np.allclose(calibrated_length(np.array([-.1, .3]), .9, -.02), [0., .125])\n", "# The same normalization constants apply to every row, and the loss averages all four errors.\n", "prediction = torch.zeros((2, 2), requires_grad=True)\n", "target = torch.tensor([[1., 4.], [3., 0.]])\n", "loss = dynamics_loss(prediction, target, torch.tensor([1., 2.]), torch.tensor([1., 2.]))\n", "assert torch.allclose(loss, torch.tensor(1.5))\n", "loss.backward()\n", "assert torch.allclose(prediction.grad, torch.tensor([[0., -.5], [-1., .5]]))\n", "# One cost per rollout, for a success, an ordinary continuation and a failure.\n", "c = rollout_cost(np.array([0., .15, .3]), np.array([True, False, False]),\n", " np.array([5., 10., 2.]), np.array([0., 1., 0.]),\n", " np.array([False, False, True]))\n", "assert c.shape == (3,) and np.allclose(c, [.1, 2.4, 4.04])\n", "# Each face sequence is normalized separately. Equal costs give equal weights,\n", "# and adding a constant to every cost leaves the weights unchanged.\n", "costs = np.array([[1., 2., 3.], [5., 5., 5.]])\n", "w = cost_weights(costs, .5)\n", "assert w.shape == costs.shape and np.allclose(w.sum(-1), 1)\n", "assert w[0, 0] > w[0, 1] > w[0, 2] and np.allclose(w[1], w[1, 0])\n", "assert np.allclose(w, cost_weights(costs + 1000, .5))\n", "assert np.allclose(cost_weights(costs[0], .5), w[0])\n", "# Different samples and times make an accidental axis swap visible.\n", "nominal = np.array([[1., 2.], [-1., 0.]])\n", "perturbations = np.arange(12, dtype=float).reshape(2, 3, 2) / 10\n", "updated = update_nominal(nominal, perturbations, w)\n", "expected = nominal + np.stack([sum(w[m, k]*perturbations[m, k] for k in range(3))\n", " for m in range(2)])\n", "assert updated.shape == nominal.shape and np.allclose(updated, expected)\n", "assert np.allclose(update_nominal(nominal[0], perturbations[0], w[0]), expected[0])\n", "print(\"All six functions passed the array checks\")" ] }, { "cell_type": "markdown", "id": "pdf-check-values-md", "metadata": {}, "source": [ "## Values for the PDF's worked checks\n", "The next cell runs your functions on the inputs of the worked checks in \u201cA fitted correction to the heuristic\u201d, \u201cLearning a transition model\u201d, \u201cPlanning for the same objective\u201d and \u201cCost-weighted MPPI updates\u201d, and prints the values to enter." ] }, { "cell_type": "code", "execution_count": null, "id": "pdf-check-values", "metadata": {}, "outputs": [], "source": [ "print(\"A fitted correction to the heuristic: calibrated_length with alpha = 0.8, beta = -0.01 m\")\n", "for e in [0.08, 0.30, 0.002]:\n", " print(f\" e = {e} m: command {float(calibrated_length(e, 0.8, -0.01)):.4f} m\")\n", "loss = dynamics_loss(torch.zeros((2, 2)), torch.tensor([[2., 8.], [6., 12.]]),\n", " torch.tensor([2., 4.]), torch.tensor([2., 4.]))\n", "print(f\"Learning a transition model: two-example dynamics_loss {float(loss):.3f}\")\n", "d0, d_H, seconds, invalid = 0.30, 0.06, 10.0, 1\n", "C = float(rollout_cost(np.array([d_H]), np.array([False]), np.array([seconds]),\n", " np.array([float(invalid)]), np.array([False]))[0])\n", "G = (d0 - d_H)/0.15 - 0.02*seconds - 0.2*invalid # PS6's rewards summed; no success or failure\n", "print(f\"Planning for the same objective: cost C {C:.3f}, predicted return {G:.3f}\")\n", "w = cost_weights(np.array([0., np.log(3)]), 1.0)\n", "z = update_nominal(np.zeros(2), np.array([[0.2, -0.4], [-0.6, 0.8]]), w)\n", "w, z = np.round(w, 3) + 0.0, np.round(z, 3) + 0.0\n", "print(f\"Cost-weighted MPPI updates: weights ({w[0]:.3f}, {w[1]:.3f}), \"\n", " f\"updated nominal ({z[0]:.3f}, {z[1]:.3f})\")" ] }, { "cell_type": "markdown", "id": "c925acc0", "metadata": {}, "source": [ "## Fitting both models\n", "`fit_models()` fits the affine correction and trains the transition network on the 886 training transitions. It does not run the simulator and takes a few seconds on a CPU. The network trains for 400 epochs, and we keep the checkpoint with the lowest prediction loss on the first eight validation episodes. Normalization statistics come from the training data only." ] }, { "cell_type": "code", "execution_count": null, "id": "f36faa7b", "metadata": {}, "outputs": [], "source": [ "# Fit both models (or load the saved fit).\n", "from model_homework import fit_models\n", "calibration, dynamics, fitting = fit_models()" ] }, { "cell_type": "code", "execution_count": null, "id": "0e845b69", "metadata": {}, "outputs": [], "source": [ "# The fitted line and its residuals, for \"A fitted correction to the heuristic\".\n", "from model_based_push import FACE_ANGLES, local, load_dynamics, diagnostics\n", "folder = MODEL_RESULTS\n", "if not (folder / \"calibration.json\").exists():\n", " print(\"Run fit_models() in the cell above first; this cell plots your fit.\")\n", "else:\n", " calibration = read(folder / \"calibration.json\")\n", " held_out = [r for r in transitions if r[\"split\"] == \"validation\" and r[\"valid\"]]\n", " x = np.array([r[\"length\"] for r in held_out])\n", " y = np.array([local(np.array(r[\"next_state\"][:2])-r[\"state\"][:2],\n", " r[\"state\"][2]+FACE_ANGLES[r[\"face\"]])[0] for r in held_out])\n", " fig, (ax, residual_ax) = plt.subplots(2, 1, figsize=(6, 7), layout=\"constrained\")\n", " ax.scatter(x, y, label=\"BC validation transitions\")\n", " grid = np.linspace(0., .125, 100)\n", " ax.plot(grid, grid-.02, label=\"nominal\")\n", " ax.plot(grid, calibration[\"alpha\"]*grid+calibration[\"beta\"], label=\"fitted affine model\")\n", " ax.set(xlabel=\"Command length (m)\", ylabel=\"Along-push displacement (m)\"); ax.legend()\n", " residual_ax.scatter(x, y-(calibration[\"alpha\"]*x+calibration[\"beta\"]))\n", " residual_ax.axhline(0, color=\"black\", linewidth=1)\n", " residual_ax.set(xlabel=\"Command length (m)\", ylabel=\"Measured minus fitted displacement (m)\")\n", " RESULTS.mkdir(parents=True, exist_ok=True)\n", " fig.savefig(RESULTS / \"fitted_displacement.png\", dpi=160)\n", " print(f\"alpha = {calibration['alpha']:.5f}, beta = {calibration['beta']:.5f} m\")\n", " print(f\"Validation RMSE (mm): nominal {1000*np.sqrt(np.mean((x-.02-y)**2)):.2f}, \"\n", " f\"fitted {1000*np.sqrt(np.mean((calibration['alpha']*x+calibration['beta']-y)**2)):.2f}\")\n", " print(\"Plot saved to\", RESULTS / \"fitted_displacement.png\")" ] }, { "cell_type": "markdown", "id": "transition-model-error", "metadata": {}, "source": [ "## Prediction error of the transition model\n", "The PDF's \u201cLearning a transition model\u201d answers use the supplied model, `data/cautious_model_based/dynamics.pt`, on the 102 diagnostic transitions. Once `fit_models()` has run, the cell also prints your own fit on the same transitions, for comparison." ] }, { "cell_type": "code", "execution_count": null, "id": "transition-model-diagnostics", "metadata": {}, "outputs": [], "source": [ "def in_mm(d):\n", " return dict(position_rmse_mm=1000*d[\"position_rmse_m\"], duration_rmse_s=d[\"time_rmse_s\"],\n", " three_step_position_rmse_mm=1000*d[\"three_step_position_rmse_m\"])\n", "\n", "supplied = diagnostics(load_dynamics(DATA / \"cautious_model_based/dynamics.pt\"), splits[\"diagnostic\"])\n", "print(\"Supplied model, diagnostic split (use these in the PDF):\")\n", "display(in_mm(supplied))\n", "if (folder / \"dynamics.pt\").exists():\n", " print(\"Your fitted model, same split (for comparison):\")\n", " display(in_mm(diagnostics(load_dynamics(folder / \"dynamics.pt\"), splits[\"diagnostic\"])))" ] }, { "cell_type": "markdown", "id": "f15b227a", "metadata": {}, "source": [ "## Running the two controllers on four goals\n", "We run the calibrated heuristic and learned-model MPPI on the first four PS6 test goals (9660000\u20139660003), with PS6's 32-command cutoff. MPPI plans with the learned model and executes each chosen command in Drake. Its settings are fixed: a horizon of two pushes, all 16 face sequences, 24 samples per sequence and three cost-weighted updates. The four goals check that both controllers behave as expected, and we do not tune anything on them. Each of the next two cells takes about a minute." ] }, { "cell_type": "code", "execution_count": null, "id": "5ced8521", "metadata": {}, "outputs": [], "source": [ "# Evaluate the calibrated heuristic (or load saved scores).\n", "from model_homework import evaluate_controller\n", "calibrated_scores, calibrated_env = evaluate_controller(\"calibrated\")" ] }, { "cell_type": "code", "execution_count": null, "id": "372300ab", "metadata": {}, "outputs": [], "source": [ "# Evaluate learned-model MPPI (or load saved scores).\n", "mppi_scores, mppi_env = evaluate_controller(\"mppi\")" ] }, { "cell_type": "code", "execution_count": null, "id": "dae4dd43", "metadata": {}, "outputs": [], "source": [ "from model_homework import score_table\n", "local_groups = {}\n", "for name in [\"calibrated\", \"mppi\"]:\n", " path = folder / f\"{name}_scores.json\"\n", " if path.exists():\n", " local_groups[name] = [read(path)[\"rows\"]]\n", "print(\"Your four-goal results (execution_seconds is total simulated time, travel_m is finger travel):\")\n", "display(score_table(local_groups).drop(columns=\"training_seed\", errors=\"ignore\"))\n", "fit_file = folder / \"fit.json\"\n", "if fit_file.exists():\n", " print(\"Your transition-network fitting time (wall-clock s):\", read(fit_file)[\"fitting\"][\"seconds\"])\n", "mppi_file = folder / \"mppi_scores.json\"\n", "if mppi_file.exists():\n", " print(\"Your mean MPPI planning time per episode (wall-clock s):\", np.mean([r[\"planning_seconds\"] for r in read(mppi_file)[\"rows\"]]))" ] }, { "cell_type": "markdown", "id": "3ff642a1", "metadata": {}, "source": [ "## The supplied 64-goal comparison\n", "The next cell analyzes a supplied run on all 64 PS6 test goals: BC and PS6's ten final scalar-residual PPO policies, plus the calibrated heuristic and MPPI with the supplied dynamics model. `paired_goal_interval` takes the difference between two methods on each goal and bootstraps over goals (5000 resamples, seed 701), averaging the ten PPO policies within each goal. The intervals describe uncertainty over goals for these particular trained controllers. The 6.4212-only section at the end looks at how much the result changes with a new fit. PPO also collected its own online experience, so the methods did not use the same amount of data.\n", "\n", "The last output lists the supplied training and planning costs. Fitting and planning times are wall-clock times on the machine that produced the reference; execution time is simulated time." ] }, { "cell_type": "code", "execution_count": null, "id": "fdb59a2b", "metadata": {}, "outputs": [], "source": [ "from model_homework import model_scores, reward_violins, paired_goal_interval\n", "reference = model_scores()\n", "display(score_table(reference))\n", "reward_violins(reference)\n", "for other in [\"bc\", \"calibrated\", \"residual_ppo\"]:\n", " interval = paired_goal_interval(reference, \"mppi\", other)\n", " print(f\"MPPI minus {other}: mean {interval['difference']:.3f}, 95% interval {np.round(interval['ci95'], 3)}\")\n", "reference_means = score_table(reference).groupby(\"method\", sort=False).mean(numeric_only=True)\n", "print(\"Mean reward by method (PPO averaged over its ten policies):\")\n", "display(reference_means[\"reward\"])\n", "print(\"Signed MPPI-minus-calibrated differences:\")\n", "display((reference_means.loc[\"mppi\"] - reference_means.loc[\"calibrated\"])[[\"pushes\", \"execution_seconds\", \"travel_m\", \"free_seconds\"]])\n", "\n", "costs = read(DATA / \"cautious_model_based\" / \"comparison.json\")[\"cost_comparison\"]\n", "print(\"Supplied training and planning costs (wall-clock seconds, except the command counts):\")\n", "display({key: costs[key] for key in [\n", " \"shared_demonstrations_and_bc_seconds\", \"dynamics_fit_seconds\", \"dynamics_extra_training_commands\",\n", " \"ppo_training_commands\", \"ppo_training_and_monitoring_seconds\",\n", " \"mppi_planning_seconds_per_episode\", \"mppi_planning_seconds_per_command\"]})" ] }, { "cell_type": "markdown", "id": "bc4e7c4e", "metadata": {}, "source": [ "## Checking one prediction\n", "The PDF's \u201cPrediction, control, and cost\u201d part uses the supplied model and the first transition of the supplied MPPI run (goal 9660000, push 1). Prediction error is often larger on the states a controller actually visits than on the demonstrations, and the first output shows this for the supplied runs. The \u201cLearning a transition model\u201d answers still come from the diagnostic split above." ] }, { "cell_type": "code", "execution_count": null, "id": "869c0ef6", "metadata": {}, "outputs": [], "source": [ "report = read(DATA / \"cautious_model_based/comparison.json\")\n", "print(\"One-step error on the states the controllers visited (not the diagnostic-split answers):\")\n", "display(report[\"deployment_prediction_error\"])\n", "trace = read(DATA / \"cautious_model_based/evaluation/mppi.json\")[\"transitions\"]\n", "first = trace[0]\n", "model = load_dynamics(DATA / \"cautious_model_based/dynamics.pt\")\n", "predicted_state, predicted_time = model.predict(np.array([first[\"state\"]]),\n", " np.array([first[\"face\"]]), np.array([first[\"length\"]]))\n", "print(\"First transition of goal\", first[\"episode\"], \"with the supplied model:\")\n", "print(\" predicted/actual position:\", predicted_state[0, :2], first[\"next_state\"][:2])\n", "print(\" predicted/actual total seconds:\", predicted_time[0], first[\"seconds\"])\n", "print(\" position prediction error (mm):\", 1000*np.linalg.norm(predicted_state[0, :2] - np.asarray(first[\"next_state\"][:2])))\n", "print(\" signed duration error, predicted minus actual (s):\", predicted_time[0] - first[\"seconds\"])" ] }, { "cell_type": "markdown", "id": "watch-one-episode", "metadata": {}, "source": [ "## Watching one MPPI episode\n", "The next cell runs MPPI with your fitted models on goal 9660000 in Drake and saves a Meshcat replay in `results/replays/`." ] }, { "cell_type": "code", "execution_count": null, "id": "92aed5af", "metadata": {}, "outputs": [], "source": [ "# Run one MPPI episode in Drake, then replay its recorded states in Meshcat.\n", "from watch_push import watch_multi_push\n", "replay_scores, replay_env = evaluate_controller(\"mppi\", folder=MODEL_RESULTS,\n", " seeds=[9660000], record=True)\n", "replay = watch_multi_push(replay_env.task, output_dir=RESULTS / \"replays\")" ] }, { "cell_type": "markdown", "id": "dynamics-fit-replication", "metadata": {}, "source": [ "## Replication over dynamics fits (6.4212 only)\n", "6.4210 students can skip this section. The comparison above uses one trained dynamics model. The supplied replication adds ten more (seeds 300\u2013309), trained the same way on the same data from different random initializations and minibatch orders. Each one drives MPPI on the same 64 goals, with the same planner seeds.\n", "\n", "Each whole fit contributes one sample, its mean reward gain over BC on the 64 goals. The bootstrap resamples the ten fits (5000 resamples, seed 816). Resampling the 640 episodes instead would treat them as 640 independently trained models. The cell also runs a two-sided t-test (\u03b1 = 0.05) on the ten gains over BC, and on the ten gains over the calibrated heuristic. All ten fits share the demonstrations, the BC policy and the goals, so these numbers measure training randomness only." ] }, { "cell_type": "code", "execution_count": null, "id": "dynamics-fit-analysis", "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "from scipy import stats\n", "from IPython.display import Image\n", "cohort_folder = DATA / \"cautious_model_based/replication\"\n", "replication = read(cohort_folder / \"analysis.json\")\n", "replication_plan = read(cohort_folder / \"plan.json\")\n", "assert replication_plan[\"seeds\"] == list(range(300, 310))\n", "fit_rewards = np.array([\n", " np.mean([r[\"reward\"] for r in read(cohort_folder / f\"seed{seed}/evaluation.json\")[\"rows\"]])\n", " for seed in replication_plan[\"seeds\"]\n", "])\n", "for name, baseline in [(\"BC\", \"bc\"), (\"the calibrated heuristic\", \"calibrated\")]:\n", " differences = fit_rewards - np.mean([r[\"reward\"] for r in reference[baseline][0]])\n", " rng = np.random.default_rng(816)\n", " boot = differences[rng.integers(len(differences), size=(5000, len(differences)))].mean(axis=1)\n", " test = stats.ttest_1samp(differences, 0.0)\n", " print(f\"MPPI minus {name}, over {len(differences)} independent dynamics fits:\")\n", " print(f\" mean reward gain {differences.mean():.3f}, training-fit SD {differences.std(ddof=1):.4f}, \"\n", " f\"whole-fit bootstrap 95% interval {np.round(np.quantile(boot, [0.025, 0.975]), 3)}\")\n", " print(f\" two-sided t-test: t({len(differences) - 1}) = {test.statistic:.2f}, p = {test.pvalue:.3g}\")\n", "display(pd.DataFrame(replication[\"per_seed\"]))\n", "costs = replication[\"costs\"]\n", "print(f\"Fitting all ten models took {costs['neural_fitting_seconds']:.2f} s. The whole replication took \"\n", " f\"{costs['calendar_seconds']/60:.2f} min, mostly to run {costs['actual_evaluation_commands']} \"\n", " f\"evaluation commands, and used {costs['training_commands']} new training commands.\")\n", "display(Image(filename=str(cohort_folder / \"replication.png\")))" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" } }, "nbformat": 4, "nbformat_minor": 5 }