{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": "# 6.4210/2 \u2014 ps6 setup\n\nOn a laptop, these Colab-specific cells detect the local runtime and skip themselves. In Colab, choose the normal Drive checkout, or set `USE_STUDENT_ZIP=True` to upload the supplied student ZIP without mounting Drive. Run setup once per runtime. ZIP mode uses temporary runtime storage: download your edits/results before deleting the runtime." }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": "try:\n import google.colab\n IN_COLAB = True\nexcept ImportError:\n IN_COLAB = False\nUSE_STUDENT_ZIP = False # True: upload the exact supplied student ZIP in Colab.\nif not IN_COLAB:\n print(\"Local runtime: skipping Colab-only setup; continue to the lesson setup cell.\")\n" }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": "if IN_COLAB and not USE_STUDENT_ZIP:\n # Mount Google Drive. This is where we will/have clone(d) your course repo to.\n from google.colab import drive\n\n drive.mount(\"/content/drive\")\n" }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": "if IN_COLAB and not USE_STUDENT_ZIP:\n # Fetch the course code into Drive and put it on the import path. With\n # the Colab secrets GH_TOKEN and PRIVATE_REPO set (see the course\n # README), this uses your private homework repo; without them, a plain\n # read-only clone of the public handouts repo.\n import pathlib\n import subprocess\n import sys\n\n from google.colab import userdata\n\n root = pathlib.Path(\"/content/drive/MyDrive/6.4210/repo\")\n public = \"https://github.com/tlpmit/6.4210-2-fall26-handouts.git\"\n try:\n private = userdata.get(\"PRIVATE_REPO\")\n origin = f\"https://github.com/{private}.git\"\n # The token goes in a credential file outside the repo, so it never\n # ends up in .git/config or in what git prints when a fetch fails.\n creds = pathlib.Path(\"/root/.git-credentials\")\n creds.write_text(\n f\"https://x-access-token:{userdata.get('GH_TOKEN')}@github.com\\n\"\n )\n creds.chmod(0o600)\n subprocess.run(\n [\"git\", \"config\", \"--global\",\n \"credential.helper\", f\"store --file={creds}\"],\n check=True,\n )\n print(f\"Colab secrets found: your repo is {private}\")\n except Exception:\n origin = None\n print(\"no Colab secrets configured: \"\n \"using the public repo, read-only\")\n if root.exists():\n print(f\"course code already in Drive at {root}\")\n else:\n print(f\"cloning {'your private repo' if origin else 'the public repo'}: \"\n f\"{origin or public}\")\n root.parent.mkdir(parents=True, exist_ok=True)\n subprocess.run([\"git\", \"clone\", origin or public, str(root)], check=True)\n if origin:\n subprocess.run(\n [\"git\", \"-C\", str(root), \"remote\", \"add\", \"upstream\", public],\n check=True,\n )\n if str(root) not in sys.path:\n sys.path.insert(0, str(root))\n print(\"done: the course code is on the import path\")\n" }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": "if IN_COLAB and USE_STUDENT_ZIP:\n from google.colab import files\n from pathlib import Path\n import tempfile, zipfile, sys\n uploaded = files.upload()\n if len(uploaded) != 1:\n raise ValueError(\"Upload exactly one student ZIP\")\n archive = Path(next(iter(uploaded)))\n workspace = Path(tempfile.mkdtemp(prefix=\"ps6-colab-\"))\n with zipfile.ZipFile(archive) as bundle:\n if any(Path(n).is_absolute() or \"..\" in Path(n).parts for n in bundle.namelist()):\n raise ValueError(\"Unsafe archive path\")\n bundle.extractall(workspace)\n root = workspace / \"ps6_student\"\n if not (root / \"ps6/homework.py\").is_file():\n raise ValueError(\"Use the supplied ps6 student ZIP\")\n sys.path.insert(0, str(root))\n print(\"Uploaded package:\", root)\n" }, { "cell_type": "markdown", "metadata": {}, "source": "In Drive mode, reopen this notebook from `My Drive \u2192 6.4210 \u2192 repo \u2192 ps6` as usual. In ZIP mode, continue here; code and data come from the uploaded archive, without cloning GitHub or granting Drive access." }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": "if IN_COLAB:\n from utils import colab_bootstrap\n colab_bootstrap.setup(\"ps6\")\n import os\n os.chdir(colab_bootstrap.ROOT / \"ps6\")\n" }, { "cell_type": "markdown", "id": "d1afbfbd", "metadata": {}, "source": [ "# HW6: Improving cautious BC with scalar residual PPO\n", "\n", "Complete `joint_log_prob`, `bc_loss`, `gae_advantages`, `ppo_actor_loss`, and `critic_loss` in `push_math.py`. The required local work is a quick BC fit and **one PPO update on a supplied real on-policy batch**. Analyze the packaged full training runs; do not repeat 8,192 pushes or the ten-run reference cohort.\n", "\n", "Run cells individually. The setup cell turns on autoreload, so edits to `push_math.py` take effect without restarting the kernel; rerun a check after each edit. The main path below needs NumPy/PyTorch/SciPy, not Drake or a GPU. Optional replay uses Drake. Results go to `results/cautious_push`; packaged data remain unchanged. Fitting/update calls reuse verified cached results; `retrain=True` archives the old result and restarts the specified exercise from its original inputs." ] }, { "cell_type": "code", "execution_count": null, "id": "89588a2c", "metadata": {}, "outputs": [], "source": [ "# Re-read edited .py files before each cell runs (%load_ext autoreload; %autoreload 2).\n", "from IPython import get_ipython\n", "if get_ipython() is not None:\n", " get_ipython().run_line_magic(\"load_ext\", \"autoreload\")\n", " get_ipython().run_line_magic(\"autoreload\", \"2\")\n", "from pathlib import Path\n", "import os, sys\n", "HERE = next((p if (p / \"residual_push.py\").exists() else p / \"ps6\"\n", " for p in [Path.cwd(), *Path.cwd().parents]\n", " if (p / \"residual_push.py\").exists() or (p / \"ps6/residual_push.py\").exists()), None)\n", "if HERE is None:\n", " raise FileNotFoundError(\"Open this notebook from the course checkout.\")\n", "sys.path.insert(0, str(HERE))\n", "os.environ.setdefault(\"OMP_NUM_THREADS\", \"1\")\n", "os.environ.setdefault(\"OPENBLAS_NUM_THREADS\", \"1\")\n", "import numpy as np\n", "import torch\n", "from torch import nn\n", "import matplotlib.pyplot as plt\n", "from IPython.display import display\n", "from residual_push import DATA, RESULTS\n", "from residual_analysis import verify_data\n", "print(\"Code:\", HERE, \"\\nData:\", DATA, \"\\nYour outputs:\", RESULTS)\n", "print(verify_data())\n", "torch.set_num_threads(1)" ] }, { "cell_type": "markdown", "id": "ec42d18c", "metadata": {}, "source": [ "## Task and cautious teacher\n", "\n", "The 0.43125 kg block must reach a goal within 2.5 cm. Actions choose a body-fixed face and a stick travel in [0, 0.125] m. The supplied planner approaches, pushes, retracts and waits for settling; policies use the 13 resting-configuration inputs listed in Problem 7 of the handout (see `LearningPush.observation` in `push_env.py`).\n", "\n", "The teacher chooses the largest body-axis goal error and caps **intended object displacement at 4 cm**. An affine calibration compensates for travel lost before the block moves. Thus a capped push commands about 8.35 cm of stick travel. The teacher succeeds but uses many short pushes. BC accurately imitates that cautious behavior; task reward instead values efficient completion.\n", "\n", "Reward is progress/0.15 + success \u2212 0.02 \u00d7 total simulated seconds, with the stated invalid/fall penalties. **32 commands is a computational rollout cutoff, not a terminal deadline.** The critic supplies continuation only for training at cutoffs. Evaluation reports actual observed reward/success/time with no added value estimate." ] }, { "cell_type": "markdown", "id": "cautious-bc-replay-intro", "metadata": {}, "source": [ "## Watch the initial BC policy in Drake (optional)\n", "\n", "Before implementing the losses, you can watch the supplied BC actor on one fixed goal. Set `RUN_DRAKE_REPLAYS=True` to run a fresh Drake episode and embed its Meshcat recording. Observe the many approach\u2013push\u2013retract cycles. Keep this recording: the last section compares it with a learned residual policy on the **same** goal and initial state. This optional simulation is separate from the short required numerical exercise; it does not call your unfinished mathematical functions." ] }, { "cell_type": "code", "id": "cautious-initial-bc-replay", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "RUN_DRAKE_REPLAYS = False # True enables both the initial and learned-policy views.\n", "REPLAY_GOAL = 9460000 # Fixed illustrative goal, not selected for a favorable outcome.\n", "initial_bc_replay = None\n", "if RUN_DRAKE_REPLAYS:\n", " from residual_push import replay\n", " initial_bc_replay = replay(\"bc\", seed=REPLAY_GOAL)" ] }, { "cell_type": "markdown", "id": "7af6ec77", "metadata": {}, "source": [ "## The residual policy\n", "\n", "For the sampled face $f$, let $e$ be the corresponding goal-error component in meters and\n", "$g=\\mathrm{clip}((|e|-0.07)/0.05,0,1)$. The latent command is\n", "\n", "$$z\\sim\\mathcal N\\Bigl(\\underbrace{\\mu_{BC}(s,f)}_{\\text{original BC mean}}+\\underbrace{\\theta g}_{\\text{PPO correction}},\\;\\bigl(\\underbrace{0.2+0.3g}_{\\text{exploration std. dev.}}\\bigr)^2\\Bigr)$$\n", "\n", "and the stick command is $0.125\\,\\mathrm{sigmoid}(z)$ m. Only the signed scalar $\\theta$ and the critic learn. BC weights, face probabilities, and exploration schedule remain fixed. The initial correction is zero; the actor may learn either sign. Pushes near the goal have zero mean correction.\n", "\n", "`joint_log_prob` returns $\\log p(f, z\\mid s)$ for each recorded action. Complete it (Problem 7.2), then run the check below.\n", "\n", "Each check cell tests one part of `push_math.py` on its own example values, not the ones in Problem 7, and prints a \"passed\" line if the code is correct. Work the Problem 7 calculations out by hand. `gae_advantages` uses ordinary NumPy arrays; the four loss/probability functions use PyTorch arrays so the training code can differentiate them. Old probabilities and targets are held fixed by the caller." ] }, { "cell_type": "code", "execution_count": null, "id": "b6bb7dcb", "metadata": {}, "outputs": [], "source": [ "from push_math import joint_log_prob\n", "cat = torch.distributions.Categorical(probs=torch.tensor([[.2, .8]]))\n", "normal = torch.distributions.Normal(torch.tensor([0.]), torch.tensor([1.]))\n", "z = torch.tensor([.3]);face = torch.tensor([0])\n", "assert torch.allclose(joint_log_prob(cat, normal, face, z), cat.log_prob(face)+normal.log_prob(z))\n", "print(\"joint_log_prob check passed.\")" ] }, { "cell_type": "markdown", "id": "9339e588", "metadata": {}, "source": [ "## Behavior cloning\n", "\n", "Complete `bc_loss` (Problem 7.3) and run its check, then fit BC once in the cell after." ] }, { "cell_type": "code", "execution_count": null, "id": "4939d8c2", "metadata": {}, "outputs": [], "source": [ "from push_math import bc_loss\n", "logits = torch.tensor([[np.log(.6), np.log(.4)], [np.log(.25), np.log(.75)]], dtype=torch.float32)\n", "assert abs(float(bc_loss(logits, torch.tensor([.03, .09]), torch.tensor([0, 1]), torch.tensor([.05, .08]))) - .4242538481) < 1e-6\n", "print(\"bc_loss check passed.\")" ] }, { "cell_type": "markdown", "id": "9fdba356", "metadata": {}, "source": [ "## Fit BC once\n", "\n", "The package contains 64 training / 16 validation episodes with 886 / 221 executed labels. Whole episodes are separated; every recorded demonstration episode finished successfully. Fit 400 inexpensive supervised epochs using face cross entropy plus squared physical-distance error normalized by 0.10 m. Report validation face accuracy and distance MAE separately.\n", "\n", "The full RL reference runs all use **one fixed supplied BC actor**. Your short supervised fit tests the BC code; it is not silently substituted into the reference study." ] }, { "cell_type": "code", "execution_count": null, "id": "4ef69b28", "metadata": {}, "outputs": [], "source": [ "from residual_push import clone_cautious\n", "cloning = clone_cautious()\n", "display(cloning[\"validation\"])\n", "print(\"Fit seconds:\", cloning[\"fit_seconds\"])" ] }, { "cell_type": "markdown", "id": "7f500ea1", "metadata": {}, "source": [ "## Generalized advantage estimation\n", "\n", "The reference learner uses GAE with gamma=1, lambda=0.95. For one episode,\n", "$\\delta_t=r_t+V_{old}(s_{t+1})-V_{old}(s_t)$ and\n", "$A_t=\\delta_t+0.95A_{t+1}$, with $A_T=0$. At a true terminal, the last next-value is zero; at a computational cutoff it is the old critic's final-state prediction.\n", "\n", "`gae_advantages` handles one episode and returns raw advantages. `batch_targets` in `residual_push.py` calls it once per episode, so GAE never carries into another episode, and then standardizes the whole batch once. Critic targets remain unstandardized Monte Carlo returns with the cutoff continuation.\n", "\n", "Complete `gae_advantages` (Problem 7.4) and run its check." ] }, { "cell_type": "code", "execution_count": null, "id": "a159fbe4", "metadata": {}, "outputs": [], "source": [ "from push_math import gae_advantages\n", "assert np.allclose(gae_advantages([1., 0., 2.], [.5, .3, .6], [.3, .6, 0.]), [2.3485, 1.63, 1.4])\n", "assert np.allclose(gae_advantages([1., 0., 2.], [.5, .3, .6], [.3, .6, .4]), [2.7095, 2.01, 1.8])\n", "print(\"gae_advantages check passed.\")" ] }, { "cell_type": "markdown", "id": "9648d819", "metadata": {}, "source": [ "## The clipped PPO update\n", "\n", "Complete `ppo_actor_loss` and `critic_loss` (Problem 7.5) and run their check. `update_from_batch` in `residual_push.py` passes `ppo_actor_loss` the standardized advantages from `batch_targets`, and adds 1/4 times `critic_loss` to it." ] }, { "cell_type": "code", "execution_count": null, "id": "98b84967", "metadata": {}, "outputs": [], "source": [ "from push_math import ppo_actor_loss, critic_loss\n", "ratios = torch.tensor([1.3, .6, 1.0, .85])\n", "logs = ratios.log().requires_grad_()\n", "advantages = torch.tensor([2., -1., 1., -3.])\n", "loss = ppo_actor_loss(logs, torch.zeros(4), advantages, .2)\n", "assert torch.allclose(loss, torch.tensor(-.0125))\n", "loss.backward()\n", "assert torch.isfinite(logs.grad).all()\n", "assert torch.allclose(critic_loss(torch.tensor([2., 0., 1.]), torch.tensor([1., 2., 1.])), torch.tensor(5 / 3))\n", "print(\"ppo_actor_loss and critic_loss checks passed.\")" ] }, { "cell_type": "markdown", "id": "dcd171bf", "metadata": {}, "source": [ "## One PPO iteration from a recorded on-policy batch\n", "\n", "`update_batch.npz` contains real complete episodes sampled by `update_start.pt`, a predetermined reference checkpoint. It includes observations, faces, latent actions, old log probabilities/values, rewards, next-values and episode boundaries. The batch was not chosen for a favorable update.\n", "\n", "The helper verifies that the checkpoint reproduces the recorded old log probabilities, computes episode-local GAE and critic targets, then performs four PPO optimization epochs. This is one collected-batch iteration, not full online training. Re-running with `retrain=True` starts again from the recorded checkpoint. Instructor batches collected by later policies cannot replace fresh on-policy interaction after arbitrary student changes." ] }, { "cell_type": "code", "execution_count": null, "id": "cf4243aa", "metadata": {}, "outputs": [], "source": [ "from residual_push import update_batch, update_from_batch\n", "batch, metadata = update_batch()\n", "print({k: metadata[k] for k in [\"transitions\", \"episodes\", \"theta\", \"collection_seconds\"]})\n", "short_update = update_from_batch()\n", "display(short_update)" ] }, { "cell_type": "markdown", "id": "954b7009", "metadata": {}, "source": [ "## Full reference learning and budget growth\n", "\n", "The three exploratory runs developed this method. Each starts from theta=0 and a fresh critic, uses 8,192 requested pushes, and saves a checkpoint at the first complete episode crossing 4,096. Adam, the critic, RNG streams and warmup count continue without resetting. The final episode can slightly exceed a requested budget. The 4,096 and 8,192 policies are evaluated on the same 64 fresh goals. Final greedy, final sampled and monitoring-selected policies are labeled separately; test goals select nothing." ] }, { "cell_type": "code", "execution_count": null, "id": "4a5e1db6", "metadata": {}, "outputs": [], "source": [ "from residual_analysis import reference, score_table, aggregate_table, paired_goal_interval, learning\n", "exploratory = reference(\"exploratory\")\n", "display(score_table(exploratory, \"final\"))\n", "display(aggregate_table(exploratory))\n", "learning(exploratory)\n", "display(paired_goal_interval(exploratory, variant=\"final\", metric=\"execution_seconds\", compare=\"midpoint\"))\n", "display(paired_goal_interval(exploratory, variant=\"sampled\", metric=\"reward\"))" ] }, { "cell_type": "markdown", "id": "22084b0d", "metadata": {}, "source": [ "## Independent training runs for inference\n", "\n", "The confirmatory cohort was fixed at **10 new PPO training seeds**, each with the same 8,192-push budget and supplied BC, before examining its outcomes. Earlier exploratory runs are excluded. All ten runs and all evaluation attempts are retained. There is no stopping or adding runs based on a p-value.\n", "\n", "For each training run, compute its mean final-greedy reward on the fixed 64-goal benchmark minus BC's mean reward on those same goals. The resulting **10 differences**, not 640 goals, are the units for training-run inference. Use the two-sided one-sample t-test of mean difference zero and a bootstrap interval that resamples whole training runs (5,000 resamples, seed 815). The helper also reports an exact sign test as a robustness check of direction. The conclusion is conditional on this BC checkpoint and benchmark; it does not measure variability across new BC fits.\n", "\n", "Use the final checkpoint for the primary test. Time, sampled deployment and budget growth are labeled secondary outcomes. Report the observed p-value and effect; do not assume the result must be significant." ] }, { "cell_type": "code", "execution_count": null, "id": "031f9b63", "metadata": {}, "outputs": [], "source": [ "from residual_analysis import training_comparison\n", "confirmation = reference(\"confirmatory\")\n", "assert len(confirmation[\"runs\"]) == 10\n", "display(score_table(confirmation, \"final\"))\n", "display(aggregate_table(confirmation))\n", "confirmation_stats = training_comparison(confirmation)\n", "display(confirmation_stats)\n", "learning(confirmation)" ] }, { "cell_type": "markdown", "id": "24c36905", "metadata": {}, "source": [ "## Watch the learned policy in Drake (optional)\n", "\n", "With `RUN_DRAKE_REPLAYS=True` above, compare the initial BC rollout and the **best monitoring-selected learned policy** from exactly the same starting state and goal. The helper shows two Meshcat recordings and a score table. Each viewer has its own playback controls; use the same playback speed and compare simulated duration, push count, approaches and retractions. The baseline recording from the beginning of the notebook is reused.\n", "\n", "\u201cBest\u201d means maximum monitoring success, then monitoring reward, with earlier steps/lower seed breaking ties. The package records that choice in `replay_policy.json`; no test-goal or replay outcome selects it. It is chosen from the ten confirmatory runs. This illustrative selection does not replace the statistical comparison of **all ten fixed final policies**. Replays run fresh Drake physics and save standalone HTML plus metrics under your results folder." ] }, { "cell_type": "code", "execution_count": null, "id": "resting-study-findings", "metadata": {}, "outputs": [], "source": [ "if RUN_DRAKE_REPLAYS:\n", " from residual_push import replay_pair\n", " comparison_replays = replay_pair(seed=REPLAY_GOAL, bc_replay=initial_bc_replay)\n", "else:\n", " print(\"Optional replay skipped; all required numerical work is above.\")" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" } }, "nbformat": 4, "nbformat_minor": 5 }