{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": "# 6.4210/2 \u2014 ps6 setup\n\nOn a laptop, these Colab-specific cells detect the local runtime and skip themselves. In Colab, choose the normal Drive checkout, or set `USE_STUDENT_ZIP=True` to upload the supplied student ZIP without mounting Drive. Run setup once per runtime. ZIP mode uses temporary runtime storage: download your edits/results before deleting the runtime." }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": "try:\n import google.colab\n IN_COLAB = True\nexcept ImportError:\n IN_COLAB = False\nUSE_STUDENT_ZIP = False # True: upload the exact supplied student ZIP in Colab.\nif not IN_COLAB:\n print(\"Local runtime: skipping Colab-only setup; continue to the lesson setup cell.\")\n" }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": "if IN_COLAB and not USE_STUDENT_ZIP:\n # Mount Google Drive. This is where we will/have clone(d) your course repo to.\n from google.colab import drive\n\n drive.mount(\"/content/drive\")\n" }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": "if IN_COLAB and not USE_STUDENT_ZIP:\n # Fetch the course code into Drive and put it on the import path. With\n # the Colab secrets GH_TOKEN and PRIVATE_REPO set (see the course\n # README), this uses your private homework repo; without them, a plain\n # read-only clone of the public handouts repo.\n import pathlib\n import subprocess\n import sys\n\n from google.colab import userdata\n\n root = pathlib.Path(\"/content/drive/MyDrive/6.4210/repo\")\n public = \"https://github.com/tlpmit/6.4210-2-fall26-handouts.git\"\n try:\n private = userdata.get(\"PRIVATE_REPO\")\n origin = f\"https://github.com/{private}.git\"\n # The token goes in a credential file outside the repo, so it never\n # ends up in .git/config or in what git prints when a fetch fails.\n creds = pathlib.Path(\"/root/.git-credentials\")\n creds.write_text(\n f\"https://x-access-token:{userdata.get('GH_TOKEN')}@github.com\\n\"\n )\n creds.chmod(0o600)\n subprocess.run(\n [\"git\", \"config\", \"--global\",\n \"credential.helper\", f\"store --file={creds}\"],\n check=True,\n )\n print(f\"Colab secrets found: your repo is {private}\")\n except Exception:\n origin = None\n print(\"no Colab secrets configured: \"\n \"using the public repo, read-only\")\n if root.exists():\n print(f\"course code already in Drive at {root}\")\n else:\n print(f\"cloning {'your private repo' if origin else 'the public repo'}: \"\n f\"{origin or public}\")\n root.parent.mkdir(parents=True, exist_ok=True)\n subprocess.run([\"git\", \"clone\", origin or public, str(root)], check=True)\n if origin:\n subprocess.run(\n [\"git\", \"-C\", str(root), \"remote\", \"add\", \"upstream\", public],\n check=True,\n )\n if str(root) not in sys.path:\n sys.path.insert(0, str(root))\n print(\"done: the course code is on the import path\")\n" }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": "if IN_COLAB and USE_STUDENT_ZIP:\n from google.colab import files\n from pathlib import Path\n import tempfile, zipfile, sys\n uploaded = files.upload()\n if len(uploaded) != 1:\n raise ValueError(\"Upload exactly one student ZIP\")\n archive = Path(next(iter(uploaded)))\n workspace = Path(tempfile.mkdtemp(prefix=\"ps6-colab-\"))\n with zipfile.ZipFile(archive) as bundle:\n if any(Path(n).is_absolute() or \"..\" in Path(n).parts for n in bundle.namelist()):\n raise ValueError(\"Unsafe archive path\")\n bundle.extractall(workspace)\n root = workspace / \"ps6_student\"\n if not (root / \"ps6/homework.py\").is_file():\n raise ValueError(\"Use the supplied ps6 student ZIP\")\n sys.path.insert(0, str(root))\n print(\"Uploaded package:\", root)\n" }, { "cell_type": "markdown", "metadata": {}, "source": "In Drive mode, reopen this notebook from `My Drive \u2192 6.4210 \u2192 repo \u2192 ps6` as usual. In ZIP mode, continue here; code and data come from the uploaded archive, without cloning GitHub or granting Drive access." }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": "if IN_COLAB:\n from utils import colab_bootstrap\n colab_bootstrap.setup(\"ps6\")\n import os\n os.chdir(colab_bootstrap.ROOT / \"ps6\")\n" }, { "cell_type": "markdown", "id": "c34153b2", "metadata": {}, "source": [ "# HW6: Cart-pole\n", "\n", "Use this notebook for the Cart-pole question. Implement `reward_to_go` and `policy_loss` in `reinforce.py`.\n", "\n", "Run the setup cell first. In VS Code select the course `.venv` Python kernel. If dependencies are missing, use `%pip install -r requirements.txt` from `ps6` (installation is separate from the experiment budget). In Colab, run the setup cells at the top first.\n", "\n", "Edit the indicated `.py` functions, then rerun the checks. The setup cell turns on autoreload, so the notebook picks up your edits without restarting the kernel. Supplied data are read-only reference material; new scores, checkpoints, and HTML replays go to `results/`. **Run cells individually as you reach them; do not use Run All.** A training, fitting, or replay cell starts that work immediately. Complete the mathematical steps and their checks first. Training/fitting and controller evaluation calls reuse saved results when the code, settings, data, and output files match. To deliberately repeat a run, add `retrain=True` to that call (for example, `run_cartpole(False, seed=100, retrain=True)`); old files are archived. Changed, interrupted, or older unverified runs ask you to choose explicitly. Replays run a fresh simulation whenever their cell is executed.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "a28eedb0", "metadata": {}, "outputs": [], "source": [ "# Re-read edited .py files before each cell runs (%load_ext autoreload; %autoreload 2).\n", "from IPython import get_ipython\n", "if get_ipython() is not None:\n", " get_ipython().run_line_magic(\"load_ext\", \"autoreload\")\n", " get_ipython().run_line_magic(\"autoreload\", \"2\")\n", "from pathlib import Path\n", "import os, sys\n", "# Works with a kernel started in ps6 or in the repository root.\n", "HERE = next((p if (p / \"homework.py\").exists() else p / \"ps6\"\n", " for p in [Path.cwd(), *Path.cwd().parents]\n", " if (p / \"homework.py\").exists() or (p / \"ps6/homework.py\").exists()), None)\n", "if HERE is None:\n", " raise FileNotFoundError(\"Open this notebook from the course checkout (including ps6/data).\")\n", "sys.path.insert(0, str(HERE))\n", "os.environ.setdefault(\"OMP_NUM_THREADS\", \"1\")\n", "os.environ.setdefault(\"OPENBLAS_NUM_THREADS\", \"1\")\n", "import numpy as np\n", "import torch\n", "import matplotlib.pyplot as plt\n", "from IPython.display import display\n", "torch.set_num_threads(1)\n", "from homework import DATA, RESULTS, read\n", "print(\"Code/data:\", HERE, \"\\nYour outputs:\", RESULTS)\n" ] }, { "cell_type": "markdown", "id": "7e3d0c0d", "metadata": {}, "source": [ "## Check your mathematical steps\n", "After editing `reward_to_go` and `policy_loss` in `reinforce.py`, run this cell. It tests your functions on its own example values, not the ones in Problem 6.1, and prints \"checks passed\" if they are correct. Work the Problem 6.1 values out by hand. These small checks need no Drake simulation.\n", "\n", "`reward_to_go` returns one future-reward sum per action in an episode. The training loop uses these sums in the actor loss and as targets for the baseline. It prepares the baseline values as fixed numbers before calling `policy_loss`; no `.detach()` is needed in your implementation. See the function docstrings for exact input and output types.\n", "\n", "`policy_loss` receives a one-dimensional array of the current policy's log probabilities for the recorded actions. `train` in `reinforce.py` handles log-softmax and recorded-action indexing. Your function implements advantage weighting and averaging over episodes. Keep the provided array type in the loss; `reward_to_go` uses ordinary NumPy.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "fa3e29d3", "metadata": {}, "outputs": [], "source": [ "from reinforce import reward_to_go, policy_loss\n", "assert np.allclose(reward_to_go([1, 2, 3]), [6, 5, 3])\n", "# Three transitions from two episodes, including positive, negative, and zero advantages.\n", "action_log_probs = torch.tensor([-.2, -1., -.4], requires_grad=True)\n", "loss = policy_loss(action_log_probs, torch.tensor([3., 1., 4.]),\n", " torch.tensor([1., 2., 4.]), episode_count=2)\n", "assert torch.allclose(loss, torch.tensor(-.3))\n", "loss.backward()\n", "assert torch.allclose(action_log_probs.grad, torch.tensor([-1., .5, 0.]))\n", "print(\"Return and advantage-weighting checks passed\")\n" ] }, { "cell_type": "markdown", "id": "5a68cbe3", "metadata": {}, "source": [ "## Compare the baseline and no-baseline policies\n", "Train once without the baseline and once with it, both with training seed **100** and the supplied training schedule. A seed initializes the random-number generators: the actor starts with the same weights in both runs, and evaluation uses the same starting states. These two runs form one **training pair**. Their policies and training trajectories can diverge; pairing does not mean identical training data.\n", "\n", "The learning plot compares mean sampled evaluation return as training progresses. The final comparison uses 20 other starting states, shared by the two policies. One training pair lets you see the process, but is not enough to establish an algorithm ranking. The next section uses many supplied pairs to assess the evidence. Run each training cell once; both belong to the shared homework compute budget.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "134785ca", "metadata": {}, "outputs": [], "source": [ "# Train without a baseline (or load this completed run).\n", "from homework import run_cartpole\n", "plain = run_cartpole(False, seed=100)" ] }, { "cell_type": "code", "execution_count": null, "id": "14e86bad", "metadata": {}, "outputs": [], "source": [ "# Train with a baseline (or load this completed run).\n", "baseline = run_cartpole(True, seed=100)" ] }, { "cell_type": "code", "execution_count": null, "id": "0cc824a4", "metadata": {}, "outputs": [], "source": [ "from learning_curves import plot_learning_curves, compare_runs\n", "local = RESULTS / \"cartpole\"\n", "if len(list(local.glob(\"reinforce_*_seed100.json\"))) == 2:\n", " plot_learning_curves(local, output_dir=local)\n", " import pandas as pd\n", " rows = []\n", " for name in [\"plain\", \"baseline\"]:\n", " run = read(local / f\"reinforce_{name}_seed100.json\")\n", " rows.append(dict(condition=name, training_seed=run[\"seed\"],\n", " evaluation_episodes=len(run[\"final\"][\"stochastic\"]),\n", " mean_final_return=np.mean(run[\"final\"][\"stochastic\"])))\n", " display(pd.DataFrame(rows))\n", "else:\n", " print(\"Complete both local conditions to plot the two learning curves and compare final returns.\")\n" ] }, { "cell_type": "markdown", "id": "52821492", "metadata": {}, "source": [ "## Supplied 30-pair study: statistical answer blanks\n", "Use **this supplied data set**, not your local pair, for the numerical answers in the statistical comparison. It contains 30 training pairs; each of the 60 final policies has 100 sampled evaluation episodes. Within each pair, average the 100 returns for each policy, then subtract **baseline minus plain**. These are the 30 independent differences for the paired analysis. The endpoint is fixed at training round 60.\n", "\n", "The learning plot shows evaluation return versus policy updates. Thin lines show individual training runs; thick lines show their means, with pointwise uncertainty bands across runs. Every run uses the same 60-update schedule and contributes at every checkpoint. Your single local pair has no confidence band.\n", "\n", "The learning-curve band resamples whole training runs (`band=\"stderr\"` instead shows one standard error). The paired bootstrap interval uses 5,000 resamples and seed 813. Keep these settings fixed for the answer blanks; the simulation is not rerun.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "2678948c", "metadata": {}, "outputs": [], "source": [ "plot_learning_curves(DATA / \"cartpole\", band=\"bootstrap\", output_dir=RESULTS / \"cartpole_reference\")" ] }, { "cell_type": "code", "execution_count": null, "id": "dfd09aec", "metadata": {}, "outputs": [], "source": [ "inference = compare_runs(DATA / \"cartpole\", show=False, output_dir=RESULTS / \"cartpole_reference\")\n", "{k: inference[k] for k in [\"n_pairs\", \"mean_difference\", \"paired_bootstrap_ci95\",\n", " \"paired_t_statistic\", \"paired_t_p\"]}\n" ] }, { "cell_type": "markdown", "id": "98fa98e6", "metadata": {}, "source": [ "For the handout's answer blanks, use `mean_difference`, `paired_bootstrap_ci95`, `paired_t_statistic`, and `paired_t_p`. `n_pairs` is the number of independent paired differences; the t-test has `n_pairs - 1` degrees of freedom. Return differences and interval endpoints are in environment steps. Use the handout's rounding instructions.\n", "\n", "The 100 evaluations of one policy describe that trained policy; they are not independent training runs. The paired analysis uses one baseline-minus-plain mean difference for each training seed.\n" ] }, { "cell_type": "markdown", "id": "5d3f5c20", "metadata": {}, "source": [ "## Watch a policy in Drake/Meshcat\n", "The helper displays an embedded animation and saves HTML under `results/replays`. Start with `CONDITION=\"random\"`: it chooses each force with probability 1/2 at every step and needs no checkpoint. Change the replay seed to see a different episode. You can also call `watch_policy(\"random\", seed=200000)` directly.\n", "\n", "Set `CONDITION` to `\"plain\"` and then `\"baseline\"` to inspect each learned policy. With one checkpoint, compare `SAMPLED=True` and `False` (greedy), keeping the replay seed fixed. `SAMPLED` does not change the explicitly random policy. Red markers at both ends of the rail show the cart-center limits, normally \u00b12.4 m; these are termination boundaries, not collision stops. The printed ending reason distinguishes a pole-angle failure, a cart-position failure, and the horizon cap. Record observations in the notebook; there is no universal local-score answer key.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "da134a74", "metadata": {}, "outputs": [], "source": [ "CONDITION = \"random\" # choose \"random\", \"plain\", or \"baseline\"\n", "SAMPLED = True # False gives greedy actions for a trained policy; random stays random\n", "if CONDITION == \"random\":\n", " policy = \"random\" # no checkpoint or training required\n", "elif CONDITION in (\"plain\", \"baseline\"):\n", " policy = RESULTS / \"cartpole\" / f\"reinforce_{CONDITION}_seed100.pt\"\n", " if not policy.exists():\n", " policy = DATA / f\"cartpole/reinforce_{CONDITION}_seed100.pt\"\n", " print(\"Using the supplied reference checkpoint; train locally to inspect your own policy.\")\n", "else:\n", " raise ValueError('CONDITION must be \"random\", \"plain\", or \"baseline\"')\n", "from visualization import watch_policy\n", "replay = watch_policy(policy, seed=200000, stochastic=SAMPLED,\n", " output_dir=RESULTS / \"replays\")\n", "print(\"Episode length:\", replay.rollout[\"length\"], \"Ending reason:\", replay.rollout[\"end_reason\"])\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" } }, "nbformat": 4, "nbformat_minor": 5 }