{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4472ab8f",
   "metadata": {},
   "source": [
    "# Neural networks from scratch, then in PyTorch\n",
    "\n",
    "Build logistic regression and a one-hidden-layer network with NumPy only (forward pass, backprop, gradient descent), check the gradients, then build the same network in PyTorch, and finish with a deeper network on handwritten digits.\n",
    "\n",
    "[Open in Google Colab](https://colab.research.google.com/github/mahmoudyosrimahmoud13/inpace-library/blob/main/courses/dl-1-neural-networks/files/dl1_neural_networks_pytorch.ipynb) · part of the inPace course library. Slides © DeepLearning.AI (CC BY-SA 2.0); this notebook: inPace (CC BY-SA 4.0).\n",
    "\n",
    "**Run it top to bottom** (Runtime → Run all in Colab). Each part says what to look for."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "478f82c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Works in Google Colab as it is (PyTorch is already installed there).\n",
    "# On your own computer:  pip install torch torchvision scikit-learn scipy matplotlib pandas\n",
    "# One framework per notebook: PyTorch and TensorFlow can clash when loaded into the same Python process.\n",
    "import numpy as np, matplotlib.pyplot as plt, torch\n",
    "from torch import nn\n",
    "torch.manual_seed(0); np.random.seed(0)\n",
    "device = \"cuda\" if torch.cuda.is_available() else \"mps\" if torch.backends.mps.is_available() else \"cpu\"\n",
    "print(\"PyTorch\", torch.__version__, \"| device:\", device)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d64d52f",
   "metadata": {},
   "source": [
    "## 1. The data: two moons\n",
    "Two classes that a straight line can't separate. 1,000 points: 800 to train, 200 to check (dev)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "01500b5f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import make_moons\n",
    "X, y = make_moons(1000, noise=0.2, random_state=0)\n",
    "X_tr, y_tr, X_dv, y_dv = X[:800], y[:800], X[800:], y[800:]\n",
    "# The course's convention: one example per COLUMN, so X has shape (n_x, m).\n",
    "Xt, Yt, Xd, Yd = X_tr.T, y_tr.reshape(1, -1), X_dv.T, y_dv.reshape(1, -1)\n",
    "print(\"X:\", Xt.shape, \" Y:\", Yt.shape)\n",
    "plt.scatter(X_tr[:, 0], X_tr[:, 1], c=y_tr, cmap=\"coolwarm\", s=8); plt.title(\"training set\"); plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2423bbdd",
   "metadata": {},
   "source": [
    "## 2. Logistic regression, vectorized (no loops over examples)\n",
    "`Z = wᵀX + b`, `A = σ(Z)`, cost = mean cross-entropy, then `dZ = A − Y`, `dw = X dZᵀ / m`, `db = mean(dZ)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1090cacb",
   "metadata": {},
   "outputs": [],
   "source": [
    "def sigmoid(z): return 1 / (1 + np.exp(-z))\n",
    "\n",
    "def logistic_regression(X, Y, lr=0.5, steps=2000):\n",
    "    w, b = np.zeros((X.shape[0], 1)), 0.0\n",
    "    m = X.shape[1]\n",
    "    for i in range(steps):\n",
    "        A = sigmoid(w.T @ X + b)                                   # forward, all m examples at once\n",
    "        cost = -np.mean(Y * np.log(A + 1e-12) + (1 - Y) * np.log(1 - A + 1e-12))\n",
    "        dZ = A - Y                                                 # backward\n",
    "        w -= lr * (X @ dZ.T) / m\n",
    "        b -= lr * dZ.mean()\n",
    "    return w, b, cost\n",
    "\n",
    "w, b, cost = logistic_regression(Xt, Yt)\n",
    "acc = lambda P, Y: (P == Y).mean()\n",
    "print(f\"final cost {cost:.3f} | train acc {acc((sigmoid(w.T @ Xt + b) > .5), Yt):.3f} | dev acc {acc((sigmoid(w.T @ Xd + b) > .5), Yd):.3f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3f78e35b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_boundary(predict, title):\n",
    "    xx, yy = np.meshgrid(np.linspace(-1.5, 2.5, 300), np.linspace(-1, 1.5, 300))\n",
    "    zz = predict(np.c_[xx.ravel(), yy.ravel()].T).reshape(xx.shape)\n",
    "    plt.contourf(xx, yy, zz, levels=20, cmap=\"coolwarm\", alpha=.5)\n",
    "    plt.scatter(X_tr[:, 0], X_tr[:, 1], c=y_tr, cmap=\"coolwarm\", s=6); plt.title(title); plt.show()\n",
    "\n",
    "plot_boundary(lambda P: sigmoid(w.T @ P + b), \"logistic regression: one straight line\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "acd4e540",
   "metadata": {},
   "source": [
    "## 3. One hidden layer, from scratch\n",
    "Shapes to keep in your head: `W1 (n_h, n_x)`, `b1 (n_h, 1)`, `W2 (1, n_h)`, `b2 (1, 1)`. tanh in the hidden layer, sigmoid at the output. Small random weights (never zeros: every hidden unit would stay identical)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b1ceb0c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def init(n_x, n_h, seed=1):\n",
    "    rng = np.random.default_rng(seed)\n",
    "    return {\"W1\": rng.normal(0, 0.5, (n_h, n_x)), \"b1\": np.zeros((n_h, 1)),\n",
    "            \"W2\": rng.normal(0, 0.5, (1, n_h)), \"b2\": np.zeros((1, 1))}\n",
    "\n",
    "def forward(p, X):\n",
    "    Z1 = p[\"W1\"] @ X + p[\"b1\"]; A1 = np.tanh(Z1)\n",
    "    Z2 = p[\"W2\"] @ A1 + p[\"b2\"]; A2 = sigmoid(Z2)\n",
    "    return A2, (A1, A2)\n",
    "\n",
    "def cost_fn(A2, Y):\n",
    "    return -np.mean(Y * np.log(A2 + 1e-12) + (1 - Y) * np.log(1 - A2 + 1e-12))\n",
    "\n",
    "def backward(p, X, Y, cache):\n",
    "    A1, A2 = cache; m = X.shape[1]\n",
    "    dZ2 = A2 - Y\n",
    "    dW2 = dZ2 @ A1.T / m;  db2 = dZ2.mean(axis=1, keepdims=True)\n",
    "    dZ1 = (p[\"W2\"].T @ dZ2) * (1 - A1 ** 2)          # tanh'(z) = 1 - tanh(z)^2\n",
    "    dW1 = dZ1 @ X.T / m;   db1 = dZ1.mean(axis=1, keepdims=True)\n",
    "    return {\"W1\": dW1, \"b1\": db1, \"W2\": dW2, \"b2\": db2}\n",
    "\n",
    "p, costs = init(2, 8), []\n",
    "for i in range(5000):\n",
    "    A2, cache = forward(p, Xt)\n",
    "    g = backward(p, Xt, Yt, cache)\n",
    "    for k in p: p[k] -= 1.0 * g[k]\n",
    "    costs.append(cost_fn(A2, Yt))\n",
    "print(f\"train acc {acc(forward(p, Xt)[0] > .5, Yt):.3f} | dev acc {acc(forward(p, Xd)[0] > .5, Yd):.3f}\")\n",
    "plt.plot(costs); plt.yscale(\"log\"); plt.title(\"cost\"); plt.xlabel(\"step\"); plt.show()\n",
    "plot_boundary(lambda P: forward(p, P)[0], \"8 tanh units: a curved boundary\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42f57671",
   "metadata": {},
   "source": [
    "## 4. Gradient check: is our backprop right?\n",
    "Compare each analytic gradient with `(J(θ+ε) − J(θ−ε)) / 2ε`. The relative difference should be around `1e-7` or smaller; `1e-3` means a bug."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "546f8e13",
   "metadata": {},
   "outputs": [],
   "source": [
    "def grad_check(p, X, Y, eps=1e-6):\n",
    "    _, cache = forward(p, X); g = backward(p, X, Y, cache)\n",
    "    num, ana = [], []\n",
    "    for k in p:\n",
    "        for idx in np.ndindex(p[k].shape):\n",
    "            old = p[k][idx]\n",
    "            p[k][idx] = old + eps; jp = cost_fn(forward(p, X)[0], Y)\n",
    "            p[k][idx] = old - eps; jm = cost_fn(forward(p, X)[0], Y)\n",
    "            p[k][idx] = old\n",
    "            num.append((jp - jm) / (2 * eps)); ana.append(g[k][idx])\n",
    "    num, ana = np.array(num), np.array(ana)\n",
    "    return np.linalg.norm(num - ana) / (np.linalg.norm(num) + np.linalg.norm(ana))\n",
    "\n",
    "print(\"relative difference:\", grad_check(init(2, 4), Xt[:, :50], Yt[:, :50]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "900f46fa",
   "metadata": {},
   "source": [
    "## 5. The same network in PyTorch\n",
    "PyTorch stores one example per **row**, so `X` is `(m, n_x)`. `loss.backward()` does section 3's backward pass for us (autograd). `BCEWithLogitsLoss` takes raw scores (logits) and is more stable than sigmoid + log."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2500fab5",
   "metadata": {},
   "outputs": [],
   "source": [
    "Xtr_t, ytr_t = torch.tensor(X_tr, dtype=torch.float32), torch.tensor(y_tr, dtype=torch.float32)\n",
    "Xdv_t, ydv_t = torch.tensor(X_dv, dtype=torch.float32), torch.tensor(y_dv, dtype=torch.float32)\n",
    "\n",
    "class ShallowNet(nn.Module):\n",
    "    def __init__(self, n_h=8):\n",
    "        super().__init__()\n",
    "        self.hidden = nn.Linear(2, n_h)\n",
    "        self.out = nn.Linear(n_h, 1)\n",
    "    def forward(self, x):\n",
    "        return self.out(torch.tanh(self.hidden(x))).squeeze(1)   # logits\n",
    "\n",
    "model = ShallowNet().to(device)\n",
    "opt = torch.optim.SGD(model.parameters(), lr=1.0)\n",
    "loss_fn = nn.BCEWithLogitsLoss()\n",
    "for step in range(3000):\n",
    "    opt.zero_grad()\n",
    "    loss = loss_fn(model(Xtr_t.to(device)), ytr_t.to(device))\n",
    "    loss.backward()        # autograd: every gradient in one call\n",
    "    opt.step()\n",
    "with torch.no_grad():\n",
    "    dev_acc = ((model(Xdv_t.to(device)) > 0).float().cpu() == ydv_t).float().mean().item()\n",
    "print(f\"final loss {loss.item():.3f} | dev acc {dev_acc:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46fbbf2b",
   "metadata": {},
   "source": [
    "## 7. A deeper network on handwritten digits\n",
    "1,797 images of 8×8 pixels, 10 classes. Two hidden ReLU layers, softmax output (inside `CrossEntropyLoss`), Adam."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "24480c92",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_digits\n",
    "from sklearn.model_selection import train_test_split\n",
    "d = load_digits()\n",
    "Xa, Xb, ya, yb = train_test_split(d.data / 16.0, d.target, test_size=0.25, random_state=0)\n",
    "Xa, Xb = torch.tensor(Xa, dtype=torch.float32), torch.tensor(Xb, dtype=torch.float32)\n",
    "ya, yb = torch.tensor(ya), torch.tensor(yb)\n",
    "deep = nn.Sequential(nn.Linear(64, 128), nn.ReLU(), nn.Linear(128, 64), nn.ReLU(), nn.Linear(64, 10))\n",
    "opt = torch.optim.Adam(deep.parameters(), lr=1e-3)\n",
    "for epoch in range(300):\n",
    "    opt.zero_grad(); loss = nn.functional.cross_entropy(deep(Xa), ya); loss.backward(); opt.step()\n",
    "with torch.no_grad():\n",
    "    print(f\"parameters: {sum(p.numel() for p in deep.parameters()):,} | test accuracy {(deep(Xb).argmax(1) == yb).float().mean():.3f}\")\n",
    "fig, ax = plt.subplots(1, 8, figsize=(10, 1.6))\n",
    "for i, a in enumerate(ax):\n",
    "    a.imshow(Xb[i].reshape(8, 8), cmap=\"gray_r\"); a.set_title(f\"pred {deep(Xb[i:i+1]).argmax().item()}\"); a.axis(\"off\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03d7240c",
   "metadata": {},
   "source": [
    "## Try this\n",
    "1. Change `n_h` in section 3 to 1, 2, 4 and 50. When does the boundary curve? When does it overfit?\n",
    "2. Replace `tanh` with ReLU in section 3 (`np.maximum(0, Z1)`, slope `Z1 > 0`). Rerun the gradient check.\n",
    "3. Initialize `W1` with zeros. What happens to the hidden units, and why?\n",
    "4. In section 7, remove one hidden layer. How much does test accuracy change?"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}