{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "61b40771",
   "metadata": {},
   "source": [
    "# Neural networks in Keras 3 and TensorFlow\n",
    "\n",
    "The same ideas the TensorFlow way: Keras 3 layers, `compile`/`fit`, callbacks, and `tf.GradientTape` when you want your own training loop. TensorFlow 2.21 is stable and mostly receives fixes now; Google recommends Keras 3, JAX or PyTorch for new generative-AI work, and Keras 3 code like this also runs on JAX or PyTorch by changing one line.\n",
    "\n",
    "[Open in Google Colab](https://colab.research.google.com/github/mahmoudyosrimahmoud13/inpace-library/blob/main/courses/dl-1-neural-networks/files/dl1_neural_networks_keras.ipynb) · part of the inPace course library. Slides © DeepLearning.AI (CC BY-SA 2.0); this notebook: inPace (CC BY-SA 4.0).\n",
    "\n",
    "**Run it top to bottom** (Runtime → Run all in Colab). Each part says what to look for."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1b96f50f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Works in Google Colab as it is (TensorFlow and Keras are already installed there).\n",
    "# On your own computer:  pip install tensorflow keras scikit-learn scipy matplotlib\n",
    "# One framework per notebook: PyTorch and TensorFlow can clash when loaded into the same Python process.\n",
    "import os\n",
    "os.environ[\"KERAS_BACKEND\"] = \"tensorflow\"   # Keras 3 also runs on \"jax\" or \"torch\": change it here\n",
    "os.environ.setdefault(\"TF_CPP_MIN_LOG_LEVEL\", \"3\")\n",
    "import numpy as np, matplotlib.pyplot as plt\n",
    "import tensorflow as tf, keras\n",
    "keras.utils.set_random_seed(0)\n",
    "print(\"TensorFlow\", tf.__version__, \"| Keras\", keras.__version__, \"| backend:\", keras.backend.backend(),\n",
    "      \"| GPU:\", bool(tf.config.list_physical_devices(\"GPU\")))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e20d62f",
   "metadata": {},
   "source": [
    "## 1. Data: two moons"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "994c1d98",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import make_moons\n",
    "X, y = make_moons(1000, noise=0.2, random_state=0)\n",
    "X_tr, y_tr, X_dv, y_dv = X[:800].astype(\"float32\"), y[:800], X[800:].astype(\"float32\"), y[800:]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "338dd605",
   "metadata": {},
   "source": [
    "## 2. One hidden layer with `fit`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "807d9be9",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.Sequential([keras.Input((2,)), keras.layers.Dense(8, activation=\"tanh\"), keras.layers.Dense(1)])\n",
    "model.compile(optimizer=keras.optimizers.SGD(1.0), loss=keras.losses.BinaryCrossentropy(from_logits=True),\n",
    "              metrics=[keras.metrics.BinaryAccuracy(threshold=0.0)])\n",
    "hist = model.fit(X_tr, y_tr, epochs=80, batch_size=800, verbose=0, validation_data=(X_dv, y_dv))\n",
    "print(\"dev accuracy:\", round(hist.history[\"val_binary_accuracy\"][-1], 3))\n",
    "plt.plot(hist.history[\"loss\"], label=\"train\"); plt.plot(hist.history[\"val_loss\"], label=\"dev\"); plt.legend(); plt.title(\"loss\"); plt.show()\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f1a938e",
   "metadata": {},
   "source": [
    "## 3. The same training with your own loop: `tf.GradientTape`\n",
    "The tape records the forward pass; `tape.gradient` gives every gradient (TensorFlow's autograd)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "839fc1d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "W1 = tf.Variable(tf.random.normal((2, 8), stddev=0.5)); b1 = tf.Variable(tf.zeros(8))\n",
    "W2 = tf.Variable(tf.random.normal((8, 1), stddev=0.5)); b2 = tf.Variable(tf.zeros(1))\n",
    "params = [W1, b1, W2, b2]\n",
    "opt = keras.optimizers.SGD(1.0)\n",
    "Xt, yt = tf.constant(X_tr), tf.constant(y_tr.reshape(-1, 1), tf.float32)\n",
    "@tf.function                      # compiles the step into a fast graph\n",
    "def step():\n",
    "    with tf.GradientTape() as tape:\n",
    "        logits = tf.tanh(Xt @ W1 + b1) @ W2 + b2\n",
    "        loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(yt, logits))\n",
    "    opt.apply_gradients(zip(tape.gradient(loss, params), params))\n",
    "    return loss\n",
    "for i in range(3000): loss = step()\n",
    "pred = (tf.tanh(X_dv @ W1 + b1) @ W2 + b2).numpy().ravel() > 0\n",
    "print(f\"final loss {loss.numpy():.3f} | dev accuracy {(pred == y_dv).mean():.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3d9a20f",
   "metadata": {},
   "source": [
    "## 4. A deeper network on digits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fbe9a2e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_digits\n",
    "from sklearn.model_selection import train_test_split\n",
    "d = load_digits()\n",
    "Xa, Xb, ya, yb = train_test_split((d.data / 16.0).astype(\"float32\"), d.target, test_size=0.25, random_state=0)\n",
    "deep = keras.Sequential([keras.Input((64,)), keras.layers.Dense(128, activation=\"relu\"),\n",
    "                         keras.layers.Dense(64, activation=\"relu\"), keras.layers.Dense(10)])\n",
    "deep.compile(\"adam\", keras.losses.SparseCategoricalCrossentropy(from_logits=True), metrics=[\"accuracy\"])\n",
    "deep.fit(Xa, ya, epochs=40, batch_size=64, verbose=0)\n",
    "print(f\"parameters: {deep.count_params():,} | test accuracy {deep.evaluate(Xb, yb, verbose=0)[1]:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b313bd2e",
   "metadata": {},
   "source": [
    "## Try this\n",
    "1. Change `KERAS_BACKEND` in the first cell to `\"jax\"` (Colab has JAX) and run sections 1, 2 and 4 again: the same code, another engine.\n",
    "2. In section 3, remove `@tf.function`. How much slower is the loop?"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}