{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "89b77529",
   "metadata": {},
   "source": [
    "# Sequence models in Keras 3 and TensorFlow\n",
    "\n",
    "The same ideas the TensorFlow way: Keras 3 layers, `compile`/`fit`, callbacks, and `tf.GradientTape` when you want your own training loop. TensorFlow 2.21 is stable and mostly receives fixes now; Google recommends Keras 3, JAX or PyTorch for new generative-AI work, and Keras 3 code like this also runs on JAX or PyTorch by changing one line.\n",
    "\n",
    "[Open in Google Colab](https://colab.research.google.com/github/mahmoudyosrimahmoud13/inpace-library/blob/main/courses/dl-5-sequence-models/files/dl5_sequence_models_keras.ipynb) · part of the inPace course library. Slides © DeepLearning.AI (CC BY-SA 2.0); this notebook: inPace (CC BY-SA 4.0).\n",
    "\n",
    "**Run it top to bottom** (Runtime → Run all in Colab). Each part says what to look for."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1dd31f66",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Works in Google Colab as it is (TensorFlow and Keras are already installed there).\n",
    "# On your own computer:  pip install tensorflow keras scikit-learn scipy matplotlib\n",
    "# One framework per notebook: PyTorch and TensorFlow can clash when loaded into the same Python process.\n",
    "import os\n",
    "os.environ[\"KERAS_BACKEND\"] = \"tensorflow\"   # Keras 3 also runs on \"jax\" or \"torch\": change it here\n",
    "os.environ.setdefault(\"TF_CPP_MIN_LOG_LEVEL\", \"3\")\n",
    "import numpy as np, matplotlib.pyplot as plt\n",
    "import tensorflow as tf, keras\n",
    "keras.utils.set_random_seed(0)\n",
    "print(\"TensorFlow\", tf.__version__, \"| Keras\", keras.__version__, \"| backend:\", keras.backend.backend(),\n",
    "      \"| GPU:\", bool(tf.config.list_physical_devices(\"GPU\")))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80ca6a27",
   "metadata": {},
   "source": [
    "## 1. RNN, GRU and LSTM forecasting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f4ba31df",
   "metadata": {},
   "outputs": [],
   "source": [
    "t = np.arange(0, 400, 0.1)\n",
    "signal = np.sin(t) + 0.5 * np.sin(3.1 * t) + 0.1 * np.random.default_rng(0).normal(size=t.size)\n",
    "W = 40\n",
    "Xs = np.stack([signal[i:i + W] for i in range(len(signal) - W)])[:, :, None].astype(\"float32\"); ys = signal[W:].astype(\"float32\")\n",
    "n = int(0.8 * len(Xs))\n",
    "for name, layer in ((\"SimpleRNN\", keras.layers.SimpleRNN), (\"GRU\", keras.layers.GRU), (\"LSTM\", keras.layers.LSTM)):\n",
    "    keras.utils.set_random_seed(0)\n",
    "    m = keras.Sequential([keras.Input((W, 1)), layer(32), keras.layers.Dense(1)])\n",
    "    m.compile(keras.optimizers.Adam(3e-3), \"mse\")\n",
    "    h = m.fit(Xs[:n], ys[:n], epochs=3, batch_size=64, verbose=0)\n",
    "    print(f\"{name:<10} test MSE {m.evaluate(Xs[n:], ys[n:], verbose=0):.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eeb94e1a",
   "metadata": {},
   "source": [
    "## 2. A tiny transformer with MultiHeadAttention\n",
    "The counting task: for every position, how many times its token appears in the whole sequence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "66f5b1f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "V_, L_ = 6, 12\n",
    "rng = np.random.default_rng(0)\n",
    "def batch(n):\n",
    "    x = rng.integers(0, V_, (n, L_)); return x, (x[:, :, None] == x[:, None, :]).sum(-1) - 1\n",
    "inp = keras.Input((L_,), dtype=\"int32\")\n",
    "h = keras.layers.Embedding(V_, 64)(inp) + keras.layers.Embedding(L_, 64)(keras.ops.arange(L_))\n",
    "for _ in range(2):\n",
    "    h = keras.layers.LayerNormalization()(h + keras.layers.MultiHeadAttention(num_heads=4, key_dim=16)(h, h))\n",
    "    h = keras.layers.LayerNormalization()(h + keras.layers.Dense(64)(keras.layers.Dense(128, activation=\"relu\")(h)))\n",
    "model = keras.Model(inp, keras.layers.Dense(L_)(h))\n",
    "model.compile(keras.optimizers.AdamW(1e-3), keras.losses.SparseCategoricalCrossentropy(from_logits=True), metrics=[\"accuracy\"])\n",
    "x, y = batch(20000); model.fit(x, y, epochs=8, batch_size=64, verbose=0)\n",
    "x, y = batch(1000); print(\"accuracy:\", round(model.evaluate(x, y, verbose=0)[1], 3))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db4d6da8",
   "metadata": {},
   "source": [
    "## 3. Text in, numbers out: TextVectorization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d8d21206",
   "metadata": {},
   "outputs": [],
   "source": [
    "vec = keras.layers.TextVectorization(max_tokens=50, output_sequence_length=6)\n",
    "vec.adapt([\"the dessert is excellent\", \"completely lacking in good taste\", \"good service\"])\n",
    "print(vec.get_vocabulary()[:12]); print(vec([\"the service is good\"]).numpy())"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}