{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3a8a2e23",
   "metadata": {},
   "source": [
    "# Activity: Improving sentiment analysis with convolutional layers\n",
    "\n",
    "## Objective\n",
    "\n",
    "In this activity, you will extend the baseline RNN sentiment classifier by adding **convolutional layers** to create a hybrid CNN-RNN architecture. Convolutional layers can capture local n-gram patterns (like \"not good\" or \"very happy\") before the recurrent layers process the sequence.\n",
    "\n",
    "**Your tasks:**\n",
    "1. Run the baseline GRU model and note its performance\n",
    "2. Modify the model architecture to include convolutional layers\n",
    "3. Train your CNN-RNN hybrid model\n",
    "4. Compare the results to the baseline\n",
    "\n",
    "---\n",
    "\n",
    "## Key terms\n",
    "\n",
    "- **Tokenization**: Splitting text into individual units (tokens), typically words. For example, \"I love this!\" becomes `[\"i\", \"love\", \"this\"]`. This is the first step in converting text to a format neural networks can process.\n",
    "\n",
    "- **Embedding**: A dense vector representation of a token. Instead of one-hot encoding (sparse, high-dimensional), embeddings map each word to a fixed-size vector (e.g., 100 dimensions) where similar words have similar vectors. We use pretrained GloVe embeddings trained on Twitter data.\n",
    "\n",
    "- **LSTM/GRU**: Variants of RNNs designed to handle long sequences better than SimpleRNN. They use \"gates\" to control information flow, solving the vanishing gradient problem. **GRU** (Gated Recurrent Unit) has 2 gates and is simpler; **LSTM** (Long Short-Term Memory) has 3 gates and more parameters. Both work fine in practice, we use GRU here for efficiency.\n",
    "\n",
    "## How the GRU processes a sequence\n",
    "\n",
    "Given input sequence `[\"i\", \"love\", \"this]`, the GRU predicts sentiment:\n",
    "\n",
    "```text\n",
    "\n",
    "                   ┌────────────◀───────────┐ 'h₁', then 'h₂'\n",
    "                   ▼                        │\n",
    "                ╔══════════════════════════════╗\n",
    "                ║  │           GRU          ▲  ║\n",
    "                ║  │                        │  ║\n",
    "  'I' then,     ║  ├──▶Reset───▶candidate──▶│  ║         ╔═══════╗ \n",
    "'love', then ══▶║  │   gate      hidden     │  ╠══'h₃'══▶║ Dense ╠══▶ Sentiment\n",
    "   'this'       ║  │              state     │  ║         ╚═══════╝\n",
    "                ║  │                        │  ║\n",
    "                ║  └─────▶Update gate───────┘  ║\n",
    "                ╚══════════════════════════════╝\n",
    "\n",
    "```\n",
    "\n",
    "- **Reset gate**: Controls how much the prior hidden state influences the new candidate hidden state. Gates are **dynamic weights**, one per hidden unit. Depends on **both** the previous hidden state AND the current input.\n",
    "- **Update gate**: Controls how much prior hidden state vs new candidate hidden state goes into the new hidden state. Also computed from previous hidden state + current input, just with different learned weights.\n",
    "- **Bidirectional recurrent layers**: Processes a sequence in both directions (forward and backward) and combines the results. This allows the model to use context from both before AND after each word, improving understanding.\n",
    "\n",
    "## External tools & resources\n",
    "\n",
    "- **Stopwords**: Common words (e.g., \"the\", \"is\", \"at\") filtered out during preprocessing to reduce noise and focus on meaningful content.\n",
    "  - *This project*: [NLTK Stopwords](https://www.nltk.org/nltk_data/) curated lists for 20+ languages\n",
    "  - *Alternatives*: [spaCy stopwords](https://spacy.io/usage/rule-based-matching#vocab-stopwords), [scikit-learn ENGLISH_STOP_WORDS](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.ENGLISH_STOP_WORDS.html)\n",
    "\n",
    "- **Tokenizers**: Tools that split text into tokens. Different tokenizers handle edge cases differently social media text requires special handling for emoticons, hashtags, and informal spelling.\n",
    "  - *This project*: [NLTK TweetTokenizer](https://www.nltk.org/api/nltk.tokenize.casual.html#nltk.tokenize.casual.TweetTokenizer) handles emoticons, hashtags, mentions, and normalizes repeated characters (e.g., \"sooooo\" → \"sooo\")\n",
    "  - *Alternatives*: [spaCy Tokenizer](https://spacy.io/usage/linguistic-features#tokenization), [Hugging Face Tokenizers](https://huggingface.co/docs/tokenizers/), [NLTK word_tokenize](https://www.nltk.org/api/nltk.tokenize.word_tokenize.html)\n",
    "\n",
    "- **Word Embeddings**: Pretrained vector representations that capture semantic relationships between words. Using pretrained embeddings transfers knowledge from large corpora to your model.\n",
    "  - *This project*: [GloVe Twitter embeddings](https://nlp.stanford.edu/projects/glove/) trained on 27B Twitter tokens, 100 dimensions\n",
    "  - *Alternatives*: [Word2Vec](https://radimrehurek.com/gensim/models/word2vec.html), [FastText](https://fasttext.cc/docs/en/english-vectors.html), [Hugging Face sentence-transformers](https://huggingface.co/sentence-transformers)\n",
    "\n",
    "\n",
    "## Notebook setup\n",
    "\n",
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2c49205e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import re\n",
    "import zipfile\n",
    "import urllib.request\n",
    "from collections import Counter\n",
    "from pathlib import Path\n",
    "\n",
    "import keras\n",
    "import nltk\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import tensorflow as tf\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.tokenize import TweetTokenizer\n",
    "from keras.models import Sequential\n",
    "from keras.layers import (\n",
    "    Input, Embedding, Dense, Dropout, Bidirectional, GRU, SpatialDropout1D\n",
    ")\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "from keras.callbacks import ModelCheckpoint, EarlyStopping, TensorBoard\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.utils.class_weight import compute_class_weight\n",
    "from sklearn.metrics import confusion_matrix, roc_curve, auc, precision_recall_curve\n",
    "from sklearn.preprocessing import label_binarize"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "664d2ca9",
   "metadata": {},
   "source": [
    "### GPU configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b8d56b31",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Configure GPU\n",
    "gpus = tf.config.list_physical_devices('GPU')\n",
    "\n",
    "if gpus:\n",
    "    try:\n",
    "        # Set memory growth on ALL GPUs (required when multiple GPUs present)\n",
    "        for gpu in gpus:\n",
    "            tf.config.experimental.set_memory_growth(gpu, True)\n",
    "\n",
    "        print(f'Using {len(gpus)} GPU(s): {[gpu.name for gpu in gpus]}')\n",
    "        \n",
    "    except RuntimeError as e:\n",
    "        print(e)\n",
    "\n",
    "else:\n",
    "    print('No GPU available, using CPU')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59731491",
   "metadata": {},
   "source": [
    "### Environment setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bcd8997d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set data directory\n",
    "data_dir = '../data'\n",
    "Path(data_dir).mkdir(parents=True, exist_ok=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1064ec60",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download NLTK resources\n",
    "os.environ[\"NLTK_DATA\"] = data_dir\n",
    "nltk.download('stopwords', quiet=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d5d0d555",
   "metadata": {},
   "outputs": [],
   "source": [
    "# GloVe Twitter embeddings download\n",
    "glove_url  = 'https://nlp.stanford.edu/data/glove.twitter.27B.zip'\n",
    "glove_file = os.path.join(data_dir, 'glove.twitter.27B.100d.txt')\n",
    "glove_zip  = os.path.join(data_dir, 'glove.twitter.27B.zip')\n",
    "\n",
    "if not os.path.exists(glove_file):\n",
    "\n",
    "    print('Downloading GloVe embeddings...')\n",
    "    urllib.request.urlretrieve(glove_url, glove_zip)\n",
    "\n",
    "    print('Extracting...')\n",
    "    with zipfile.ZipFile(glove_zip, 'r') as zip_ref:\n",
    "        zip_ref.extract('glove.twitter.27B.100d.txt', data_dir)\n",
    "\n",
    "    os.remove(glove_zip)\n",
    "    print('Done!')\n",
    "\n",
    "else:\n",
    "    print(f'GloVe file already exists: {glove_file}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed0c2110",
   "metadata": {},
   "source": [
    "### Shared hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "01105d9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "hidden_dim              = 64      # Number of hidden units in the GRU layers\n",
    "dropout                 = 0.3     # Spatial dropout rate\n",
    "l2_reg                  = 0.02    # L2 regularization strength\n",
    "learning_rate           = 0.0001  # Low LR for fine-tuning embeddings\n",
    "batch_size              = 128\n",
    "epochs                  = 200\n",
    "early_stopping_patience = 20\n",
    "verbose                 = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc9b3cef",
   "metadata": {},
   "source": [
    "## 1. Data preparation\n",
    "\n",
    "### 1.1. Load"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "08292db5",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_parquet('../data/twitter-2016.parquet')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52be2521",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b4c04aa",
   "metadata": {},
   "source": [
    "### 1.2. Clean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bf67ece0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Clean data - drop rows with missing scores\n",
    "df_clean = df.dropna(subset=['score']).copy()\n",
    "df_clean['score'] = df_clean['score'].astype(int)\n",
    "\n",
    "# Shift scores to 0-based index: [-2,-1,0,1,2] -> [0,1,2,3,4]\n",
    "score_min = df_clean['score'].min()\n",
    "df_clean['score_shifted'] = df_clean['score'] - score_min\n",
    "\n",
    "print(f'Cleaned dataset: {len(df_clean):,} rows ({len(df) - len(df_clean):,} dropped)')\n",
    "print(f'Score distribution: {df_clean[\"score_shifted\"].value_counts().sort_index().to_dict()}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b45c1eb",
   "metadata": {},
   "source": [
    "### 1.3. Tokenize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d3966114",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set up stop words and tokenizer\n",
    "stop_words      = set(stopwords.words('english'))\n",
    "tweet_tokenizer = TweetTokenizer(preserve_case=False, reduce_len=True, strip_handles=True)\n",
    "\n",
    "def clean_text(text):\n",
    "    '''Clean and tokenize tweet text using TweetTokenizer.'''\n",
    "\n",
    "    if not isinstance(text, str):\n",
    "        return []\n",
    "\n",
    "    # Remove URLs before tokenizing\n",
    "    text = re.sub(r'http\\S+|www\\S+|https\\S+', '', text)\n",
    "\n",
    "    # TweetTokenizer handles: lowercase, @mentions, repeated chars (e.g., sooooo -> sooo)\n",
    "    tokens = tweet_tokenizer.tokenize(text)\n",
    "\n",
    "    # Remove '#' symbols from hashtags (keep the word)\n",
    "    tokens = [t.lstrip('#') for t in tokens]\n",
    "\n",
    "    # Remove stopwords and single characters\n",
    "    tokens = [t for t in tokens if t not in stop_words and len(t) > 1]\n",
    "\n",
    "    return tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b7efdeb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Apply tokenization\n",
    "df_clean['tokens'] = df_clean['text'].apply(clean_text)\n",
    "df_clean[['text', 'tokens', 'score']].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a70b43c",
   "metadata": {},
   "source": [
    "### 1.4. Build vocabulary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d8efd85",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build vocabulary\n",
    "def build_vocab(token_lists, min_freq=2):\n",
    "    '''Build vocabulary from token lists.'''\n",
    "\n",
    "    counter = Counter()\n",
    "\n",
    "    for tokens in token_lists:\n",
    "        counter.update(tokens)\n",
    "    \n",
    "    # Filter by minimum frequency and create vocab\n",
    "    vocab = {'<PAD>': 0, '<UNK>': 1}\n",
    "\n",
    "    for word, freq in counter.items():\n",
    "        if freq >= min_freq:\n",
    "            vocab[word] = len(vocab)\n",
    "\n",
    "    return vocab\n",
    "\n",
    "vocab = build_vocab(df_clean['tokens'])\n",
    "print(f'Vocabulary size: {len(vocab)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc6059bd",
   "metadata": {},
   "source": [
    "### 1.5. Convert to indices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "951ab5c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "def tokens_to_indices(tokens_list, vocab, max_len=50):\n",
    "    '''Convert token lists to padded index sequences.'''\n",
    "\n",
    "    sequences = []\n",
    "\n",
    "    for tokens in tokens_list:\n",
    "        indices = [vocab.get(token, vocab['<UNK>']) for token in tokens]\n",
    "        sequences.append(indices)\n",
    "\n",
    "    # Pad sequences to max_len\n",
    "    return pad_sequences(sequences, maxlen=max_len, padding='post', value=vocab['<PAD>'])\n",
    "\n",
    "# Convert all data to indices\n",
    "max_len = 50\n",
    "X = tokens_to_indices(df_clean['tokens'].tolist(), vocab, max_len)\n",
    "y = df_clean['score_shifted'].values\n",
    "\n",
    "print(f'X shape: {X.shape}')\n",
    "print(f'y shape: {y.shape}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5118247a",
   "metadata": {},
   "source": [
    "### 1.6. Train/test split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c941eef5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train/test split\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y,\n",
    "    test_size=0.2,\n",
    "    random_state=42,\n",
    "    stratify=y\n",
    ")\n",
    "\n",
    "print(f'X_train shape: {X_train.shape}')\n",
    "print(f'X_test shape: {X_test.shape}')\n",
    "print(f'Score distribution (train): {Counter(y_train)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45f38bea",
   "metadata": {},
   "source": [
    "## 2. Prepare GloVe embeddings\n",
    "\n",
    "GloVe (Global Vectors) provides pretrained word embeddings trained on large text corpora. Using these gives our model a head start - words already have meaningful representations instead of random vectors.\n",
    "\n",
    "### 2.1. Load embeddings into dictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5cf57e38",
   "metadata": {},
   "outputs": [],
   "source": [
    "glove_vectors = {}\n",
    "\n",
    "with open(glove_file, 'r', encoding='utf-8') as f:\n",
    "    for line in f:\n",
    "\n",
    "        values = line.split()\n",
    "        word = values[0]\n",
    "        vector = np.array(values[1:], dtype='float32')\n",
    "        glove_vectors[word] = vector\n",
    "\n",
    "embedding_dim = len(next(iter(glove_vectors.values())))\n",
    "\n",
    "print(f'Loaded {len(glove_vectors):,} word vectors')\n",
    "print(f'Embedding dimension: {embedding_dim}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "229ff779",
   "metadata": {},
   "source": [
    "### 2.2. Create embedding look-up table for our vocabulary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d1cc20ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab_size = len(vocab)\n",
    "\n",
    "embedding_matrix = np.zeros((vocab_size, embedding_dim))\n",
    "words_found = 0\n",
    "\n",
    "for word, idx in vocab.items():\n",
    "    if word in glove_vectors:\n",
    "        embedding_matrix[idx] = glove_vectors[word]\n",
    "        words_found += 1\n",
    "\n",
    "    else:\n",
    "        # Random initialization for words not in GloVe\n",
    "        embedding_matrix[idx] = np.random.normal(scale=0.6, size=(embedding_dim,))\n",
    "\n",
    "coverage = words_found / vocab_size\n",
    "print(f'Vocabulary coverage: {words_found:,} / {vocab_size:,} ({coverage:.1%})')\n",
    "print(f'Embedding matrix shape: {embedding_matrix.shape}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e889cccf",
   "metadata": {},
   "source": [
    "## 3. Baseline: Bidirectional GRU model\n",
    "\n",
    "First, let's establish our baseline performance with a pure recurrent architecture.\n",
    "\n",
    "### 3.1. Build model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d90c1cf5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert numpy embedding matrix into a Constant initializer\n",
    "glove_initializer = keras.initializers.Constant(embedding_matrix)\n",
    "\n",
    "# Build Bidirectional GRU model with pretrained GloVe embeddings\n",
    "baseline_model = Sequential([\n",
    "    Input(shape=(max_len,)),\n",
    "    Embedding(\n",
    "        input_dim=vocab_size, \n",
    "        output_dim=embedding_dim,\n",
    "        embeddings_initializer=glove_initializer,\n",
    "        trainable=True  # Set to True to make embeddings trainable\n",
    "    ),\n",
    "    SpatialDropout1D(dropout),\n",
    "    Bidirectional(GRU(hidden_dim, return_sequences=True, name='GRU_1'), name='Bidirectional_1'),\n",
    "    Bidirectional(GRU(hidden_dim, return_sequences=False, name='GRU_2'), name='Bidirectional_2'),\n",
    "    Dropout(dropout),\n",
    "    Dense(len(set(y_train)), activation='softmax', kernel_regularizer=keras.regularizers.l2(l2_reg))\n",
    "])\n",
    "\n",
    "baseline_model.compile(\n",
    "    optimizer=keras.optimizers.Adam(learning_rate=learning_rate),\n",
    "    loss='sparse_categorical_crossentropy',\n",
    "    metrics=['accuracy']\n",
    ")\n",
    "\n",
    "baseline_params = baseline_model.count_params()\n",
    "baseline_model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f4d6bcf",
   "metadata": {},
   "source": [
    "### 3.2. Define training callbacks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e1fb763",
   "metadata": {},
   "outputs": [],
   "source": [
    "callbacks = [\n",
    "    ModelCheckpoint(\n",
    "        '../models/activity_baseline_rnn_classifier.keras',\n",
    "        save_best_only=True,\n",
    "        monitor='val_accuracy',\n",
    "        mode='max',\n",
    "        verbose=verbose\n",
    "    ),\n",
    "    EarlyStopping(\n",
    "        monitor='val_loss',\n",
    "        patience=early_stopping_patience,\n",
    "        restore_best_weights=True,\n",
    "        verbose=verbose\n",
    "    ),\n",
    "    TensorBoard(\n",
    "        log_dir='../logs/activity_baseline_rnn_classifier',\n",
    "        histogram_freq=1,\n",
    "        write_graph=True\n",
    "    )\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9cae247",
   "metadata": {},
   "source": [
    "### 3.3. Calculate class weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f10adb3a",
   "metadata": {},
   "outputs": [],
   "source": [
    "class_weights = compute_class_weight('balanced', classes=np.unique(y_train), y=y_train)\n",
    "class_weight_dict = dict(enumerate(class_weights))\n",
    "print(f'Class weights: {class_weight_dict}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0791aaeb",
   "metadata": {},
   "source": [
    "### 3.4. Train the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9fb97944",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "# Training with class weighting and early stopping\n",
    "history = baseline_model.fit(\n",
    "    X_train, y_train,\n",
    "    validation_data=(X_test, y_test),\n",
    "    epochs=epochs,\n",
    "    batch_size=batch_size,\n",
    "    class_weight=class_weight_dict,\n",
    "    callbacks=callbacks,\n",
    "    verbose=verbose\n",
    ")\n",
    "\n",
    "print()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c654a0e",
   "metadata": {},
   "source": [
    "### 3.5. Baseline model training curves"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "afccdc5f",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(1, 2, figsize=(10, 4))\n",
    "\n",
    "axes[0].plot(history.history['loss'], label='Train')\n",
    "axes[0].plot(history.history['val_loss'], label='Validation')\n",
    "axes[0].set_title('Loss')\n",
    "axes[0].set_xlabel('Epoch')\n",
    "axes[0].set_ylabel('Loss')\n",
    "axes[0].legend()\n",
    "\n",
    "axes[1].plot(history.history['accuracy'], label='Train')\n",
    "axes[1].plot(history.history['val_accuracy'], label='Validation')\n",
    "axes[1].set_title('Accuracy')\n",
    "axes[1].set_xlabel('Epoch')\n",
    "axes[1].set_ylabel('Accuracy')\n",
    "axes[1].legend()\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72de8d3b",
   "metadata": {},
   "source": [
    "## 4. Baseline evaluation\n",
    "\n",
    "Record the baseline metrics below so you can compare them to your CNN-RNN model later.\n",
    "\n",
    "### 4.1. Make test set predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7342f545",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load best model and get predictions\n",
    "baseline_model = keras.saving.load_model('../models/activity_baseline_rnn_classifier.keras')\n",
    "\n",
    "# Get predicted probabilities and classes\n",
    "y_prob_baseline = baseline_model.predict(X_test)\n",
    "y_pred_baseline = y_prob_baseline.argmax(axis=1)\n",
    "\n",
    "# Class labels\n",
    "class_names = ['-2', '-1', '0', '1', '2']\n",
    "n_classes = len(class_names)\n",
    "\n",
    "print(f'Test samples: {len(y_test)}')\n",
    "print(f'Prediction distribution:\\n')\n",
    "print(f'  Actual:    {Counter(y_test)}')\n",
    "print(f'  Predicted: {Counter(y_pred_baseline)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "408d5b20",
   "metadata": {},
   "source": [
    "### 4.2. Per-class accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89df382c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Per-class accuracy table\n",
    "per_class_stats = []\n",
    "\n",
    "for i, class_name in enumerate(class_names):\n",
    "    mask = y_test == i\n",
    "    n_samples = mask.sum()\n",
    "    n_correct = (y_pred_baseline[mask] == i).sum()\n",
    "    acc = n_correct / n_samples if n_samples > 0 else 0\n",
    "    \n",
    "    per_class_stats.append({\n",
    "        'Class': class_name,\n",
    "        'Samples': n_samples,\n",
    "        'Correct': n_correct,\n",
    "        'Accuracy': f'{acc:.1%}'\n",
    "    })\n",
    "\n",
    "# Add overall accuracy\n",
    "baseline_acc = (y_pred_baseline == y_test).mean()\n",
    "\n",
    "per_class_stats.append({\n",
    "    'Class': 'Overall',\n",
    "    'Samples': len(y_test),\n",
    "    'Correct': (y_pred_baseline == y_test).sum(),\n",
    "    'Accuracy': f'{baseline_acc:.1%}'\n",
    "})\n",
    "\n",
    "accuracy_df = pd.DataFrame(per_class_stats)\n",
    "accuracy_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "296a3976",
   "metadata": {},
   "source": [
    "### 4.3. Confusion matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "02d44a72",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot confusion matrix\n",
    "fig, ax = plt.subplots(figsize=(8, 6))\n",
    "\n",
    "ax.set_title('Confusion matrix')\n",
    "cm = confusion_matrix(y_test, y_pred_baseline)\n",
    "\n",
    "sns.heatmap(\n",
    "    cm, \n",
    "    annot=True, fmt='d', cmap='Blues', \n",
    "    xticklabels=class_names, yticklabels=class_names, ax=ax\n",
    ")\n",
    "\n",
    "ax.set_xlabel('Predicted')\n",
    "ax.set_ylabel('Actual')\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7895bdbc",
   "metadata": {},
   "source": [
    "### 4.4. Predicted probability distributions by class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "91f28172",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predicted probability distributions for each true class\n",
    "fig, axes = plt.subplots(1, n_classes, figsize=(10, 2.5))\n",
    "\n",
    "plt.suptitle('Predicted probability distributions by true class', y=1.02)\n",
    "\n",
    "for i, class_name in enumerate(class_names):\n",
    "\n",
    "    # Get predictions for samples of this true class\n",
    "    mask = y_test == i\n",
    "    probs_for_class = y_prob_baseline[mask]\n",
    "    \n",
    "    # Plot distribution of predicted probabilities for each predicted class\n",
    "    ax = axes[i]\n",
    "    ax.boxplot([probs_for_class[:, j] for j in range(n_classes)], tick_labels=class_names)\n",
    "    ax.set_title(f'True class: {class_name}')\n",
    "    ax.set_xlabel('Predicted class')\n",
    "    ax.set_ylabel('Probability')\n",
    "    ax.set_ylim(0, 1)\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fa8cfff",
   "metadata": {},
   "source": [
    "### 4.5. Evaluation curves"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a0b7d21",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Binarize labels for multiclass ROC/PR\n",
    "y_test_bin = label_binarize(y_test, classes=range(n_classes))\n",
    "\n",
    "# Compute ROC and PR curves for each class\n",
    "fig, axes = plt.subplots(1, 2, figsize=(10, 4))\n",
    "\n",
    "# ROC curves\n",
    "ax_roc = axes[0]\n",
    "ax_roc.set_title('ROC curves (one-vs-rest)')\n",
    "\n",
    "for i, class_name in enumerate(class_names):\n",
    "    fpr, tpr, _ = roc_curve(y_test_bin[:, i], y_prob_baseline[:, i])\n",
    "    roc_auc = auc(fpr, tpr)\n",
    "    ax_roc.plot(fpr, tpr, label=f'Class {class_name} (AUC={roc_auc:.2f})')\n",
    "\n",
    "ax_roc.plot([0, 1], [0, 1], 'k--', alpha=0.5)\n",
    "ax_roc.set_xlabel('False positive rate')\n",
    "ax_roc.set_ylabel('True positive rate')\n",
    "ax_roc.legend(loc='lower right')\n",
    "ax_roc.set_xlim([0, 1])\n",
    "ax_roc.set_ylim([0, 1.05])\n",
    "\n",
    "# PR curves\n",
    "ax_pr = axes[1]\n",
    "ax_pr.set_title('Precision-recall curves (one-vs-rest)')\n",
    "\n",
    "for i, class_name in enumerate(class_names):\n",
    "    precision, recall, _ = precision_recall_curve(y_test_bin[:, i], y_prob_baseline[:, i])\n",
    "    pr_auc = auc(recall, precision)\n",
    "    ax_pr.plot(recall, precision, label=f'Class {class_name} (AUC={pr_auc:.2f})')\n",
    "\n",
    "ax_pr.set_xlabel('Recall')\n",
    "ax_pr.set_ylabel('Precision')\n",
    "ax_pr.set_xlim([0, 1])\n",
    "ax_pr.set_ylim([0, 1.05])\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a59b2536",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 5. Activity: Build a CNN-RNN hybrid model\n",
    "\n",
    "Now it's your turn! Modify the model architecture to include **convolutional layers** before the recurrent layers.\n",
    "\n",
    "### Why add convolutions?\n",
    "\n",
    "- **Local pattern detection**: Conv1D layers can detect n-gram patterns (e.g., \"not good\", \"very happy\") that are important for sentiment\n",
    "- **Dimensionality reduction**: Convolutions can reduce sequence length before the RNN, making training faster\n",
    "- **Feature extraction**: The CNN extracts local features, then the RNN captures long-range dependencies\n",
    "\n",
    "### Hints\n",
    "\n",
    "1. **Import the layers you'll need** - look at `tensorflow.keras.layers` for `Conv1D`, `MaxPooling1D`, or `GlobalMaxPooling1D`\n",
    "\n",
    "2. **Where to add convolutions** - think about the data flow:\n",
    "   ```\n",
    "   Embedding → ??? → GRU → Dense\n",
    "   ```\n",
    "   \n",
    "3. **Conv1D parameters to consider**:\n",
    "   - `filters`: How many feature detectors? (try 64-128)\n",
    "   - `kernel_size`: What n-gram size? (try 3-5 for trigrams to 5-grams)\n",
    "   - `activation`: What activation function works well with text?\n",
    "   - `padding`: Should the output be the same length as input?\n",
    "\n",
    "4. **Pooling options**:\n",
    "   - `MaxPooling1D(pool_size=2)`: Reduces sequence length by half\n",
    "   - `GlobalMaxPooling1D()`: Collapses to a single vector (use this if you want to skip the RNN entirely for comparison)\n",
    "\n",
    "5. **Architecture ideas to try**:\n",
    "   - Single conv layer before the GRU\n",
    "   - Multiple conv layers with increasing filter sizes\n",
    "   - Parallel conv layers with different kernel sizes (requires Functional API)\n",
    "\n",
    "### Your task\n",
    "\n",
    "In the cell below, build your CNN-RNN model. Start simple and iterate!\n",
    "\n",
    "### 5.1. Define the CNN-RNN model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ff1c7488",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: Import any additional layers you need\n",
    "\n",
    "# TODO: Build your CNN-RNN model\n",
    "# Hint: Start with the same structure as the baseline, then add conv layers\n",
    "\n",
    "cnn_rnn_model = Sequential([\n",
    "    Input(shape=(max_len,)),\n",
    "    Embedding(\n",
    "        input_dim=vocab_size, \n",
    "        output_dim=embedding_dim,\n",
    "        embeddings_initializer=glove_initializer,\n",
    "        trainable=True  # Set to True to make embeddings trainable\n",
    "    ),\n",
    "    SpatialDropout1D(dropout),\n",
    "    \n",
    "    # TODO: Add your convolutional layer(s) here\n",
    "    # Conv1D(filters=???, kernel_size=???, activation=???, padding=???),\n",
    "    \n",
    "    # Keep or modify the recurrent layers\n",
    "    Bidirectional(GRU(hidden_dim, return_sequences=True, name='GRU_1'), name='Bidirectional_1'),\n",
    "    Bidirectional(GRU(hidden_dim, return_sequences=False, name='GRU_2'), name='Bidirectional_2'),\n",
    "    Dropout(dropout),\n",
    "    Dense(len(set(y_train)), activation='softmax', kernel_regularizer=keras.regularizers.l2(l2_reg))\n",
    "])\n",
    "\n",
    "cnn_rnn_model.compile(\n",
    "    optimizer=tf.keras.optimizers.Adam(learning_rate=learning_rate),\n",
    "    loss='sparse_categorical_crossentropy',\n",
    "    metrics=['accuracy']\n",
    ")\n",
    "\n",
    "cnn_rnn_params = cnn_rnn_model.count_params()\n",
    "cnn_rnn_model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f472a3ee",
   "metadata": {},
   "source": [
    "### 5.2. Define training callbacks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d3f18542",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define callbacks for CNN-RNN model (saves to different file)\n",
    "cnn_rnn_callbacks = [\n",
    "    ModelCheckpoint(\n",
    "        '../models/cnn_rnn_classifier.keras',\n",
    "        save_best_only=True,\n",
    "        monitor='val_accuracy',\n",
    "        mode='max',\n",
    "        verbose=verbose\n",
    "    ),\n",
    "    EarlyStopping(\n",
    "        monitor='val_loss',\n",
    "        patience=early_stopping_patience,\n",
    "        restore_best_weights=True,\n",
    "        verbose=verbose\n",
    "    ),\n",
    "    TensorBoard(\n",
    "        log_dir='../logs/cnn-rnn_classifier',\n",
    "        histogram_freq=1,\n",
    "        write_graph=True\n",
    "    )\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4dfef75",
   "metadata": {},
   "source": [
    "### 5.3. Train your CNN-RNN model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "35d2245e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define callbacks for CNN-RNN model (saves to different file)\n",
    "cnn_rnn_callbacks = [\n",
    "    ModelCheckpoint(\n",
    "        '../models/cnn_rnn_classifier.keras',\n",
    "        save_best_only=True,\n",
    "        monitor='val_accuracy',\n",
    "        verbose=1\n",
    "    ),\n",
    "    EarlyStopping(\n",
    "        monitor='val_loss',\n",
    "        patience=5,\n",
    "        restore_best_weights=True,\n",
    "        verbose=1\n",
    "    )\n",
    "]\n",
    "\n",
    "# Train the CNN-RNN model\n",
    "cnn_rnn_history = cnn_rnn_model.fit(\n",
    "    X_train, y_train,\n",
    "    validation_data=(X_test, y_test),\n",
    "    epochs=30,\n",
    "    batch_size=64,\n",
    "    class_weight=class_weight_dict,\n",
    "    callbacks=cnn_rnn_callbacks\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2e7ad0a",
   "metadata": {},
   "source": [
    "### 5.4. CNN-RNN model learning curves"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18805323",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(1, 2, figsize=(10, 4))\n",
    "\n",
    "axes[0].plot(history.history['loss'], label='Train')\n",
    "axes[0].plot(history.history['val_loss'], label='Validation')\n",
    "axes[0].set_title('Loss')\n",
    "axes[0].set_xlabel('Epoch')\n",
    "axes[0].set_ylabel('Loss')\n",
    "axes[0].legend()\n",
    "\n",
    "axes[1].plot(history.history['accuracy'], label='Train')\n",
    "axes[1].plot(history.history['val_accuracy'], label='Validation')\n",
    "axes[1].set_title('Accuracy')\n",
    "axes[1].set_xlabel('Epoch')\n",
    "axes[1].set_ylabel('Accuracy')\n",
    "axes[1].legend()\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7951f272",
   "metadata": {},
   "source": [
    "### 5.5. Evaluate your CNN-RNN model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3bc6b841",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load best CNN-RNN model and get predictions\n",
    "cnn_rnn_model = keras.saving.load_model('../models/cnn_rnn_classifier.keras')\n",
    "\n",
    "# Get predicted probabilities and classes\n",
    "y_prob_cnn_rnn = cnn_rnn_model.predict(X_test)\n",
    "y_pred_cnn_rnn = y_prob_cnn_rnn.argmax(axis=1)\n",
    "\n",
    "# Calculate accuracy\n",
    "cnn_rnn_acc = (y_pred_cnn_rnn == y_test).mean()\n",
    "print(f'CNN-RNN model accuracy: {cnn_rnn_acc:.1%}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5dee91fe",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 6. Compare models\n",
    "\n",
    "Now let's compare your CNN-RNN model to the baseline GRU model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "03720dce",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create comparison dataframe\n",
    "comparison = pd.DataFrame({\n",
    "    'Model': ['Baseline GRU', 'CNN-RNN'],\n",
    "    'Accuracy': [f'{baseline_acc:.1%}', f'{cnn_rnn_acc:.1%}'],\n",
    "    'Parameters': [baseline_params, cnn_rnn_params]\n",
    "})\n",
    "\n",
    "print('Model comparison:')\n",
    "comparison"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "53cf2d4d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Side-by-side confusion matrices\n",
    "fig, axes = plt.subplots(1, 2, figsize=(14, 5))\n",
    "\n",
    "for ax, y_pred, title in zip(\n",
    "    axes, \n",
    "    [y_pred_baseline, y_pred_cnn_rnn], \n",
    "    ['Baseline GRU', 'CNN-RNN']\n",
    "):\n",
    "    cm = confusion_matrix(y_test, y_pred)\n",
    "    sns.heatmap(\n",
    "        cm, annot=True, fmt='d', cmap='Blues',\n",
    "        xticklabels=class_names, yticklabels=class_names, ax=ax\n",
    "    )\n",
    "    ax.set_title(f'{title} (Acc: {(y_pred == y_test).mean():.1%})')\n",
    "    ax.set_xlabel('Predicted')\n",
    "    ax.set_ylabel('Actual')\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d87f777",
   "metadata": {},
   "source": [
    "## 7. Reflection\n",
    "\n",
    "Answer the following questions:\n",
    "\n",
    "1. **Did your CNN-RNN model outperform the baseline?** Why or why not?\n",
    "\n",
    "2. **How did the number of parameters change?** Is there a trade-off between model complexity and performance?\n",
    "\n",
    "3. **Which classes improved the most?** Look at the per-class accuracy changes.\n",
    "\n",
    "4. **What other architectures could you try?** Consider:\n",
    "   - Multiple conv layers with different kernel sizes\n",
    "   - Using only CNN (no RNN) with GlobalMaxPooling\n",
    "   - LSTM instead of GRU\n",
    "   - Attention mechanisms"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d23f8d6",
   "metadata": {},
   "source": [
    "### Your notes\n",
    "\n",
    "*Write your observations and answers here...*"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}