{ "cells": [ { "cell_type": "markdown", "id": "7ecf4494", "metadata": {}, "source": [ "# Lesson 23: MovieLens recommendation system demonstration\n", "\n", "\n", "\n", "This notebook uses the MovieLens 100K dataset to demonstrate core recommendation system techniques.\n", "\n", "\n", "\n", "Dataset source: [MovieLens 100K](https://grouplens.org/datasets/movielens/100k/)\n", "\n", "\n", "\n", "1. **Notebook set up**\n", "\n", "\n", "\n", "2. **Collaborative filtering**\n", "\n", " - Memory-based: item-item cosine similarity\n", "\n", " - Model-based: matrix factorization with SVD\n", "\n", "\n", "\n", "3. **Content-based filtering**: genre cosine similarity\n", "\n", "\n", "\n", "4. **Hybrid filtering**: weighted combination of both approaches\n", "\n", "\n", "\n", "5. **Evaluation**: GetTopN and hit rate\n", "\n", "\n", "\n", "6. **Export for web app deployment**\n", "\n", "\n", "\n", "7. **Challenges**: cold start, implicit vs explicit feedback\n", "\n", "\n", "\n", "8. **More advanced techniques**\n", "\n", "\n", "\n", "## 1. Notebook set up\n", "\n", "\n", "\n", "### Imports" ] }, { "cell_type": "code", "execution_count": 1, "id": "d79cd983", "metadata": {}, "outputs": [], "source": [ "import io\n", "import os\n", "import pickle\n", "import zipfile\n", "\n", "import matplotlib.pyplot as plt\n", "import numpy as np\n", "import pandas as pd\n", "import requests\n", "\n", "from sklearn.decomposition import TruncatedSVD\n", "from sklearn.metrics.pairwise import cosine_similarity" ] }, { "cell_type": "markdown", "id": "b9031a07", "metadata": {}, "source": [ "### Dataset\n", "\n", "The [MovieLens 100K dataset](https://grouplens.org/datasets/movielens/100k/) contains 100,000 ratings given by 943 users to 1,682 movies on a 1-5 star scale. It was collected by the GroupLens Research Project at the University of Minnesota and is a standard benchmark for recommendation system research.\n", "\n", "The dataset has two tables:\n", "- **Ratings**: one row per (user, movie) pair with a 1-5 star rating.\n", "- **Movies**: one row per movie with title and 19 binary genre flags." ] }, { "cell_type": "code", "execution_count": 2, "id": "ff212d3e", "metadata": {}, "outputs": [], "source": [ "# Use Python's requests library to get zip archive of data\n", "url = 'https://files.grouplens.org/datasets/movielens/ml-100k.zip'\n", "response = requests.get(url)\n", "\n", "# Create read-only zipfile object in memory from response content\n", "z = zipfile.ZipFile(io.BytesIO(response.content))\n", "\n", "# Load ratings\n", "ratings = pd.read_csv(\n", " z.open('ml-100k/u.data'),\n", " sep='\\t',\n", " names=['user_id', 'movie_id', 'rating', 'timestamp']\n", ").drop(columns=['timestamp'])\n", "\n", "# Load genre/name list\n", "genre_names = pd.read_csv(\n", " z.open('ml-100k/u.genre'),\n", " sep='|',\n", " names=['genre', 'genre_id'],\n", " encoding='latin-1'\n", ")['genre'].tolist()" ] }, { "cell_type": "code", "execution_count": 3, "id": "07936f7a", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
user_idmovie_idrating
01962423
11863023
2223771
3244512
41663461
\n", "
" ], "text/plain": [ " user_id movie_id rating\n", "0 196 242 3\n", "1 186 302 3\n", "2 22 377 1\n", "3 244 51 2\n", "4 166 346 1" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Example ratings data\n", "ratings.head()" ] }, { "cell_type": "code", "execution_count": 4, "id": "cbd543e9", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['unknown',\n", " 'Action',\n", " 'Adventure',\n", " 'Animation',\n", " \"Children's\",\n", " 'Comedy',\n", " 'Crime',\n", " 'Documentary',\n", " 'Drama',\n", " 'Fantasy']" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# First 10 genre names\n", "genre_names[:10]" ] }, { "cell_type": "code", "execution_count": 5, "id": "63339afd", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Ratings: 100,000 rows\n", "Movies: 1,682 rows\n", "Users: 943\n", "Rating scale: 1-5\n" ] } ], "source": [ "# Load movie data\n", "movie_cols = ['movie_id', 'title', 'release_date', 'video_release', 'imdb_url'] + genre_names\n", "\n", "movies = pd.read_csv(\n", " z.open('ml-100k/u.item'),\n", " sep='|',\n", " names=movie_cols,\n", " encoding='latin-1'\n", ")[['movie_id', 'title'] + genre_names]\n", "\n", "print(f'Ratings: {len(ratings):,} rows')\n", "print(f'Movies: {len(movies):,} rows')\n", "print(f'Users: {ratings[\"user_id\"].nunique():,}')\n", "print(f'Rating scale: {ratings[\"rating\"].min()}-{ratings[\"rating\"].max()}')" ] }, { "cell_type": "code", "execution_count": 6, "id": "071fb27b", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
user_idmovie_idrating
01962423
11863023
2223771
3244512
41663461
\n", "
" ], "text/plain": [ " user_id movie_id rating\n", "0 196 242 3\n", "1 186 302 3\n", "2 22 377 1\n", "3 244 51 2\n", "4 166 346 1" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ratings.head()" ] }, { "cell_type": "code", "execution_count": 7, "id": "d208be45", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
01234
movie_id12345
titleToy Story (1995)GoldenEye (1995)Four Rooms (1995)Get Shorty (1995)Copycat (1995)
unknown00000
Action01010
Adventure01000
Animation10000
Children's10000
Comedy10010
Crime00001
Documentary00000
Drama00011
Fantasy00000
Film-Noir00000
Horror00000
Musical00000
Mystery00000
Romance00000
Sci-Fi00000
Thriller01101
War00000
Western00000
\n", "
" ], "text/plain": [ " 0 1 2 \\\n", "movie_id 1 2 3 \n", "title Toy Story (1995) GoldenEye (1995) Four Rooms (1995) \n", "unknown 0 0 0 \n", "Action 0 1 0 \n", "Adventure 0 1 0 \n", "Animation 1 0 0 \n", "Children's 1 0 0 \n", "Comedy 1 0 0 \n", "Crime 0 0 0 \n", "Documentary 0 0 0 \n", "Drama 0 0 0 \n", "Fantasy 0 0 0 \n", "Film-Noir 0 0 0 \n", "Horror 0 0 0 \n", "Musical 0 0 0 \n", "Mystery 0 0 0 \n", "Romance 0 0 0 \n", "Sci-Fi 0 0 0 \n", "Thriller 0 1 1 \n", "War 0 0 0 \n", "Western 0 0 0 \n", "\n", " 3 4 \n", "movie_id 4 5 \n", "title Get Shorty (1995) Copycat (1995) \n", "unknown 0 0 \n", "Action 1 0 \n", "Adventure 0 0 \n", "Animation 0 0 \n", "Children's 0 0 \n", "Comedy 1 0 \n", "Crime 0 1 \n", "Documentary 0 0 \n", "Drama 1 1 \n", "Fantasy 0 0 \n", "Film-Noir 0 0 \n", "Horror 0 0 \n", "Musical 0 0 \n", "Mystery 0 0 \n", "Romance 0 0 \n", "Sci-Fi 0 0 \n", "Thriller 0 1 \n", "War 0 0 \n", "Western 0 0 " ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "movies.head().T" ] }, { "cell_type": "code", "execution_count": 8, "id": "7b343fdb", "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeoAAAEiCAYAAAA21pHjAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjksIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvJkbTWQAAAAlwSFlzAAAPYQAAD2EBqD+naQAAKstJREFUeJzt3XtclHW+B/DPcJnhflFuXriTIhBeUDhTWV5INHLFPOWyZui65gUs08zlnJO37YRW2+YqYZ0S2nPOZtpJc01RRJEt8YZQgkq6YbjKRVMYJRhg+J0/XJ51BBUGdH7I5/16zeu1z/N853m+z8+Nzzy3GZUQQoCIiIikZGHuBoiIiOj2GNREREQSY1ATERFJjEFNREQkMQY1ERGRxBjUREREEmNQExERSYxBTUREJDEGNRERkcQY1ETdwIoVK6BSqczdhpFRo0Zh1KhRyvS5c+egUqmQkZFxz7edkZEBlUqFc+fOKfP8/Pzw9NNP3/NtA0BOTg5UKhVycnLuy/aoZ2NQE5mgJShaXlZWVujXrx9mzJiBCxcumLTOn3/+GStWrOhxf/zff//9+xLuppC5N+o5VPyub6KOy8jIwMyZM7Fq1Sr4+/ujvr4ehw4dQkZGBvz8/FBUVAQbG5sOrfPy5ctwd3fH8uXLsWLFCqNlTU1NaGpq6vA676WWo+mWDxZCCOj1elhbW8PS0rLd6wkLC4Obm1uHPqAYDAY0NjZCo9EoZxr8/PwQFhaGHTt2tHs9pvbW3NyMhoYGqNVqWFjweIfuLStzN0DUnU2YMAHDhw8HAPzmN7+Bm5sb1qxZg+3bt+O5557rsu1YWVnBykru/1xVKtU9/yBRW1sLe3t7WFpadujDQFezsLCQ6kMTPdj4UZCoC40cORIA8Le//U2Z19DQgGXLliEiIgLOzs6wt7fHyJEjsX//fqXm3LlzcHd3BwCsXLlSOaXecmTd1jVqlUqFpKQkbNu2DWFhYdBoNAgNDUVmZmarvnJycjB8+HDY2NggMDAQH3zwQYeue3/44YcIDAyEra0tIiMj8de//rVVTVvXqCsqKjBz5kz0798fGo0Gffr0waRJk5Rry35+figuLsaBAweUfW45Um+5vHDgwAHMnz8fHh4e6N+/v9Gym69Rt9izZw+GDBkCGxsbhISE4IsvvjBafrv9vnWdd+rtdteot2zZgoiICNja2sLNzQ3PP/98q0shM2bMgIODAy5cuIC4uDg4ODjA3d0dr776KgwGw23+Bagnk/sjOlE30/JH3tXVVZmn0+nw0UcfIT4+HrNnz8a1a9fw8ccfIyYmBkeOHMGQIUPg7u6OtLQ0zJs3D5MnT8YzzzwDAAgPD7/j9r7++mt88cUXmD9/PhwdHfHHP/4RU6ZMQVlZGXr37g0AKCgowPjx49GnTx+sXLkSBoMBq1atUj4Y3M3HH3+MOXPm4JFHHsHChQvxww8/4Be/+AV69eoFb2/vO753ypQpKC4uxoIFC+Dn54eqqipkZWWhrKwMfn5+eO+997BgwQI4ODjg3//93wEAnp6eRuuYP38+3N3dsWzZMtTW1t5xe2fOnMHUqVMxd+5cJCQkID09Hc8++ywyMzPx5JNPtmt/W7Snt5u1XA4ZMWIEUlJSUFlZibVr1+Kbb75BQUEBXFxclFqDwYCYmBhERUXhnXfewd69e/H73/8egYGBmDdvXof6pB5AEFGHpaenCwBi79694tKlS+L8+fPi888/F+7u7kKj0Yjz588rtU1NTUKv1xu9/+rVq8LT01P8+te/VuZdunRJABDLly9vtb3ly5eLW/9zBSDUarU4e/asMu/bb78VAMS6deuUeRMnThR2dnbiwoULyrwzZ84IKyurVuu8VUNDg/Dw8BBDhgwx2ocPP/xQABBPPPGEMq+0tFQAEOnp6co+AhBvv/32HbcRGhpqtJ4WLWP82GOPiaampjaXlZaWKvN8fX0FAPF///d/yryamhrRp08fMXToUGVeW2N5u3Xerrf9+/cLAGL//v1CiH+OU1hYmKirq1PqduzYIQCIZcuWKfMSEhIEALFq1SqjdQ4dOlRERES02hYRT30TdUJ0dDTc3d3h7e2Nf/3Xf4W9vT22b9+unKIFAEtLS6jVagA3bkK6cuUKmpqaMHz4cBw/frzT2w8MDFSmw8PD4eTkhB9++AHAjSO3vXv3Ii4uDn379lXqgoKCMGHChLuu/9ixY6iqqsLcuXOVfQBunL51dna+43ttbW2hVquRk5ODq1evdnTXFLNnz2739ei+ffti8uTJyrSTkxNeeOEFFBQUoKKiwuQe7qZlnObPn2907To2NhbBwcH46quvWr1n7ty5RtMjR45U/t2IbsagJuqE1NRUZGVl4fPPP8dTTz2Fy5cvQ6PRtKr75JNPEB4eDhsbG/Tu3Rvu7u746quvUFNT06nt+/j4tJrn6uqqBGNVVRXq6uoQFBTUqq6tebf68ccfAQAPPfSQ0Xxra2sEBATc8b0ajQZr1qzBrl274OnpiccffxxvvfVWhwPT39+/3bVBQUGtrj8PGDAAANq8nt1VWsZp4MCBrZYFBwcry1vY2Ni0uvRw878b0c0Y1ESdEBkZiejoaEyZMgXbt29HWFgYfvWrX+H69etKzf/8z/9gxowZCAwMxMcff4zMzExkZWVhzJgxaG5u7tT2b3ekKSR56nLhwoX4/vvvkZKSAhsbG7z++usYNGgQCgoK2r0OW1vbLu3pdjfQ3c8bucx5xzp1Pwxqoi5iaWmJlJQUXLx4EevXr1fmf/755wgICMAXX3yB6dOnIyYmBtHR0aivrzd6/7345jEPDw/Y2Njg7NmzrZa1Ne9Wvr6+AG7cpHWzxsZGlJaWtquHwMBALF68GHv27EFRUREaGhrw+9//Xlnelft99uzZVh9Svv/+ewA37uIG/nmjX3V1tVHdrUe9HemtZZxKSkpaLSspKVGWE5mCQU3UhUaNGoXIyEi89957ShC3HD3dHCCHDx9GXl6e0Xvt7OwAtA6QzrC0tER0dDS2bduGixcvKvPPnj2LXbt23fX9w4cPh7u7OzZs2ICGhgZlfkZGxl37/Pnnn1t9GAkMDISjoyP0er0yz97evsv2+eLFi9i6dasyrdPp8Kc//QlDhgyBl5eX0gMA5ObmKnW1tbX45JNPWq2vvb0NHz4cHh4e2LBhg9G+7dq1C6dOnUJsbKypu0TEx7OIutqSJUvw7LPPIiMjA3PnzsXTTz+NL774ApMnT0ZsbCxKS0uxYcMGhISEGJ0it7W1RUhICD777DMMGDAAvXr1QlhYGMLCwjrVz4oVK7Bnzx48+uijmDdvHgwGA9avX4+wsDAUFhbe8b3W1tZ44403MGfOHIwZMwZTp05FaWkp0tPT73qN+vvvv8fYsWPx3HPPISQkBFZWVti6dSsqKyvxy1/+UqmLiIhAWloa3njjDQQFBcHDwwNjxowxaV8HDBiAWbNm4ejRo/D09MTGjRtRWVmJ9PR0pWbcuHHw8fHBrFmzsGTJElhaWmLjxo1wd3dHWVmZ0fra25u1tTXWrFmDmTNn4oknnkB8fLzyeJafnx9eeeUVk/aHCAAfzyIyRcujPEePHm21zGAwiMDAQBEYGCiamppEc3OzePPNN4Wvr6/QaDRi6NChYseOHSIhIUH4+voavffgwYMiIiJCqNVqo0e1bvd4VmJiYqvt+/r6ioSEBKN52dnZYujQoUKtVovAwEDx0UcficWLFwsbG5t27e/7778v/P39hUajEcOHDxe5ubniiSeeuOPjWZcvXxaJiYkiODhY2NvbC2dnZxEVFSU2b95stO6KigoRGxsrHB0djR75utMY3+7xrNjYWLF7924RHh4uNBqNCA4OFlu2bGn1/vz8fBEVFSXUarXw8fER7777bpvrvF1vtz6e1eKzzz4TQ4cOFRqNRvTq1UtMmzZN/P3vfzeqSUhIEPb29q16ut1jY0T8rm+iHiouLg7FxcWtrj8TkVx4jZqoB6irqzOaPnPmDHbu3Gn0M5VEJCceURP1AH369MGMGTMQEBCAH3/8EWlpadDr9SgoKGj1jDQRyYU3kxH1AOPHj8enn36KiooKaDQaaLVavPnmmwxpom6AR9REREQS4zVqIiIiiTGoiYiIJMZr1F2kubkZFy9ehKOj4z35KkgiInpwCCFw7do19O3bFxYWdz5mZlB3kYsXL8Lb29vcbRARUTdy/vx5o5/FbQuDuos4OjoCuDHoTk5OZu6GiIhkptPp4O3trWTHnTCou0jL6W4nJycGNRERtUt7LpXyZjIiIiKJMaiJiIgkxqAmIiKSGIOaiIhIYgxqIiIiiTGoiYiIJMagJiIikhifoyYi6kKyf4UwfzCx++ERNRERkcQY1ERERBJjUBMREUmMQU1ERCQxBjUREZHEGNREREQSY1ATERFJjEFNREQkMQY1ERGRxBjUREREEmNQExERSYxBTUREJDEGNRERkcQY1ERERBJjUBMREUmMQU1ERCQxBjUREZHEGNREREQSM2tQp6WlITw8HE5OTnBycoJWq8WuXbuU5fX19UhMTETv3r3h4OCAKVOmoLKy0mgdZWVliI2NhZ2dHTw8PLBkyRI0NTUZ1eTk5GDYsGHQaDQICgpCRkZGq15SU1Ph5+cHGxsbREVF4ciRI/dkn4mIiDrCrEHdv39/rF69Gvn5+Th27BjGjBmDSZMmobi4GADwyiuv4C9/+Qu2bNmCAwcO4OLFi3jmmWeU9xsMBsTGxqKhoQEHDx7EJ598goyMDCxbtkypKS0tRWxsLEaPHo3CwkIsXLgQv/nNb7B7926l5rPPPsOiRYuwfPlyHD9+HIMHD0ZMTAyqqqru32AQERG1RUjG1dVVfPTRR6K6ulpYW1uLLVu2KMtOnTolAIi8vDwhhBA7d+4UFhYWoqKiQqlJS0sTTk5OQq/XCyGEeO2110RoaKjRNqZOnSpiYmKU6cjISJGYmKhMGwwG0bdvX5GSktLuvmtqagQAUVNT07EdJqIHCgCpXySHjmSGNNeoDQYDNm3ahNraWmi1WuTn56OxsRHR0dFKTXBwMHx8fJCXlwcAyMvLw8MPPwxPT0+lJiYmBjqdTjkqz8vLM1pHS03LOhoaGpCfn29UY2FhgejoaKWGiIjIXKzM3cCJEyeg1WpRX18PBwcHbN26FSEhISgsLIRarYaLi4tRvaenJyoqKgAAFRUVRiHdsrxl2Z1qdDod6urqcPXqVRgMhjZrTp8+fdu+9Xo99Hq9Mq3T6Tq240RERO1g9iPqgQMHorCwEIcPH8a8efOQkJCAkydPmrutu0pJSYGzs7Py8vb2NndLRET0ADJ7UKvVagQFBSEiIgIpKSkYPHgw1q5dCy8vLzQ0NKC6utqovrKyEl5eXgAALy+vVneBt0zfrcbJyQm2trZwc3ODpaVlmzUt62hLcnIyampqlNf58+dN2n8iIqI7MXtQ36q5uRl6vR4RERGwtrZGdna2sqykpARlZWXQarUAAK1WixMnThjdnZ2VlQUnJyeEhIQoNTevo6WmZR1qtRoRERFGNc3NzcjOzlZq2qLRaJTHylpeREREXe4+3Nx2W7/97W/FgQMHRGlpqfjuu+/Eb3/7W6FSqcSePXuEEELMnTtX+Pj4iH379oljx44JrVYrtFqt8v6mpiYRFhYmxo0bJwoLC0VmZqZwd3cXycnJSs0PP/wg7OzsxJIlS8SpU6dEamqqsLS0FJmZmUrNpk2bhEajERkZGeLkyZPixRdfFC4uLkZ3k98N7/omIiF41ze1T0cyw6z/ar/+9a+Fr6+vUKvVwt3dXYwdO1YJaSGEqKurE/Pnzxeurq7Czs5OTJ48WZSXlxut49y5c2LChAnC1tZWuLm5icWLF4vGxkajmv3794shQ4YItVotAgICRHp6eqte1q1bJ3x8fIRarRaRkZHi0KFDHdoXBjURCcGgpvbpSGaohBCiY8fg1BadTgdnZ2fU1NTwNDhRD6ZSqczdwh3xT74cOpIZ0l2jJiIion9iUBMREUmMQU1ERCQxBjUREZHEGNREREQSY1ATERFJjEFNREQkMQY1ERGRxBjUREREEmNQExERSczK3A0QkXxk/hpMfgUm9TQ8oiYiIpIYg5qIiEhiDGoiIiKJMaiJiIgkxqAmIiKSGIOaiIhIYgxqIiIiiTGoiYiIJMagJiIikhiDmoiISGIMaiIiIokxqImIiCTGoCYiIpIYg5qIiEhiDGoiIiKJMaiJiIgkxqAmIiKSGIOaiIhIYgxqIiIiiTGoiYiIJMagJiIikhiDmoiISGIMaiIiIokxqImIiCTGoCYiIpIYg5qIiEhiDGoiIiKJMaiJiIgkxqAmIiKSGIOaiIhIYgxqIiIiiTGoiYiIJMagJiIikphJQR0QEICffvqp1fzq6moEBAR0uikiIup5VCqV1C9zMSmoz507B4PB0Gq+Xq/HhQsXOt0UERER3dChoN6+fTu2b98OANi9e7cyvX37dmzduhW/+93v4Ofn1+71paSkYMSIEXB0dISHhwfi4uJQUlJiVFNfX4/ExET07t0bDg4OmDJlCiorK41qysrKEBsbCzs7O3h4eGDJkiVoamoyqsnJycGwYcOg0WgQFBSEjIyMVv2kpqbCz88PNjY2iIqKwpEjR9q9L0RERPeE6ACVSiVUKpWwsLBQ/nfLS61WiwEDBoi//OUv7V5fTEyMSE9PF0VFRaKwsFA89dRTwsfHR1y/fl2pmTt3rvD29hbZ2dni2LFj4l/+5V/EI488oixvamoSYWFhIjo6WhQUFIidO3cKNzc3kZycrNT88MMPws7OTixatEicPHlSrFu3TlhaWorMzEylZtOmTUKtVouNGzeK4uJiMXv2bOHi4iIqKyvbtS81NTUCgKipqWn3/hPJCoC0L9mZe3y68/iZe2zu59h1JDNM2rKfn5+4dOmSKW+9o6qqKgFAHDhwQAghRHV1tbC2thZbtmxRak6dOiUAiLy8PCGEEDt37hQWFhaioqJCqUlLSxNOTk5Cr9cLIYR47bXXRGhoqNG2pk6dKmJiYpTpyMhIkZiYqEwbDAbRt29fkZKS0q7eGdT0IDH3H8TuGjRCyD12so+fucdG1qA26Rp1aWkp3NzcTHnrHdXU1AAAevXqBQDIz89HY2MjoqOjlZrg4GD4+PggLy8PAJCXl4eHH34Ynp6eSk1MTAx0Oh2Ki4uVmpvX0VLTso6Ghgbk5+cb1VhYWCA6OlqpISIiMgcrU9+YnZ2N7OxsVFVVobm52WjZxo0bO7y+5uZmLFy4EI8++ijCwsIAABUVFVCr1XBxcTGq9fT0REVFhVJzc0i3LG9ZdqcanU6Huro6XL16FQaDoc2a06dPt9mvXq+HXq9XpnU6XQf3mIiI6O5MOqJeuXIlxo0bh+zsbFy+fBlXr141epkiMTERRUVF2LRpk0nvv99SUlLg7OysvLy9vc3dEhERPYBMOqLesGEDMjIyMH369C5pIikpCTt27EBubi769++vzPfy8kJDQwOqq6uNjqorKyvh5eWl1Nx6d3bLXeE319x6p3hlZSWcnJxga2sLS0tLWFpatlnTso5bJScnY9GiRcq0TqdjWBMRUZcz6Yi6oaEBjzzySKc3LoRAUlIStm7din379sHf399oeUREBKytrZGdna3MKykpQVlZGbRaLQBAq9XixIkTqKqqUmqysrLg5OSEkJAQpebmdbTUtKxDrVYjIiLCqKa5uRnZ2dlKza00Gg2cnJyMXkRERF3OlLvVXnvtNbFq1SpT3mpk3rx5wtnZWeTk5Ijy8nLl9fPPPys1c+fOFT4+PmLfvn3i2LFjQqvVCq1WqyxveTxr3LhxorCwUGRmZgp3d/c2H89asmSJOHXqlEhNTW3z8SyNRiMyMjLEyZMnxYsvvihcXFyM7ia/E971TQ8SSHCH7e1esjP3+HTn8TP32NzPsbvnj2e99NJLwsXFRTz++OMiKSlJvPLKK0av9rrdYKSnpys1dXV1Yv78+cLV1VXY2dmJyZMni/LycqP1nDt3TkyYMEHY2toKNzc3sXjxYtHY2GhUs3//fjFkyBChVqtFQECA0TZarFu3Tvj4+Ai1Wi0iIyPFoUOH2r0vDGp6kJj7D2J3DRoh5B472cfP3GMja1Cr/jE4HTJ69OjbLlOpVNi3b19HV9nt6XQ6ODs7o6amhqfBqdsz5/ca340Jf7LuK5nHDpB7/HrS2HUkM0y6mWz//v0mNUZEREQdw5+5JCIikphJR9SjR4++4ymKnnjqm4iI6F4wKaiHDBliNN3Y2IjCwkIUFRUhISGhK/oiIiIimBjUf/jDH9qcv2LFCly/fr1TDREREdE/dek16ueff96k7/kmIiKitnVpUOfl5cHGxqYrV0lERNSjmXTq+5lnnjGaFkKgvLwcx44dw+uvv94ljREREZGJQe3s7Gw0bWFhgYEDB2LVqlUYN25clzRGREREJgZ1enp6V/dBREREbTApqFvk5+fj1KlTAIDQ0FAMHTq0S5oiIiKiG0wK6qqqKvzyl79ETk6O8jvR1dXVGD16NDZt2gR3d/eu7JGIiKjHMumu7wULFuDatWsoLi7GlStXcOXKFRQVFUGn0+Gll17q6h6JiIh6LJN+PcvZ2Rl79+7FiBEjjOYfOXIE48aNQ3V1dVf1123w17PoQSLzrxjJ/OtPgNxjB8g9fj1p7DqSGSYdUTc3N8Pa2rrVfGtrazQ3N5uySiIiImqDSUE9ZswYvPzyy7h48aIy78KFC3jllVcwduzYLmuOiIiopzMpqNevXw+dTgc/Pz8EBgYiMDAQ/v7+0Ol0WLduXVf3SERE1GOZdNe3t7c3jh8/jr179+L06dMAgEGDBiE6OrpLmyMiIurpOnREvW/fPoSEhECn00GlUuHJJ5/EggULsGDBAowYMQKhoaH461//eq96JSIi6nE6FNTvvfceZs+e3eYdas7OzpgzZw7efffdLmuOiIiop+tQUH/77bcYP378bZePGzcO+fn5nW6KiIiIbuhQUFdWVrb5WFYLKysrXLp0qdNNERER0Q0dCup+/fqhqKjotsu/++479OnTp9NNERER0Q0dCuqnnnoKr7/+Ourr61stq6urw/Lly/H00093WXNEREQ9XYe+QrSyshLDhg2DpaUlkpKSMHDgQADA6dOnkZqaCoPBgOPHj8PT0/OeNSwrfoUoPUhk/ipHmb8CE5B77AC5x68njV1HMqNDz1F7enri4MGDmDdvHpKTk5WmVSoVYmJikJqa2iNDmoiI6F7p8Bee+Pr6YufOnbh69SrOnj0LIQQeeughuLq63ov+iIiIejSTvpkMAFxdXVv9ehYRERF1LZO+65uIiIjuDwY1ERGRxBjUREREEmNQExERSYxBTUREJDEGNRERkcQY1ERERBJjUBMREUmMQU1ERCQxBjUREZHEGNREREQSY1ATERFJjEFNREQkMQY1ERGRxBjUREREEjP596iJZKZSqczdwh0JIczdAhF1EzyiJiIikhiDmoiISGJmDerc3FxMnDgRffv2hUqlwrZt24yWCyGwbNky9OnTB7a2toiOjsaZM2eMaq5cuYJp06bByckJLi4umDVrFq5fv25U891332HkyJGwsbGBt7c33nrrrVa9bNmyBcHBwbCxscHDDz+MnTt3dvn+EhERdZRZg7q2thaDBw9Gampqm8vfeust/PGPf8SGDRtw+PBh2NvbIyYmBvX19UrNtGnTUFxcjKysLOzYsQO5ubl48cUXleU6nQ7jxo2Dr68v8vPz8fbbb2PFihX48MMPlZqDBw8iPj4es2bNQkFBAeLi4hAXF4eioqJ7t/NERETtISQBQGzdulWZbm5uFl5eXuLtt99W5lVXVwuNRiM+/fRTIYQQJ0+eFADE0aNHlZpdu3YJlUolLly4IIQQ4v333xeurq5Cr9crNUuXLhUDBw5Upp977jkRGxtr1E9UVJSYM2dOu/uvqakRAERNTU2730P3DgCpX7Iz9/hw7Hrm+Jl7bO7n2HUkM6S9Rl1aWoqKigpER0cr85ydnREVFYW8vDwAQF5eHlxcXDB8+HClJjo6GhYWFjh8+LBS8/jjj0OtVis1MTExKCkpwdWrV5Wam7fTUtOyHSIiInOR9vGsiooKAICnp6fRfE9PT2VZRUUFPDw8jJZbWVmhV69eRjX+/v6t1tGyzNXVFRUVFXfcTlv0ej30er0yrdPpOrJ7RERE7SLtEbXsUlJS4OzsrLy8vb3N3RIRET2ApA1qLy8vAEBlZaXR/MrKSmWZl5cXqqqqjJY3NTXhypUrRjVtrePmbdyupmV5W5KTk1FTU6O8zp8/39FdJCIiuitpg9rf3x9eXl7Izs5W5ul0Ohw+fBharRYAoNVqUV1djfz8fKVm3759aG5uRlRUlFKTm5uLxsZGpSYrKwsDBw6Eq6urUnPzdlpqWrbTFo1GAycnJ6MXERFRl+vS29g66Nq1a6KgoEAUFBQIAOLdd98VBQUF4scffxRCCLF69Wrh4uIivvzyS/Hdd9+JSZMmCX9/f1FXV6esY/z48WLo0KHi8OHD4uuvvxYPPfSQiI+PV5ZXV1cLT09PMX36dFFUVCQ2bdok7OzsxAcffKDUfPPNN8LKykq888474tSpU2L58uXC2tpanDhxot37wru+5QIJ7hC900t25h4fjl3PHD9zj839HLuOZIZZ/9X279/f5mAkJCQIIW48ovX6668LT09PodFoxNixY0VJSYnROn766ScRHx8vHBwchJOTk5g5c6a4du2aUc23334rHnvsMaHRaES/fv3E6tWrW/WyefNmMWDAAKFWq0VoaKj46quvOrQvDGq5mPs/6O78x1IIucdPduYen+48fuYem/s5dh3JDNU/Boc6SafTwdnZGTU1NTwNLgH+KEfnyDx+HLvOkXn8etLYdSQzpL1GTURERAxqIiIiqTGoiYiIJMagJiIikhiDmoiISGIMaiIiIokxqImIiCTGoCYiIpIYg5qIiEhiDGoiIiKJMaiJiIgkxqAmIiKSGIOaiIhIYgxqIiIiiTGoiYiIJMagJiIikpiVuRug2+tJP6JORERt4xE1ERGRxBjUREREEmNQExERSYxBTUREJDEGNRERkcQY1ERERBJjUBMREUmMQU1ERCQxBjUREZHEGNREREQSY1ATERFJjEFNREQkMQY1ERGRxBjUREREEmNQExERSYxBTUREJDEGNRERkcQY1ERERBJjUBMREUmMQU1ERCQxBjUREZHEGNREREQSY1ATERFJjEFNREQkMQY1ERGRxBjUREREEmNQExERSYxBTUREJDEGNRERkcQY1LdITU2Fn58fbGxsEBUVhSNHjpi7JSIi6sEY1Df57LPPsGjRIixfvhzHjx/H4MGDERMTg6qqKnO3RkREPRSD+ibvvvsuZs+ejZkzZyIkJAQbNmyAnZ0dNm7caO7WiIioh2JQ/0NDQwPy8/MRHR2tzLOwsEB0dDTy8vLM2BkREfVkVuZuQBaXL1+GwWCAp6en0XxPT0+cPn26Vb1er4der1ema2pqAAA6ne7eNiqRnrSvXY1jZzqOXedw/EzXlWPXsi4hxF1rGdQmSklJwcqVK1vN9/b2NkM35uHs7GzuFrotjp3pOHadw/Ez3b0Yu2vXrt11vQzqf3Bzc4OlpSUqKyuN5ldWVsLLy6tVfXJyMhYtWqRMNzc348qVK+jduzdUKtU977ejdDodvL29cf78eTg5OZm7nW6FY9c5HD/TcexMJ/vYCSFw7do19O3b9661DOp/UKvViIiIQHZ2NuLi4gDcCN/s7GwkJSW1qtdoNNBoNEbzXFxc7kOnnePk5CTl/2m7A45d53D8TMexM53MY9feI3QG9U0WLVqEhIQEDB8+HJGRkXjvvfdQW1uLmTNnmrs1IiLqoRjUN5k6dSouXbqEZcuWoaKiAkOGDEFmZmarG8yIiIjuFwb1LZKSkto81d3daTQaLF++vNXpero7jl3ncPxMx7Ez3YM0dirRnnvDiYiIyCz4hSdEREQSY1ATERFJjEFNREQkMQb1Ay43NxcTJ05E3759oVKpsG3bNnO31G2kpKRgxIgRcHR0hIeHB+Li4lBSUmLutrqFtLQ0hIeHK8+warVa7Nq1y9xtdUurV6+GSqXCwoULzd1Kt7BixQqoVCqjV3BwsLnb6hQG9QOutrYWgwcPRmpqqrlb6XYOHDiAxMREHDp0CFlZWWhsbMS4ceNQW1tr7tak179/f6xevRr5+fk4duwYxowZg0mTJqG4uNjcrXUrR48exQcffIDw8HBzt9KthIaGory8XHl9/fXX5m6pU/h41gNuwoQJmDBhgrnb6JYyMzONpjMyMuDh4YH8/Hw8/vjjZuqqe5g4caLR9H/+538iLS0Nhw4dQmhoqJm66l6uX7+OadOm4b/+67/wxhtvmLudbsXKyqrNr37urnhETdROLb+Q1qtXLzN30r0YDAZs2rQJtbW10Gq15m6n20hMTERsbKzRT+9S+5w5cwZ9+/ZFQEAApk2bhrKyMnO31Ck8oiZqh+bmZixcuBCPPvoowsLCzN1Ot3DixAlotVrU19fDwcEBW7duRUhIiLnb6hY2bdqE48eP4+jRo+ZupduJiopCRkYGBg4ciPLycqxcuRIjR45EUVERHB0dzd2eSRjURO2QmJiIoqKibn+t634aOHAgCgsLUVNTg88//xwJCQk4cOAAw/ouzp8/j5dffhlZWVmwsbExdzvdzs2X+sLDwxEVFQVfX19s3rwZs2bNMmNnpmNQE91FUlISduzYgdzcXPTv39/c7XQbarUaQUFBAICIiAgcPXoUa9euxQcffGDmzuSWn5+PqqoqDBs2TJlnMBiQm5uL9evXQ6/Xw9LS0owddi8uLi4YMGAAzp49a+5WTMagJroNIQQWLFiArVu3IicnB/7+/uZuqVtrbm6GXq83dxvSGzt2LE6cOGE0b+bMmQgODsbSpUsZ0h10/fp1/O1vf8P06dPN3YrJGNQPuOvXrxt9kiwtLUVhYSF69eoFHx8fM3Ymv8TERPz5z3/Gl19+CUdHR1RUVAC48Ruytra2Zu5ObsnJyZgwYQJ8fHxw7do1/PnPf0ZOTg52795t7tak5+jo2Oo+CHt7e/Tu3Zv3R7TDq6++iokTJ8LX1xcXL17E8uXLYWlpifj4eHO3ZjIG9QPu2LFjGD16tDK9aNEiAEBCQgIyMjLM1FX3kJaWBgAYNWqU0fz09HTMmDHj/jfUjVRVVeGFF15AeXk5nJ2dER4ejt27d+PJJ580d2v0gPv73/+O+Ph4/PTTT3B3d8djjz2GQ4cOwd3d3dytmYy/nkVERCQxPkdNREQkMQY1ERGRxBjUREREEmNQExERSYxBTUREJDEGNRERkcQY1ERERBJjUBMREUmMQU1E91xOTg5UKhWqq6vN3QpRt8OgJiLFjBkzoFKpoFKpYG1tDX9/f7z22muor69v9zpGjRqFhQsXGs175JFHlK8TJaKO4Xd9E5GR8ePHIz09HY2NjcjPz0dCQgJUKhXWrFlj8jrVajW8vLy6sEuinoNH1ERkRKPRwMvLC97e3oiLi0N0dDSysrIAAD/99BPi4+PRr18/2NnZ4eGHH8ann36qvHfGjBk4cOAA1q5dqxyZnzt3rtWp74yMDLi4uGD37t0YNGgQHBwcMH78eJSXlyvrampqwksvvQQXFxf07t0bS5cuRUJCAuLi4u7ncBCZHYOaiG6rqKgIBw8ehFqtBgDU19cjIiICX331FYqKivDiiy9i+vTpOHLkCABg7dq10Gq1mD17NsrLy1FeXg5vb+821/3zzz/jnXfewX//938jNzcXZWVlePXVV5Xla9aswf/+7/8iPT0d33zzDXQ6HbZt23bP95lINjz1TURGduzYAQcHBzQ1NUGv18PCwgLr168HAPTr188oTBcsWIDdu3dj8+bNiIyMhLOzM9RqNezs7O56qruxsREbNmxAYGAgACApKQmrVq1Slq9btw7JycmYPHkyAGD9+vXYuXNnV+8ukfQY1ERkZPTo0UhLS0NtbS3+8Ic/wMrKClOmTAEAGAwGvPnmm9i8eTMuXLiAhoYG6PV62NnZdXg7dnZ2SkgDQJ8+fVBVVQUAqKmpQWVlJSIjI5XllpaWiIiIQHNzcyf3kKh74alvIjJib2+PoKAgDB48GBs3bsThw4fx8ccfAwDefvttrF27FkuXLsX+/ftRWFiImJgYNDQ0dHg71tbWRtMqlQpCiC7ZB6IHCYOaiG7LwsIC//Zv/4b/+I//QF1dHb755htMmjQJzz//PAYPHoyAgAB8//33Ru9Rq9UwGAyd2q6zszM8PT1x9OhRZZ7BYMDx48c7tV6i7ohBTUR39Oyzz8LS0hKpqal46KGHkJWVhYMHD+LUqVOYM2cOKisrjer9/Pxw+PBhnDt3DpcvXzb5VPWCBQuQkpKCL7/8EiUlJXj55Zdx9epVqFSqrtgtom6DQU1Ed2RlZYWkpCS89dZbWLx4MYYNG4aYmBiMGjUKXl5erR6XevXVV2FpaYmQkBC4u7ujrKzMpO0uXboU8fHxeOGFF6DVauHg4ICYmBjY2Nh0wV4RdR8qwYtCRNQNNDc3Y9CgQXjuuefwu9/9ztztEN03vOubiKT0448/Ys+ePXjiiSeg1+uxfv16lJaW4le/+pW5WyO6r3jqm4ikZGFhgYyMDIwYMQKPPvooTpw4gb1792LQoEHmbo3ovuKpbyIiIonxiJqIiEhiDGoiIiKJMaiJiIgkxqAmIiKSGIOaiIhIYgxqIiIiiTGoiYiIJMagJiIikhiDmoiISGL/Dwn9PHHjMZjlAAAAAElFTkSuQmCC", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "ratings['rating'].value_counts().sort_index().plot(kind='bar', color='black', figsize=(5, 3))\n", "plt.title('Rating distribution')\n", "plt.xlabel('Rating')\n", "plt.ylabel('Count')\n", "plt.xticks(rotation=0)\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "ca7f1d26", "metadata": {}, "source": [ "## 2. Collaborative filtering\n", "\n", "\n", "\n", "Collaborative filtering recommends items based on the collective behaviour of all users. It does not use item content; it relies entirely on the pattern of who rated what and how highly.\n", "\n", "\n", "\n", "### 2.1. Memory-based collaborative filtering\n", "\n", "\n", "\n", "Memory-based collaborative filtering compares items directly using a similarity metric computed from the rating matrix. No model is trained; all computation happens at query time from the stored similarity matrix.\n", "\n", "\n", "\n", "Similarity between two items is computed with cosine similarity over their rating vectors across all users:\n", "\n", "\n", "\n", "$$\\text{sim}(i, j) = \\frac{r_i \\cdot r_j}{\\|r_i\\| \\|r_j\\|}$$\n", "\n", "\n", "\n", "A value of 1 means identical rating patterns; 0 means no shared users." ] }, { "cell_type": "code", "execution_count": 9, "id": "2b80aaa1", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "User-item matrix shape: (943, 1682)\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
movie_id12345678910...1673167416751676167716781679168016811682
user_id
15.03.04.03.03.05.04.01.05.03.0...0.00.00.00.00.00.00.00.00.00.0
24.00.00.00.00.00.00.00.00.02.0...0.00.00.00.00.00.00.00.00.00.0
30.00.00.00.00.00.00.00.00.00.0...0.00.00.00.00.00.00.00.00.00.0
40.00.00.00.00.00.00.00.00.00.0...0.00.00.00.00.00.00.00.00.00.0
54.03.00.00.00.00.00.00.00.00.0...0.00.00.00.00.00.00.00.00.00.0
\n", "

5 rows × 1682 columns

\n", "
" ], "text/plain": [ "movie_id 1 2 3 4 5 6 7 8 9 10 ... \\\n", "user_id ... \n", "1 5.0 3.0 4.0 3.0 3.0 5.0 4.0 1.0 5.0 3.0 ... \n", "2 4.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.0 ... \n", "3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... \n", "4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... \n", "5 4.0 3.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... \n", "\n", "movie_id 1673 1674 1675 1676 1677 1678 1679 1680 1681 1682 \n", "user_id \n", "1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 \n", "2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 \n", "3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 \n", "4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 \n", "5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 \n", "\n", "[5 rows x 1682 columns]" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Build user-item matrix (rows = users, columns = movies)\n", "user_item_matrix = ratings.pivot_table(\n", " index='user_id',\n", " columns='movie_id',\n", " values='rating'\n", ")\n", "\n", "# Fill unrated entries with 0 for cosine similarity\n", "user_item_filled = user_item_matrix.fillna(0)\n", "\n", "print(f'User-item matrix shape: {user_item_filled.shape}')\n", "user_item_filled.head()" ] }, { "cell_type": "code", "execution_count": 10, "id": "d4214e51", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Item similarity matrix shape: (1682, 1682)\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
movie_id12345678910...1673167416751676167716781679168016811682
movie_id
11.0000000.4023820.3302450.4549380.2867140.1163440.6209790.4811140.4962880.273935...0.0353870.00.0000000.0000000.0353870.00.00.00.0471830.047183
20.4023821.0000000.2730690.5025710.3188360.0835630.3834030.3370020.2552520.171082...0.0000000.00.0000000.0000000.0000000.00.00.00.0782990.078299
30.3302450.2730691.0000000.3248660.2129570.1067220.3729210.2007940.2736690.158104...0.0000000.00.0000000.0000000.0322920.00.00.00.0000000.096875
40.4549380.5025710.3248661.0000000.3342390.0903080.4892830.4902360.4190440.252561...0.0000000.00.0940220.0940220.0376090.00.00.00.0564130.075218
50.2867140.3188360.2129570.3342391.0000000.0372990.3347690.2591610.2724480.055453...0.0000000.00.0000000.0000000.0000000.00.00.00.0000000.094211
\n", "

5 rows × 1682 columns

\n", "
" ], "text/plain": [ "movie_id 1 2 3 4 5 6 \\\n", "movie_id \n", "1 1.000000 0.402382 0.330245 0.454938 0.286714 0.116344 \n", "2 0.402382 1.000000 0.273069 0.502571 0.318836 0.083563 \n", "3 0.330245 0.273069 1.000000 0.324866 0.212957 0.106722 \n", "4 0.454938 0.502571 0.324866 1.000000 0.334239 0.090308 \n", "5 0.286714 0.318836 0.212957 0.334239 1.000000 0.037299 \n", "\n", "movie_id 7 8 9 10 ... 1673 1674 \\\n", "movie_id ... \n", "1 0.620979 0.481114 0.496288 0.273935 ... 0.035387 0.0 \n", "2 0.383403 0.337002 0.255252 0.171082 ... 0.000000 0.0 \n", "3 0.372921 0.200794 0.273669 0.158104 ... 0.000000 0.0 \n", "4 0.489283 0.490236 0.419044 0.252561 ... 0.000000 0.0 \n", "5 0.334769 0.259161 0.272448 0.055453 ... 0.000000 0.0 \n", "\n", "movie_id 1675 1676 1677 1678 1679 1680 1681 1682 \n", "movie_id \n", "1 0.000000 0.000000 0.035387 0.0 0.0 0.0 0.047183 0.047183 \n", "2 0.000000 0.000000 0.000000 0.0 0.0 0.0 0.078299 0.078299 \n", "3 0.000000 0.000000 0.032292 0.0 0.0 0.0 0.000000 0.096875 \n", "4 0.094022 0.094022 0.037609 0.0 0.0 0.0 0.056413 0.075218 \n", "5 0.000000 0.000000 0.000000 0.0 0.0 0.0 0.000000 0.094211 \n", "\n", "[5 rows x 1682 columns]" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Transpose so items are rows, then compute pairwise cosine similarity\n", "item_similarity = cosine_similarity(user_item_filled.T)\n", "\n", "item_similarity_df = pd.DataFrame(\n", " item_similarity,\n", " index=user_item_matrix.columns,\n", " columns=user_item_matrix.columns\n", ")\n", "\n", "print(f'Item similarity matrix shape: {item_similarity_df.shape}')\n", "item_similarity_df.head()" ] }, { "cell_type": "code", "execution_count": 11, "id": "afc9d8f0", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Memory-based recommendations for \"Toy Story (1995)\":\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
titlesimilarity
movie_id
50Star Wars (1977)0.7346
181Return of the Jedi (1983)0.6999
121Independence Day (ID4) (1996)0.6898
117Rock, The (1996)0.6646
405Mission: Impossible (1996)0.6413
\n", "
" ], "text/plain": [ " title similarity\n", "movie_id \n", "50 Star Wars (1977) 0.7346\n", "181 Return of the Jedi (1983) 0.6999\n", "121 Independence Day (ID4) (1996) 0.6898\n", "117 Rock, The (1996) 0.6646\n", "405 Mission: Impossible (1996) 0.6413" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def get_title(movie_id):\n", " '''Return movie title for a given ID.'''\n", "\n", " result = movies.loc[movies['movie_id'] == movie_id, 'title']\n", " return result.values[0] if len(result) > 0 else f'Unknown ({movie_id})'\n", "\n", "\n", "def memory_recommendations(movie_id, n=5):\n", " '''Return the top n movies most similar to movie_id by item-item cosine similarity.'''\n", "\n", " scores = item_similarity_df[movie_id].sort_values(ascending=False).iloc[1:n + 1]\n", "\n", " return pd.DataFrame({\n", " 'title': [get_title(mid) for mid in scores.index],\n", " 'similarity': scores.values.round(4)\n", " }, index=scores.index)\n", "\n", "# Toy Story (movie_id = 1)\n", "print(f'Memory-based recommendations for \"{get_title(1)}\":')\n", "memory_recommendations(1)" ] }, { "cell_type": "markdown", "id": "a56e4f68", "metadata": {}, "source": [ "### 2.2. Model-based collaborative filtering\n", "\n", "\n", "\n", "Model-based collaborative filtering uses matrix factorization to learn a compact latent representation of users and items. Truncated SVD decomposes the user-item matrix into latent factors that capture hidden patterns (e.g. genre preferences), then reconstructs predicted ratings for user-movie pairs the user has not yet rated.\n", "\n", "\n", "\n", "$$X \\approx U_k \\Sigma_k V_k^T$$\n", "\n", "\n", "\n", "Where $U_k$ contains user latent factors, $\\Sigma_k$ scales each component, and $V_k^T$ contains item latent factors. The product fills in predicted ratings across the full matrix." ] }, { "cell_type": "code", "execution_count": 12, "id": "115b16db", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Predicted ratings matrix shape: (943, 1682)\n", "Variance explained by 50 components: 52.4%\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
movie_id12345678910...1673167416751676167716781679168016811682
user_id
16.5062342.5861961.5746002.5845131.4201021.5311084.4159291.3382771.0789052.846275...0.051324-0.041984-0.051310-0.0342060.015757-0.002322-0.006965-0.0046430.0352490.054830
22.3997420.4622020.0488990.3390380.2145520.5172750.015426-0.3632132.1046120.610305...0.022276-0.033809-0.024924-0.016616-0.0048990.0010880.0032630.0021750.006681-0.016521
30.171814-0.191129-0.131494-0.1503850.093206-0.0668130.0160070.6778930.1685390.212851...-0.016763-0.0183990.0691300.046087-0.0037690.0155630.0466890.0311260.004082-0.000249
40.391900-0.2944680.036593-0.0005040.0241110.0828170.236818-0.2165440.083964-0.117373...0.000497-0.022420-0.014679-0.0097860.0073210.0056970.0170900.011393-0.005285-0.016400
54.1365411.8881610.1544751.0075560.5550100.2929421.6773590.3446100.4686230.242909...0.007956-0.0824700.0090440.006029-0.057786-0.002230-0.006691-0.004461-0.046427-0.001119
\n", "

5 rows × 1682 columns

\n", "
" ], "text/plain": [ "movie_id 1 2 3 4 5 6 \\\n", "user_id \n", "1 6.506234 2.586196 1.574600 2.584513 1.420102 1.531108 \n", "2 2.399742 0.462202 0.048899 0.339038 0.214552 0.517275 \n", "3 0.171814 -0.191129 -0.131494 -0.150385 0.093206 -0.066813 \n", "4 0.391900 -0.294468 0.036593 -0.000504 0.024111 0.082817 \n", "5 4.136541 1.888161 0.154475 1.007556 0.555010 0.292942 \n", "\n", "movie_id 7 8 9 10 ... 1673 1674 \\\n", "user_id ... \n", "1 4.415929 1.338277 1.078905 2.846275 ... 0.051324 -0.041984 \n", "2 0.015426 -0.363213 2.104612 0.610305 ... 0.022276 -0.033809 \n", "3 0.016007 0.677893 0.168539 0.212851 ... -0.016763 -0.018399 \n", "4 0.236818 -0.216544 0.083964 -0.117373 ... 0.000497 -0.022420 \n", "5 1.677359 0.344610 0.468623 0.242909 ... 0.007956 -0.082470 \n", "\n", "movie_id 1675 1676 1677 1678 1679 1680 \\\n", "user_id \n", "1 -0.051310 -0.034206 0.015757 -0.002322 -0.006965 -0.004643 \n", "2 -0.024924 -0.016616 -0.004899 0.001088 0.003263 0.002175 \n", "3 0.069130 0.046087 -0.003769 0.015563 0.046689 0.031126 \n", "4 -0.014679 -0.009786 0.007321 0.005697 0.017090 0.011393 \n", "5 0.009044 0.006029 -0.057786 -0.002230 -0.006691 -0.004461 \n", "\n", "movie_id 1681 1682 \n", "user_id \n", "1 0.035249 0.054830 \n", "2 0.006681 -0.016521 \n", "3 0.004082 -0.000249 \n", "4 -0.005285 -0.016400 \n", "5 -0.046427 -0.001119 \n", "\n", "[5 rows x 1682 columns]" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "svd = TruncatedSVD(n_components=50, random_state=315)\n", "user_factors = svd.fit_transform(user_item_filled)\n", "\n", "# Reconstruct the full ratings matrix from latent factors\n", "predicted_ratings = np.dot(user_factors, svd.components_)\n", "\n", "predicted_ratings_df = pd.DataFrame(\n", " predicted_ratings,\n", " index=user_item_matrix.index,\n", " columns=user_item_matrix.columns\n", ")\n", "\n", "print(f'Predicted ratings matrix shape: {predicted_ratings_df.shape}')\n", "print(f'Variance explained by 50 components: {svd.explained_variance_ratio_.sum():.1%}')\n", "predicted_ratings_df.head()" ] }, { "cell_type": "code", "execution_count": 13, "id": "2b381b32", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Model-based recommendations for user 1:\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
titlepredicted_rating
movie_id
276Leaving Las Vegas (1995)3.310
403Batman (1989)3.069
423E.T. the Extra-Terrestrial (1982)2.938
732Dave (1993)2.739
509My Left Foot (1989)2.711
\n", "
" ], "text/plain": [ " title predicted_rating\n", "movie_id \n", "276 Leaving Las Vegas (1995) 3.310\n", "403 Batman (1989) 3.069\n", "423 E.T. the Extra-Terrestrial (1982) 2.938\n", "732 Dave (1993) 2.739\n", "509 My Left Foot (1989) 2.711" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def model_recommendations(user_id, n=5):\n", " '''Return the top n unrated movies by predicted rating for a given user.'''\n", "\n", " user_actual = user_item_matrix.loc[user_id]\n", " unrated = user_actual[user_actual.isna()].index\n", " predicted = predicted_ratings_df.loc[user_id, unrated].sort_values(ascending=False).iloc[:n]\n", "\n", " return pd.DataFrame({\n", " 'title': [get_title(mid) for mid in predicted.index],\n", " 'predicted_rating': predicted.values.round(3)\n", " }, index=predicted.index)\n", "\n", "\n", "user_id = user_item_matrix.index[0]\n", "print(f'Model-based recommendations for user {user_id}:')\n", "model_recommendations(user_id)" ] }, { "cell_type": "markdown", "id": "62dc7863", "metadata": {}, "source": [ "## 3. Content-based filtering\n", "\n", "\n", "\n", "Content-based filtering recommends items similar to ones a user has already liked, using item attributes rather than other users' behaviour. Here each movie is represented as a binary genre vector with one entry per genre flag in the MovieLens metadata.\n", "\n", "\n", "\n", "Similarity between two movies is cosine similarity over their genre vectors:\n", "\n", "\n", "\n", "$$\\text{sim}(i, j) = \\frac{g_i \\cdot g_j}{\\|g_i\\| \\|g_j\\|}$$\n", "\n", "\n", "\n", "A score of 1 means both movies share exactly the same genres; 0 means they share no genres." ] }, { "cell_type": "code", "execution_count": 14, "id": "511289dd", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Genre similarity matrix shape: (1682, 1682)\n" ] } ], "source": [ "# Build genre feature matrix (rows = movies, columns = genres)\n", "genre_matrix = movies[genre_names].values\n", "\n", "genre_similarity = cosine_similarity(genre_matrix)\n", "\n", "genre_similarity_df = pd.DataFrame(\n", " genre_similarity,\n", " index=movies['movie_id'],\n", " columns=movies['movie_id']\n", ")\n", "\n", "print(f'Genre similarity matrix shape: {genre_similarity_df.shape}')" ] }, { "cell_type": "code", "execution_count": 15, "id": "7279329f", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
movie_id12345678910...1673167416751676167716781679168016811682
movie_id
11.0000000.0000000.000000.3333330.0000000.000000.0000000.6666670.000000.000000...0.0000000.000000.000000.000000.000000.000000.0000000.0000000.577350.00000
20.0000001.0000000.577350.3333330.3333330.000000.0000000.0000000.000000.000000...0.8164970.000000.000000.000000.000000.000000.4082480.0000000.000000.00000
30.0000000.5773501.000000.0000000.5773500.000000.0000000.0000000.000000.000000...0.7071070.000000.000000.000000.000000.000000.7071070.0000000.000000.00000
40.3333330.3333330.000001.0000000.3333330.577350.4082480.6666670.577350.408248...0.4082480.577350.577350.577350.577350.577350.0000000.4082480.577350.57735
50.0000000.3333330.577350.3333331.0000000.577350.4082480.3333330.577350.408248...0.4082480.577350.577350.577350.577350.577350.4082480.4082480.000000.57735
\n", "

5 rows × 1682 columns

\n", "
" ], "text/plain": [ "movie_id 1 2 3 4 5 6 7 \\\n", "movie_id \n", "1 1.000000 0.000000 0.00000 0.333333 0.000000 0.00000 0.000000 \n", "2 0.000000 1.000000 0.57735 0.333333 0.333333 0.00000 0.000000 \n", "3 0.000000 0.577350 1.00000 0.000000 0.577350 0.00000 0.000000 \n", "4 0.333333 0.333333 0.00000 1.000000 0.333333 0.57735 0.408248 \n", "5 0.000000 0.333333 0.57735 0.333333 1.000000 0.57735 0.408248 \n", "\n", "movie_id 8 9 10 ... 1673 1674 1675 \\\n", "movie_id ... \n", "1 0.666667 0.00000 0.000000 ... 0.000000 0.00000 0.00000 \n", "2 0.000000 0.00000 0.000000 ... 0.816497 0.00000 0.00000 \n", "3 0.000000 0.00000 0.000000 ... 0.707107 0.00000 0.00000 \n", "4 0.666667 0.57735 0.408248 ... 0.408248 0.57735 0.57735 \n", "5 0.333333 0.57735 0.408248 ... 0.408248 0.57735 0.57735 \n", "\n", "movie_id 1676 1677 1678 1679 1680 1681 1682 \n", "movie_id \n", "1 0.00000 0.00000 0.00000 0.000000 0.000000 0.57735 0.00000 \n", "2 0.00000 0.00000 0.00000 0.408248 0.000000 0.00000 0.00000 \n", "3 0.00000 0.00000 0.00000 0.707107 0.000000 0.00000 0.00000 \n", "4 0.57735 0.57735 0.57735 0.000000 0.408248 0.57735 0.57735 \n", "5 0.57735 0.57735 0.57735 0.408248 0.408248 0.00000 0.57735 \n", "\n", "[5 rows x 1682 columns]" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "genre_similarity_df.head()" ] }, { "cell_type": "code", "execution_count": 16, "id": "881f1014", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Content-based recommendations for \"Toy Story (1995)\"\n", "Target genres: Animation, Children's, Comedy\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
titlegenressimilarity
0Aladdin and the King of Thieves (1996)Animation, Children's, Comedy1.0000
1Aladdin (1992)Animation, Children's, Comedy, Musical0.8660
2Goofy Movie, A (1995)Animation, Children's, Comedy, Romance0.8660
3Jungle2Jungle (1997)Children's, Comedy0.8165
4Angels in the Outfield (1994)Children's, Comedy0.8165
\n", "
" ], "text/plain": [ " title \\\n", "0 Aladdin and the King of Thieves (1996) \n", "1 Aladdin (1992) \n", "2 Goofy Movie, A (1995) \n", "3 Jungle2Jungle (1997) \n", "4 Angels in the Outfield (1994) \n", "\n", " genres similarity \n", "0 Animation, Children's, Comedy 1.0000 \n", "1 Animation, Children's, Comedy, Musical 0.8660 \n", "2 Animation, Children's, Comedy, Romance 0.8660 \n", "3 Children's, Comedy 0.8165 \n", "4 Children's, Comedy 0.8165 " ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def content_recommendations(movie_id, n=5):\n", " '''Return the top n movies with the most similar genre profile.'''\n", "\n", " target_row = movies.loc[movies['movie_id'] == movie_id, genre_names].values[0]\n", " target_genres = ', '.join([g for g, v in zip(genre_names, target_row) if v == 1])\n", "\n", " scores = genre_similarity_df[movie_id].sort_values(ascending=False).iloc[1:n + 1]\n", "\n", " results = []\n", "\n", " for mid, score in scores.items():\n", " row = movies.loc[movies['movie_id'] == mid, genre_names].values[0]\n", " genres = ', '.join([g for g, v in zip(genre_names, row) if v == 1])\n", " results.append({'title': get_title(mid), 'genres': genres, 'similarity': round(score, 4)})\n", "\n", " print(f'Content-based recommendations for \"{get_title(movie_id)}\"')\n", " print(f'Target genres: {target_genres}')\n", "\n", " return pd.DataFrame(results)\n", "\n", "\n", "content_recommendations(1)" ] }, { "cell_type": "markdown", "id": "5dcfb40c", "metadata": {}, "source": [ "## 4. Hybrid filtering\n", "\n", "\n", "\n", "Hybrid filtering combines collaborative and content-based scores into a single ranking. Because the two score distributions differ, both are normalised to $[0, 1]$ before combining. The parameter $\\alpha$ controls the balance between the two approaches:\n", "\n", "\n", "\n", "$$s_{\\text{hybrid}} = \\alpha \\cdot \\hat{s}_{\\text{collab}} + (1 - \\alpha) \\cdot \\hat{s}_{\\text{content}}$$\n", "\n", "\n", "\n", "Setting $\\alpha = 1$ reduces to pure collaborative filtering; $\\alpha = 0$ reduces to pure content-based filtering." ] }, { "cell_type": "code", "execution_count": 17, "id": "f474dbd3", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Hybrid recommendations for \"Toy Story (1995)\" (alpha=0.5):\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
titlehybrid_score
0Aladdin (1992)0.8291
1Willy Wonka and the Chocolate Factory (1971)0.7677
2Beauty and the Beast (1991)0.7106
3Lion King, The (1994)0.7096
4Beavis and Butt-head Do America (1996)0.7015
\n", "
" ], "text/plain": [ " title hybrid_score\n", "0 Aladdin (1992) 0.8291\n", "1 Willy Wonka and the Chocolate Factory (1971) 0.7677\n", "2 Beauty and the Beast (1991) 0.7106\n", "3 Lion King, The (1994) 0.7096\n", "4 Beavis and Butt-head Do America (1996) 0.7015" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def normalize(series):\n", " '''Scale a series to [0, 1].'''\n", "\n", " lo, hi = series.min(), series.max()\n", "\n", " return (series - lo) / (hi - lo) if hi > lo else series * 0\n", "\n", "\n", "def hybrid_recommendations(movie_id, n=5, alpha=0.5):\n", " '''Return the top n movies by weighted combination of collab and content scores.'''\n", "\n", " collab = item_similarity_df[movie_id] if movie_id in item_similarity_df.columns else pd.Series(dtype=float)\n", " content = genre_similarity_df[movie_id]\n", "\n", " common = collab.index.intersection(content.index)\n", " common = common[common != movie_id]\n", "\n", " hybrid = alpha * normalize(collab[common]) + (1 - alpha) * normalize(content[common])\n", " top = hybrid.sort_values(ascending=False).iloc[:n]\n", "\n", " results = [{'title': get_title(mid), 'hybrid_score': round(score, 4)} for mid, score in top.items()]\n", " print(f'Hybrid recommendations for \"{get_title(movie_id)}\" (alpha={alpha}):')\n", "\n", " return pd.DataFrame(results)\n", "\n", "hybrid_recommendations(1)" ] }, { "cell_type": "markdown", "id": "745c19ab", "metadata": {}, "source": [ "## 5. Evaluation\n", "\n", "\n", "\n", "Evaluating a recommendation system requires simulating the real-world scenario where some ratings are known and others are withheld. A **leave-one-out** strategy holds out one randomly chosen rating per user as the test item and trains the model on the remaining ratings.\n", "\n", "\n", "\n", "One important detail: only **positively-rated items** (rating ≥ 4) should be held out. If we hold out a movie the user rated 1 or 2 stars, finding it in the top-10 is not a success; the model would be recommending something the user disliked. Users with no positive ratings have no test item and are excluded from the evaluation.\n", "\n", "\n", "\n", "**Note on production practice:** two common refinements are used in rigorous benchmarks. First, a **time-based split** holds out each user's *most recent* positive interaction rather than a random one, preventing the model from using future data to predict the past. Second, many papers use a **sampled ranking** protocol (the model ranks the 1 test item against 100 randomly sampled unrated items) instead of ranking across all items, which is much cheaper to compute at scale.\n", "\n", "\n", "\n", "### 5.1. Train/test split" ] }, { "cell_type": "code", "execution_count": 18, "id": "6c25b19a", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Train ratings: 99,058\n", "Test ratings: 942 (one positively-rated item per user)\n" ] } ], "source": [ "# Only hold out positively-rated items; a hit on a disliked movie is not meaningful\n", "positive_ratings = ratings[ratings['rating'] >= 4]\n", "held_out = positive_ratings.groupby('user_id').sample(n=1, random_state=315)\n", "test_ratings = held_out.reset_index(drop=True)\n", "train_ratings = ratings.drop(index=held_out.index)\n", "\n", "print(f'Train ratings: {len(train_ratings):,}')\n", "print(f'Test ratings: {len(test_ratings):,} (one positively-rated item per user)')" ] }, { "cell_type": "code", "execution_count": 19, "id": "a4f7705e", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Training predictions shape: (943, 1682)\n" ] } ], "source": [ "# Retrain SVD on training data only\n", "train_matrix = train_ratings.pivot_table(\n", " index='user_id', columns='movie_id', values='rating'\n", ").fillna(0)\n", "\n", "svd_eval = TruncatedSVD(n_components=50, random_state=315)\n", "\n", "train_preds = pd.DataFrame(\n", " np.dot(svd_eval.fit_transform(train_matrix), svd_eval.components_),\n", " index=train_matrix.index,\n", " columns=train_matrix.columns\n", ")\n", "\n", "print(f'Training predictions shape: {train_preds.shape}')" ] }, { "cell_type": "markdown", "id": "4a7f143b", "metadata": {}, "source": [ "### 5.2. GetTopN\n", "\n", "\n", "\n", "`GetTopN` generates the ranked recommendation list for each user: it identifies items the user has not rated in the training data, ranks them by predicted rating, and returns the top-N. It is the bridge between raw model predictions and a usable list of recommendations." ] }, { "cell_type": "code", "execution_count": 20, "id": "2c38d8f9", "metadata": {}, "outputs": [], "source": [ "def get_top_n(predicted_df, train_matrix, n=10):\n", " '''For each user, return the top-N unrated movie IDs ranked by predicted rating.'''\n", "\n", " top_n = {}\n", "\n", " for user_id in predicted_df.index:\n", " rated = train_matrix.loc[user_id][train_matrix.loc[user_id] > 0].index\n", " preds = predicted_df.loc[user_id].drop(index=rated, errors='ignore')\n", " top_n[user_id] = preds.sort_values(ascending=False).iloc[:n].index.tolist()\n", "\n", " return top_n" ] }, { "cell_type": "code", "execution_count": 21, "id": "b7b7a33c", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Generated top-10 lists for 943 users\n", "\n", "Example: top-10 for user 1:\n", " Leaving Las Vegas (1995)\n", " Dr. Strangelove or: How I Learned to Stop Worrying and Love the Bomb (1963)\n", " Dave (1993)\n", " Batman (1989)\n", " Tombstone (1993)\n", " E.T. the Extra-Terrestrial (1982)\n", " Glory (1989)\n", " Piano, The (1993)\n", " My Left Foot (1989)\n", " Like Water For Chocolate (Como agua para chocolate) (1992)\n" ] } ], "source": [ "top_n_recs = get_top_n(train_preds, train_matrix, n=10)\n", "\n", "example_user = list(top_n_recs.keys())[0]\n", "print(f'Generated top-10 lists for {len(top_n_recs)} users')\n", "print(f'\\nExample: top-10 for user {example_user}:')\n", "\n", "for title in [get_title(mid) for mid in top_n_recs[example_user]]:\n", " print(f' {title}')" ] }, { "cell_type": "markdown", "id": "b0c754dd", "metadata": {}, "source": [ "### 5.3. Hit rate\n", "\n", "\n", "\n", "Hit rate measures how often the model's top-N list contains the held-out test item:\n", "\n", "\n", "\n", "$$\\text{Hit Rate@}N = \\frac{|\\{u : \\text{test item} \\in \\text{top-}N(u)\\}|}{|U|}$$\n", "\n", "\n", "\n", "Where $U$ is the set of users with a positive test item and $\\text{top-}N(u)$ is the set of recommended items for user $u$. A hit rate of 0.05 at $N = 10$ means 5% of those users had their held-out movie appear in their top-10 list.\n", "\n", "\n", "\n", "Because we only held out positively-rated items, a hit genuinely means the model surfaced something the user liked. Without this filter, the metric would credit the model for recommending movies the user had already rated poorly.\n", "\n", "\n", "\n", "Hit rate rewards the model for placing the test item anywhere in the top-N. It is often paired with NDCG (normalised discounted cumulative gain), which additionally rewards higher placements within the list." ] }, { "cell_type": "code", "execution_count": 22, "id": "0d347d09", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Hit rate @ 10: 0.377\n", "About 1 in 3 users had their held-out movie in the top-10 list\n" ] } ], "source": [ "def hit_rate(top_n_dict, test_df):\n", " '''Fraction of users whose held-out test movie appears in their top-N list.'''\n", "\n", " eligible = test_df[test_df['user_id'].isin(top_n_dict)]\n", " hits = eligible.apply(lambda row: row['movie_id'] in top_n_dict[row['user_id']], axis=1).sum()\n", "\n", " return hits / len(eligible) if len(eligible) > 0 else 0.0\n", "\n", "\n", "hr = hit_rate(top_n_recs, test_ratings)\n", "print(f'Hit rate @ 10: {hr:.3f}')\n", "print(f'About 1 in {1/hr:.0f} users had their held-out movie in the top-10 list')" ] }, { "cell_type": "markdown", "id": "433298ef", "metadata": {}, "source": [ "## 6. Export for web app deployment\n", "\n", "\n", "\n", "The three pre-computed similarity matrices and supporting data are saved to a single pickle file. A web app can load this file at startup and serve recommendations without re-running any computation. The file contains everything needed to:\n", "\n", "\n", "\n", "- Look up a movie by title and return memory-based recommendations.\n", "\n", "- Return model-based (SVD) recommendations for a given user.\n", "\n", "- Return content-based recommendations by genre profile.\n", "\n", "- Return hybrid recommendations with a configurable `alpha`." ] }, { "cell_type": "code", "execution_count": 23, "id": "b9677b1c", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Saved movie_recommender.pkl (67.7 MB)\n", "Keys: ['item_similarity_df', 'genre_similarity_df', 'predicted_ratings_df', 'user_item_matrix', 'movies', 'genre_names']\n" ] } ], "source": [ "export = {\n", " 'item_similarity_df': item_similarity_df,\n", " 'genre_similarity_df': genre_similarity_df,\n", " 'predicted_ratings_df': predicted_ratings_df,\n", " 'user_item_matrix': user_item_matrix,\n", " 'movies': movies,\n", " 'genre_names': genre_names,\n", "}\n", "\n", "output_path = 'movie_recommender.pkl'\n", "\n", "with open(output_path, 'wb') as f:\n", " pickle.dump(export, f)\n", "\n", "size_mb = os.path.getsize(output_path) / (1024 * 1024)\n", "print(f'Saved {output_path} ({size_mb:.1f} MB)')\n", "print(f'Keys: {list(export.keys())}')" ] }, { "cell_type": "markdown", "id": "219a012c", "metadata": {}, "source": [ "## 7. Challenges\n", "\n", "\n", "\n", "Several fundamental challenges arise when building recommendation systems in practice." ] }, { "cell_type": "markdown", "id": "4036ba28", "metadata": {}, "source": [ "### 7.1. Cold start problem\n", "\n", "\n", "\n", "The cold start problem occurs when the system has insufficient data to make reliable recommendations.\n", "\n", "\n", "\n", "**New user cold start**: a first-time user has no rating history. Collaborative filtering cannot compute user similarity or generate predictions.\n", "\n", "\n", "\n", "- Ask users to rate a small set of popular items during onboarding\n", "\n", "- Use demographic information\n", "\n", "- Default to popularity-based recommendations until enough history is collected.\n", "\n", "\n", "\n", "**New item cold start**: a newly added item has no user ratings and will not appear in collaborative filtering results.\n", "\n", "\n", "\n", "- Use content features (genre, description, cast) to seed similarity via content-based filtering\n", "\n", "- Use a hybrid model that blends collaborative and content signals\n", "\n", "- Apply knowledge transfer from similar established items" ] }, { "cell_type": "markdown", "id": "5814f171", "metadata": {}, "source": [ "### 7.2. Implicit vs explicit feedback\n", "\n", "\n", "\n", "**Explicit feedback** is a direct expression of preference (e.g. a 1-5 star rating or a thumbs up/down). It is unambiguous but rare: most users never rate what they consume.\n", "\n", "\n", "\n", "**Implicit feedback** is collected passively from behavior: clicks, views, play counts, purchase history, and dwell time. It is abundant but noisy; a song played once may have been background noise, and a product viewed many times may have been considered and rejected.\n", "\n", "\n", "\n", "The key challenge with implicit feedback is the absence of a negative signal. An unobserved (user, item) pair could mean the user dislikes it, has never been exposed to it, or simply hasn't discovered it yet. Models must handle this uncertainty explicitly.\n", "\n", "\n", "\n", "| | Explicit | Implicit |\n", "|---|---|---|\n", "| Examples | Star ratings, thumbs up/down | Clicks, views, play counts |\n", "| Volume | Low | High |\n", "| Negative signal | Direct (low ratings) | Absent |\n", "| Noise | Low | High |\n", "| Common models | SVD, NMF | BPR, WALS, EASE |" ] }, { "cell_type": "markdown", "id": "dded5260", "metadata": {}, "source": [ "### 7.3. Other challenges\n", "\n", "\n", "\n", "- **Scalability**: computing pairwise similarities over millions of users and items is expensive. Production systems use approximate nearest-neighbor (ANN) search and distributed matrix factorization.\n", "\n", "\n", "\n", "- **Filter bubbles and diversity**: accuracy-optimized systems tend to recommend more of what a user has already seen. Real-world systems add diversity, novelty, and serendipity constraints to broaden the recommendation space.\n", "\n", "\n", "\n", "- **Temporal drift**: user preferences and item relevance change over time. Models need regular retraining or online updates to stay current." ] }, { "cell_type": "markdown", "id": "23c0feda", "metadata": {}, "source": [ "## 8. More advanced techniques\n", "\n", "\n", "\n", "The approaches in this notebook are strong baselines, but production recommendation systems typically use more powerful methods:\n", "\n", "\n", "\n", "- **Alternating Least Squares (ALS)**: a matrix factorization algorithm designed for implicit feedback that scales to millions of users and items. Available in PySpark MLlib and the `implicit` Python library.\n", "\n", "- **Weighted Alternating Least Squares (WALS)**: extends matrix factorization to implicit feedback by assigning confidence weights to all (user, item) pairs. Unobserved pairs get a low but non-zero weight rather than being ignored; observed interactions get a higher weight proportional to frequency.\n", "\n", "- **Bayesian Personalized Ranking (BPR)**: optimizes directly for ranking rather than rating prediction, using pairwise comparisons between observed and unobserved items.\n", "\n", "- **EASE (Embarrassingly Shallow Autoencoders)**: learns a sparse item-item weight matrix from the interaction data using a closed-form solution, with no latent factors or neural networks. Simple but surprisingly competitive on standard benchmarks.\n", "\n", "- **Factorization Machines (FM)**: generalizes matrix factorization to arbitrary feature interactions, making it straightforward to incorporate user and item side information (demographics, genres, context).\n", "\n", "- **Neural Collaborative Filtering (NCF)**: replaces the dot product with a neural network, allowing the model to learn non-linear user-item interactions.\n", "\n", "- **Sequence-aware models** (SASRec, BERT4Rec): model the temporal order of interactions, capturing session-level intent and recent context.\n", "\n", "- **Graph Neural Networks** (LightGCN, NGCF): represent users and items as nodes in an interaction graph and propagate signals through multi-hop connections to capture higher-order collaborative patterns." ] } ], "metadata": { "kernelspec": { "display_name": ".venv (3.12.3)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.3" } }, "nbformat": 4, "nbformat_minor": 5 }