{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### 计算传播应用\n",
"***\n",
"***\n",
"# 推荐系统简介\n",
"***\n",
"***\n",
"\n",
"王成军\n",
"\n",
"wangchengjun@nju.edu.cn\n",
"\n",
"计算传播网 http://computational-communication.com"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# 集体智慧编程\n",
"\n",
"> 集体智慧是指为了创造新想法，将一群人的行为、偏好或思想组合在一起。一般基于聪明的算法（Netflix, Google）或者提供内容的用户(Wikipedia)。\n",
"\n",
"集体智慧编程所强调的是前者，即通过编写计算机程序、构造具有智能的算法收集并分析用户的数据，发现新的信息甚至是知识。\n",
"\n",
"- Netflix\n",
"- Google\n",
"- Wikipedia\n",
"\n",
"Toby Segaran, 2007, Programming Collective Intelligence. O'Reilly. \n",
"\n",
"https://github.com/computational-class/programming-collective-intelligence-code/blob/master/chapter2/recommendations.py"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# 推荐系统\n",
"\n",
"- 目前互联网世界最常见的智能产品形式。\n",
"- 从信息时代过渡到注意力时代：\n",
" - 信息过载（information overload）\n",
" - 注意力稀缺\n",
"- 推荐系统的基本任务是联系用户和物品，帮助用户快速发现有用信息，解决信息过载的问题。\n",
" - 针对长尾分布问题，找到个性化需求，优化资源配置\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# 推荐系统的类型\n",
"- 社会化推荐（Social Recommendation）\n",
" - 让朋友帮助推荐物品\n",
"- 基于内容的推荐 （Content-based filtering）\n",
" - 基于用户已经消费的物品内容，推荐新的物品。例如根据看过的电影的导演和演员，推荐新影片。\n",
"- 基于协同过滤的推荐（collaborative filtering）\n",
" - 找到和某用户的历史兴趣一致的用户，根据这些用户之间的相似性或者他们所消费物品的相似性，为该用户推荐物品"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# 协同过滤算法\n",
"\n",
"- 基于邻域的方法（neighborhood-based method）\n",
" - 基于用户的协同过滤（user-based filtering）\n",
" - 基于物品的协同过滤 （item-based filtering）\n",
"- 隐语义模型（latent factor model）\n",
"- 基于图的随机游走算法（random walk on graphs）"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# UserCF和ItemCF的比较\n",
"\n",
"- UserCF较为古老， 1992年应用于电子邮件个性化推荐系统Tapestry, 1994年应用于Grouplens新闻个性化推荐， 后来被Digg采用\n",
" - 推荐那些与个体有共同兴趣爱好的用户所喜欢的物品（群体热点，社会化）\n",
" - 反映用户所在小型群体中物品的热门程度\n",
"- ItemCF相对较新，应用于电子商务网站Amazon和DVD租赁网站Netflix\n",
" - 推荐那些和用户之前喜欢的物品相似的物品 （历史兴趣，个性化）\n",
" - 反映了用户自己的兴趣传承\n",
"- 新闻更新快，物品数量庞大，相似度变化很快，不利于维护一张物品相似度的表格，电影、音乐、图书则可以。\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# 推荐系统评测\n",
"- 用户满意度\n",
"- 预测准确度\n",
"\n",
" $r_{ui}$用户实际打分， $\\hat{r_{ui}}$推荐算法预测打分\n",
"\n",
" - 均方根误差RMSE\n",
" \n",
" $RMSE = \\sqrt{\\frac{\\sum_{u, i \\in T} (r_{ui} - \\hat{r_{ui}})}{\\left | T \\right |}^2} $\n",
" \n",
" - 平均绝对误差MAE\n",
" $ MAE = \\frac{\\sum_{u, i \\in T} \\left | r_{ui} - \\hat{r_{ui}} \\right|}{\\left | T \\right|}$"
]
},
{
"cell_type": "code",
"execution_count": 70,
"metadata": {
"collapsed": true,
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [],
"source": [
"# A dictionary of movie critics and their ratings of a small\n",
"# set of movies\n",
"critics={'Lisa Rose': {'Lady in the Water': 2.5, 'Snakes on a Plane': 3.5,\n",
" 'Just My Luck': 3.0, 'Superman Returns': 3.5, 'You, Me and Dupree': 2.5,\n",
" 'The Night Listener': 3.0},\n",
" 'Gene Seymour': {'Lady in the Water': 3.0, 'Snakes on a Plane': 3.5,\n",
" 'Just My Luck': 1.5, 'Superman Returns': 5.0, 'The Night Listener': 3.0,\n",
" 'You, Me and Dupree': 3.5},\n",
" 'Michael Phillips': {'Lady in the Water': 2.5, 'Snakes on a Plane': 3.0,\n",
" 'Superman Returns': 3.5, 'The Night Listener': 4.0},\n",
" 'Claudia Puig': {'Snakes on a Plane': 3.5, 'Just My Luck': 3.0,\n",
" 'The Night Listener': 4.5, 'Superman Returns': 4.0,\n",
" 'You, Me and Dupree': 2.5},\n",
" 'Mick LaSalle': {'Lady in the Water': 3.0, 'Snakes on a Plane': 4.0,\n",
" 'Just My Luck': 2.0, 'Superman Returns': 3.0, 'The Night Listener': 3.0,\n",
" 'You, Me and Dupree': 2.0},\n",
" 'Jack Matthews': {'Lady in the Water': 3.0, 'Snakes on a Plane': 4.0,\n",
" 'The Night Listener': 3.0, 'Superman Returns': 5.0, 'You, Me and Dupree': 3.5},\n",
" 'Toby': {'Snakes on a Plane':4.5,'You, Me and Dupree':1.0,'Superman Returns':4.0}}"
]
},
{
"cell_type": "code",
"execution_count": 39,
"metadata": {
"collapsed": false,
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"data": {
"text/plain": [
"2.5"
]
},
"execution_count": 39,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"critics['Lisa Rose']['Lady in the Water']"
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {
"collapsed": true,
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [],
"source": [
"critics['Toby']['Snakes on a Plane']=4.5"
]
},
{
"cell_type": "code",
"execution_count": 41,
"metadata": {
"collapsed": false,
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"data": {
"text/plain": [
"{'Snakes on a Plane': 4.5, 'Superman Returns': 4.0, 'You, Me and Dupree': 1.0}"
]
},
"execution_count": 41,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"critics['Toby']"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"***\n",
"# 1. User-based filtering\n",
"***\n",
"## 1.0 Finding similar users"
]
},
{
"cell_type": "code",
"execution_count": 48,
"metadata": {
"collapsed": false,
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"data": {
"text/plain": [
"3.1622776601683795"
]
},
"execution_count": 48,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# 欧几里得距离\n",
"import numpy as np\n",
"np.sqrt(np.power(5-4, 2) + np.power(4-1, 2))"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"This formula calculates the distance, which will be smaller for people who are more similar. However, you need a function that gives higher values for people who are similar. This can be done by adding 1 to the function (so you don’t get a division-by- zero error) and inverting it:"
]
},
{
"cell_type": "code",
"execution_count": 49,
"metadata": {
"collapsed": false,
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"data": {
"text/plain": [
"0.2402530733520421"
]
},
"execution_count": 49,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"1.0 /(1 + np.sqrt(np.power(5-4, 2) + np.power(4-1, 2)) )"
]
},
{
"cell_type": "code",
"execution_count": 53,
"metadata": {
"collapsed": false,
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"# Returns a distance-based similarity score for person1 and person2\n",
"def sim_distance(prefs,person1,person2):\n",
" # Get the list of shared_items\n",
" si={}\n",
" for item in prefs[person1]:\n",
" if item in prefs[person2]:\n",
" si[item]=1\n",
" # if they have no ratings in common, return 0\n",
" if len(si)==0: return 0\n",
" # Add up the squares of all the differences\n",
" sum_of_squares=np.sum([np.power(prefs[person1][item]-prefs[person2][item],2)\n",
" for item in prefs[person1] if item in prefs[person2]])\n",
" return 1/(1+np.sqrt(sum_of_squares) )"
]
},
{
"cell_type": "code",
"execution_count": 54,
"metadata": {
"collapsed": false,
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"data": {
"text/plain": [
"0.14814814814814814"
]
},
"execution_count": 54,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sim_distance(critics, 'Lisa Rose','Gene Seymour')"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Pearson correlation coefficient"
]
},
{
"cell_type": "code",
"execution_count": 59,
"metadata": {
"collapsed": true,
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"# Returns the Pearson correlation coefficient for p1 and p2\n",
"def sim_pearson(prefs,p1,p2):\n",
" # Get the list of mutually rated items\n",
" si={}\n",
" for item in prefs[p1]:\n",
" if item in prefs[p2]: si[item]=1\n",
" # Find the number of elements\n",
" n=len(si)\n",
" # if they are no ratings in common, return 0\n",
" if n==0: return 0\n",
" # Add up all the preferences\n",
" sum1=np.sum([prefs[p1][it] for it in si])\n",
" sum2=np.sum([prefs[p2][it] for it in si])\n",
" # Sum up the squares\n",
" sum1Sq=np.sum([np.power(prefs[p1][it],2) for it in si])\n",
" sum2Sq=np.sum([np.power(prefs[p2][it],2) for it in si])\n",
" # Sum up the products\n",
" pSum=np.sum([prefs[p1][it]*prefs[p2][it] for it in si])\n",
" # Calculate Pearson score\n",
" num=pSum-(sum1*sum2/n)\n",
" den=np.sqrt((sum1Sq-np.power(sum1,2)/n)*(sum2Sq-np.power(sum2,2)/n))\n",
" if den==0: return 0\n",
" return num/den"
]
},
{
"cell_type": "code",
"execution_count": 60,
"metadata": {
"collapsed": false,
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"data": {
"text/plain": [
"0.39605901719066977"
]
},
"execution_count": 60,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sim_pearson(critics, 'Lisa Rose','Gene Seymour')"
]
},
{
"cell_type": "code",
"execution_count": 75,
"metadata": {
"collapsed": true,
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"# Returns the best matches for person from the prefs dictionary.\n",
"# Number of results and similarity function are optional params.\n",
"def topMatches(prefs,person,n=5,similarity=sim_pearson):\n",
" scores=[(similarity(prefs,person,other),other)\n",
" for other in prefs if other!=person]\n",
" # Sort the list so the highest scores appear at the top \n",
" scores.sort( )\n",
" scores.reverse( )\n",
" return scores[0:n]"
]
},
{
"cell_type": "code",
"execution_count": 77,
"metadata": {
"collapsed": false,
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"data": {
"text/plain": [
"[(0.99124070716192991, 'Lisa Rose'),\n",
" (0.92447345164190486, 'Mick LaSalle'),\n",
" (0.89340514744156474, 'Claudia Puig')]"
]
},
"execution_count": 77,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"topMatches(critics,'Toby',n=3) # topN"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## 1.1 Recommending Items"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"\n",
" \n",
" item_id \n",
" rating \n",
" user_id \n",
" \n",
" \n",
" a \n",
" 1 \n",
" 0 \n",
" \n",
" \n",
" b \n",
" 3 \n",
" 0 \n",
" \n",
" \n",
" c \n",
" 2 \n",
" 0 \n",
" \n",
" \n",
" a \n",
" 5 \n",
" 1 \n",
" \n",
" \n",
" b \n",
" 4 \n",
" 1 \n",
" \n",
" \n",
" b \n",
" 1 \n",
" 2 \n",
" \n",
" \n",
" c \n",
" 4 \n",
" 2 \n",
" \n",
" \n",
" d \n",
" 3 \n",
" 2 \n",
" \n",
"
\n",
"[8 rows x 3 columns]
\n",
"
Recsys training: model = ranking_factorization_recommender "
],
"text/plain": [
"Recsys training: model = ranking_factorization_recommender"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"Preparing data set. "
],
"text/plain": [
"Preparing data set."
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" Data has 8 observations with 3 users and 4 items. "
],
"text/plain": [
" Data has 8 observations with 3 users and 4 items."
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" Data prepared in: 0.008394s "
],
"text/plain": [
" Data prepared in: 0.008394s"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"Training ranking_factorization_recommender for recommendations. "
],
"text/plain": [
"Training ranking_factorization_recommender for recommendations."
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"+--------------------------------+--------------------------------------------------+----------+ "
],
"text/plain": [
"+--------------------------------+--------------------------------------------------+----------+"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"| Parameter | Description | Value | "
],
"text/plain": [
"| Parameter | Description | Value |"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"+--------------------------------+--------------------------------------------------+----------+ "
],
"text/plain": [
"+--------------------------------+--------------------------------------------------+----------+"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"| num_factors | Factor Dimension | 32 | "
],
"text/plain": [
"| num_factors | Factor Dimension | 32 |"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"| regularization | L2 Regularization on Factors | 1e-09 | "
],
"text/plain": [
"| regularization | L2 Regularization on Factors | 1e-09 |"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"| solver | Solver used for training | sgd | "
],
"text/plain": [
"| solver | Solver used for training | sgd |"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"| linear_regularization | L2 Regularization on Linear Coefficients | 1e-09 | "
],
"text/plain": [
"| linear_regularization | L2 Regularization on Linear Coefficients | 1e-09 |"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"| ranking_regularization | Rank-based Regularization Weight | 0.25 | "
],
"text/plain": [
"| ranking_regularization | Rank-based Regularization Weight | 0.25 |"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"| max_iterations | Maximum Number of Iterations | 25 | "
],
"text/plain": [
"| max_iterations | Maximum Number of Iterations | 25 |"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"+--------------------------------+--------------------------------------------------+----------+ "
],
"text/plain": [
"+--------------------------------+--------------------------------------------------+----------+"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" Optimizing model using SGD; tuning step size. "
],
"text/plain": [
" Optimizing model using SGD; tuning step size."
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" Using 8 / 8 points for tuning the step size. "
],
"text/plain": [
" Using 8 / 8 points for tuning the step size."
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"+---------+-------------------+------------------------------------------+ "
],
"text/plain": [
"+---------+-------------------+------------------------------------------+"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"| Attempt | Initial Step Size | Estimated Objective Value | "
],
"text/plain": [
"| Attempt | Initial Step Size | Estimated Objective Value |"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"+---------+-------------------+------------------------------------------+ "
],
"text/plain": [
"+---------+-------------------+------------------------------------------+"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"| 0 | 25 | Not Viable | "
],
"text/plain": [
"| 0 | 25 | Not Viable |"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"| 1 | 6.25 | Not Viable | "
],
"text/plain": [
"| 1 | 6.25 | Not Viable |"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"| 2 | 1.5625 | Not Viable | "
],
"text/plain": [
"| 2 | 1.5625 | Not Viable |"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"| 3 | 0.390625 | 2.88256 | "
],
"text/plain": [
"| 3 | 0.390625 | 2.88256 |"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"| 4 | 0.195312 | 2.79407 | "
],
"text/plain": [
"| 4 | 0.195312 | 2.79407 |"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"| 5 | 0.0976562 | 2.80084 | "
],
"text/plain": [
"| 5 | 0.0976562 | 2.80084 |"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"| 6 | 0.0488281 | 3.00496 | "
],
"text/plain": [
"| 6 | 0.0488281 | 3.00496 |"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"| 7 | 0.0244141 | 3.28286 | "
],
"text/plain": [
"| 7 | 0.0244141 | 3.28286 |"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"+---------+-------------------+------------------------------------------+ "
],
"text/plain": [
"+---------+-------------------+------------------------------------------+"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"| Final | 0.195312 | 2.79407 | "
],
"text/plain": [
"| Final | 0.195312 | 2.79407 |"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"+---------+-------------------+------------------------------------------+ "
],
"text/plain": [
"+---------+-------------------+------------------------------------------+"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"Starting Optimization. "
],
"text/plain": [
"Starting Optimization."
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"+---------+--------------+-------------------+-----------------------+-------------+ "
],
"text/plain": [
"+---------+--------------+-------------------+-----------------------+-------------+"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"| Iter. | Elapsed Time | Approx. Objective | Approx. Training RMSE | Step Size | "
],
"text/plain": [
"| Iter. | Elapsed Time | Approx. Objective | Approx. Training RMSE | Step Size |"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"+---------+--------------+-------------------+-----------------------+-------------+ "
],
"text/plain": [
"+---------+--------------+-------------------+-----------------------+-------------+"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"| Initial | 46us | 3.89999 | 1.3637 | | "
],
"text/plain": [
"| Initial | 46us | 3.89999 | 1.3637 | |"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"+---------+--------------+-------------------+-----------------------+-------------+ "
],
"text/plain": [
"+---------+--------------+-------------------+-----------------------+-------------+"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"| 1 | 772us | 4.1149 | 1.6432 | 0.195312 | "
],
"text/plain": [
"| 1 | 772us | 4.1149 | 1.6432 | 0.195312 |"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"| 2 | 1.711ms | 3.04554 | 1.37292 | 0.116134 | "
],
"text/plain": [
"| 2 | 1.711ms | 3.04554 | 1.37292 | 0.116134 |"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"| 3 | 2.214ms | 2.71319 | 1.24268 | 0.0856819 | "
],
"text/plain": [
"| 3 | 2.214ms | 2.71319 | 1.24268 | 0.0856819 |"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"| 4 | 2.749ms | 2.51048 | 1.18246 | 0.0580668 | "
],
"text/plain": [
"| 4 | 2.749ms | 2.51048 | 1.18246 | 0.0580668 |"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"| 5 | 3.255ms | 2.44767 | 1.18285 | 0.0491185 | "
],
"text/plain": [
"| 5 | 3.255ms | 2.44767 | 1.18285 | 0.0491185 |"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"| 6 | 3.777ms | 2.39372 | 1.17296 | 0.042841 | "
],
"text/plain": [
"| 6 | 3.777ms | 2.39372 | 1.17296 | 0.042841 |"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"| 11 | 6.085ms | 2.27879 | 1.09218 | 0.0271912 | "
],
"text/plain": [
"| 11 | 6.085ms | 2.27879 | 1.09218 | 0.0271912 |"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"+---------+--------------+-------------------+-----------------------+-------------+ "
],
"text/plain": [
"+---------+--------------+-------------------+-----------------------+-------------+"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"Optimization Complete: Maximum number of passes through the data reached. "
],
"text/plain": [
"Optimization Complete: Maximum number of passes through the data reached."
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"Computing final objective value and training RMSE. "
],
"text/plain": [
"Computing final objective value and training RMSE."
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" Final objective value: 2.80467 "
],
"text/plain": [
" Final objective value: 2.80467"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
" Final training RMSE: 1.04538 "
],
"text/plain": [
" Final training RMSE: 1.04538"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"+---------+---------+---------------+------+\n",
"| user_id | item_id | score | rank |\n",
"+---------+---------+---------------+------+\n",
"| 0 | d | 1.24744260311 | 1 |\n",
"| 1 | c | 3.97898635268 | 1 |\n",
"| 1 | d | 3.09123837948 | 2 |\n",
"| 2 | a | 2.52528711408 | 1 |\n",
"+---------+---------+---------------+------+\n",
"[4 rows x 4 columns]\n",
"\n"
]
}
],
"source": [
"m = graphlab.recommender.create(sf, target='rating')\n",
"recs = m.recommend()\n",
"print recs"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"collapsed": false,
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"data": {
"text/plain": [
"{'intercept': 2.875, 'item_id': Columns:\n",
" \titem_id\tstr\n",
" \tlinear_terms\tfloat\n",
" \tfactors\tarray\n",
" \n",
" Rows: 4\n",
" \n",
" Data:\n",
" +---------+------------------+-------------------------------+\n",
" | item_id | linear_terms | factors |\n",
" +---------+------------------+-------------------------------+\n",
" | a | -0.0691520795226 | [0.000691717606969, -0.001... |\n",
" | b | -0.0680508315563 | [-0.000844004040118, -0.00... |\n",
" | c | 0.277156561613 | [4.07315637858e-05, 0.0025... |\n",
" | d | -0.61060488224 | [0.00015950971283, 0.00215... |\n",
" +---------+------------------+-------------------------------+\n",
" [4 rows x 3 columns], 'user_id': Columns:\n",
" \tuser_id\tstr\n",
" \tlinear_terms\tfloat\n",
" \tfactors\tarray\n",
" \n",
" Rows: 3\n",
" \n",
" Data:\n",
" +---------+----------------+-------------------------------+\n",
" | user_id | linear_terms | factors |\n",
" +---------+----------------+-------------------------------+\n",
" | 0 | -1.01692664623 | [-0.000924227701034, -0.00... |\n",
" | 1 | 0.826922476292 | [0.000409463362303, -0.002... |\n",
" | 2 | -0.28046759963 | [0.000454146531411, 0.0036... |\n",
" +---------+----------------+-------------------------------+\n",
" [3 rows x 3 columns]}"
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"m['coefficients']"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"After importing GraphLab Create, we can download data directly from S3. We have placed a preprocessed version of the [Million Song Dataset](http://labrosa.ee.columbia.edu/millionsong/) on S3. This data set was used for a [Kaggle challenge](https://www.kaggle.com/c/msdchallenge) and includes data from [The Echo Nest](http://the.echonest.com/), [SecondHandSongs](http://www.secondhandsongs.com/), [musiXmatch](http://musixmatch.com/), and [Last.fm](http://www.last.fm/). This file includes data for a subset of 10000 songs."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## The CourseTalk dataset: loading and first look\n",
"\n",
"Loading of the CourseTalk database."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": false,
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"------------------------------------------------------\n",
"Inferred types from first line of file as \n",
"column_type_hints=[int,int,float]\n",
"If parsing fails due to incorrect types, you can correct\n",
"the inferred type list above and pass it to read_csv in\n",
"the column_type_hints argument\n",
"------------------------------------------------------\n",
"PROGRESS: Finished parsing file /Users/chengjun/GitHub/cjc2016/data/ratings.dat\n",
"PROGRESS: Parsing completed. Parsed 2773 lines in 0.0105 secs.\n"
]
},
{
"data": {
"application/javascript": [
"$(\"head\").append($(\""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"result = gl.recommender.util.compare_models(test_set, [popularity_model, item_sim_model, factorization_machine_model],\n",
" user_sample=.1, skip_set=train_set)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Now let's ask the item similarity model for song recommendations on several users. We first create a list of users and create a subset of observations, `users_ratings`, that pertain to these users."
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {
"collapsed": false,
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [],
"source": [
"K = 10\n",
"users = gl.SArray(sf['user_id'].unique().head(100))"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Next we use the `recommend()` function to query the model we created for recommendations. The returned object has four columns: `user_id`, `song_id`, the `score` that the algorithm gave this user for this song, and the song's rank (an integer from 0 to K-1). To see this we can grab the top few rows of `recs`:"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"collapsed": false,
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" user_id \n",
" course_id \n",
" score \n",
" rank \n",
" \n",
" \n",
" 232 \n",
" 15 \n",
" 5.0 \n",
" 1 \n",
" \n",
" \n",
" 232 \n",
" 14 \n",
" 5.0 \n",
" 2 \n",
" \n",
" \n",
" 232 \n",
" 13 \n",
" 5.0 \n",
" 3 \n",
" \n",
" \n",
" 232 \n",
" 12 \n",
" 5.0 \n",
" 4 \n",
" \n",
" \n",
" 232 \n",
" 11 \n",
" 5.0 \n",
" 5 \n",
" \n",
" \n",
" 232 \n",
" 10 \n",
" 5.0 \n",
" 6 \n",
" \n",
" \n",
" 232 \n",
" 9 \n",
" 5.0 \n",
" 7 \n",
" \n",
" \n",
" 232 \n",
" 8 \n",
" 5.0 \n",
" 8 \n",
" \n",
" \n",
" 232 \n",
" 7 \n",
" 5.0 \n",
" 9 \n",
" \n",
" \n",
" 232 \n",
" 4 \n",
" 5.0 \n",
" 10 \n",
" \n",
"
\n",
"[10 rows x 4 columns]
\n",
"
API docs .)\n",
"- Toby Segaran, 2007, Programming Collective Intelligence. O'Reilly. Chapter 2 Making Recommendations\n",
" - programming-collective-intelligence-code/blob/master/chapter2/recommendations.py\n",
"- 项亮 2012 推荐系统实践 人民邮电出版社"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
}
],
"metadata": {
"celltoolbar": "Slideshow",
"kernelspec": {
"display_name": "Python 2",
"language": "python",
"name": "python2"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 2
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.11"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
\n",
"\n",
"- Toby相似的五个用户（Rose, Reymour, Puig, LaSalle, Matthews）及相似度（依次为0.99， 0.38， 0.89， 0.92）\n",
"- 这五个用户看过的三个电影（Night,Lady, Luck）及其评分\n",
" - 例如，Rose对Night评分是3.0\n",
"- S.xNight是用户相似度与电影评分的乘积\n",
" - 例如，Toby于Rose相似度对Night评分是3.0*0.99 = 2.97\n",
"- 可以得到每部电影的得分\n",
" - 例如，Night的得分是12.89 = 2.97+1.14+4.02+2.77+1.99\n",
"- 电影得分需要使用用户相似度之和进行加权\n",
" - 例如，Night电影的预测得分是3.35 = 12.89/3.84\n"
]
},
{
"cell_type": "code",
"execution_count": 66,
"metadata": {
"collapsed": false,
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"# Gets recommendations for a person by using a weighted average\n",
"# of every other user's rankings\n",
"def getRecommendations(prefs,person,similarity=sim_pearson):\n",
" totals={}\n",
" simSums={}\n",
" for other in prefs:\n",
" # don't compare me to myself\n",
" if other==person: continue\n",
" sim=similarity(prefs,person,other)\n",
" # ignore scores of zero or lower\n",
" if sim<=0: continue\n",
" for item in prefs[other]: \n",
" # only score movies I haven't seen yet\n",
" if item not in prefs[person] or prefs[person][item]==0:\n",
" # Similarity * Score\n",
" totals.setdefault(item,0)\n",
" totals[item]+=prefs[other][item]*sim\n",
" # Sum of similarities\n",
" simSums.setdefault(item,0)\n",
" simSums[item]+=sim\n",
" # Create the normalized list\n",
" rankings=[(total/simSums[item],item) for item,total in totals.items()]\n",
" # Return the sorted list\n",
" rankings.sort()\n",
" rankings.reverse()\n",
" return rankings"
]
},
{
"cell_type": "code",
"execution_count": 78,
"metadata": {
"collapsed": false,
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"data": {
"text/plain": [
"[(3.3477895267131013, 'The Night Listener'),\n",
" (2.8325499182641614, 'Lady in the Water'),\n",
" (2.5309807037655645, 'Just My Luck')]"
]
},
"execution_count": 78,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Now you can find out what movies I should watch next:\n",
"getRecommendations(critics,'Toby')"
]
},
{
"cell_type": "code",
"execution_count": 79,
"metadata": {
"collapsed": false,
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"data": {
"text/plain": [
"[(3.5002478401415877, 'The Night Listener'),\n",
" (2.7561242939959363, 'Lady in the Water'),\n",
" (2.4619884860743739, 'Just My Luck')]"
]
},
"execution_count": 79,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# You’ll find that the results are only affected very slightly by the choice of similarity metric.\n",
"getRecommendations(critics,'Toby',similarity=sim_distance)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"***\n",
"# 2. Item-based filtering\n",
"***\n",
"\n",
"Now you know how to find similar people and recommend products for a given person\n",
"\n",
"### But what if you want to see which products are similar to each other? \n",
"This is actually the same method we used earlier to determine similarity between people—"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"#### 将item-user字典的键值翻转"
]
},
{
"cell_type": "code",
"execution_count": 85,
"metadata": {
"collapsed": false,
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"# you just need to swap the people and the items. \n",
"def transformPrefs(prefs):\n",
" result={}\n",
" for person in prefs:\n",
" for item in prefs[person]:\n",
" result.setdefault(item,{})\n",
" # Flip item and person\n",
" result[item][person]=prefs[person][item]\n",
" return result\n",
"\n",
"movies = transformPrefs(critics)\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"#### 计算item的相似性"
]
},
{
"cell_type": "code",
"execution_count": 83,
"metadata": {
"collapsed": false,
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"data": {
"text/plain": [
"[(0.65795169495976946, 'You, Me and Dupree'),\n",
" (0.48795003647426888, 'Lady in the Water'),\n",
" (0.11180339887498941, 'Snakes on a Plane'),\n",
" (-0.17984719479905439, 'The Night Listener'),\n",
" (-0.42289003161103106, 'Just My Luck')]"
]
},
"execution_count": 83,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"topMatches(movies,'Superman Returns')"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"#### 给item推荐user"
]
},
{
"cell_type": "code",
"execution_count": 86,
"metadata": {
"collapsed": false,
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"data": {
"text/plain": [
"{'Just My Luck': [(0.22222222222222221, 'Lady in the Water'),\n",
" (0.18181818181818182, 'You, Me and Dupree'),\n",
" (0.15384615384615385, 'The Night Listener'),\n",
" (0.10526315789473684, 'Snakes on a Plane'),\n",
" (0.064516129032258063, 'Superman Returns')],\n",
" 'Lady in the Water': [(0.40000000000000002, 'You, Me and Dupree'),\n",
" (0.2857142857142857, 'The Night Listener'),\n",
" (0.22222222222222221, 'Snakes on a Plane'),\n",
" (0.22222222222222221, 'Just My Luck'),\n",
" (0.090909090909090912, 'Superman Returns')],\n",
" 'Snakes on a Plane': [(0.22222222222222221, 'Lady in the Water'),\n",
" (0.18181818181818182, 'The Night Listener'),\n",
" (0.16666666666666666, 'Superman Returns'),\n",
" (0.10526315789473684, 'Just My Luck'),\n",
" (0.05128205128205128, 'You, Me and Dupree')],\n",
" 'Superman Returns': [(0.16666666666666666, 'Snakes on a Plane'),\n",
" (0.10256410256410256, 'The Night Listener'),\n",
" (0.090909090909090912, 'Lady in the Water'),\n",
" (0.064516129032258063, 'Just My Luck'),\n",
" (0.053333333333333337, 'You, Me and Dupree')],\n",
" 'The Night Listener': [(0.2857142857142857, 'Lady in the Water'),\n",
" (0.18181818181818182, 'Snakes on a Plane'),\n",
" (0.15384615384615385, 'Just My Luck'),\n",
" (0.14814814814814814, 'You, Me and Dupree'),\n",
" (0.10256410256410256, 'Superman Returns')],\n",
" 'You, Me and Dupree': [(0.40000000000000002, 'Lady in the Water'),\n",
" (0.18181818181818182, 'Just My Luck'),\n",
" (0.14814814814814814, 'The Night Listener'),\n",
" (0.053333333333333337, 'Superman Returns'),\n",
" (0.05128205128205128, 'Snakes on a Plane')]}"
]
},
"execution_count": 86,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def calculateSimilarItems(prefs,n=10):\n",
" # Create a dictionary of items showing which other items they\n",
" # are most similar to.\n",
" result={}\n",
" # Invert the preference matrix to be item-centric\n",
" itemPrefs=transformPrefs(prefs)\n",
" c=0\n",
" for item in itemPrefs:\n",
" # Status updates for large datasets\n",
" c+=1\n",
" if c%100==0: \n",
" print \"%d / %d\" % (c,len(itemPrefs))\n",
" # Find the most similar items to this one\n",
" scores=topMatches(itemPrefs,item,n=n,similarity=sim_distance)\n",
" result[item]=scores\n",
" return result\n",
"\n",
"itemsim=calculateSimilarItems(critics) \n",
"itemsim"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"
\n",
"\n",
"- Toby看过三个电影（snakes、Superman、dupree）和评分（依次是4.5、4.0、1.0）\n",
"- 表格2-3给出这三部电影与另外三部电影的相似度\n",
" - 例如superman与night的相似度是0.103\n",
"- R.xNight表示Toby对自己看过的三部定影的评分与Night这部电影相似度的乘积\n",
" - 例如，0.412 = 4.0*0.103\n",
" \n",
" \n",
"- 那么Toby对于Night的评分可以表达为0.818+0.412+0.148 = 1.378\n",
" - 已经知道Night相似度之和是0.182+0.103+0.148 = 0.433\n",
" - 那么Toby对Night的最终评分可以表达为：1.378/0.433 = 3.183\n"
]
},
{
"cell_type": "code",
"execution_count": 88,
"metadata": {
"collapsed": false,
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"data": {
"text/plain": [
"[(3.182634730538922, 'The Night Listener'),\n",
" (2.5983318700614575, 'Just My Luck'),\n",
" (2.4730878186968837, 'Lady in the Water')]"
]
},
"execution_count": 88,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def getRecommendedItems(prefs,itemMatch,user):\n",
" userRatings=prefs[user]\n",
" scores={}\n",
" totalSim={}\n",
" # Loop over items rated by this user\n",
" for (item,rating) in userRatings.items( ):\n",
" # Loop over items similar to this one\n",
" for (similarity,item2) in itemMatch[item]:\n",
" # Ignore if this user has already rated this item\n",
" if item2 in userRatings: continue\n",
" # Weighted sum of rating times similarity\n",
" scores.setdefault(item2,0)\n",
" scores[item2]+=similarity*rating\n",
" # Sum of all the similarities\n",
" totalSim.setdefault(item2,0)\n",
" totalSim[item2]+=similarity\n",
" # Divide each total score by total weighting to get an average\n",
" rankings=[(score/totalSim[item],item) for item,score in scores.items( )]\n",
" # Return the rankings from highest to lowest\n",
" rankings.sort( )\n",
" rankings.reverse( )\n",
" return rankings\n",
"\n",
"getRecommendedItems(critics,itemsim,'Toby')"
]
},
{
"cell_type": "code",
"execution_count": 81,
"metadata": {
"collapsed": false,
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"data": {
"text/plain": [
"[(4.0, 'Michael Phillips'), (3.0, 'Jack Matthews')]"
]
},
"execution_count": 81,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"getRecommendations(movies,'Just My Luck')"
]
},
{
"cell_type": "code",
"execution_count": 84,
"metadata": {
"collapsed": false,
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"data": {
"text/plain": [
"[(3.1637361366111816, 'Michael Phillips')]"
]
},
"execution_count": 84,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"getRecommendations(movies, 'You, Me and Dupree')"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"
\n",
"\n",
"# 基于物品的协同过滤算法的网络表示方法"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# 基于图的模型\n",
"\n",
"使用二分图表示用户行为，因此基于图的算法可以应用到推荐系统当中。\n",
"\n",
"
"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"collapsed": false,
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"# https://github.com/ParticleWave/RecommendationSystemStudy/blob/d1960056b96cfaad62afbfe39225ff680240d37e/PersonalRank.py\n",
"import os\n",
"import random\n",
"\n",
"class Graph:\n",
" def __init__(self):\n",
" self.G = dict()\n",
" \n",
" def addEdge(self, p, q):\n",
" if p not in self.G: self.G[p] = dict()\n",
" if q not in self.G: self.G[q] = dict()\n",
" self.G[p][q] = 1\n",
" self.G[q][p] = 1\n",
"\n",
" def getGraphMatrix(self):\n",
" return self.G"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"collapsed": false,
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"['A', 'a', 'c', 'B', 'd', 'C', 'b']\n"
]
}
],
"source": [
"graph = Graph()\n",
"graph.addEdge('A', 'a')\n",
"graph.addEdge('A', 'c')\n",
"graph.addEdge('B', 'a')\n",
"graph.addEdge('B', 'b')\n",
"graph.addEdge('B', 'c')\n",
"graph.addEdge('B', 'd')\n",
"graph.addEdge('C', 'c')\n",
"graph.addEdge('C', 'd')\n",
"G = graph.getGraphMatrix()\n",
"print(G.keys())"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"collapsed": false,
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"data": {
"text/plain": [
"{'A': {'a': 1, 'c': 1},\n",
" 'B': {'a': 1, 'b': 1, 'c': 1, 'd': 1},\n",
" 'C': {'c': 1, 'd': 1},\n",
" 'a': {'A': 1, 'B': 1},\n",
" 'b': {'B': 1},\n",
" 'c': {'A': 1, 'B': 1, 'C': 1},\n",
" 'd': {'B': 1, 'C': 1}}"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"G"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"collapsed": true,
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"def PersonalRank(G, alpha, root, max_step):\n",
" # G is the biparitite graph of users' ratings on items\n",
" # alpha is the probability of random walk forward\n",
" # root is the studied User\n",
" # max_step if the steps of iterations.\n",
" rank = dict()\n",
" rank = {x:0.0 for x in G.keys()}\n",
" rank[root] = 1.0\n",
" for k in range(max_step):\n",
" tmp = {x:0.0 for x in G.keys()}\n",
" for i,ri in G.items():\n",
" for j,wij in ri.items():\n",
" if j not in tmp: tmp[j] = 0.0 #\n",
" tmp[j] += alpha * rank[i] / (len(ri)*1.0)\n",
" if j == root: tmp[j] += 1.0 - alpha\n",
" rank = tmp\n",
" print(k, rank)\n",
" return rank"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"collapsed": false,
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"(0, {'A': 0.3999999999999999, 'a': 0.4, 'c': 0.4, 'B': 0.0, 'd': 0.0, 'C': 0.0, 'b': 0.0})\n",
"(1, {'A': 0.6666666666666666, 'a': 0.15999999999999998, 'c': 0.15999999999999998, 'B': 0.2666666666666667, 'd': 0.0, 'C': 0.10666666666666669, 'b': 0.0})\n",
"(2, {'A': 0.5066666666666666, 'a': 0.32, 'c': 0.3626666666666667, 'B': 0.10666666666666665, 'd': 0.09600000000000003, 'C': 0.04266666666666666, 'b': 0.053333333333333344})\n",
"(3, {'A': 0.624711111111111, 'a': 0.22399999999999998, 'c': 0.24106666666666665, 'B': 0.30577777777777787, 'd': 0.03839999999999999, 'C': 0.13511111111111113, 'b': 0.02133333333333333})\n",
"(4, {'A': 0.5538844444444444, 'a': 0.31104, 'c': 0.36508444444444443, 'B': 0.1863111111111111, 'd': 0.11520000000000002, 'C': 0.07964444444444443, 'b': 0.061155555555555574})\n",
"(5, {'A': 0.6217718518518518, 'a': 0.258816, 'c': 0.29067377777777775, 'B': 0.31677629629629633, 'd': 0.06911999999999999, 'C': 0.14343585185185187, 'b': 0.03726222222222222})\n",
"(6, {'A': 0.5810394074074073, 'a': 0.312064, 'c': 0.3694383407407408, 'B': 0.2384971851851852, 'd': 0.12072960000000002, 'C': 0.1051610074074074, 'b': 0.06335525925925926})\n",
"(7, {'A': 0.6233424908641975, 'a': 0.2801152, 'c': 0.322179602962963, 'B': 0.322318538271605, 'd': 0.08976384000000001, 'C': 0.14680873086419757, 'b': 0.047699437037037044})\n",
"(8, {'A': 0.5979606407901235, 'a': 0.313800704, 'c': 0.372524196345679, 'B': 0.27202572641975314, 'd': 0.12318720000000004, 'C': 0.12182009679012348, 'b': 0.06446370765432101})\n",
"(9, {'A': 0.6248600672921809, 'a': 0.2935894016, 'c': 0.34231744031604944, 'B': 0.32570591341563787, 'd': 0.10313318400000002, 'C': 0.14861466569218107, 'b': 0.05440514528395063})\n",
"(10, {'A': 0.6087204113909463, 'a': 0.31508520959999997, 'c': 0.3745310758768724, 'B': 0.29349780121810704, 'd': 0.12458704896, 'C': 0.13253792435094652, 'b': 0.06514118268312757})\n",
"(11, {'A': 0.6259090374071659, 'a': 0.3021877248, 'c': 0.3552028945403786, 'B': 0.3278568031376681, 'd': 0.11171472998400003, 'C': 0.14970977315116596, 'b': 0.05869956024362141})\n",
"(12, {'A': 0.6155958617974342, 'a': 0.31593497559039996, 'c': 0.37581888485086634, 'B': 0.3072414019859315, 'd': 0.125455269888, 'C': 0.13940666387103434, 'b': 0.06557136062753362})\n",
"(13, {'A': 0.6265923595297243, 'a': 0.30768662511615996, 'c': 0.3634492906645737, 'B': 0.32923155598695125, 'd': 0.11721094594560004, 'C': 0.15040047724876437, 'b': 0.0614482803971863})\n",
"(14, {'A': 0.6199944608903503, 'a': 0.31648325500928, 'c': 0.37664344590878573, 'B': 0.3160374635863394, 'd': 0.126006502096896, 'C': 0.14380418922212634, 'b': 0.06584631119739025})\n",
"(15, {'A': 0.6270315542460547, 'a': 0.311205277073408, 'c': 0.36872695276225853, 'B': 0.3301112040427255, 'd': 0.12072916840611841, 'C': 0.1508408530811013, 'b': 0.06320749271726787})\n",
"(16, {'A': 0.6228092982326321, 'a': 0.316834862506967, 'c': 0.37717120373940755, 'B': 0.3216669597688938, 'd': 0.12635858204098563, 'C': 0.14661885476571632, 'b': 0.0660222408085451})\n",
"(17, {'A': 0.6273129326666287, 'a': 0.3134571112468316, 'c': 0.37210465315311814, 'B': 0.3306741581298591, 'd': 0.1229809338600653, 'C': 0.15112242048023627, 'b': 0.06433339195377877})\n",
"(18, {'A': 0.6246107520062307, 'a': 0.3170600046926233, 'c': 0.3775089728847178, 'B': 0.32526983911327995, 'd': 0.12658379981806633, 'C': 0.1484202810515243, 'b': 0.06613483162597182})\n",
"(19, {'A': 0.627493061312974, 'a': 0.3148982686251483, 'c': 0.37426638104575805, 'B': 0.3310344465409781, 'd': 0.1244220802432657, 'C': 0.15130257936315128, 'b': 0.06505396782265599})\n",
"{'A': 0.627493061312974, 'a': 0.3148982686251483, 'c': 0.37426638104575805, 'B': 0.3310344465409781, 'd': 0.1244220802432657, 'C': 0.15130257936315128, 'b': 0.06505396782265599}\n"
]
}
],
"source": [
"print(PersonalRank(G, 0.8, 'A', 20))\n",
"# print(PersonalRank(G, 0.8, 'B', 20))\n",
"# print(PersonalRank(G, 0.8, 'C', 20))"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"***\n",
"# 3. MovieLens Recommender\n",
"***\n",
"MovieLens是一个电影评价的真实数据，由明尼苏达州立大学的GroupLens项目组开发。\n",
"\n",
"### 数据下载\n",
"http://grouplens.org/datasets/movielens/1m/\n",
"\n",
"> These files contain 1,000,209 anonymous ratings of approximately 3,900 movies \n",
"made by 6,040 MovieLens users who joined MovieLens in 2000.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"# 数据格式\n",
"All ratings are contained in the file \"ratings.dat\" and are in the following format:\n",
"\n",
"UserID::MovieID::Rating::Timestamp\n",
"\n",
"1::1193::5::978300760\n",
"\n",
"1::661::3::978302109\n",
"\n",
"1::914::3::978301968\n"
]
},
{
"cell_type": "code",
"execution_count": 91,
"metadata": {
"collapsed": true,
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"def loadMovieLens(path='/Users/chengjun/bigdata/ml-1m/'):\n",
" # Get movie titles\n",
" movies={}\n",
" for line in open(path+'movies.dat'):\n",
" (id,title)=line.split('::')[0:2]\n",
" movies[id]=title\n",
" \n",
" # Load data\n",
" prefs={}\n",
" for line in open(path+'/ratings.dat'):\n",
" (user,movieid,rating,ts)=line.split('::')\n",
" prefs.setdefault(user,{})\n",
" prefs[user][movies[movieid]]=float(rating)\n",
" return prefs"
]
},
{
"cell_type": "code",
"execution_count": 92,
"metadata": {
"collapsed": false,
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"data": {
"text/plain": [
"{'Alice in Wonderland (1951)': 1.0,\n",
" 'Army of Darkness (1993)': 3.0,\n",
" 'Bad Boys (1995)': 5.0,\n",
" 'Benji (1974)': 1.0,\n",
" 'Brady Bunch Movie, The (1995)': 1.0,\n",
" 'Braveheart (1995)': 5.0,\n",
" 'Buffalo 66 (1998)': 1.0,\n",
" 'Chambermaid on the Titanic, The (1998)': 1.0,\n",
" 'Cowboy Way, The (1994)': 1.0,\n",
" 'Cyrano de Bergerac (1990)': 4.0,\n",
" 'Dear Diary (Caro Diario) (1994)': 1.0,\n",
" 'Die Hard (1988)': 3.0,\n",
" 'Diebinnen (1995)': 1.0,\n",
" 'Dr. No (1962)': 1.0,\n",
" 'Escape from the Planet of the Apes (1971)': 1.0,\n",
" 'Fast, Cheap & Out of Control (1997)': 1.0,\n",
" 'Faster Pussycat! Kill! Kill! (1965)': 1.0,\n",
" 'From Russia with Love (1963)': 1.0,\n",
" 'Fugitive, The (1993)': 5.0,\n",
" 'Get Shorty (1995)': 1.0,\n",
" 'Gladiator (2000)': 5.0,\n",
" 'Goldfinger (1964)': 5.0,\n",
" 'Good, The Bad and The Ugly, The (1966)': 4.0,\n",
" 'Hunt for Red October, The (1990)': 5.0,\n",
" 'Hurricane, The (1999)': 5.0,\n",
" 'Indiana Jones and the Last Crusade (1989)': 4.0,\n",
" 'Jaws (1975)': 5.0,\n",
" 'Jurassic Park (1993)': 5.0,\n",
" 'King Kong (1933)': 1.0,\n",
" 'King of New York (1990)': 1.0,\n",
" 'Last of the Mohicans, The (1992)': 1.0,\n",
" 'Lethal Weapon (1987)': 5.0,\n",
" 'Longest Day, The (1962)': 1.0,\n",
" 'Man with the Golden Gun, The (1974)': 5.0,\n",
" 'Mask of Zorro, The (1998)': 5.0,\n",
" 'Matrix, The (1999)': 5.0,\n",
" \"On Her Majesty's Secret Service (1969)\": 1.0,\n",
" 'Out of Sight (1998)': 1.0,\n",
" 'Palookaville (1996)': 1.0,\n",
" 'Planet of the Apes (1968)': 1.0,\n",
" 'Pope of Greenwich Village, The (1984)': 1.0,\n",
" 'Princess Bride, The (1987)': 3.0,\n",
" 'Raiders of the Lost Ark (1981)': 4.0,\n",
" 'Rock, The (1996)': 5.0,\n",
" 'Rocky (1976)': 5.0,\n",
" 'Saving Private Ryan (1998)': 4.0,\n",
" 'Shanghai Noon (2000)': 1.0,\n",
" 'Speed (1994)': 1.0,\n",
" 'Star Wars: Episode IV - A New Hope (1977)': 5.0,\n",
" 'Star Wars: Episode V - The Empire Strikes Back (1980)': 5.0,\n",
" 'Taking of Pelham One Two Three, The (1974)': 1.0,\n",
" 'Terminator 2: Judgment Day (1991)': 5.0,\n",
" 'Terminator, The (1984)': 4.0,\n",
" 'Thelma & Louise (1991)': 1.0,\n",
" 'True Romance (1993)': 1.0,\n",
" 'U-571 (2000)': 5.0,\n",
" 'Untouchables, The (1987)': 5.0,\n",
" 'Westworld (1973)': 1.0,\n",
" 'X-Men (2000)': 4.0}"
]
},
"execution_count": 92,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"prefs=loadMovieLens()\n",
"prefs['87']"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### user-based filtering"
]
},
{
"cell_type": "code",
"execution_count": 93,
"metadata": {
"collapsed": false,
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"data": {
"text/plain": [
"[(5.0, 'Time of the Gypsies (Dom za vesanje) (1989)'),\n",
" (5.0, 'Tigrero: A Film That Was Never Made (1994)'),\n",
" (5.0, 'Schlafes Bruder (Brother of Sleep) (1995)'),\n",
" (5.0, 'Return with Honor (1998)'),\n",
" (5.0, 'Lured (1947)'),\n",
" (5.0, 'Identification of a Woman (Identificazione di una donna) (1982)'),\n",
" (5.0, 'I Am Cuba (Soy Cuba/Ya Kuba) (1964)'),\n",
" (5.0, 'Hour of the Pig, The (1993)'),\n",
" (5.0, 'Gay Deceivers, The (1969)'),\n",
" (5.0, 'Gate of Heavenly Peace, The (1995)'),\n",
" (5.0, 'Foreign Student (1994)'),\n",
" (5.0, 'Dingo (1992)'),\n",
" (5.0, 'Dangerous Game (1993)'),\n",
" (5.0, 'Callej\\xf3n de los milagros, El (1995)'),\n",
" (5.0, 'Bittersweet Motel (2000)'),\n",
" (4.8204601017229889, 'Apple, The (Sib) (1998)'),\n",
" (4.7389561849363862, 'Lamerica (1994)'),\n",
" (4.6818165414673958, 'Bells, The (1926)'),\n",
" (4.6649580725222339, 'Hurricane Streets (1998)'),\n",
" (4.6507418408045593, 'Sanjuro (1962)'),\n",
" (4.6499741726003458, 'On the Ropes (1999)'),\n",
" (4.6368254087395071, 'Shawshank Redemption, The (1994)'),\n",
" (4.627888709544556, 'For All Mankind (1989)'),\n",
" (4.5820483492805089, 'Midaq Alley (Callej\\xf3n de los milagros, El) (1995)'),\n",
" (4.5797786468711532, \"Schindler's List (1993)\"),\n",
" (4.5751999410373871,\n",
" 'Seven Samurai (The Magnificent Seven) (Shichinin no samurai) (1954)'),\n",
" (4.5749049884034561, 'Godfather, The (1972)'),\n",
" (4.5746840191882345, \"Ed's Next Move (1996)\"),\n",
" (4.5585190371478284, 'Hanging Garden, The (1997)'),\n",
" (4.5277600427755909, 'Close Shave, A (1995)')]"
]
},
"execution_count": 93,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"getRecommendations(prefs,'87')[0:30]"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Item-based filtering"
]
},
{
"cell_type": "code",
"execution_count": 94,
"metadata": {
"collapsed": false,
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"100 / 3706\n",
"200 / 3706\n",
"300 / 3706\n",
"400 / 3706\n",
"500 / 3706\n",
"600 / 3706\n",
"700 / 3706\n",
"800 / 3706\n",
"900 / 3706\n",
"1000 / 3706\n",
"1100 / 3706\n",
"1200 / 3706\n",
"1300 / 3706\n",
"1400 / 3706\n",
"1500 / 3706\n",
"1600 / 3706\n",
"1700 / 3706\n",
"1800 / 3706\n",
"1900 / 3706\n",
"2000 / 3706\n",
"2100 / 3706\n",
"2200 / 3706\n",
"2300 / 3706\n",
"2400 / 3706\n",
"2500 / 3706\n",
"2600 / 3706\n",
"2700 / 3706\n",
"2800 / 3706\n",
"2900 / 3706\n",
"3000 / 3706\n",
"3100 / 3706\n",
"3200 / 3706\n",
"3300 / 3706\n",
"3400 / 3706\n",
"3500 / 3706\n",
"3600 / 3706\n",
"3700 / 3706\n"
]
}
],
"source": [
"itemsim=calculateSimilarItems(prefs,n=50)"
]
},
{
"cell_type": "code",
"execution_count": 95,
"metadata": {
"collapsed": false,
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"data": {
"text/plain": [
"[(5.0, 'Uninvited Guest, An (2000)'),\n",
" (5.0, 'Two Much (1996)'),\n",
" (5.0, 'Two Family House (2000)'),\n",
" (5.0, 'Trial by Jury (1994)'),\n",
" (5.0, 'Tom & Viv (1994)'),\n",
" (5.0, 'This Is My Father (1998)'),\n",
" (5.0, 'Something to Sing About (1937)'),\n",
" (5.0, 'Slappy and the Stinkers (1998)'),\n",
" (5.0, 'Running Free (2000)'),\n",
" (5.0, 'Roula (1995)'),\n",
" (5.0, 'Prom Night IV: Deliver Us From Evil (1992)'),\n",
" (5.0, 'Project Moon Base (1953)'),\n",
" (5.0, 'Price Above Rubies, A (1998)'),\n",
" (5.0, 'Open Season (1996)'),\n",
" (5.0, 'Only Angels Have Wings (1939)'),\n",
" (5.0, 'Onegin (1999)'),\n",
" (5.0, 'Once Upon a Time... When We Were Colored (1995)'),\n",
" (5.0, 'Office Killer (1997)'),\n",
" (5.0, 'N\\xe9nette et Boni (1996)'),\n",
" (5.0, 'No Looking Back (1998)'),\n",
" (5.0, 'Never Met Picasso (1996)'),\n",
" (5.0, 'Music From Another Room (1998)'),\n",
" (5.0, \"Mummy's Tomb, The (1942)\"),\n",
" (5.0, 'Modern Affair, A (1995)'),\n",
" (5.0, 'Machine, The (1994)'),\n",
" (5.0, 'Lured (1947)'),\n",
" (5.0, 'Low Life, The (1994)'),\n",
" (5.0, 'Lodger, The (1926)'),\n",
" (5.0, 'Loaded (1994)'),\n",
" (5.0, 'Line King: Al Hirschfeld, The (1996)')]"
]
},
"execution_count": 95,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"getRecommendedItems(prefs,itemsim,'87')[0:30]"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# Buiding Recommendation System with GraphLab"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"In this notebook we will import [GraphLab Create](https://dato.com/products/create) and use it to\n",
"\n",
"- train two models that can be used for recommending new songs to users \n",
"- compare the performance of the two models\n"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": true,
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"# set product key using GraphLab Create API\n",
"#import graphlab\n",
"#graphlab.product_key.set_product_key('4972-65DF-8E02-816C-AB15-021C-EC1B-0367')\n"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"collapsed": false,
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"%matplotlib inline\n",
"import graphlab\n",
"# set canvas to show sframes and sgraphs in ipython notebook\n",
"gl.canvas.set_target('ipynb')\n",
"import matplotlib.pyplot as plt\n"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"collapsed": false,
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"2016-05-21 09:54:26,565 [INFO] graphlab.cython.cy_server, 176: GraphLab Create v1.8.5 started. Logging: /tmp/graphlab_server_1463795663.log\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"This non-commercial license of GraphLab Create is assigned to wangchengjun@nju.edu.cn and will expire on July 31, 2016. For commercial licensing options, visit https://dato.com/buy/.\n"
]
},
{
"data": {
"text/html": [
"