{ "metadata": { "name": "", "signature": "sha256:385227659ff4511c03f3655dac3d2046d42294a9b8dd56a4494d79efe6ade2af" }, "nbformat": 3, "nbformat_minor": 0, "worksheets": [ { "cells": [ { "cell_type": "heading", "level": 1, "metadata": {}, "source": [ "The Five-Line Recommender, Explained" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Building a recommender system is easy with GraphLab Create. Simply import graphlab, load data, create a recommender model, and start making recommendations. Let's walk through this line by line.\n", "\n", "

Step 1: Import GraphLab into Python

" ] }, { "cell_type": "code", "collapsed": false, "input": [ "import graphlab" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 1 }, { "cell_type": "markdown", "metadata": {}, "source": [ "

Step 2: Load the dataset

\n", "\n", "The data is sitting on an AWS S3 bucket as a csv file. We can load it into a GraphLab SFrame with `read_csv()`, specifying the \"rating\" column to be loaded as integers. For other ways of creating an SFrame and doing data munging, see the SFrame tutorial." ] }, { "cell_type": "code", "collapsed": false, "input": [ "data = graphlab.SFrame.read_csv(\"http://s3.amazonaws.com/dato-datasets/movie_ratings/training_data.csv\", column_type_hints={\"rating\":int})\n", "data.head()" ], "language": "python", "metadata": {}, "outputs": [ { "html": [ "
PROGRESS: Downloading http://s3.amazonaws.com/dato-datasets/movie_ratings/training_data.csv to /var/tmp/graphlab-zach/39733/000000.csv
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: Downloading http://s3.amazonaws.com/dato-datasets/movie_ratings/training_data.csv to /var/tmp/graphlab-zach/39733/000000.csv" ] }, { "html": [ "
PROGRESS: Finished parsing file http://s3.amazonaws.com/dato-datasets/movie_ratings/training_data.csv
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: Finished parsing file http://s3.amazonaws.com/dato-datasets/movie_ratings/training_data.csv" ] }, { "html": [ "
PROGRESS: Parsing completed. Parsed 100 lines in 0.109583 secs.
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: Parsing completed. Parsed 100 lines in 0.109583 secs." ] }, { "html": [ "
PROGRESS: Finished parsing file http://s3.amazonaws.com/dato-datasets/movie_ratings/training_data.csv
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: Finished parsing file http://s3.amazonaws.com/dato-datasets/movie_ratings/training_data.csv" ] }, { "html": [ "
PROGRESS: Parsing completed. Parsed 82068 lines in 0.101536 secs.
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: Parsing completed. Parsed 82068 lines in 0.101536 secs." ] }, { "html": [ "
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
usermovierating
Jacob SmithFlirting with Disaster4
Jacob SmithIndecent Proposal3
Jacob SmithRunaway Bride2
Jacob SmithSwiss Family Robinson1
Jacob SmithThe Mexican2
Jacob SmithMaid in Manhattan4
Jacob SmithA Charlie Brown
Thanksgiving / The ...		3
Jacob SmithBrazil1
Jacob SmithForrest Gump3
Jacob SmithIt Happened One Night4
\n", "[10 rows x 3 columns]
\n", "
" ], "metadata": {}, "output_type": "pyout", "prompt_number": 2, "text": [ "Columns:\n", "\tuser\tstr\n", "\tmovie\tstr\n", "\trating\tint\n", "\n", "Rows: 10\n", "\n", "Data:\n", "+-------------+-------------------------------+--------+\n", "| user | movie | rating |\n", "+-------------+-------------------------------+--------+\n", "| Jacob Smith | Flirting with Disaster | 4 |\n", "| Jacob Smith | Indecent Proposal | 3 |\n", "| Jacob Smith | Runaway Bride | 2 |\n", "| Jacob Smith | Swiss Family Robinson | 1 |\n", "| Jacob Smith | The Mexican | 2 |\n", "| Jacob Smith | Maid in Manhattan | 4 |\n", "| Jacob Smith | A Charlie Brown Thanksgivi... | 3 |\n", "| Jacob Smith | Brazil | 1 |\n", "| Jacob Smith | Forrest Gump | 3 |\n", "| Jacob Smith | It Happened One Night | 4 |\n", "+-------------+-------------------------------+--------+\n", "[10 rows x 3 columns]" ] } ], "prompt_number": 2 }, { "cell_type": "markdown", "metadata": {}, "source": [ "

Step 3: Build a model

\n", "\n", "We have the data. It's time to build a model. Now, have you ever tried typing \"item similarity, matrix factorization, factorization machine, popularity\" ten times in a row? We have, and we don't recommend it. (Harhar.) There are many good models for making recommendations, but sometimes even knowing the right names can be a challenge, much less typing them time after time.\n", "\n", "This is why GraphLab Create provides a default recommender called ... `recommender`. You can build a default recommender with `recommender.create()`. It requires a dataset to use for training the model, as well as the names of the columns that contain the user IDs, the item IDs, and the ratings (if present)." ] }, { "cell_type": "code", "collapsed": false, "input": [ "# Build a default recommender (a Matrix Factorization model)\n", "# The data needs to contain at least three columns: user, item, and rating.\n", "# All other columns in the dataset are ignored by the default recommender.\n", "model = graphlab.recommender.create(data, user_id=\"user\", item_id=\"movie\", target=\"rating\")" ], "language": "python", "metadata": {}, "outputs": [ { "html": [ "
PROGRESS: Recsys training: model = ranking_factorization_recommender
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: Recsys training: model = ranking_factorization_recommender" ] }, { "html": [ "
PROGRESS: Preparing data set.
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: Preparing data set." ] }, { "html": [ "
PROGRESS:     Data has 82068 observations with 334 users and 7714 items.
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: Data has 82068 observations with 334 users and 7714 items." ] }, { "html": [ "
PROGRESS:     Data prepared in: 0.204213s
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: Data prepared in: 0.204213s" ] }, { "html": [ "
PROGRESS: Training ranking_factorization_recommender for recommendations.
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: Training ranking_factorization_recommender for recommendations." ] }, { "html": [ "
PROGRESS: +--------------------------------+--------------------------------------------------+----------+
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: +--------------------------------+--------------------------------------------------+----------+" ] }, { "html": [ "
PROGRESS: | Parameter                      | Description                                      | Value    |
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: | Parameter | Description | Value |" ] }, { "html": [ "
PROGRESS: +--------------------------------+--------------------------------------------------+----------+
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: +--------------------------------+--------------------------------------------------+----------+" ] }, { "html": [ "
PROGRESS: | num_factors                    | Factor Dimension                                 | 32       |
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: | num_factors | Factor Dimension | 32 |" ] }, { "html": [ "
PROGRESS: | regularization                 | L2 Regularization on Factors                     | 1e-09    |
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: | regularization | L2 Regularization on Factors | 1e-09 |" ] }, { "html": [ "
PROGRESS: | solver                         | Solver used for training                         | sgd      |
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: | solver | Solver used for training | sgd |" ] }, { "html": [ "
PROGRESS: | linear_regularization          | L2 Regularization on Linear Coefficients         | 1e-09    |
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: | linear_regularization | L2 Regularization on Linear Coefficients | 1e-09 |" ] }, { "html": [ "
PROGRESS: | ranking_regularization         | Rank-based Regularization Weight                 | 0.25     |
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: | ranking_regularization | Rank-based Regularization Weight | 0.25 |" ] }, { "html": [ "
PROGRESS: | max_iterations                 | Maximum Number of Iterations                     | 25       |
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: | max_iterations | Maximum Number of Iterations | 25 |" ] }, { "html": [ "
PROGRESS: +--------------------------------+--------------------------------------------------+----------+
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: +--------------------------------+--------------------------------------------------+----------+" ] }, { "html": [ "
PROGRESS:   Optimizing model using SGD; tuning step size.
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: Optimizing model using SGD; tuning step size." ] }, { "html": [ "
PROGRESS:   Using 10258 / 82068 points for tuning the step size.
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: Using 10258 / 82068 points for tuning the step size." ] }, { "html": [ "
PROGRESS: +---------+-------------------+------------------------------------------+
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: +---------+-------------------+------------------------------------------+" ] }, { "html": [ "
PROGRESS: | Attempt | Initial Step Size | Estimated Objective Value                |
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: | Attempt | Initial Step Size | Estimated Objective Value |" ] }, { "html": [ "
PROGRESS: +---------+-------------------+------------------------------------------+
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: +---------+-------------------+------------------------------------------+" ] }, { "html": [ "
PROGRESS: | 0       | 25                | Not Viable                               |
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: | 0 | 25 | Not Viable |" ] }, { "html": [ "
PROGRESS: | 1       | 6.25              | Not Viable                               |
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: | 1 | 6.25 | Not Viable |" ] }, { "html": [ "
PROGRESS: | 2       | 1.5625            | Not Viable                               |
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: | 2 | 1.5625 | Not Viable |" ] }, { "html": [ "
PROGRESS: | 3       | 0.390625          | Not Viable                               |
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: | 3 | 0.390625 | Not Viable |" ] }, { "html": [ "
PROGRESS: | 4       | 0.0976562         | 1.61865                                  |
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: | 4 | 0.0976562 | 1.61865 |" ] }, { "html": [ "
PROGRESS: | 5       | 0.0488281         | 1.66185                                  |
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: | 5 | 0.0488281 | 1.66185 |" ] }, { "html": [ "
PROGRESS: | 6       | 0.0244141         | 1.72837                                  |
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: | 6 | 0.0244141 | 1.72837 |" ] }, { "html": [ "
PROGRESS: | 7       | 0.012207          | 1.80785                                  |
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: | 7 | 0.012207 | 1.80785 |" ] }, { "html": [ "
PROGRESS: +---------+-------------------+------------------------------------------+
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: +---------+-------------------+------------------------------------------+" ] }, { "html": [ "
PROGRESS: | Final   | 0.0976562         | 1.61865                                  |
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: | Final | 0.0976562 | 1.61865 |" ] }, { "html": [ "
PROGRESS: +---------+-------------------+------------------------------------------+
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: +---------+-------------------+------------------------------------------+" ] }, { "html": [ "
PROGRESS: Starting Optimization.
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: Starting Optimization." ] }, { "html": [ "
PROGRESS: +---------+--------------+-------------------+-----------------------+-------------+
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: +---------+--------------+-------------------+-----------------------+-------------+" ] }, { "html": [ "
PROGRESS: | Iter.   | Elapsed Time | Approx. Objective | Approx. Training RMSE | Step Size   |
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: | Iter. | Elapsed Time | Approx. Objective | Approx. Training RMSE | Step Size |" ] }, { "html": [ "
PROGRESS: +---------+--------------+-------------------+-----------------------+-------------+
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: +---------+--------------+-------------------+-----------------------+-------------+" ] }, { "html": [ "
PROGRESS: | Initial | 339us        | 2.40069           | 1.10654               |             |
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: | Initial | 339us | 2.40069 | 1.10654 | |" ] }, { "html": [ "
PROGRESS: +---------+--------------+-------------------+-----------------------+-------------+
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: +---------+--------------+-------------------+-----------------------+-------------+" ] }, { "html": [ "
PROGRESS: | 1       | 121.209ms    | 2.01682           | 1.13516               | 0.0976562   |
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: | 1 | 121.209ms | 2.01682 | 1.13516 | 0.0976562 |" ] }, { "html": [ "
PROGRESS: | 2       | 236.228ms    | 1.76884           | 1.06319               | 0.0580668   |
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: | 2 | 236.228ms | 1.76884 | 1.06319 | 0.0580668 |" ] }, { "html": [ "
PROGRESS: | 3       | 344.786ms    | 1.55833           | 0.983183              | 0.042841    |
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: | 3 | 344.786ms | 1.55833 | 0.983183 | 0.042841 |" ] }, { "html": [ "
PROGRESS: | 4       | 455.018ms    | 1.36545           | 0.906899              | 0.0345267   |
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: | 4 | 455.018ms | 1.36545 | 0.906899 | 0.0345267 |" ] }, { "html": [ "
PROGRESS: | 5       | 558.047ms    | 1.19205           | 0.832933              | 0.029206    |
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: | 5 | 558.047ms | 1.19205 | 0.832933 | 0.029206 |" ] }, { "html": [ "
PROGRESS: | 6       | 688.083ms    | 1.04541           | 0.765651              | 0.0254734   |
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: | 6 | 688.083ms | 1.04541 | 0.765651 | 0.0254734 |" ] }, { "html": [ "
PROGRESS: | 10      | 1.14s        | 0.718906          | 0.605205              | 0.017366    |
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: | 10 | 1.14s | 0.718906 | 0.605205 | 0.017366 |" ] }, { "html": [ "
PROGRESS: | 11      | 1.23s        | 0.674778          | 0.583149              | 0.016168    |
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: | 11 | 1.23s | 0.674778 | 0.583149 | 0.016168 |" ] }, { "html": [ "
PROGRESS: | 20      | 2.22s        | 0.518888          | 0.497694              | 0.0103259   |
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: | 20 | 2.22s | 0.518888 | 0.497694 | 0.0103259 |" ] }, { "html": [ "
PROGRESS: +---------+--------------+-------------------+-----------------------+-------------+
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: +---------+--------------+-------------------+-----------------------+-------------+" ] }, { "html": [ "
PROGRESS: Optimization Complete: Maximum number of passes through the data reached.
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: Optimization Complete: Maximum number of passes through the data reached." ] }, { "html": [ "
PROGRESS: Computing final objective value and training RMSE.
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: Computing final objective value and training RMSE." ] }, { "html": [ "
PROGRESS:        Final objective value: 0.448549
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: Final objective value: 0.448549" ] }, { "html": [ "
PROGRESS:        Final training RMSE: 0.417582
" ], "metadata": {}, "output_type": "display_data", "text": [ "PROGRESS: Final training RMSE: 0.417582" ] } ], "prompt_number": 3 }, { "cell_type": "markdown", "metadata": {}, "source": [ "Details (and the small devil therein):\n", "\n", "Under the hood, the type of recommender is chosen based on the provided data and whether the desired task is ranking (default) or rating prediction. The default recommender for this type of data and the default ranking task is a matrix factorization model, implemented on top of the disk-backed SFrame data structure. The default solver is stochastic gradient descent, and the recommender model used is the RankingFactorizationModel , which balances rating prediction with a ranking objective. The default `create()` function does not allow changes to the default parameters of a specific model, but it is just as easy to build a specific recommender with your own parameters using the appropriate model-specific `create()` function. \n", "\n", "

Step 4: Make recommendations

\n", "\n", "The trained model can now make recommendations of new items for users. To do so, call `recommend()` with an SArray of user ids. If `users` is set to None, then `recommend()` will make recommendations for all the users seen during training, automatically excluding the items that are observed for each user. In other words, if `data` contains a row \"Alice, The Great Gatsby\", then `recommend()` will not recommend \"The Great Gatsby\" for user \"Alice\". It will return at most `k` new items for each user, sorted by their rank. It will return fewer than `k` items if there are not enough items that the user has not already rated or seen.\n", "\n", "The `score` column of the output contains the *unnormalized* prediction scores for each user-item pair. The semantic meanings of these scores may differ between models. For the linear regression model, for instance, a higher average score for a user means that the model thinks that this user is generally more enthusiastic than others. See the Basic Recommender Functionalities tutorial for more details." ] }, { "cell_type": "code", "collapsed": false, "input": [ "# You can now make recommendations for all the users you've just trained on\n", "results = model.recommend(users=None, k=5)\n", "results.head()" ], "language": "python", "metadata": {}, "outputs": [ { "html": [ "
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
usermoviescorerank
Jacob SmithSex and the City: Season
2 ...		5.116391406421
Jacob SmithSex and the City: Season
1 ...		5.026847348572
Jacob SmithSex and the City: Season
6: Part 2 ...		4.854491935143
Jacob SmithSex and the City: Season
3 ...		4.676988826164
Jacob SmithDoctor Zhivago4.65450023215
Mason SmithMulholland Drive6.079183802971
Mason SmithRushmore5.845500693682
Mason SmithThe Sound of Music5.733602271443
Mason SmithNapoleon Dynamite5.483698115714
Mason SmithSix Feet Under: Season 15.384902701745
\n", "[10 rows x 4 columns]
\n", "
" ], "metadata": {}, "output_type": "pyout", "prompt_number": 4, "text": [ "Columns:\n", "\tuser\tstr\n", "\tmovie\tstr\n", "\tscore\tfloat\n", "\trank\tint\n", "\n", "Rows: 10\n", "\n", "Data:\n", "+-------------+-------------------------------+---------------+------+\n", "| user | movie | score | rank |\n", "+-------------+-------------------------------+---------------+------+\n", "| Jacob Smith | Sex and the City: Season 2 | 5.11639140642 | 1 |\n", "| Jacob Smith | Sex and the City: Season 1 | 5.02684734857 | 2 |\n", "| Jacob Smith | Sex and the City: Season 6... | 4.85449193514 | 3 |\n", "| Jacob Smith | Sex and the City: Season 3 | 4.67698882616 | 4 |\n", "| Jacob Smith | Doctor Zhivago | 4.6545002321 | 5 |\n", "| Mason Smith | Mulholland Drive | 6.07918380297 | 1 |\n", "| Mason Smith | Rushmore | 5.84550069368 | 2 |\n", "| Mason Smith | The Sound of Music | 5.73360227144 | 3 |\n", "| Mason Smith | Napoleon Dynamite | 5.48369811571 | 4 |\n", "| Mason Smith | Six Feet Under: Season 1 | 5.38490270174 | 5 |\n", "+-------------+-------------------------------+---------------+------+\n", "[10 rows x 4 columns]" ] } ], "prompt_number": 4 }, { "cell_type": "markdown", "metadata": {}, "source": [ "

Step 5: Save the model

\n", "\n", "The model can be saved for later use, either on the local machine or in an AWS S3 bucket. The saved model sits in its own directory, and can be loaded back in later to make more predictions.\n" ] }, { "cell_type": "code", "collapsed": false, "input": [ "# Save the model for later use\n", "model.save(\"my_model\")" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 5 }, { "cell_type": "markdown", "metadata": {}, "source": [ "Et voilà! You've just built your first recommender with GraphLab Create. Congratulations!\n", "\n", "(Looking for more details about the modules and functions? Check out the API docs.)" ] } ], "metadata": {} } ] }