{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "I'm currently reading [Dataclysm](http://dataclysm.org/), a book by one of the OkCupid founders, Christian Rudder. He's the one behind the [OkTrends blog](http://blog.okcupid.com/), which gives you a taste of what sort of data analysis the book is about. About halfway through the book, Rudder analyzes essays written by the users about themselves. To find meaning in the data across the different categories (white, black, asian, hispanic), he makes us of [quantile-quantile plots](https://en.wikipedia.org/wiki/Q%E2%80%93Q_plot). This struck me as an excellent application of interactive visualization using Bokeh and the Kaggle What's Cooking challenge data, which [I have previously investigated](http://flothesof.github.io/kaggle-whats-cooking-machine-learning.html). " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Loading the data and counting it " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We will start by loading the data, as usual:" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": true }, "outputs": [], "source": [ "import pandas as pd" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": false }, "outputs": [], "source": [ "df = pd.read_json('train.json')" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
cuisineidingredients
0greek10259[romaine lettuce, black olives, grape tomatoes...
1southern_us25693[plain flour, ground pepper, salt, tomatoes, g...
2filipino20130[eggs, pepper, salt, mayonaise, cooking oil, g...
3indian22213[water, vegetable oil, wheat, salt]
4indian13162[black pepper, shallots, cornflour, cayenne pe...
\n", "
" ], "text/plain": [ " cuisine id ingredients\n", "0 greek 10259 [romaine lettuce, black olives, grape tomatoes...\n", "1 southern_us 25693 [plain flour, ground pepper, salt, tomatoes, g...\n", "2 filipino 20130 [eggs, pepper, salt, mayonaise, cooking oil, g...\n", "3 indian 22213 [water, vegetable oil, wheat, salt]\n", "4 indian 13162 [black pepper, shallots, cornflour, cayenne pe..." ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To produce the sort of plot we want, we need to select one of the categories, say greek cuisine, compute the counts of all ingredients and their ranks and then do the same with the data from all other cuisines. To do this, we'll build ourselves a helper function that provides a count of all ingredients in a part of the dataframe. " ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "collapsed": false }, "outputs": [], "source": [ "from collections import Counter\n", "\n", "def count_ingredients(df):\n", " \"\"\"Counts ingredients in given df.\"\"\"\n", " \n", " c = Counter()\n", " for recipe in df['ingredients']:\n", " for ingredient in recipe:\n", " c.update([ingredient])\n", " \n", " return c" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's test this on a single recipe, using the first recipe in the dataset:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "Counter({'black olives': 1,\n", " 'feta cheese crumbles': 1,\n", " 'garbanzo beans': 1,\n", " 'garlic': 1,\n", " 'grape tomatoes': 1,\n", " 'pepper': 1,\n", " 'purple onion': 1,\n", " 'romaine lettuce': 1,\n", " 'seasoning': 1})" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "count_ingredients(df.iloc[0:1])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The input recipe was:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "['romaine lettuce',\n", " 'black olives',\n", " 'grape tomatoes',\n", " 'garlic',\n", " 'pepper',\n", " 'purple onion',\n", " 'seasoning',\n", " 'garbanzo beans',\n", " 'feta cheese crumbles']" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.iloc[0]['ingredients']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "So this seems to work. Let's count all greek recipes using this mechanism:" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "collapsed": false }, "outputs": [], "source": [ "is_greek = df['cuisine'] == 'greek'\n", "greek_counts = count_ingredients(df[is_greek])" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "[('chopped green bell pepper', 8),\n", " ('fresh bay leaves', 2),\n", " ('canned low sodium chicken broth', 5),\n", " ('cucumber', 187),\n", " ('thin pizza crust', 2),\n", " ('cooking spray', 67),\n", " ('top sirloin steak', 1),\n", " ('bone in skin on chicken thigh', 1),\n", " ('granulated garlic', 3),\n", " ('chopped parsley', 18)]" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "[(key, greek_counts[key]) for key in greek_counts.keys()][:10]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The last thing we need now is just to use the sorted count results as their ranks. We can do this using the counter's `most_common` method:" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "[('salt', 572),\n", " ('olive oil', 504),\n", " ('dried oregano', 267),\n", " ('garlic cloves', 254),\n", " ('feta cheese crumbles', 252),\n", " ('extra-virgin olive oil', 229),\n", " ('fresh lemon juice', 222),\n", " ('ground black pepper', 221),\n", " ('garlic', 216),\n", " ('pepper', 203)]" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "greek_counts.most_common(10)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Actually, we also need a count for all non-greek ingredients. Let's make it here:" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "collapsed": true }, "outputs": [], "source": [ "non_greek_counts = count_ingredients(df[~is_greek])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Visualization using Bokeh " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's now move on to the qq-plot. To do this interactively, I want to use Bokeh. It turns out that Bokeh can easily plot points as glyphs on a plane and add hover labels ([see here](http://bokeh.pydata.org/en/0.10.0/docs/user_guide/tools.html):" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ " \n", "\n", "\n", " \n", "\n", "
\n", " \n", " BokehJS successfully loaded.\n", "
" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", "
\n", "\n" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from bokeh.plotting import output_notebook, figure, show\n", "from bokeh.models import HoverTool, BoxSelectTool\n", "\n", "output_notebook()\n", "\n", "TOOLS = [BoxSelectTool(), HoverTool()]\n", "\n", "p = figure(plot_width=600, plot_height=400, title='A test scatter plot with hover labels', tools=TOOLS)\n", "\n", "p.circle([1, 2, 3, 4, 5], [2, 5, 8, 2, 7], size=10)\n", "\n", "show(p)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's write a function that does our qq-plot:" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "collapsed": true }, "outputs": [], "source": [ "from bokeh.plotting import ColumnDataSource\n", "from bokeh.models import BoxZoomTool, ResetTool\n", "\n", "def qqplot(x_list, y_list, specific_cuisine_name):\n", " \"\"\"Makes a qqplot using the x_list for abscissa and y_list for ordinate ranks.\n", " Expects x_list and y_list to be sorted.\"\"\"\n", " coords = []\n", " labels = []\n", " for y_index, y in enumerate(y_list):\n", " try:\n", " x_index = x_list.index(y)\n", " except ValueError:\n", " x_index = len(x_list)\n", " coords.append([x_index, y_index])\n", " labels.append(y)\n", " \n", " x_coords = [(len(x_list) - item[0]) / len(x_list) for item in coords]\n", " y_coords = [(len(y_list) - item[1]) / len(y_list) for item in coords]\n", " \n", " source = ColumnDataSource(\n", " data=dict(\n", " x=x_coords,\n", " y=y_coords,\n", " desc=labels,\n", " )\n", " )\n", " \n", " hover = HoverTool(\n", " tooltips=\"\"\"\n", "
\n", "
\n", " @desc\n", " [$index]\n", "
\n", "
\n", " Location\n", " ($x, $y)\n", "
\n", "
\n", " \"\"\"\n", " )\n", " \n", " TOOLS = [BoxZoomTool(), ResetTool(), hover]\n", "\n", " p = figure(plot_width=600, plot_height=400, title='QQ plot for {} cuisine'.format(specific_cuisine_name), tools=TOOLS)\n", " \n", " p.circle('x', 'y', size=5, source=source)\n", " p.line([0, 1], [0, 1], line_width=2)\n", " p.xaxis.axis_label = \"rank in all other cuisines\"\n", " p.yaxis.axis_label = \"rank in {} cuisine\".format(specific_cuisine_name)\n", " show(p)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's now use that function to display ingredients typical for some cuisines and not others:" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "collapsed": false }, "outputs": [], "source": [ "sorted_greek_ingredients = [item[0] for item in greek_counts.most_common()]" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "collapsed": true }, "outputs": [], "source": [ "sorted_non_greek_ingredients = [item[0] for item in non_greek_counts.most_common()]" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "\n", "
\n", "\n" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "qqplot(sorted_non_greek_ingredients, sorted_greek_ingredients, 'greek')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "What's interesting in the above graph is that everything that stands out above the reference line is typical for the specific cuisine. We can distinguish the parts that are very common in greek cuisine and typical (top right above the line) from the ones that are typical of greek cuisine, but uncommon (top left above the line).\n", "\n", "Examples include:\n", "\n", "- typical and common:\n", " - kalamata (olives)\n", " - tahini (sesame paste)\n", " - pita bread\n", " - tzatziki\n", " - bulgur\n", "- typical and not so common:\n", " - roe\n", " - tarama\n", " - myzithra\n", " - sliced kalamata olives\n", " - greek-style vinaigrette\n", " \n", " \n", "After looking at the data, I can't help but think that this raw data also shows the dataset is not very clean, as a lot of labels on the left could be clustered by humans with similar labels found on the right. For example, the dataset contains \"sliced kalamata olives\" as well as \"kalamata\". Both labels refer to the same thing but are not classified in the same way. This is exemplary for how hard it is to work with real world data." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Visualization of the whole dataset " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Finally, we can produce the same sort of plots for all cuisines:" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "collapsed": false, "scrolled": false }, "outputs": [ { "data": { "text/html": [ "\n", "
\n", "\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", "
\n", "\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", "
\n", "\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", "
\n", "\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", "
\n", "\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", "
\n", "\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", "
\n", "\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", "
\n", "\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", "
\n", "\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", "
\n", "\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", "
\n", "\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", "
\n", "\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", "
\n", "\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", "
\n", "\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", "
\n", "\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", "
\n", "\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", "
\n", "\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", "
\n", "\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", "
\n", "\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", "
\n", "\n" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "for cuisine in df.cuisine.unique():\n", " is_cuisine = df.cuisine == cuisine\n", " cuisine_counts = count_ingredients(df[is_cuisine])\n", " other_counts = count_ingredients(df[~is_cuisine])\n", " sorted_cuisine = [item[0] for item in cuisine_counts.most_common()]\n", " sorted_other = [item[0] for item in other_counts.most_common()]\n", " qqplot(sorted_other, sorted_cuisine, cuisine)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.5.1" } }, "nbformat": 4, "nbformat_minor": 0 }