![](\"dataphilly.jpeg\")
\n",
"\n",
"[DataPhilly](www.meetup.com/DataPhilly) is a local data meetup group I started back in 2012. I had attended a few data science conferences and I was really disappointed about the lack of a local meetup group for people interested in data science. And so DataPhilly was born!\n",
"\n",
"# Jupyter Notebook\n",
"The Jupyter Notebook is a web application that allows you to create and share documents that contain live code, equations, visualizations and explanatory text. Uses include: data cleaning and transformation, numerical simulation, statistical modeling, machine learning and much more.\n",
"\n",
"![](\"jupyterpreview.png\")
\n",
"\n",
"Through Jupyter's kernel and messaging architecture, the Notebook allows code to be run in a range of different programming languages. For each notebook document that a user opens, the web application starts a kernel that runs the code for that notebook. Each kernel is capable of running code in a single programming language and there are kernels available in the following languages\n",
"\n",
"* Python(https://github.com/ipython/ipython)\n",
"* Julia (https://github.com/JuliaLang/IJulia.jl)\n",
"* R (https://github.com/takluyver/IRkernel)\n",
"* Ruby (https://github.com/minrk/iruby)\n",
"* Haskell (https://github.com/gibiansky/IHaskell)\n",
"* Scala (https://github.com/Bridgewater/scala-notebook)\n",
"* node.js (https://gist.github.com/Carreau/4279371)\n",
"* Go (https://github.com/takluyver/igo)\n",
"\n",
"The default kernel runs Python code. The notebook provides a simple way for users to pick which of these kernels is used for a given notebook.\n",
"\n",
"**Jupyter examples and tutorials can be found in the Jupyter github repo [here](https://github.com/jupyter/notebook/blob/master/docs/source/examples/Notebook/Examples%20and%20Tutorials%20Index.ipynb).**"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# The task\n",
"The task I'll be walking you through today will demonstrate how to use Python for exploratory data analysis. The dataset I'll use is one I created by querying the Meetup API for the DataPhilly meetup. I'll walk you through using Jupyter notebook (The webapp we're using now), Pandas (an excel like tool for data exploration) and scikit-learn (a Python machine learning library) to explore the DataPhilly dataset. I won't go in depth into these tools but my hope is that you'll leave my talk wanting to learn more about using Python for exploratory data analysis and that you'll learn some interesting things about DataPhilly in the process."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Initializing our environment"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"First let's start off by initializing our environment\n",
"* ``%matplotlib inline`` initializes [matplotlib](http://matplotlib.org/) so that we can display graphs and charts in our notebook.\n",
"* ``import seaborn as sns`` imports [seaborn](https://web.stanford.edu/~mwaskom/software/seaborn/) a graphing library built on top of matplotlib.\n",
"* ``import pandas as pd`` imports [pandas](http://pandas.pydata.org/) a tool I'll explain in the next section."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"*Hint: If you've installed Jupyter Notebook and you're running this on your machine, you can use the run button in the toolbar at the top of the page to execute each cell*"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Click on the cell above and the cell below. You'll notice that the cell above is [Markdown](https://daringfireball.net/projects/markdown/). You can edit it by double clicking on it. The cell below contains Python code which can be modified and executed. If the code has any output it will be printed out below the cell with **Out [n]:** in front of it."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"%matplotlib inline\n",
"import seaborn as sns\n",
"import pandas as pd\n",
"from matplotlib import rcParams\n",
"\n",
"# Modify aesthetics for visibility during presentation\n",
"sns.set_style('darkgrid', {'axes.facecolor': '#C2C2C8'})\n",
"sns.set_palette('colorblind')\n",
"\n",
"# Make everything bigger for visibility during presentation\n",
"rcParams['figure.figsize'] = 20, 10\n",
"rcParams['axes.titlesize'] = 'xx-large'\n",
"rcParams['axes.labelsize'] = 'x-large'\n",
"rcParams['xtick.labelsize'] = 'x-large'\n",
"rcParams['ytick.labelsize'] = 'x-large'\n",
"rcParams['legend.fontsize'] = 'xx-large'\n",
"rcParams['lines.linewidth'] = 4.0\n",
"rcParams['grid.linewidth'] = 2.0\n",
"\n",
"# Hide warnings in the notebook\n",
"import warnings\n",
"warnings.filterwarnings('ignore')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Pandas"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![](\"pandas_logo.png\"){kind=logo}
\n",
"\n",
"Pandas is a library that provides data analysis tools for the Python programming language. You can think of it as Excel on steroids, but in Python."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To start off, I've used the [meetup API](http://www.meetup.com/meetup_api/docs/) to gather a bunch of data on members of the [DataPhilly meetup group](http://www.meetup.com/DataPhilly/). First let's start off by looking at the events we've had over the past few years. I've loaded the data into a pandas ``DataFrame`` and stored it in the file ``events.pkl``. A ``DataFrame`` is a table similar to an Excel spreadsheet. Let's load it and see what it looks like:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## DataPhilly events dataset"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" created name \\\n",
"0 1351948193000 Meet and greet \n",
"1 1357781071000 DataPhilly January 2013 Meetup - An Introducti... \n",
"\n",
" rating time waitlist_count \\\n",
"0 {u'count': 3, u'average': 5} 1352934000000 0 \n",
"1 {u'count': 6, u'average': 4.17000007629} 1359588600000 0 \n",
"\n",
" yes_rsvp_count id total_RSVP_count \n",
"0 17 89769502 17 \n",
"1 61 98833672 61 "
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"events_df.head(2)"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"import datetime\n",
"def get_datetime_from_epoch(epoch):\n",
" return datetime.datetime.fromtimestamp(epoch/1000.0)"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"0 2012-11-14 18:00:00\n",
"1 2013-01-30 18:30:00\n",
"2 2013-02-19 18:30:00\n",
"3 2013-03-27 18:30:00\n",
"4 2013-04-16 18:30:00\n",
"5 2013-06-05 18:30:00\n",
"6 2013-08-20 18:30:00\n",
"7 2013-09-26 18:30:00\n",
"8 2013-10-23 18:00:00\n",
"9 2013-11-18 18:00:00\n",
"10 2014-01-29 18:00:00\n",
"11 2014-03-12 18:00:00\n",
"12 2014-04-16 18:00:00\n",
"13 2014-05-21 18:00:00\n",
"14 2014-10-07 18:00:00\n",
"15 2014-11-06 18:00:00\n",
"16 2014-12-16 18:00:00\n",
"17 2015-02-19 18:00:00\n",
"18 2015-03-19 18:00:00\n",
"19 2015-04-16 18:00:00\n",
"20 2015-10-22 18:00:00\n",
"Name: time, dtype: datetime64[ns]"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"events_df['time'] = events_df['time'].apply(get_datetime_from_epoch)\n",
"events_df['time']"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next let's make the ``time`` column the index of the ``DataFrame`` using the ``set_index`` method and then re-``plot`` our data."
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"| \n", " | created | \n", "name | \n", "rating | \n", "time | \n", "waitlist_count | \n", "yes_rsvp_count | \n", "id | \n", "total_RSVP_count | \n", "
|---|---|---|---|---|---|---|---|---|
| 0 | \n", "1351948193000 | \n", "Meet and greet | \n", "{u'count': 3, u'average': 5} | \n", "1352934000000 | \n", "0 | \n", "17 | \n", "89769502 | \n", "17 | \n", "
| 1 | \n", "1357781071000 | \n", "DataPhilly January 2013 Meetup - An Introducti... | \n", "{u'count': 6, u'average': 4.17000007629} | \n", "1359588600000 | \n", "0 | \n", "61 | \n", "98833672 | \n", "61 | \n", "
\n",
"\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" anon_id anon_name bio \\\n",
"0 0 James NaN \n",
"1 1 Vijay java software developer in center city philly \n",
"2 2 Justin NaN \n",
"\n",
" city country gender hometown joined lat \\\n",
"0 Philadelphia us male Philadelphia 2015-10-10 21:43:33 39.94 \n",
"1 Philadelphia us male Philadelphia 2013-11-22 22:32:04 39.96 \n",
"2 Philadelphia us male NaN 2015-06-10 16:18:43 40.00 \n",
"\n",
" lon membership_count state \\\n",
"0 -75.23 0 PA \n",
"1 -75.20 0 PA \n",
"2 -75.14 63 PA \n",
"\n",
" topics visited \n",
"0 [{u'name': u'Poker', u'urlkey': u'poker', u'id... 2015-10-10 21:43:33 \n",
"1 [] 2015-04-13 20:20:24 \n",
"2 [{u'name': u'Extreme Programming', u'urlkey': ... 2015-10-22 15:42:23 "
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"members_df = pd.read_pickle('members.pkl')\n",
"for column in ['joined', 'visited']:\n",
" members_df[column] = members_df[column].apply(get_datetime_from_epoch)\n",
"members_df.head(3)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You'll notice that I've anonymized the meetup member_id and the member's name. I've also used the python module [``SexMachine``](https://pypi.python.org/pypi/SexMachine/) to infer members gender based on their first name. I ran SexMachine on the original names before I anonymized them. Let's have a closer look at the gender breakdown of our members:"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"male 716\n",
"andy 257\n",
"female 175\n",
"mostly_male 91\n",
"mostly_female 35\n",
"Name: gender, dtype: int64"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"gender_counts = members_df['gender'].value_counts()\n",
"gender_counts"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next let's use the ``hist`` method to plot a histogram of ``membership_count``. This is the number of groups each member is in."
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"| \n", " | anon_id | \n", "anon_name | \n", "bio | \n", "city | \n", "country | \n", "gender | \n", "hometown | \n", "joined | \n", "lat | \n", "lon | \n", "membership_count | \n", "state | \n", "topics | \n", "visited | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "0 | \n", "James | \n", "NaN | \n", "Philadelphia | \n", "us | \n", "male | \n", "Philadelphia | \n", "2015-10-10 21:43:33 | \n", "39.94 | \n", "-75.23 | \n", "0 | \n", "PA | \n", "[{u'name': u'Poker', u'urlkey': u'poker', u'id... | \n", "2015-10-10 21:43:33 | \n", "
| 1 | \n", "1 | \n", "Vijay | \n", "java software developer in center city philly | \n", "Philadelphia | \n", "us | \n", "male | \n", "Philadelphia | \n", "2013-11-22 22:32:04 | \n", "39.96 | \n", "-75.20 | \n", "0 | \n", "PA | \n", "[] | \n", "2015-04-13 20:20:24 | \n", "
| 2 | \n", "2 | \n", "Justin | \n", "NaN | \n", "Philadelphia | \n", "us | \n", "male | \n", "NaN | \n", "2015-06-10 16:18:43 | \n", "40.00 | \n", "-75.14 | \n", "63 | \n", "PA | \n", "[{u'name': u'Extreme Programming', u'urlkey': ... | \n", "2015-10-22 15:42:23 | \n", "
\n",
"\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" membership_count city country state\n",
"301 1838 Berlin de NaN\n",
"25 816 San Francisco us CA\n",
"141 651 Jerusalem il NaN\n",
"67 303 Philadelphia us PA\n",
"420 295 Baltimore us MD\n",
"1178 278 Princeton us NJ\n",
"257 241 New York us NY\n",
"223 207 Scarsdale us NY\n",
"150 197 Philadelphia us PA\n",
"174 166 Philadelphia us PA\n",
"86 166 West Chester us PA\n",
"449 146 Exton us PA\n",
"154 119 San Francisco us CA\n",
"1158 119 Philadelphia us PA\n",
"1022 113 Levittown us PA\n",
"868 106 Seattle us WA\n",
"987 102 San Francisco us CA"
]
},
"execution_count": 23,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"all_the_meetups = members_df[members_df['membership_count'] > 100]\n",
"filtered = all_the_meetups[['membership_count', 'city', 'country', 'state']]\n",
"filtered.sort_values(by='membership_count', ascending=False)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The people from Philly might actually be legitimate members, let's use a compound mask to filter them out as well:"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/html": [
"| \n", " | membership_count | \n", "city | \n", "country | \n", "state | \n", "
|---|---|---|---|---|
| 301 | \n", "1838 | \n", "Berlin | \n", "de | \n", "NaN | \n", "
| 25 | \n", "816 | \n", "San Francisco | \n", "us | \n", "CA | \n", "
| 141 | \n", "651 | \n", "Jerusalem | \n", "il | \n", "NaN | \n", "
| 67 | \n", "303 | \n", "Philadelphia | \n", "us | \n", "PA | \n", "
| 420 | \n", "295 | \n", "Baltimore | \n", "us | \n", "MD | \n", "
| 1178 | \n", "278 | \n", "Princeton | \n", "us | \n", "NJ | \n", "
| 257 | \n", "241 | \n", "New York | \n", "us | \n", "NY | \n", "
| 223 | \n", "207 | \n", "Scarsdale | \n", "us | \n", "NY | \n", "
| 150 | \n", "197 | \n", "Philadelphia | \n", "us | \n", "PA | \n", "
| 174 | \n", "166 | \n", "Philadelphia | \n", "us | \n", "PA | \n", "
| 86 | \n", "166 | \n", "West Chester | \n", "us | \n", "PA | \n", "
| 449 | \n", "146 | \n", "Exton | \n", "us | \n", "PA | \n", "
| 154 | \n", "119 | \n", "San Francisco | \n", "us | \n", "CA | \n", "
| 1158 | \n", "119 | \n", "Philadelphia | \n", "us | \n", "PA | \n", "
| 1022 | \n", "113 | \n", "Levittown | \n", "us | \n", "PA | \n", "
| 868 | \n", "106 | \n", "Seattle | \n", "us | \n", "WA | \n", "
| 987 | \n", "102 | \n", "San Francisco | \n", "us | \n", "CA | \n", "
\n",
"\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" membership_count city country state\n",
"301 1838 Berlin de NaN\n",
"25 816 San Francisco us CA\n",
"141 651 Jerusalem il NaN\n",
"420 295 Baltimore us MD\n",
"1178 278 Princeton us NJ\n",
"257 241 New York us NY\n",
"223 207 Scarsdale us NY\n",
"86 166 West Chester us PA\n",
"449 146 Exton us PA\n",
"154 119 San Francisco us CA\n",
"1022 113 Levittown us PA\n",
"868 106 Seattle us WA\n",
"987 102 San Francisco us CA"
]
},
"execution_count": 24,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"all_the_meetups = members_df[\n",
" (members_df['membership_count'] > 100) & (members_df['city'] != 'Philadelphia')\n",
"]\n",
"filtered = all_the_meetups[['membership_count', 'city', 'country', 'state']]\n",
"filtered.sort_values(by='membership_count', ascending=False)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"That's strange, I don't think we've ever had any members from Berlin, San Francisco, or Jerusalem in attendance :-)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## The RSVP dataset"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Moving on, we also have all the events that each member RSVPed to:"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/html": [
"| \n", " | membership_count | \n", "city | \n", "country | \n", "state | \n", "
|---|---|---|---|---|
| 301 | \n", "1838 | \n", "Berlin | \n", "de | \n", "NaN | \n", "
| 25 | \n", "816 | \n", "San Francisco | \n", "us | \n", "CA | \n", "
| 141 | \n", "651 | \n", "Jerusalem | \n", "il | \n", "NaN | \n", "
| 420 | \n", "295 | \n", "Baltimore | \n", "us | \n", "MD | \n", "
| 1178 | \n", "278 | \n", "Princeton | \n", "us | \n", "NJ | \n", "
| 257 | \n", "241 | \n", "New York | \n", "us | \n", "NY | \n", "
| 223 | \n", "207 | \n", "Scarsdale | \n", "us | \n", "NY | \n", "
| 86 | \n", "166 | \n", "West Chester | \n", "us | \n", "PA | \n", "
| 449 | \n", "146 | \n", "Exton | \n", "us | \n", "PA | \n", "
| 154 | \n", "119 | \n", "San Francisco | \n", "us | \n", "CA | \n", "
| 1022 | \n", "113 | \n", "Levittown | \n", "us | \n", "PA | \n", "
| 868 | \n", "106 | \n", "Seattle | \n", "us | \n", "WA | \n", "
| 987 | \n", "102 | \n", "San Francisco | \n", "us | \n", "CA | \n", "
\n",
"\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" 102502622 106043892 107740582 120425212 133803672 138415912 \\\n",
"0 0 0 0 0 0 0 \n",
"1 0 0 0 0 0 0 \n",
"2 1 0 0 0 0 0 \n",
"\n",
" 144769822 149515412 160323532 168747852 ... 206754182 \\\n",
"0 0 0 0 0 ... 1 \n",
"1 0 0 0 0 ... 0 \n",
"2 0 0 0 0 ... 0 \n",
"\n",
" 215265722 219055217 219840555 220526799 221245827 225488147 89769502 \\\n",
"0 0 0 1 0 0 0 0 \n",
"1 0 0 0 0 0 1 0 \n",
"2 0 0 0 0 0 0 0 \n",
"\n",
" 98833672 member_id \n",
"0 0 151 \n",
"1 0 370 \n",
"2 1 157 \n",
"\n",
"[3 rows x 22 columns]"
]
},
"execution_count": 25,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"rsvps_df = pd.read_pickle('rsvps.pkl')\n",
"rsvps_df.head(3)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"| \n", " | 102502622 | \n", "106043892 | \n", "107740582 | \n", "120425212 | \n", "133803672 | \n", "138415912 | \n", "144769822 | \n", "149515412 | \n", "160323532 | \n", "168747852 | \n", "... | \n", "206754182 | \n", "215265722 | \n", "219055217 | \n", "219840555 | \n", "220526799 | \n", "221245827 | \n", "225488147 | \n", "89769502 | \n", "98833672 | \n", "member_id | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "... | \n", "1 | \n", "0 | \n", "0 | \n", "1 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "151 | \n", "
| 1 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "... | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "1 | \n", "0 | \n", "0 | \n", "370 | \n", "
| 2 | \n", "1 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "... | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "1 | \n", "157 | \n", "
3 rows × 22 columns
\n", "![](\"inner_join.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can utilize the pandas ``merge`` method to join our members ``DataFrame`` and our rsvps ``DataFrame``:"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" anon_id anon_name bio \\\n",
"0 4 Edward NaN \n",
"1 8 John CTO as SnipSnap, the coupon app \n",
"2 11 John Founder and CEO of Azavea, a firm that builds ... \n",
"\n",
" city country gender hometown joined lat \\\n",
"0 Downingtown us male Philadelphia 2015-05-20 05:24:59 40.02 \n",
"1 Woodbury us male NaN 2013-01-29 22:49:12 39.83 \n",
"2 Philadelphia us male NaN 2012-11-06 12:18:12 39.95 \n",
"\n",
" lon ... 206754182 215265722 219055217 219840555 220526799 \\\n",
"0 -75.71 ... 0 0 0 0 0 \n",
"1 -75.13 ... 0 0 0 0 0 \n",
"2 -75.16 ... 0 0 0 0 0 \n",
"\n",
" 221245827 225488147 89769502 98833672 member_id \n",
"0 0 1 0 0 4 \n",
"1 0 0 0 0 8 \n",
"2 0 1 0 1 11 \n",
"\n",
"[3 rows x 36 columns]"
]
},
"execution_count": 26,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"joined_with_rsvps_df = pd.merge(members_df, rsvps_df, left_on='anon_id', right_on='member_id')\n",
"joined_with_rsvps_df.head(3)"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"Index([ u'anon_id', u'anon_name', u'bio',\n",
" u'city', u'country', u'gender',\n",
" u'hometown', u'joined', u'lat',\n",
" u'lon', u'membership_count', u'state',\n",
" u'topics', u'visited', u'102502622',\n",
" u'106043892', u'107740582', u'120425212',\n",
" u'133803672', u'138415912', u'144769822',\n",
" u'149515412', u'160323532', u'168747852',\n",
" u'175993712', u'182860422', u'206754182',\n",
" u'215265722', u'219055217', u'219840555',\n",
" u'220526799', u'221245827', u'225488147',\n",
" u'89769502', u'98833672', u'member_id'],\n",
" dtype='object')"
]
},
"execution_count": 27,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"joined_with_rsvps_df.columns"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we have a ton of data, let's see what kind of interesting things we can discover.\n",
"Let's look at the some stats on male attendees vs. female attendees:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"First we can use the ``isin`` method to make ``DataFrame``s for male and female members."
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/html": [
"| \n", " | anon_id | \n", "anon_name | \n", "bio | \n", "city | \n", "country | \n", "gender | \n", "hometown | \n", "joined | \n", "lat | \n", "lon | \n", "... | \n", "206754182 | \n", "215265722 | \n", "219055217 | \n", "219840555 | \n", "220526799 | \n", "221245827 | \n", "225488147 | \n", "89769502 | \n", "98833672 | \n", "member_id | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "4 | \n", "Edward | \n", "NaN | \n", "Downingtown | \n", "us | \n", "male | \n", "Philadelphia | \n", "2015-05-20 05:24:59 | \n", "40.02 | \n", "-75.71 | \n", "... | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "1 | \n", "0 | \n", "0 | \n", "4 | \n", "
| 1 | \n", "8 | \n", "John | \n", "CTO as SnipSnap, the coupon app | \n", "Woodbury | \n", "us | \n", "male | \n", "NaN | \n", "2013-01-29 22:49:12 | \n", "39.83 | \n", "-75.13 | \n", "... | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "8 | \n", "
| 2 | \n", "11 | \n", "John | \n", "Founder and CEO of Azavea, a firm that builds ... | \n", "Philadelphia | \n", "us | \n", "male | \n", "NaN | \n", "2012-11-06 12:18:12 | \n", "39.95 | \n", "-75.16 | \n", "... | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "1 | \n", "0 | \n", "1 | \n", "11 | \n", "
3 rows × 36 columns
\n", "\n",
"\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" anon_id anon_name bio city country gender hometown \\\n",
"578 1261 Tom NaN Philadelphia us male NaN \n",
"579 1262 Daniel NaN Philadelphia us male NaN \n",
"583 1271 Chris NaN Philadelphia us male NaN \n",
"\n",
" joined lat lon ... 206754182 215265722 \\\n",
"578 2015-10-06 17:50:59 39.96 -75.20 ... 0 0 \n",
"579 2015-10-07 15:48:58 39.97 -75.17 ... 0 0 \n",
"583 2015-10-20 15:00:55 39.96 -75.20 ... 0 0 \n",
"\n",
" 219055217 219840555 220526799 221245827 225488147 89769502 98833672 \\\n",
"578 0 0 0 0 1 0 0 \n",
"579 0 0 0 0 1 0 0 \n",
"583 0 0 0 0 1 0 0 \n",
"\n",
" member_id \n",
"578 1261 \n",
"579 1262 \n",
"583 1271 \n",
"\n",
"[3 rows x 36 columns]"
]
},
"execution_count": 28,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"male_attendees = joined_with_rsvps_df[joined_with_rsvps_df['gender'].isin(['male', 'mostly_male'])]\n",
"male_attendees.tail(3)"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/html": [
"| \n", " | anon_id | \n", "anon_name | \n", "bio | \n", "city | \n", "country | \n", "gender | \n", "hometown | \n", "joined | \n", "lat | \n", "lon | \n", "... | \n", "206754182 | \n", "215265722 | \n", "219055217 | \n", "219840555 | \n", "220526799 | \n", "221245827 | \n", "225488147 | \n", "89769502 | \n", "98833672 | \n", "member_id | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 578 | \n", "1261 | \n", "Tom | \n", "NaN | \n", "Philadelphia | \n", "us | \n", "male | \n", "NaN | \n", "2015-10-06 17:50:59 | \n", "39.96 | \n", "-75.20 | \n", "... | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "1 | \n", "0 | \n", "0 | \n", "1261 | \n", "
| 579 | \n", "1262 | \n", "Daniel | \n", "NaN | \n", "Philadelphia | \n", "us | \n", "male | \n", "NaN | \n", "2015-10-07 15:48:58 | \n", "39.97 | \n", "-75.17 | \n", "... | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "1 | \n", "0 | \n", "0 | \n", "1262 | \n", "
| 583 | \n", "1271 | \n", "Chris | \n", "NaN | \n", "Philadelphia | \n", "us | \n", "male | \n", "NaN | \n", "2015-10-20 15:00:55 | \n", "39.96 | \n", "-75.20 | \n", "... | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "1 | \n", "0 | \n", "0 | \n", "1271 | \n", "
3 rows × 36 columns
\n", "\n",
"\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" anon_id anon_name bio city country gender hometown \\\n",
"580 1265 Erin NaN Philadelphia us female NaN \n",
"581 1268 Anne NaN Philadelphia us female Philadelphia \n",
"582 1269 Stacey NaN Philadelphia us female NaN \n",
"\n",
" joined lat lon ... 206754182 215265722 \\\n",
"580 2015-10-13 18:13:37 39.95 -75.16 ... 0 0 \n",
"581 2015-10-18 15:29:42 39.96 -75.20 ... 0 0 \n",
"582 2015-10-20 09:55:35 39.96 -75.20 ... 0 0 \n",
"\n",
" 219055217 219840555 220526799 221245827 225488147 89769502 98833672 \\\n",
"580 0 0 0 0 1 0 0 \n",
"581 0 0 0 0 1 0 0 \n",
"582 0 0 0 0 1 0 0 \n",
"\n",
" member_id \n",
"580 1265 \n",
"581 1268 \n",
"582 1269 \n",
"\n",
"[3 rows x 36 columns]"
]
},
"execution_count": 29,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"female_attendees = joined_with_rsvps_df[joined_with_rsvps_df['gender'].isin(['female', 'mostly_female'])]\n",
"female_attendees.tail(3)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next we can use the ``sum`` method to count the number of male and female attendees per event and create a ``Series`` for each."
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"102502622 30\n",
"106043892 35\n",
"107740582 33\n",
"dtype: float64"
]
},
"execution_count": 30,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"event_ids = [\n",
" '102502622', '106043892', '107740582', '120425212', '133803672', '138415912', '144769822', '149515412',\n",
" '160323532', '168747852', '175993712', '182860422', '206754182', '215265722', '219055217', '219840555',\n",
" '220526799', '221245827', '225488147', '89769502', '98833672'\n",
"]\n",
"male_attendees[event_ids].sum().head(3)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can then recombine the male and female ``Series``' into a new ``DataFrame``."
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/html": [
"| \n", " | anon_id | \n", "anon_name | \n", "bio | \n", "city | \n", "country | \n", "gender | \n", "hometown | \n", "joined | \n", "lat | \n", "lon | \n", "... | \n", "206754182 | \n", "215265722 | \n", "219055217 | \n", "219840555 | \n", "220526799 | \n", "221245827 | \n", "225488147 | \n", "89769502 | \n", "98833672 | \n", "member_id | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 580 | \n", "1265 | \n", "Erin | \n", "NaN | \n", "Philadelphia | \n", "us | \n", "female | \n", "NaN | \n", "2015-10-13 18:13:37 | \n", "39.95 | \n", "-75.16 | \n", "... | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "1 | \n", "0 | \n", "0 | \n", "1265 | \n", "
| 581 | \n", "1268 | \n", "Anne | \n", "NaN | \n", "Philadelphia | \n", "us | \n", "female | \n", "Philadelphia | \n", "2015-10-18 15:29:42 | \n", "39.96 | \n", "-75.20 | \n", "... | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "1 | \n", "0 | \n", "0 | \n", "1268 | \n", "
| 582 | \n", "1269 | \n", "Stacey | \n", "NaN | \n", "Philadelphia | \n", "us | \n", "female | \n", "NaN | \n", "2015-10-20 09:55:35 | \n", "39.96 | \n", "-75.20 | \n", "... | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "1 | \n", "0 | \n", "0 | \n", "1269 | \n", "
3 rows × 36 columns
\n", "\n",
"\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" female male\n",
"102502622 2 30\n",
"106043892 6 35\n",
"107740582 3 33"
]
},
"execution_count": 31,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"gender_attendance = pd.DataFrame({'male': male_attendees[event_ids].sum(), 'female': female_attendees[event_ids].sum()})\n",
"gender_attendance.head(3)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And then we can use ``merge`` again to combine this with our events ``DataFrame``."
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/html": [
"| \n", " | female | \n", "male | \n", "
|---|---|---|
| 102502622 | \n", "2 | \n", "30 | \n", "
| 106043892 | \n", "6 | \n", "35 | \n", "
| 107740582 | \n", "3 | \n", "33 | \n", "
\n",
"\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" created \\\n",
"time \n",
"2012-11-14 18:00:00 1351948193000 \n",
"2013-01-30 18:30:00 1357781071000 \n",
"2013-02-19 18:30:00 1359732939000 \n",
"\n",
" name \\\n",
"time \n",
"2012-11-14 18:00:00 Meet and greet \n",
"2013-01-30 18:30:00 DataPhilly January 2013 Meetup - An Introducti... \n",
"2013-02-19 18:30:00 DataPhilly February 2013 Meetup - Data Science... \n",
"\n",
" rating waitlist_count \\\n",
"time \n",
"2012-11-14 18:00:00 {u'count': 3, u'average': 5} 0 \n",
"2013-01-30 18:30:00 {u'count': 6, u'average': 4.17000007629} 0 \n",
"2013-02-19 18:30:00 {u'count': 5, u'average': 5} 0 \n",
"\n",
" yes_rsvp_count id total_RSVP_count female male \n",
"time \n",
"2012-11-14 18:00:00 17 89769502 17 1 8 \n",
"2013-01-30 18:30:00 61 98833672 61 1 36 \n",
"2013-02-19 18:30:00 47 102502622 47 2 30 "
]
},
"execution_count": 32,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"events_with_gender_df = pd.merge(events_df, gender_attendance, left_on='id', right_index=True)\n",
"events_with_gender_df.head(3)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The we can ``plot`` the attendance by gender over time"
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"| \n", " | created | \n", "name | \n", "rating | \n", "waitlist_count | \n", "yes_rsvp_count | \n", "id | \n", "total_RSVP_count | \n", "female | \n", "male | \n", "
|---|---|---|---|---|---|---|---|---|---|
| time | \n", "\n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " |
| 2012-11-14 18:00:00 | \n", "1351948193000 | \n", "Meet and greet | \n", "{u'count': 3, u'average': 5} | \n", "0 | \n", "17 | \n", "89769502 | \n", "17 | \n", "1 | \n", "8 | \n", "
| 2013-01-30 18:30:00 | \n", "1357781071000 | \n", "DataPhilly January 2013 Meetup - An Introducti... | \n", "{u'count': 6, u'average': 4.17000007629} | \n", "0 | \n", "61 | \n", "98833672 | \n", "61 | \n", "1 | \n", "36 | \n", "
| 2013-02-19 18:30:00 | \n", "1359732939000 | \n", "DataPhilly February 2013 Meetup - Data Science... | \n", "{u'count': 5, u'average': 5} | \n", "0 | \n", "47 | \n", "102502622 | \n", "47 | \n", "2 | \n", "30 | \n", "
\n",
"\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" 0 2 3 4 5 6 8 9 11 \\\n",
"20's & 30's Social NaN NaN NaN NaN NaN NaN NaN NaN NaN \n",
"Adventure NaN NaN NaN 1 NaN NaN NaN NaN NaN \n",
"Art NaN NaN NaN NaN NaN NaN NaN NaN NaN \n",
"\n",
" 12 ... 1261 1262 1264 1266 1267 1268 1269 \\\n",
"20's & 30's Social NaN ... NaN 1 NaN NaN NaN NaN NaN \n",
"Adventure NaN ... NaN NaN NaN NaN NaN NaN NaN \n",
"Art NaN ... NaN NaN NaN NaN NaN NaN NaN \n",
"\n",
" 1271 1272 1273 \n",
"20's & 30's Social NaN NaN NaN \n",
"Adventure NaN NaN NaN \n",
"Art NaN NaN NaN \n",
"\n",
"[3 rows x 1088 columns]"
]
},
"execution_count": 37,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"top_100_topics = set([t[0] for t in topic_counter.most_common(100)])\n",
"topic_member_map = {}\n",
"for i, m in members_df.iterrows():\n",
" if m['topics']:\n",
" top_topic_count = {}\n",
" for topic in m['topics']:\n",
" if topic['name'] in top_100_topics:\n",
" top_topic_count[topic['name']] = 1\n",
" topic_member_map[m['anon_id']] = top_topic_count\n",
" \n",
"top_topic_df = pd.DataFrame(topic_member_map)\n",
"top_topic_df.head(3)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Okay for what I'm going to do next, I want the rows to be the members and the columns to be the topics. We can use the ``T`` (transpose) method to fix this."
]
},
{
"cell_type": "code",
"execution_count": 38,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/html": [
"| \n", " | 0 | \n", "2 | \n", "3 | \n", "4 | \n", "5 | \n", "6 | \n", "8 | \n", "9 | \n", "11 | \n", "12 | \n", "... | \n", "1261 | \n", "1262 | \n", "1264 | \n", "1266 | \n", "1267 | \n", "1268 | \n", "1269 | \n", "1271 | \n", "1272 | \n", "1273 | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 20's & 30's Social | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "... | \n", "NaN | \n", "1 | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "
| Adventure | \n", "NaN | \n", "NaN | \n", "NaN | \n", "1 | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "... | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "
| Art | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "... | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "
3 rows × 1088 columns
\n", "\n",
"\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" 20's & 30's Social Adventure Art Bicycling Big Data \\\n",
"0 NaN NaN NaN NaN NaN \n",
"2 NaN NaN NaN NaN NaN \n",
"3 NaN NaN NaN NaN NaN \n",
"\n",
" Big Data Analytics Board Games Book Club Business Intelligence \\\n",
"0 NaN 1 NaN NaN \n",
"2 NaN NaN NaN NaN \n",
"3 NaN NaN NaN NaN \n",
"\n",
" Business Strategy ... Watching Movies Web Design \\\n",
"0 NaN ... NaN NaN \n",
"2 NaN ... NaN 1 \n",
"3 NaN ... NaN NaN \n",
"\n",
" Web Development Web Technology Wellness Wine Women in Technology \\\n",
"0 NaN NaN NaN NaN NaN \n",
"2 1 1 NaN NaN NaN \n",
"3 NaN NaN NaN NaN NaN \n",
"\n",
" Writing Young Professionals iOS Development \n",
"0 NaN NaN NaN \n",
"2 NaN NaN 1 \n",
"3 NaN NaN NaN \n",
"\n",
"[3 rows x 100 columns]"
]
},
"execution_count": 38,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"top_topic_df = top_topic_df.T\n",
"top_topic_df.head(3)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next we can use the ``fillna`` method to fill in the missing values with zeros."
]
},
{
"cell_type": "code",
"execution_count": 39,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/html": [
"| \n", " | 20's & 30's Social | \n", "Adventure | \n", "Art | \n", "Bicycling | \n", "Big Data | \n", "Big Data Analytics | \n", "Board Games | \n", "Book Club | \n", "Business Intelligence | \n", "Business Strategy | \n", "... | \n", "Watching Movies | \n", "Web Design | \n", "Web Development | \n", "Web Technology | \n", "Wellness | \n", "Wine | \n", "Women in Technology | \n", "Writing | \n", "Young Professionals | \n", "iOS Development | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "1 | \n", "NaN | \n", "NaN | \n", "NaN | \n", "... | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "
| 2 | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "... | \n", "NaN | \n", "1 | \n", "1 | \n", "1 | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "1 | \n", "
| 3 | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "... | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "
3 rows × 100 columns
\n", "\n",
"\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" 20's & 30's Social Adventure Art Bicycling Big Data \\\n",
"0 0 0 0 0 0 \n",
"2 0 0 0 0 0 \n",
"3 0 0 0 0 0 \n",
"\n",
" Big Data Analytics Board Games Book Club Business Intelligence \\\n",
"0 0 1 0 0 \n",
"2 0 0 0 0 \n",
"3 0 0 0 0 \n",
"\n",
" Business Strategy ... Watching Movies Web Design \\\n",
"0 0 ... 0 0 \n",
"2 0 ... 0 1 \n",
"3 0 ... 0 0 \n",
"\n",
" Web Development Web Technology Wellness Wine Women in Technology \\\n",
"0 0 0 0 0 0 \n",
"2 1 1 0 0 0 \n",
"3 0 0 0 0 0 \n",
"\n",
" Writing Young Professionals iOS Development \n",
"0 0 0 0 \n",
"2 0 0 1 \n",
"3 0 0 0 \n",
"\n",
"[3 rows x 100 columns]"
]
},
"execution_count": 39,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"top_topic_df.fillna(0, inplace=True)\n",
"top_topic_df.head(3)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next let's use a [clustering algorithm](https://en.wikipedia.org/wiki/Cluster_analysis) to see if there are any patterns in the topics members are interested in. A clustering algorithm groups a set of data points so that similar objects are in the same group. This is a classic type of [unsupervised machine learning](https://en.wikipedia.org/wiki/Unsupervised_learning). Below you can find visualisations of how different clustering algorithms perform on various kinds of data:\n",
"\n",
"| \n", " | 20's & 30's Social | \n", "Adventure | \n", "Art | \n", "Bicycling | \n", "Big Data | \n", "Big Data Analytics | \n", "Board Games | \n", "Book Club | \n", "Business Intelligence | \n", "Business Strategy | \n", "... | \n", "Watching Movies | \n", "Web Design | \n", "Web Development | \n", "Web Technology | \n", "Wellness | \n", "Wine | \n", "Women in Technology | \n", "Writing | \n", "Young Professionals | \n", "iOS Development | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "1 | \n", "0 | \n", "0 | \n", "0 | \n", "... | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "
| 2 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "... | \n", "0 | \n", "1 | \n", "1 | \n", "1 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "1 | \n", "
| 3 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "... | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "
3 rows × 100 columns
\n", "![](\"plot_cluster_comparison_001.png\")
\n",
"\n",
"[Kmeans clustering](http://scikit-learn.org/stable/modules/clustering.html#k-means) is quick and can scale well to larger datasets. Let's see how it performs on our dataset:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# scikit-learn\n",
"![](\"scikit-learn-logo-notext.png\"){kind=logo}
\n",
"\n",
"We'll use a python machine learning library called scikit-learn to do the clustering."
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"array([2, 1, 2, ..., 2, 0, 2], dtype=int32)"
]
},
"execution_count": 40,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from sklearn.cluster import MiniBatchKMeans as KMeans\n",
"X = top_topic_df.as_matrix()\n",
"n_clusters = 3\n",
"k_means = KMeans(init='k-means++', n_clusters=n_clusters, n_init=10, random_state=47)\n",
"k_means.fit(X)\n",
"k_means.labels_"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We've grouped our members into 3 clusters, let's see how many members are in each cluster"
]
},
{
"cell_type": "code",
"execution_count": 41,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"[(2, 637), (1, 293), (0, 158)]"
]
},
"execution_count": 41,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"Counter(list(k_means.labels_)).most_common()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next let's see which topics are most popular in each cluster:"
]
},
{
"cell_type": "code",
"execution_count": 42,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Cluster 0\n",
"Outdoors 104\n",
"Live Music 98\n",
"Dining Out 93\n",
"Hiking 86\n",
"Fitness 84\n",
"Travel 77\n",
"Watching Movies 60\n",
"Adventure 58\n",
"New Technology 57\n",
"Intellectual Discussion 54\n",
"dtype: float64\n",
"\n",
"Cluster 1\n",
"Computer programming 248\n",
"Open Source 236\n",
"New Technology 231\n",
"Software Development 231\n",
"Technology 214\n",
"Web Development 211\n",
"Big Data 190\n",
"Startup Businesses 177\n",
"Data Analytics 166\n",
"Mobile Technology 158\n",
"dtype: float64\n",
"\n",
"Cluster 2\n",
"Big Data 291\n",
"Data Analytics 286\n",
"Data Mining 201\n",
"Computer programming 171\n",
"New Technology 162\n",
"Data Visualization 148\n",
"Startup Businesses 130\n",
"Python 130\n",
"Open Source 123\n",
"Big Data Analytics 115\n",
"dtype: float64\n",
"\n"
]
}
],
"source": [
"from collections import defaultdict\n",
"\n",
"cluster_index_map = defaultdict(list)\n",
"for i in range(k_means.labels_.shape[0]):\n",
" cluster_index_map[k_means.labels_[i]].append(top_topic_df.index[i])\n",
"\n",
"for cluster_num in range(n_clusters):\n",
" print 'Cluster {}'.format(cluster_num)\n",
" f = top_topic_df[top_topic_df.index.isin(cluster_index_map[cluster_num])].sum()\n",
" f2 = f[f > 0]\n",
" f3 = f2.sort_values(ascending=False)\n",
" print f3[:10]\n",
" print"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"So it looks like our biggest cluster (#2) contains members whose primary interest is data science.\n",
"\n",
"The second biggest cluster (#1) contains members whose primary interests are technology, and data science just happens to be one of those interests.\n",
"\n",
"The smallest cluster (#0) contains members whose primary interests are around socializing."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Based on this information we might be able to engage members in the \"social\" (#0) cluster by having more socially oriented events. We might be able to engaged with the members in cluster (#1) by having more events geared toward beginners."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Conclusion\n",
"Hopefully you learned a little bit about DataPhilly and doing exploratory analysis in Python. There's tons of extra data in our datasets that I don't even have time to get into today. If you feel like you missed anything and would like to revist it, you can find this Notebook and instructions for how to use it in my github repo [http://github.com/mdbecker/](http://github.com/mdbecker/). If you find something interesting in the data and you'd like to share it with me I'm [@beckerfuffle](https://twitter.com/beckerfuffle) on Twitter, and you can always [contact me](http://goo.gl/RvxB6J) through the DataPhilly Meetup page."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 2",
"language": "python",
"name": "python2"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 2
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.10"
}
},
"nbformat": 4,
"nbformat_minor": 0
}