{ "metadata": { "name": "", "signature": "sha256:06896b2a57caf4e9059d9e3f5b8006bf3222d117e21054a6856d622f260e621b" }, "nbformat": 3, "nbformat_minor": 0, "worksheets": [ { "cells": [ { "cell_type": "heading", "level": 1, "metadata": {}, "source": [ "Goals" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The focus of this notebook is on baby names that have been given to both male and female. " ] }, { "cell_type": "code", "collapsed": false, "input": [ "%matplotlib inline" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "import matplotlib.pyplot as plt\n", "import numpy as np\n", "\n", "from pylab import figure, show\n", "\n", "from pandas import DataFrame, Series\n", "import pandas as pd" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "try:\n", " import mpld3\n", " from mpld3 import enable_notebook\n", " from mpld3 import plugins\n", " enable_notebook()\n", "except Exception as e:\n", " print \"Attempt to import and enable mpld3 failed\", e" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "# what would seaborn do?\n", "try:\n", " import seaborn as sns\n", "except Exception as e:\n", " print \"Attempt to import and enable seaborn failed\", e" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "heading", "level": 1, "metadata": {}, "source": [ "Preliminaries: Assumed location of pydata-book files" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To make it more practical for me to look at your homework, I'm again going to assume a relative placement of files. I placed the files from \n", "\n", "https://github.com/pydata/pydata-book\n", "\n", "in a local directory, which in my case is \"/Users/raymondyee/D/Document/Working_with_Open_Data/pydata-book/\" \n", "\n", "and then symbolically linked (`ln -s`) to the the pydata-book from the root directory of the working-open-data folder. i.e., on OS X\n", "\n", " cd /Users/raymondyee/D/Document/Working_with_Open_Data/working-open-data\n", " ln -s /Users/raymondyee/D/Document/Working_with_Open_Data/pydata-book/ pydata-book\n", "\n", "That way the files from the pydata-book repository look like they sit in the working-open-data directory -- without having to actually copy the files.\n", "\n", "With this arrangment, I should then be able to drop your notebook into my own notebooks directory and run them without having to mess around with paths." ] }, { "cell_type": "code", "collapsed": false, "input": [ "import os\n", "\n", "NAMES_DIR = os.path.join(os.pardir, \"pydata-book\", \"ch02\", \"names\")\n", "\n", "assert os.path.exists(NAMES_DIR)" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Please make sure the above assertion works.**" ] }, { "cell_type": "heading", "level": 1, "metadata": {}, "source": [ "Baby names dataset" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "discussed in p. 35 of `PfDA` book" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To download all the data, including that for 2011 and 2012: [Popular Baby Names](http://www.ssa.gov/OACT/babynames/limits.html) --> includes state by state data." ] }, { "cell_type": "heading", "level": 1, "metadata": {}, "source": [ "Loading all data into Pandas" ] }, { "cell_type": "code", "collapsed": false, "input": [ "# show the first five files in the NAMES_DIR\n", "\n", "import glob\n", "glob.glob(NAMES_DIR + \"/*\")[:5]" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "# 2010 is the last available year in the pydata-book repo\n", "import os\n", "\n", "years = range(1880, 2011)\n", "\n", "pieces = []\n", "columns = ['name', 'sex', 'births']\n", "\n", "for year in years:\n", " path = os.path.join(NAMES_DIR, 'yob%d.txt' % year)\n", " frame = pd.read_csv(path, names=columns)\n", "\n", " frame['year'] = year\n", " pieces.append(frame)\n", "\n", "# Concatenate everything into a single DataFrame\n", "names = pd.concat(pieces, ignore_index=True)\n", "\n", "# why floats? I'm not sure.\n", "names.describe()" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "# how many people, names, males and females represented in names?\n", "\n", "names.births.sum()" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "# F vs M\n", "\n", "names.groupby('sex')['births'].sum()" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "# total number of names\n", "\n", "len(names.groupby('name'))" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "# use pivot_table to collect records by year (rows) and sex (columns)\n", "\n", "total_births = names.pivot_table('births', rows='year', cols='sex', aggfunc=sum)\n", "total_births.head()" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "# You can use groupy to get equivalent pivot_table calculation\n", "\n", "names.groupby('year').apply(lambda s: s.groupby('sex').agg('sum')).unstack()['births']" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "# how to calculate the total births / year\n", "\n", "names.groupby('year').sum().plot(title=\"total births by year\")" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "names.groupby('year').apply(lambda s: s.groupby('sex').agg('sum')).unstack()['births'].plot(title=\"births (M/F) by year\")" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "# from book: add prop to names\n", "\n", "def add_prop(group):\n", " # Integer division floors\n", " births = group.births.astype(float)\n", "\n", " group['prop'] = births / births.sum()\n", " return group\n", "\n", "names = names.groupby(['year', 'sex']).apply(add_prop)\n" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "# verify prop --> all adds up to 1\n", "\n", "np.allclose(names.groupby(['year', 'sex']).prop.sum(), 1)" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "# number of records in full names dataframe\n", "\n", "len(names)" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "heading", "level": 1, "metadata": {}, "source": [ "How to do top1000 calculation" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This section on the top1000 calculation is kept in here to provide some inspiration on how to work with baby names" ] }, { "cell_type": "code", "collapsed": false, "input": [ "# from book: useful to work with top 1000 for each year/sex combo\n", "# can use groupby/apply\n", "\n", "names.groupby(['year', 'sex']).apply(lambda g: g.sort_index(by='births', ascending=False)[:1000])" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "def get_top1000(group):\n", " return group.sort_index(by='births', ascending=False)[:1000]\n", "\n", "grouped = names.groupby(['year', 'sex'])\n", "top1000 = grouped.apply(get_top1000)\n", "top1000.head()" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "# Do pivot table: row: year and cols= names for top 1000\n", "\n", "top_births = top1000.pivot_table('births', rows='year', cols='name', aggfunc=np.sum)\n", "top_births.tail()" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "# is your name in the top_births list?\n", "\n", "top_births['Raymond'].plot(title='plot for Raymond')" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "# for Aaden, which shows up at the end\n", "\n", "top_births.Aaden.plot(xlim=[1880,2010])" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "# number of names represented in top_births\n", "\n", "len(top_births.columns)" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "# how to get the most popular name of all time in top_births?\n", "\n", "most_common_names = top_births.sum()\n", "most_common_names.sort(ascending=False)\n", "\n", "most_common_names.head()" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "# as of mpl v 0.1 (2014.03.04), the name labeling doesn't work -- so disble mpld3 for this figure\n", "\n", "mpld3.disable_notebook()\n", "plt.figure()\n", "most_common_names[:50][::-1].plot(kind='barh', figsize=(10,10))" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "# turn mpld3 back on\n", "\n", "mpld3.enable_notebook()" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "heading", "level": 1, "metadata": {}, "source": [ "all_births pivot table" ] }, { "cell_type": "code", "collapsed": false, "input": [ "# instead of top_birth -- get all_births\n", "\n", "all_births = names.pivot_table('births', rows='year', cols='name', aggfunc=sum)" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "all_births = all_births.fillna(0)\n", "all_births.tail()" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "# set up to do start/end calculation\n", "\n", "all_births_cumsum = all_births.apply(lambda s: s.cumsum(), axis=0)" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "all_births_cumsum.tail()" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "heading", "level": 2, "metadata": {}, "source": [ "Names that are both M and F" ] }, { "cell_type": "code", "collapsed": false, "input": [ "# remind ourselves of what's in names\n", "\n", "names.head()" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "# columns in names\n", "\n", "names.columns" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "heading", "level": 2, "metadata": {}, "source": [ "Calculating ambigendered names" ] }, { "cell_type": "code", "collapsed": false, "input": [ "# calculate set of male_only, female_only, ambigender names\n", "\n", "def calc_of_sex_of_names():\n", "\n", " k = names.groupby('sex').apply(lambda s: set(list(s['name'])))\n", " male_only_names = k['M'] - k['F']\n", " female_only_names = k['F'] - k['M']\n", " ambi_names = k['F'] & k['M'] # intersection of two \n", " return {'male_only_names': male_only_names, \n", " 'female_only_names': female_only_names,\n", " 'ambi_names': ambi_names }\n", " \n", "names_by_sex = calc_of_sex_of_names() \n", "ambi_names_array = np.array(list(names_by_sex['ambi_names']))\n", "\n", "[(k, len(v)) for (k,v) in names_by_sex.items()]" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "# total number of people in names\n", "names.births.sum()" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "# pivot table of ambigendered names to aggregate \n", "\n", "names_ambi = names[np.in1d(names.name,ambi_names_array)]\n", "ambi_names_pt = names_ambi.pivot_table('births',\n", " rows='year', \n", " cols=['name','sex'], \n", " aggfunc='sum')" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "# total number of people in k1 -- almost everyone!\n", "\n", "ambi_names_pt.sum().sum()" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "# fill n/a with 0 and look at the table at the end\n", "\n", "ambi_names_pt=ambi_names_pt.fillna(0L)\n", "ambi_names_pt.tail()" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "# plot M, F in ambigender_names over time\n", "ambi_names_pt.T.xs('M',level='sex').sum().cumsum()" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "ambi_names_pt.T.xs('F',level='sex').sum().cumsum()" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "# don't know what pivot table has type float\n", "# https://github.com/pydata/pandas/issues/3283\n", "ambi_names_pt['Raymond', 'M'].dtype" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "# calculate proportion of males for given name\n", "\n", "def prop_male(name):\n", " return (ambi_names_pt[name]['M']/ \\\n", " ((ambi_names_pt[name]['M'] + ambi_names_pt[name]['F'])))\n", "\n", "def prop_c_male(name):\n", " return (ambi_names_pt[name]['M'].cumsum()/ \\\n", " ((ambi_names_pt[name]['M'].cumsum() + ambi_names_pt[name]['F'].cumsum())))" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "prop_c_male('Leslie').plot()" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "# I couldn't figure out a way of iterating over the names rather than names/sex combo in\n", "# a vectorized way. \n", "\n", "from itertools import islice\n", "\n", "names_to_calc = list(islice(list(ambi_names_pt.T.index.levels[0]),None))\n", "\n", "m = [(name_, ambi_names_pt[name_]['M']/(ambi_names_pt[name_]['F'] + ambi_names_pt[name_]['M'])) \\\n", " for name_ in names_to_calc]\n", "p_m_instant = DataFrame(dict(m))\n", "p_m_instant.tail()" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "# similar calculation except instead of looking at the proportions for a given year only,\n", "# we look at the cumulative number of male/female babies for given name\n", "\n", "from itertools import islice\n", "\n", "names_to_calc = list(islice(list(ambi_names_pt.T.index.levels[0]),None))\n", "\n", "m = [(name_, ambi_names_pt[name_]['M'].cumsum()/(ambi_names_pt[name_]['F'].cumsum() + ambi_names_pt[name_]['M'].cumsum())) \\\n", " for name_ in names_to_calc]\n", "p_m_cum = DataFrame(dict(m))\n", "p_m_cum.tail()" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "p_m_cum['Donnie'].plot()" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "# some metrics that attempt to measure how a time series s has changed\n", "\n", "def min_max_range(s):\n", " \"\"\"range of s signed -- positive if slope between two points p +ve and negative\n", " otherwise; 0 if slope is 0\"\"\"\n", " # note np.argmax, np.argmin returns the position of first occurence of global max, min\n", " sign = np.sign(np.argmax(s) - np.argmin(s))\n", " if sign == 0:\n", " return 0.0\n", " else:\n", " return sign*(np.max(s) - np.min(s))\n", "\n", "def last_first_diff(s):\n", " \"\"\"difference between latest and earliest value\"\"\"\n", " s0 = s.dropna()\n", " return (s0.iloc[-1] - s0.iloc[0])\n", " \n" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "# population distributions of ambinames \n", "# might want to remove from consideration instances when total ratio is too great\n", "# or range of existence of a name/sex combo too short\n", "\n", "total_pop_ambiname = all_births.sum()[np.in1d(all_births.sum().index, ambi_names_array)]\n", "total_pop_ambiname.sort(ascending=False)\n", "total_pop_ambiname.plot(logy=True)" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "# now calculate a DataFrame to visualize results\n", "\n", "# calculate the total population, the change in p_m from last to first appearance, \n", "# the change from max to min in p_m, and the percentage of males overall for name\n", "\n", "df = DataFrame()\n", "df['total_pop'] = total_pop_ambiname\n", "df['last_first_diff'] = p_m_cum.apply(last_first_diff)\n", "df['min_max_range'] = p_m_cum.apply(min_max_range)\n", "df['abs_min_max_range'] = np.abs(df.min_max_range)\n", "df['p_m'] = p_m_cum.iloc[-1]\n", "\n", "# distance from full ambigender -- p_m=0.5 leads to 1, p_m=1 or 0 -> 0\n", "df['ambi_index'] = df.p_m.apply(lambda p: 1 - 2* np.abs(p-0.5))\n", "\n", "df.head()" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "# plot: x -> log10 of total population, y->how p_m has changed from first to last\n", "# turn off d3 for this plot\n", "\n", "mpld3.disable_notebook()\n", "plt.scatter(np.log10(df.total_pop), df.last_first_diff, s=1)" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "# turn d3 back on\n", "\n", "mpld3.enable_notebook()" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "# general directionality counts -- looking for over asymmetry\n", "\n", "df.groupby(np.sign(df.last_first_diff)).count()" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "# let's concentrate on more populous names that have seen big swings in the cumulative p_m\n", "\n", "# you can play with the population and range filter\n", "popular_names_with_shifts = df[(df.total_pop>5000) & (df.abs_min_max_range >0.7)]\n", "popular_names_with_shifts.sort_index(by=\"abs_min_max_range\", ascending=False)\n" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "popular_names_with_shifts.groupby(np.sign(df.last_first_diff)).count()" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "#popular_names_with_shifts.to_pickle('popular_names_with_shifts.pickle')" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "fig, ax = plt.subplots(subplot_kw=dict(axisbg='#EEEEEE'))\n", "x = np.log10(popular_names_with_shifts.total_pop)\n", "y = popular_names_with_shifts.min_max_range \n", "\n", "scatter = ax.scatter(x, y)\n", "\n", "ax.grid(color='white', linestyle='solid')\n", "ax.set_title(\"Populous Names with Major Sex Shift\", size=20)\n", "ax.set_xlabel('log10(total_pop)')\n", "ax.set_ylabel('min_max_range')\n", "\n", "#labels = ['point {0}'.format(i + 1) for i in range(len(x))]\n", "labels = list(popular_names_with_shifts.index)\n", "tooltip = plugins.PointLabelTooltip(scatter, labels=labels)\n", "plugins.connect(fig, tooltip)" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "prop_c_male('Leslie').plot()" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [], "language": "python", "metadata": {}, "outputs": [] } ], "metadata": {} } ] }