{ "metadata": { "name": "", "signature": "sha256:6c7e72aac55581fc1d765007dbd591dcecd846eaac6ea8dcbee626783e04095d" }, "nbformat": 3, "nbformat_minor": 0, "worksheets": [ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "*Working with Open Data* Midterm (March 18, 2014)\n", "\n", "There are **94** points in this exam: 2 each for the **47 questions**. The questions are either **multiple choice** or **short answers**. For **multiple choice**, just write the **number** of the choice selected.\n", "\n", "\n", "\n", "Name: `______________________________________`\n", "\n", "\n", "`" ] }, { "cell_type": "heading", "level": 1, "metadata": {}, "source": [ "World Population" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Consider this code to construct a DataFrame of populations of countries." ] }, { "cell_type": "code", "collapsed": false, "input": [ "import json\n", "import requests\n", "from pandas import DataFrame\n", "\n", "# read population in from JSON-formatted data derived from the Wikipedia\n", "pop_json_url = \"https://gist.github.com/rdhyee/8511607/\" + \\\n", " \"raw/f16257434352916574473e63612fcea55a0c1b1c/population_of_countries.json\"\n", "pop_list= requests.get(pop_json_url).json()\n", "\n", "df = DataFrame(pop_list)\n", "df[:5]" ], "language": "python", "metadata": {}, "outputs": [ { "html": [ "
\n", " | 0 | \n", "1 | \n", "2 | \n", "
---|---|---|---|
0 | \n", "1 | \n", "China | \n", "1385566537 | \n", "
1 | \n", "2 | \n", "India | \n", "1252139596 | \n", "
2 | \n", "3 | \n", "United States | \n", "320050716 | \n", "
3 | \n", "4 | \n", "Indonesia | \n", "249865631 | \n", "
4 | \n", "5 | \n", "Brazil | \n", "200361925 | \n", "
5 rows \u00d7 3 columns
\n", "\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q2**: What is the relationship between `s2` and the population of China, where `s2` is defined by:\n", "\n", " s2 = sum(df[df[1].str.startswith('X')][2])\n", " \n", "1. `s2` is **greater** than the population of China\n", "1. `s2` is the **same** as the population of China\n", "1. `s2` is **less** than the population of China\n", "1. `s2` is not a number.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A2**:\n", "
\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q3**: What happens when the following statement is run?\n", "\n", " df.columns = ['Number','Country','Population']\n", " \n", "1. Nothing\n", "1. `df` gets a new attribute called `columns`\n", "1. `df`'s columns are renamed based on the list\n", "1. Throws an exception" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A3**:\n", "
\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q4**: This statement does the following\n", "\n", " df.columns = ['Number','Country']\n", " \n", "1. Nothing\n", "1. `df` gets a new attribute called `columns`\n", "1. `df`'s columns are renamed based on the list\n", "1. Throws an exception" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A4**:\n", "
\n", "\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q5**: How would you rewrite the following statement to get the same result as\n", "\n", " s = sum(df[df[1].str.startswith('C')][2])\n", "\n", "after running:\n", "\n", " df.columns = ['Number','Country','Population']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A5**:\n", "
\n", "\n", "\n", "\n", "\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q6**. What is\n", "\n", "```Python\n", " len(df[df[\"Population\"] > 1000000000])\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A6**:\n", "
\n", "\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q7**. What is\n", "\n", "```Python\n", " \";\".join(df[df['Population']>1000000000]['Country'].apply(lambda s: s[0]))\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A7**:\n", "
\n", "\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q8**. What is\n", "\n", " ```Python\n", " len(\";\".join(df[df['Population']>1000000000]['Country'].apply(lambda s: s[0])))\n", " ```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A8**:\n", "
\n", "\n", "\n", "" ] }, { "cell_type": "heading", "level": 1, "metadata": {}, "source": [ "Pandas Series" ] }, { "cell_type": "code", "collapsed": false, "input": [ "from pandas import DataFrame, Series\n", "import numpy as np\n", "\n", "s1 = Series(np.arange(-1,4))\n", "s1" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 11, "text": [ "0 -1\n", "1 0\n", "2 1\n", "3 2\n", "4 3\n", "dtype: int64" ] } ], "prompt_number": 11 }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q9**: What is\n", "\n", "```Python\n", " s1 + 1\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A9**:\n", "
\n", "\n", "\n", "\n", "\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q10**: What is\n", "\n", "```Python\n", "s1.apply(lambda k: 2*k).sum()\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A10**:\n", "
\n", "\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q11**: What is \n", "\n", "```Python\n", " s1.cumsum()[3]\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A11**:\n", "
\n", "\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q12**: What is\n", "\n", " s1.cumsum() - s1.cumsum()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A12**:\n", "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q13**. What is\n", "\n", "```Python\n", " len(s1.cumsum() - s1.cumsum())\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A13**:\n", "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q14**: What is\n", "\n", " np.any(s1 > 2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A14**:\n", "
\n", "\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q15**. What is\n", "\n", "```Python\n", " np.all(s1<3)\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A15**:\n", "
\n", "\n", "\n", "" ] }, { "cell_type": "heading", "level": 1, "metadata": {}, "source": [ "Census API" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Consider the following code to load population(s) from the Census API." ] }, { "cell_type": "code", "collapsed": false, "input": [ "from census import Census\n", "from us import states\n", "\n", "import settings\n", "\n", "c = Census(settings.CENSUS_KEY)\n", "c.sf1.get(('NAME', 'P0010001'), {'for': 'state:%s' % states.CA.fips})" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 19, "text": [ "[{u'NAME': u'California', u'P0010001': u'37253956', u'state': u'06'}]" ] } ], "prompt_number": 19 }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q16**: What is the purpose of `settings.CENSUS_KEY`?\n", "\n", "1. It is the password for the Census Python package\n", "1. It is an API Access key for authentication with the Census API\n", "1. It is an API Access key for authentication with Github\n", "1. It is key shared by all users of the Census API" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A16**:\n", "
\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q17**. When we run\n", "\n", "```bash\n", "pip install census\n", "```\n", "\n", "we are:\n", "\n", "1. installing a Python module from PyPI\n", "1. installing the Python module census from continuum.io's repository\n", "1. signing ourselves up for a census API key\n", "1. None of the above" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A17**:\n", "
\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Consider `r1` and `r2`:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q18**: What is the difference between `r1` and `r2`?\n", "\n", "```Python\n", " r1 = c.sf1.get(('NAME', 'P0010001'), {'for': 'county:*', 'in': 'state:%s' % states.CA.fips})\n", " r2 = c.sf1.get(('NAME', 'P0010001'), {'for': 'county:*', 'in': 'state:*' })\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A18**:\n", "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q19**. What's the relationship between `len(r1)` and `len(r2)`?\n", "\n", "1. `len(r1)` is less than `len(r2)`\n", "1. `len(r1)` equals `len(r2)`\n", "1. `len(r1)` is greater than `len(r2)`\n", "1. None of the above" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A19**:\n", "
\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q20**: Which is a correct geographic hierarchy?\n", "\n", "Nation > States = Nation is subdivided into States\n", "\n", "1. Counties > States\n", "1. Counties > Block Groups > Census Tracks\n", "1. Census Tracts > Block Groups > Census Blocks\n", "1. Places > Counties\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A20**:\n", "
\n", "\n", "" ] }, { "cell_type": "code", "collapsed": false, "input": [ "from pandas import DataFrame\n", "import numpy as np\n", "from census import Census\n", "from us import states\n", "\n", "import settings\n", "\n", "c = Census(settings.CENSUS_KEY)\n", "\n", "r = c.sf1.get(('NAME', 'P0010001'), {'for': 'state:*'})\n", "df1 = DataFrame(r)\n", "\n", "df1.head()" ], "language": "python", "metadata": {}, "outputs": [ { "html": [ "
\n", " | NAME | \n", "P0010001 | \n", "state | \n", "
---|---|---|---|
0 | \n", "Alabama | \n", "4779736 | \n", "01 | \n", "
1 | \n", "Alaska | \n", "710231 | \n", "02 | \n", "
2 | \n", "Arizona | \n", "6392017 | \n", "04 | \n", "
3 | \n", "Arkansas | \n", "2915918 | \n", "05 | \n", "
4 | \n", "California | \n", "37253956 | \n", "06 | \n", "
5 rows \u00d7 3 columns
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Consider the two following expressions:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "print df1.P0010001.sum()\n", "print\n", "print df1.P0010001.astype(int).sum()" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "477973671023163920172915918372539565029196357409789793460172318801310968765313603011567582128306326483802304635528531184339367453337213283615773552654762998836405303925296729759889279894151826341270055113164708791894205917919378102953548367259111536504375135138310741270237910525674625364814180634610525145561276388562574180010246724540185299456869865636263725789\n", "\n", "312471327\n" ] } ], "prompt_number": 23 }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q22**: Why is `df1.P0010001.sum()` different from `df1.P0010001.astype(int).sum()`? " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A22**:\n", "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q23**: Describe the output of the following:\n", "\n", "```Python\n", "df1.P0010001 = df1.P0010001.astype(int)\n", "df1[['NAME','P0010001']].sort('P0010001', ascending=True).head()\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A23**:\n", "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q24**: After running:\n", "\n", "```Python\n", " df1.set_index('NAME', inplace=True)\n", "```\n", "\n", "how would you access the Series for the state of Nebraska?\n", "\n", "1. `df1['Nebraska']`\n", "1. `df1[1]`\n", "1. `df1.ix['Nebraska']`\n", "1. `df1[df1['NAME'] == 'Nebraska']`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A24**:\n", "
\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q25**. What is `len(states.STATES)`?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A25**:\n", "
\n", "\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q26**. What is\n", "\n", "```Python\n", "len(df1[np.in1d(df1.state, [s.fips for s in states.STATES])])\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A26**:\n", "
\n", "\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In the next question, we will make use of the negation operator `~`. Take a look at a specific example" ] }, { "cell_type": "code", "collapsed": false, "input": [ "~Series([True, True, False, True])" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 28, "text": [ "0 False\n", "1 False\n", "2 True\n", "3 False\n", "dtype: bool" ] } ], "prompt_number": 28 }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q27**. What is\n", "\n", "```Python\n", " list(df1[~np.in1d(df1.state, [s.fips for s in states.STATES])].index)[0]\n", "``` " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A27**:\n", "
\n", "\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Consider `pop1` and `pop2`:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "pop1 = df1['P0010001'].astype('int').sum() \n", "pop2 = df1[np.in1d(df1.state, [s.fips for s in states.STATES])]['P0010001'].astype('int').sum()\n", "\n", "pop1-pop2" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 30, "text": [ "3725789" ] } ], "prompt_number": 30 }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q28**. What does `pop11 - pop2` represent?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A28**:\n", "
\n", "\n", "\n", "\n", "\n", "\n", "" ] }, { "cell_type": "heading", "level": 1, "metadata": {}, "source": [ "Generator and range" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q29**. Given that\n", "\n", " range(10)\n", " \n", "is\n", "\n", " [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]\n", "\n", "How to get the total of every integer from 1 to 100?\n", "\n", "1. `sum(range(1, 101))`\n", "1. `sum(range(100))`\n", "1. `sum(range(1, 100))`\n", "1. None of the above" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A29**:\n", "
\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q30**. What output is produced from\n", "\n", "```Python\n", "# itertools is a great library\n", "# http://docs.python.org/2/library/itertools.html#itertools.count\n", "# itertools.count(start=0, step=1):\n", "# \"Make an iterator that returns evenly spaced values starting with step.\"\n", "\n", "from itertools import islice, count\n", "c = count(0, 1)\n", "print c.next()\n", "print c.next()\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A30**:\n", "
\n", "\n", "\n", "\n", "\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q31**. Recalling that \n", "\n", " 1+2+3+...+100 = 5050\n", " \n", "what is:\n", "\n", "```Python\n", "(2*Series(np.arange(101))).sum()\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A31**:\n", "
\n", "\n", "\n", "" ] }, { "cell_type": "heading", "level": 1, "metadata": {}, "source": [ "Census Places" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Consider the follow generator that we used to query for census places." ] }, { "cell_type": "code", "collapsed": false, "input": [ "import pandas as pd\n", "from pandas import DataFrame\n", "\n", "import census\n", "import settings\n", "import us\n", "\n", "from itertools import islice\n", "\n", "c=census.Census(settings.CENSUS_KEY)\n", "\n", "def places(variables=\"NAME\"):\n", " \n", " for state in us.states.STATES:\n", " geo = {'for':'place:*', 'in':'state:{s_fips}'.format(s_fips=state.fips)}\n", " for place in c.sf1.get(variables, geo=geo):\n", " yield place\n", " " ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 34 }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we compute a DataFrame for the places: `places_df`" ] }, { "cell_type": "code", "collapsed": false, "input": [ "r = list(islice(places(\"NAME,P0010001\"), None))\n", "places_df = DataFrame(r)\n", "places_df.P0010001 = places_df.P0010001.astype('int')\n", "\n", "print \"number of places\", len(places_df)\n", "print \"total pop\", places_df.P0010001.sum()\n", "places_df.head()\n" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "number of places 29261\n", "total pop 228457238\n" ] }, { "html": [ "
\n", " | NAME | \n", "P0010001 | \n", "place | \n", "state | \n", "
---|---|---|---|---|
0 | \n", "Abanda CDP | \n", "192 | \n", "00100 | \n", "01 | \n", "
1 | \n", "Abbeville city | \n", "2688 | \n", "00124 | \n", "01 | \n", "
2 | \n", "Adamsville city | \n", "4522 | \n", "00460 | \n", "01 | \n", "
3 | \n", "Addison town | \n", "758 | \n", "00484 | \n", "01 | \n", "
4 | \n", "Akron town | \n", "356 | \n", "00676 | \n", "01 | \n", "
5 rows \u00d7 4 columns
\n", "\n", " | NAME | \n", "P0010001 | \n", "place | \n", "state | \n", "
---|---|---|---|---|
2714 | \n", "Los Angeles city | \n", "3792621 | \n", "44000 | \n", "06 | \n", "
3112 | \n", "San Diego city | \n", "1307402 | \n", "66000 | \n", "06 | \n", "
3122 | \n", "San Jose city | \n", "945942 | \n", "68000 | \n", "06 | \n", "
3116 | \n", "San Francisco city | \n", "805235 | \n", "67000 | \n", "06 | \n", "
2425 | \n", "Fresno city | \n", "494665 | \n", "27000 | \n", "06 | \n", "
5 rows \u00d7 4 columns
\n", "\n", " | NAME | \n", "P0010001 | \n", "place | \n", "state | \n", "
---|---|---|---|---|
2714 | \n", "Los Angeles city | \n", "3792621 | \n", "44000 | \n", "06 | \n", "
3112 | \n", "San Diego city | \n", "1307402 | \n", "66000 | \n", "06 | \n", "
3122 | \n", "San Jose city | \n", "945942 | \n", "68000 | \n", "06 | \n", "
3116 | \n", "San Francisco city | \n", "805235 | \n", "67000 | \n", "06 | \n", "
2425 | \n", "Fresno city | \n", "494665 | \n", "27000 | \n", "06 | \n", "
5 rows \u00d7 4 columns
\n", "\n", "\n", "\n", "\n", "\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q33**. What is\n", "\n", "```Python\n", "places_df[\"NAME\"][3122]\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A33**:\n", "
\n", "\n", "\n", "" ] }, { "cell_type": "heading", "level": 1, "metadata": {}, "source": [ "Alphabet and apply" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now let's set up a DataFrame with some letters and properties of letters." ] }, { "cell_type": "code", "collapsed": false, "input": [ "# numpy and pandas related imports \n", "\n", "import numpy as np\n", "from pandas import Series, DataFrame\n", "import pandas as pd\n", "\n", "# for example, using lower and uppercase English letters\n", "\n", "import string\n", "\n", "lower = Series(list(string.lowercase), name='lower')\n", "upper = Series(list(string.uppercase), name='upper')\n", "\n", "df2 = pd.concat((lower, upper), axis=1)\n", "df2['ord'] = df2['lower'].apply(ord)\n", "df2.head()" ], "language": "python", "metadata": {}, "outputs": [ { "html": [ "
\n", " | lower | \n", "upper | \n", "ord | \n", "
---|---|---|---|
0 | \n", "a | \n", "A | \n", "97 | \n", "
1 | \n", "b | \n", "B | \n", "98 | \n", "
2 | \n", "c | \n", "C | \n", "99 | \n", "
3 | \n", "d | \n", "D | \n", "100 | \n", "
4 | \n", "e | \n", "E | \n", "101 | \n", "
5 rows \u00d7 3 columns
\n", "\n", "\n", "\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q35**. What is\n", "\n", "```Python\n", "df2.apply(lambda s: s['lower'] + s['upper'], axis=1)[6]\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A35**:\n", "
\n", "\n", "\n", "\n", "\n", "" ] }, { "cell_type": "heading", "level": 1, "metadata": {}, "source": [ "Berkeley I School generator" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Please remind yourself what `enumerate` does." ] }, { "cell_type": "code", "collapsed": false, "input": [ "words = ['Berkeley', 'I', 'School']\n", "\n", "for (i, word) in islice(enumerate(words),1):\n", " print (i, word)" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "(0, 'Berkeley')\n" ] } ], "prompt_number": 43 }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q36**. What is \n", "\n", "```Python\n", "list(enumerate(words))[2][1]\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A36**:\n", "
\n", "\n", "\n", "\n", "\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now consider the generator `g2`" ] }, { "cell_type": "code", "collapsed": false, "input": [ "def g2():\n", " words = ['Berkeley', 'I', 'School']\n", " for word in words:\n", " if word != 'I':\n", " for letter in list(word):\n", " yield letter\n", " \n", "my_g2 = g2()" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 45 }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q37**. What is\n", "\n", "```Python\n", "len(list(my_g2))\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A37**:\n", "
\n", "\n", "\n", "" ] }, { "cell_type": "code", "collapsed": false, "input": [ "def g3():\n", " words = ['Berkeley', 'I', 'School']\n", " for word in words:\n", " yield words\n" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 47 }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q38**. What is\n", "\n", "```Python\n", "len(list(g3()))\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A38**:\n", "
\n", "\n", "\n", "" ] }, { "cell_type": "heading", "level": 1, "metadata": {}, "source": [ "Groupby" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Consider using `groupby` with a DataFrame with states." ] }, { "cell_type": "code", "collapsed": false, "input": [ "import us\n", "import census\n", "import settings\n", "\n", "import pandas as pd\n", "import numpy as np\n", "from pandas import DataFrame, Series\n", "from itertools import islice\n", "\n", "c = census.Census(settings.CENSUS_KEY)\n", "\n", "def states(variables='NAME'):\n", " geo={'for':'state:*'}\n", " \n", " states_fips = set([state.fips for state in us.states.STATES])\n", " # need to filter out non-states\n", " for r in c.sf1.get(variables, geo=geo, year=2010):\n", " if r['state'] in states_fips:\n", " yield r\n", " \n", "# make a dataframe from the total populations of states in the 2010 Census\n", "\n", "df = DataFrame(states('NAME,P0010001'))\n", "df.P0010001 = df.P0010001.astype('int')\n", "df['first_letter'] = df.NAME.apply(lambda s:s[0])\n", "\n", "df.head()\n" ], "language": "python", "metadata": {}, "outputs": [ { "html": [ "
\n", " | NAME | \n", "P0010001 | \n", "state | \n", "first_letter | \n", "
---|---|---|---|---|
0 | \n", "Alabama | \n", "4779736 | \n", "01 | \n", "A | \n", "
1 | \n", "Alaska | \n", "710231 | \n", "02 | \n", "A | \n", "
2 | \n", "Arizona | \n", "6392017 | \n", "04 | \n", "A | \n", "
3 | \n", "Arkansas | \n", "2915918 | \n", "05 | \n", "A | \n", "
4 | \n", "California | \n", "37253956 | \n", "06 | \n", "C | \n", "
5 rows \u00d7 4 columns
\n", "\n", "\n", "\n", "\n", "\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q40**. What is\n", "\n", "\n", "```Python\n", "df.groupby('first_letter').apply(lambda g:len(g.NAME))['A']\n", "```\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A40**:\n", "
\n", "\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q41**. What is\n", "\n", "\n", "```Python\n", "df.groupby('first_letter').agg('count')['first_letter']['P']\n", "```\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A41**:\n", "
\n", "\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q42**. What is\n", "\n", "```Python\n", "len(df.groupby('NAME'))\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A42**:\n", "
\n", "\n", "\n", "" ] }, { "cell_type": "heading", "level": 1, "metadata": {}, "source": [ "Diversity Index" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Recall the code from the diversity calculations" ] }, { "cell_type": "code", "collapsed": false, "input": [ "def normalize(s):\n", " \"\"\"take a Series and divide each item by the sum so that the new series adds up to 1.0\"\"\"\n", " total = np.sum(s)\n", " return s.astype('float') / total\n", "\n", "def entropy(series):\n", " \"\"\"Normalized Shannon Index\"\"\"\n", " # a series in which all the entries are equal should result in normalized entropy of 1.0\n", " \n", " # eliminate 0s\n", " series1 = series[series!=0]\n", "\n", " # if len(series) < 2 (i.e., 0 or 1) then return 0\n", " \n", " if len(series1) > 1:\n", " # calculate the maximum possible entropy for given length of input series\n", " max_s = -np.log(1.0/len(series))\n", " \n", " total = float(sum(series1))\n", " p = series1.astype('float')/float(total)\n", " return sum(-p*np.log(p))/max_s\n", " else:\n", " return 0.0\n", "\n", "def gini_simpson(s):\n", " # https://en.wikipedia.org/wiki/Diversity_index#Gini.E2.80.93Simpson_index\n", " s1 = normalize(s)\n", " return 1-np.sum(s1*s1)" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 55 }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q43**. Suppose you have 10 people and 5 categories, how you would you maximize the Shannon entropy?\n", "\n", "1. Regardless of how you distribute the people, you'll get the same entropy.\n", "1. Put 10 people in any single category, and then 0 in the rest.\n", "1. Distribute the people evenly over all the categories.\n", "1. Put 5 people in each category." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A43**:\n", "
\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q44**. What is\n", "\n", "```Python\n", "entropy(Series([0,0,10,0,0]))\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A44**:\n", "
\n", "\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q45**. What is\n", "\n", "```Python\n", "entropy(Series([10,0,0,0,0]))\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A45**:\n", "
\n", "\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q46**. What is\n", "\n", "```Python\n", "entropy(Series([1,1,1,1,1]))\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A46**:\n", "
\n", "\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q47**. What is\n", "\n", "```Python\n", "gini_simpson(Series([2,2,2,2,2]))\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "**A47**:\n", "
\n", "\n", "\n", "" ] } ], "metadata": {} } ] }