{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# PyCon 2019: Data Science Best Practices with pandas ([video](https://www.youtube.com/watch?v=dPwLlJkSHLo&list=PL5-da3qGB5ICCsgW1MxlZ0Hq8LL5U3u9y&index=36))\n", "\n", "### GitHub repository: https://github.com/justmarkham/pycon-2019-tutorial\n", "\n", "### Instructor: Kevin Markham\n", "\n", "- Website: https://www.dataschool.io\n", "- YouTube: https://www.youtube.com/dataschool\n", "- Patreon: https://www.patreon.com/dataschool\n", "- Twitter: https://twitter.com/justmarkham\n", "- GitHub: https://github.com/justmarkham" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. Introduction to the TED Talks dataset\n", "\n", "https://www.kaggle.com/rounakbanik/ted-talks" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'0.24.2'" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import pandas as pd\n", "pd.__version__" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "%matplotlib inline" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "ted = pd.read_csv('ted.csv')" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "scrolled": false }, "outputs": [ { "data": { "text/html": [ "
\n", " | comments | \n", "description | \n", "duration | \n", "event | \n", "film_date | \n", "languages | \n", "main_speaker | \n", "name | \n", "num_speaker | \n", "published_date | \n", "ratings | \n", "related_talks | \n", "speaker_occupation | \n", "tags | \n", "title | \n", "url | \n", "views | \n", "
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | \n", "4553 | \n", "Sir Ken Robinson makes an entertaining and pro... | \n", "1164 | \n", "TED2006 | \n", "1140825600 | \n", "60 | \n", "Ken Robinson | \n", "Ken Robinson: Do schools kill creativity? | \n", "1 | \n", "1151367060 | \n", "[{'id': 7, 'name': 'Funny', 'count': 19645}, {... | \n", "[{'id': 865, 'hero': 'https://pe.tedcdn.com/im... | \n", "Author/educator | \n", "['children', 'creativity', 'culture', 'dance',... | \n", "Do schools kill creativity? | \n", "https://www.ted.com/talks/ken_robinson_says_sc... | \n", "47227110 | \n", "
1 | \n", "265 | \n", "With the same humor and humanity he exuded in ... | \n", "977 | \n", "TED2006 | \n", "1140825600 | \n", "43 | \n", "Al Gore | \n", "Al Gore: Averting the climate crisis | \n", "1 | \n", "1151367060 | \n", "[{'id': 7, 'name': 'Funny', 'count': 544}, {'i... | \n", "[{'id': 243, 'hero': 'https://pe.tedcdn.com/im... | \n", "Climate advocate | \n", "['alternative energy', 'cars', 'climate change... | \n", "Averting the climate crisis | \n", "https://www.ted.com/talks/al_gore_on_averting_... | \n", "3200520 | \n", "
2 | \n", "124 | \n", "New York Times columnist David Pogue takes aim... | \n", "1286 | \n", "TED2006 | \n", "1140739200 | \n", "26 | \n", "David Pogue | \n", "David Pogue: Simplicity sells | \n", "1 | \n", "1151367060 | \n", "[{'id': 7, 'name': 'Funny', 'count': 964}, {'i... | \n", "[{'id': 1725, 'hero': 'https://pe.tedcdn.com/i... | \n", "Technology columnist | \n", "['computers', 'entertainment', 'interface desi... | \n", "Simplicity sells | \n", "https://www.ted.com/talks/david_pogue_says_sim... | \n", "1636292 | \n", "
3 | \n", "200 | \n", "In an emotionally charged talk, MacArthur-winn... | \n", "1116 | \n", "TED2006 | \n", "1140912000 | \n", "35 | \n", "Majora Carter | \n", "Majora Carter: Greening the ghetto | \n", "1 | \n", "1151367060 | \n", "[{'id': 3, 'name': 'Courageous', 'count': 760}... | \n", "[{'id': 1041, 'hero': 'https://pe.tedcdn.com/i... | \n", "Activist for environmental justice | \n", "['MacArthur grant', 'activism', 'business', 'c... | \n", "Greening the ghetto | \n", "https://www.ted.com/talks/majora_carter_s_tale... | \n", "1697550 | \n", "
4 | \n", "593 | \n", "You've never seen data presented like this. Wi... | \n", "1190 | \n", "TED2006 | \n", "1140566400 | \n", "48 | \n", "Hans Rosling | \n", "Hans Rosling: The best stats you've ever seen | \n", "1 | \n", "1151440680 | \n", "[{'id': 9, 'name': 'Ingenious', 'count': 3202}... | \n", "[{'id': 2056, 'hero': 'https://pe.tedcdn.com/i... | \n", "Global health expert; data visionary | \n", "['Africa', 'Asia', 'Google', 'demo', 'economic... | \n", "The best stats you've ever seen | \n", "https://www.ted.com/talks/hans_rosling_shows_t... | \n", "12005869 | \n", "
\n", " | comments | \n", "description | \n", "duration | \n", "event | \n", "film_date | \n", "languages | \n", "main_speaker | \n", "name | \n", "num_speaker | \n", "published_date | \n", "ratings | \n", "related_talks | \n", "speaker_occupation | \n", "tags | \n", "title | \n", "url | \n", "views | \n", "
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1787 | \n", "2673 | \n", "Our consciousness is a fundamental aspect of o... | \n", "1117 | \n", "TED2014 | \n", "1395100800 | \n", "33 | \n", "David Chalmers | \n", "David Chalmers: How do you explain consciousness? | \n", "1 | \n", "1405350484 | \n", "[{'id': 25, 'name': 'OK', 'count': 280}, {'id'... | \n", "[{'id': 1308, 'hero': 'https://pe.tedcdn.com/i... | \n", "Philosopher | \n", "['brain', 'consciousness', 'neuroscience', 'ph... | \n", "How do you explain consciousness? | \n", "https://www.ted.com/talks/david_chalmers_how_d... | \n", "2162764 | \n", "
201 | \n", "2877 | \n", "Jill Bolte Taylor got a research opportunity f... | \n", "1099 | \n", "TED2008 | \n", "1204070400 | \n", "49 | \n", "Jill Bolte Taylor | \n", "Jill Bolte Taylor: My stroke of insight | \n", "1 | \n", "1205284200 | \n", "[{'id': 22, 'name': 'Fascinating', 'count': 14... | \n", "[{'id': 184, 'hero': 'https://pe.tedcdn.com/im... | \n", "Neuroanatomist | \n", "['biology', 'brain', 'consciousness', 'global ... | \n", "My stroke of insight | \n", "https://www.ted.com/talks/jill_bolte_taylor_s_... | \n", "21190883 | \n", "
644 | \n", "3356 | \n", "Questions of good and evil, right and wrong ar... | \n", "1386 | \n", "TED2010 | \n", "1265846400 | \n", "39 | \n", "Sam Harris | \n", "Sam Harris: Science can answer moral questions | \n", "1 | \n", "1269249180 | \n", "[{'id': 8, 'name': 'Informative', 'count': 923... | \n", "[{'id': 666, 'hero': 'https://pe.tedcdn.com/im... | \n", "Neuroscientist, philosopher | \n", "['culture', 'evolutionary psychology', 'global... | \n", "Science can answer moral questions | \n", "https://www.ted.com/talks/sam_harris_science_c... | \n", "3433437 | \n", "
0 | \n", "4553 | \n", "Sir Ken Robinson makes an entertaining and pro... | \n", "1164 | \n", "TED2006 | \n", "1140825600 | \n", "60 | \n", "Ken Robinson | \n", "Ken Robinson: Do schools kill creativity? | \n", "1 | \n", "1151367060 | \n", "[{'id': 7, 'name': 'Funny', 'count': 19645}, {... | \n", "[{'id': 865, 'hero': 'https://pe.tedcdn.com/im... | \n", "Author/educator | \n", "['children', 'creativity', 'culture', 'dance',... | \n", "Do schools kill creativity? | \n", "https://www.ted.com/talks/ken_robinson_says_sc... | \n", "47227110 | \n", "
96 | \n", "6404 | \n", "Richard Dawkins urges all atheists to openly s... | \n", "1750 | \n", "TED2002 | \n", "1012608000 | \n", "42 | \n", "Richard Dawkins | \n", "Richard Dawkins: Militant atheism | \n", "1 | \n", "1176689220 | \n", "[{'id': 3, 'name': 'Courageous', 'count': 3236... | \n", "[{'id': 86, 'hero': 'https://pe.tedcdn.com/ima... | \n", "Evolutionary biologist | \n", "['God', 'atheism', 'culture', 'religion', 'sci... | \n", "Militant atheism | \n", "https://www.ted.com/talks/richard_dawkins_on_m... | \n", "4374792 | \n", "
\n", " | comments | \n", "description | \n", "duration | \n", "event | \n", "film_date | \n", "languages | \n", "main_speaker | \n", "name | \n", "num_speaker | \n", "published_date | \n", "ratings | \n", "related_talks | \n", "speaker_occupation | \n", "tags | \n", "title | \n", "url | \n", "views | \n", "comments_per_view | \n", "
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
954 | \n", "2492 | \n", "Janet Echelman found her true voice as an arti... | \n", "566 | \n", "TED2011 | \n", "1299110400 | \n", "35 | \n", "Janet Echelman | \n", "Janet Echelman: Taking imagination seriously | \n", "1 | \n", "1307489760 | \n", "[{'id': 23, 'name': 'Jaw-dropping', 'count': 3... | \n", "[{'id': 453, 'hero': 'https://pe.tedcdn.com/im... | \n", "Artist | \n", "['art', 'cities', 'culture', 'data', 'design',... | \n", "Taking imagination seriously | \n", "https://www.ted.com/talks/janet_echelman | \n", "1832930 | \n", "0.001360 | \n", "
694 | \n", "1502 | \n", "Filmmaker Sharmeen Obaid-Chinoy takes on a ter... | \n", "489 | \n", "TED2010 | \n", "1265760000 | \n", "32 | \n", "Sharmeen Obaid-Chinoy | \n", "Sharmeen Obaid-Chinoy: Inside a school for sui... | \n", "1 | \n", "1274865960 | \n", "[{'id': 23, 'name': 'Jaw-dropping', 'count': 3... | \n", "[{'id': 171, 'hero': 'https://pe.tedcdn.com/im... | \n", "Filmmaker | \n", "['TED Fellows', 'children', 'culture', 'film',... | \n", "Inside a school for suicide bombers | \n", "https://www.ted.com/talks/sharmeen_obaid_chino... | \n", "1057238 | \n", "0.001421 | \n", "
96 | \n", "6404 | \n", "Richard Dawkins urges all atheists to openly s... | \n", "1750 | \n", "TED2002 | \n", "1012608000 | \n", "42 | \n", "Richard Dawkins | \n", "Richard Dawkins: Militant atheism | \n", "1 | \n", "1176689220 | \n", "[{'id': 3, 'name': 'Courageous', 'count': 3236... | \n", "[{'id': 86, 'hero': 'https://pe.tedcdn.com/ima... | \n", "Evolutionary biologist | \n", "['God', 'atheism', 'culture', 'religion', 'sci... | \n", "Militant atheism | \n", "https://www.ted.com/talks/richard_dawkins_on_m... | \n", "4374792 | \n", "0.001464 | \n", "
803 | \n", "834 | \n", "David Bismark demos a new system for voting th... | \n", "422 | \n", "TEDGlobal 2010 | \n", "1279065600 | \n", "36 | \n", "David Bismark | \n", "David Bismark: E-voting without fraud | \n", "1 | \n", "1288685640 | \n", "[{'id': 25, 'name': 'OK', 'count': 111}, {'id'... | \n", "[{'id': 803, 'hero': 'https://pe.tedcdn.com/im... | \n", "Voting system designer | \n", "['culture', 'democracy', 'design', 'global iss... | \n", "E-voting without fraud | \n", "https://www.ted.com/talks/david_bismark_e_voti... | \n", "543551 | \n", "0.001534 | \n", "
744 | \n", "649 | \n", "Hours before New York lawmakers rejected a key... | \n", "453 | \n", "New York State Senate | \n", "1259712000 | \n", "0 | \n", "Diane J. Savino | \n", "Diane J. Savino: The case for same-sex marriage | \n", "1 | \n", "1282062180 | \n", "[{'id': 25, 'name': 'OK', 'count': 100}, {'id'... | \n", "[{'id': 217, 'hero': 'https://pe.tedcdn.com/im... | \n", "Senator | \n", "['God', 'LGBT', 'culture', 'government', 'law'... | \n", "The case for same-sex marriage | \n", "https://www.ted.com/talks/diane_j_savino_the_c... | \n", "292395 | \n", "0.002220 | \n", "
\n", " | comments | \n", "description | \n", "duration | \n", "event | \n", "film_date | \n", "languages | \n", "main_speaker | \n", "name | \n", "num_speaker | \n", "published_date | \n", "ratings | \n", "related_talks | \n", "speaker_occupation | \n", "tags | \n", "title | \n", "url | \n", "views | \n", "comments_per_view | \n", "views_per_comment | \n", "
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
744 | \n", "649 | \n", "Hours before New York lawmakers rejected a key... | \n", "453 | \n", "New York State Senate | \n", "1259712000 | \n", "0 | \n", "Diane J. Savino | \n", "Diane J. Savino: The case for same-sex marriage | \n", "1 | \n", "1282062180 | \n", "[{'id': 25, 'name': 'OK', 'count': 100}, {'id'... | \n", "[{'id': 217, 'hero': 'https://pe.tedcdn.com/im... | \n", "Senator | \n", "['God', 'LGBT', 'culture', 'government', 'law'... | \n", "The case for same-sex marriage | \n", "https://www.ted.com/talks/diane_j_savino_the_c... | \n", "292395 | \n", "0.002220 | \n", "450.531587 | \n", "
803 | \n", "834 | \n", "David Bismark demos a new system for voting th... | \n", "422 | \n", "TEDGlobal 2010 | \n", "1279065600 | \n", "36 | \n", "David Bismark | \n", "David Bismark: E-voting without fraud | \n", "1 | \n", "1288685640 | \n", "[{'id': 25, 'name': 'OK', 'count': 111}, {'id'... | \n", "[{'id': 803, 'hero': 'https://pe.tedcdn.com/im... | \n", "Voting system designer | \n", "['culture', 'democracy', 'design', 'global iss... | \n", "E-voting without fraud | \n", "https://www.ted.com/talks/david_bismark_e_voti... | \n", "543551 | \n", "0.001534 | \n", "651.739808 | \n", "
96 | \n", "6404 | \n", "Richard Dawkins urges all atheists to openly s... | \n", "1750 | \n", "TED2002 | \n", "1012608000 | \n", "42 | \n", "Richard Dawkins | \n", "Richard Dawkins: Militant atheism | \n", "1 | \n", "1176689220 | \n", "[{'id': 3, 'name': 'Courageous', 'count': 3236... | \n", "[{'id': 86, 'hero': 'https://pe.tedcdn.com/ima... | \n", "Evolutionary biologist | \n", "['God', 'atheism', 'culture', 'religion', 'sci... | \n", "Militant atheism | \n", "https://www.ted.com/talks/richard_dawkins_on_m... | \n", "4374792 | \n", "0.001464 | \n", "683.134291 | \n", "
694 | \n", "1502 | \n", "Filmmaker Sharmeen Obaid-Chinoy takes on a ter... | \n", "489 | \n", "TED2010 | \n", "1265760000 | \n", "32 | \n", "Sharmeen Obaid-Chinoy | \n", "Sharmeen Obaid-Chinoy: Inside a school for sui... | \n", "1 | \n", "1274865960 | \n", "[{'id': 23, 'name': 'Jaw-dropping', 'count': 3... | \n", "[{'id': 171, 'hero': 'https://pe.tedcdn.com/im... | \n", "Filmmaker | \n", "['TED Fellows', 'children', 'culture', 'film',... | \n", "Inside a school for suicide bombers | \n", "https://www.ted.com/talks/sharmeen_obaid_chino... | \n", "1057238 | \n", "0.001421 | \n", "703.886818 | \n", "
954 | \n", "2492 | \n", "Janet Echelman found her true voice as an arti... | \n", "566 | \n", "TED2011 | \n", "1299110400 | \n", "35 | \n", "Janet Echelman | \n", "Janet Echelman: Taking imagination seriously | \n", "1 | \n", "1307489760 | \n", "[{'id': 23, 'name': 'Jaw-dropping', 'count': 3... | \n", "[{'id': 453, 'hero': 'https://pe.tedcdn.com/im... | \n", "Artist | \n", "['art', 'cities', 'culture', 'data', 'design',... | \n", "Taking imagination seriously | \n", "https://www.ted.com/talks/janet_echelman | \n", "1832930 | \n", "0.001360 | \n", "735.525682 | \n", "
\n", " | event | \n", "film_datetime | \n", "
---|---|---|
831 | \n", "TEDWomen 2010 | \n", "2010-12-08 | \n", "
2464 | \n", "TED2017 | \n", "2017-04-24 | \n", "
2392 | \n", "TEDxBeaconStreet | \n", "2016-11-19 | \n", "
1307 | \n", "TEDxUCL | \n", "2012-06-03 | \n", "
2234 | \n", "TED2016 | \n", "2016-02-17 | \n", "
\n", " | count | \n", "mean | \n", "
---|---|---|
event | \n", "\n", " | \n", " |
TEDxNorrkoping | \n", "1 | \n", "6569493.0 | \n", "
TEDxCreativeCoast | \n", "1 | \n", "8444981.0 | \n", "
TEDxBloomington | \n", "2 | \n", "9484259.5 | \n", "
TEDxHouston | \n", "2 | \n", "16140250.5 | \n", "
TEDxPuget Sound | \n", "1 | \n", "34309432.0 | \n", "
\n", " | count | \n", "mean | \n", "sum | \n", "
---|---|---|---|
event | \n", "\n", " | \n", " | \n", " |
TED2006 | \n", "45 | \n", "3.274345e+06 | \n", "147345533 | \n", "
TED2015 | \n", "75 | \n", "2.011017e+06 | \n", "150826305 | \n", "
TEDGlobal 2013 | \n", "66 | \n", "2.584163e+06 | \n", "170554736 | \n", "
TED2014 | \n", "84 | \n", "2.072874e+06 | \n", "174121423 | \n", "
TED2013 | \n", "77 | \n", "2.302700e+06 | \n", "177307937 | \n", "