{ "cells": [ { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Learning objectives\n", "\n", "We will learn the basic lingo around data and specifically talk about two types of data: quantitative and qualitative.\n", "\n", "We then go over how to succinctly describe both types of data with summary statistics. \n", "\n", "While basic, this lesson will give us a common vocabulary that we will use for the rest of the class." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# How We begin\n", "\n", "We begin the journey into data science with a single assumption, we have data. While this may sound like a must, it is actually not always the case. Sometimes you will begin without any data and you will be asked to rectify that problem in any number of ways: collecting it, purchasing, scraping, etc. \n", "\n", "So while the first stage of data science might actually be collecting data, this is not where **we** will start. First we will get a feel for data. Try to understand what we can do with it and what is a 'good dataset'. Then hopefully that first problem will work itself out. " ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## The data assumption\n", "\n", "So as you will come to expect, let's start off by listing the assumptions we have made (and as you will further come to expect we will do it in graph form):" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAARUAAAA7CAYAAABVPx4QAAAAAXNSR0IArs4c6QAAFMlJREFUeAHt\nnQWwJDUTx3PH4Ryuh7sdHO5euLu7OxRWFBQUUGhROBTufri7u/vhzuEOh0O+/nVV75ednVl5O7s7\nuy9dlZfszLxMppP5J93p7hngvR/uIkUORA5EDuTEgQECKj6numI1kQORA5EDbmDkQeRA5EDkQJ4c\niKCSJzdjXZEDkQNxpRLHQORA5EC+HBiUb3WxtmY58N9//7lff/3V/fLLL6XE71GjRrnffvtN87D8\nxx9/uN9//92RJ9Nff/3lLP39999l5X///deR/vnnn7Kc+6Nms5yypQEDBri0NHDgQDfaaKNpGjRo\nUEU+xhhjONLoo4+uuf0ec8wx3VhjjVWWxh57bP09zjjjuHHHHdclc44NHjxY0/jjj6859UYqDgei\norYFfcGL/M0337jvv/9e03fffVcqc+zHH3/U9NNPP5XlP//8swJHNd05LyEvlr1wvHT2YtoLab/t\n5U3m9uIDBgYIVk7+BizsnIENYATokGqVOW/Alsz//PPPEhCGwEg5CaDcK4sAJ4BmggkmcBNOOGEp\nt/LEE0/skmmSSSZxk046qf5fVr3xeN84EEGlTr4BAF9++aWmr776qqz89ddfu2+//VaBBDABHJLE\ni24De6KJJiob/OHLYLNwMjcQ4QXvj8QqjBVachVnKzr6JwnS/P7hhx9KgJ7WLwAS4DLZZJOV0hRT\nTOFIU045pSYrTz755Aqw/ZH/jTxzBBXhFiuJTz75pJQ+++wzN3LkSE2ff/655gzokJgFGXQ2ABmU\n4eCkTDIgYUURqbMcQNQzkKHPmQBI4YTAbyYPJg4mC1ZaRqzw6POpp55a05AhQ9w000zjpp12Wjfd\ndNNp4hjX9WfqF6DCDPXee++5Dz/8sCIBJiy1jVghMEDCgcNAsWRAwgwXqbc5gMgF+AAyX3zxhbMJ\nhtzKn376qQKQiayIi4yVGWaYwc0444xlaeaZZ9Zz6KV6mXoGVJDX3333XffWW29pogyQkDPjGLGE\npbOt08kBEWYbEqJIpMiBRjjA2GN1C8CQPvroI01MYpQ5hqIcYsU6yyyzaJp11lndbLPN5uacc043\nxxxzOMTiXqCuAxU65+2333avv/66e+211zR/88033QcffKBLVWYBlqR0lnUeObMEKYohvTBsu+sZ\nEKEAlvfff18nOiY7m/DIUVhDiNKAy9xzz+3mmWceN3ToUE2I2t1EhQYVxJaXX37ZvfTSS6UEgAAs\nyK2ABYynE2affXbtEMAEpWakyIFu4ACAw2qGiZJV9ogRI9wbb7yhCSU0xAp6vvnmc/PPP79bYIEF\nNGd1XVQqDKgAFK+88op75pln3LPPPqsJRiOrIrLA1GHDhmkOigMibJVGihzoVQ6g72NFzntBYoJF\nnEfXw8bAIossUkqLLrpoYcSnjoEKW4NPPPGEe/zxxzUBJChMAZCFFlqoLE011VS9Om7ic0UONMQB\n3hHA5fnnn9f03HPP6SqHSuaaay631FJLuSWXXNIts8wybvrpp2+o7rwubhuosBJ58skn3QMPPKAJ\nEAFx5513XrfEEku4xRdfXPOZZpopr2eL9UQO9AsOYI/z9NNPu6eeekrfMcqITugQV1hhBU0rrrii\nmji0gyEtBRW24m6//XZ31113ufvuu08Nl5ALl19+ebfsssu6pZdeWo3A2vGg8R6RA/2FA9jjvPji\ni+6RRx5xDz/8sOZYKS+88MJu9dVXd2ussYZbcMEFW8aO3EEFOfD66693N954oyInBmCrrbaaW2WV\nVRxoiSwYKXIgcqB9HGDLGynh3nvvdXfeeafqZ1D0rr/++m6DDTZQcSlP25lcQIVdmuHDh7srrrjC\nPfroo7rsorFrr722W2yxxaJpc/vGT7xT5EBNDjDx33rrre7mm2/WlQwmGFtssYXbaqutdAe1ZgU1\nLmgKVNjqPfvss91VV12lRmM0bJNNNmnp0qrG88TTkQORAw1wAMPQ6667Tt9hdDIoeHfbbTddxfTV\n+7tPoMIy6vjjj1dZba211nI77bSTW3XVVdXlvYHniZdGDkQOFIgD2MlceOGF7uKLL1bP9/3339/t\nvPPOjdt9iR1I3fTYY495EWe8WKX6ffbZx4uFYN3/Gy+MHIgc6A4OyLa1P+ecc7y4EXjRifrTTjvN\ni16m7sZjXFaTxGPTb7zxxl6c6Py+++7rZVen5v/ECyIHIge6mwOyi+Rl1eLFP04B5qGHHqrrgWqK\nPyh0tt9+ezUNPuuss9SnplMrNtzW77777orbY/CDKXO3E1uB2O9gt9MqYpufZe5yyy3XqlvUXS+2\nFOjjcLzD5WLzzTfv175ZbAOzW1o0E3y2o4899lh34oknqjh00kknaRS/zI6uBj3HHHOMrk5OOeWU\nape19RyrJHkh+KyIF+cr//HHH3sxomtrG1pxM4kG56XjvOyktaJ6Lwo5LzKyl+hwfu+9927JPRqp\nVIDNSxgJnQHF3UL7U4y1vIQYaKSanrpWDET9rrvu6sW+pJDP9cILL3j6SOzLvEQwzGxjpvjDAEd3\ncv/992f+c6dOnHzyyToIi/ByXHrppU2zQdzmvSi8PcDSKpIVkBf/kcLwTWyXtD08L4C34447attk\nVdwqFnRFvYgc8ObVV18tZHslvowXNxovfkdeVpqpbUwFlQcffNBL2EIv1rCp/9Tpg6Kh1gF42GGH\ndbQp8EmCOTXdBvRVF110UdP11KpAXOwLASrit+LFpqmsuRL0SMccq8/+TmJ9rhsiReWDqCG8uNP4\nHXbYIbWJqXHvRBnrtt12WzXnzZSbCniCmBVY8u61117qQn7LLbeofIr9jMV2JZgOeiL24jFjvuee\nezTKmzDIETiaCF/Uga/SSiutpGEVREGlVog8MlaIyLwcW2eddTS6/LnnnqsRvdheFy5rvTh9EQWM\n+BjUk0XoUO644w53wQUXlC4hTowsNfU3day88spqdk2IQ2wHBISqy7SlmvIr4KJ/7bXXahBrakWH\nRXAhWcmqDxde5KR6iMBYuPCHhNMopuNFCsWYdz/UMz7hCZbnvIOMQ8Zb0Yj4LoxX/IpkF1hjv5S1\nMQk14mqts5kojZKnCvM7baUiQOHFBUDbjg5ou+2282uuuab+RpSDmB0lupbqFZBdWWqLL4Rew3LO\nts3EOliPCeNKz3zkkUfqMQEhPSaGf168QfWeaMX5DR1yyCH+/PPP17J4kOoyUX9k/BHLYy+DqOLs\nJZdcovfbcsst9ZwoqL34S3mJp1pxbb0Hml2poO8RB1BtlwQX0tuKktxfc8019Tah6nXoWI466qiq\n17T7ZF79UM/4DJ9N7EO8AG94qHBl8Yr2Bx98cEW7KsQfMd3VQSPRyysuLsqBNFChbTygIGaZHoiO\nkRmw1HReUvFz8ICnEWIU/8fePGTAGoIKg4JrDFS4bt111/UyY1NUQmHMvj4gY3T00UdbMTXHFmDr\nrbdOPUdbJQq/f+eddxQgm93KbxZUaKQEEFLlPYP+tttu84cffnhq2xs9iHJSzMUz5fRG68vz+rz6\noZ7xae3GNkRWbZ4+KyptuOGGfqONNqpoXsX3HvgeChTGddUDXfAH8QVC5DAixgS+DkZEhWOJTbQ4\nI+lsPYbfUqMUOmJRJngUrgqIXtABBxyQWSWOXoTBzIoXIwNLvbgJC8G2PuEGO03wU4DEyWrMnX76\n6U4AuekmIVpRJ2LpeOON13R9eVeQVz/UMz6t7cRKxsSAcJNFJTDC8CJsYwWo4B7NhTgI9gKhkxAo\nrfooxK3FqYrPMzRKIajwv2eeeabjy3myilHZmA+HZREfFuOFssGWvI7Pe8hKRyO6E9SqKHTQQQc5\n9CLop2h/swTw7rfffmoL1Wxdrfj/VvZD1vg0cIXHRSTahecz0QeSVAEqfHqCTsbYpagPlHyIZn/L\nElM/w9CXAFFJUEFZiRHT7rvvrh6gKCQBjzTicx8ovSwWafIaglihxMXTG4UYhmtFIGJ0sGLBiE50\nTU016bzzzlMwwaO9qNSJfsDQEyqqUSfjkdU+mxNJqgAVLjjwwAM1IhsdzXdPep3wzuQLeKLY1Ue1\nHQiOVSMAJZypAafLL79cP6WJ9TGAYLtJWfXQMVmipiicdYcJq1NEJXasOk2svE444QR3ww03aHuw\nsrSdqkbbdtNNN+kqUnRKZf/KrlyRqBP9wLhhfPE5maLRoYceqkHXGOustJKUCipciHzLpzpxhSYA\ndZGI+C2Q5dY2+80LaMTX53jZQxEIWZWo/Ea8ILKzUgIVIvKzvJddDScWuzoj4x4OEe6BmQtCF8Lq\nAb0In19ARBFlb+lebAVjdk3KIqLfsXWZJAIesyLYZpttdGChuyD+BTFr+ko2+2WBJR6pRAZj6zqL\n2K5H/8GKFk91RAPZadOPxGf9T9pxtqIBJ7buERlJ6C522WUXJ4ZfpX8B8AnwjA6nGiGSiQFd6RL4\nR4RBluhGyWs4XuuZ8+yHescn7SLCPuOHd7AoxHtE/5xxxhk6YRKAPpUqVLfBAbH59xK4xYt858UW\no+Pm8JgGs10s6K07Mey0yMD2ooj18gKqQY48pFpnYu599dVXe9Fv6LVHHHGExwxamOIFNP2ee+7p\nZUXmN910U7VmlQ4Pntx7dn5ENNFn32yzzdR0mt0JHCoFZPVadnnQ0HOdKC09/BKg0ToFhLzM4jV3\nR3gmCfbtbYuWijGqE1DzIoaWeH7llVfqc7AbZFvWZQ2u8UMifnlRIGsd3I864FFImGDDP/HtCA9r\nGf7AM4z9BJD1GI6mbKvzPxL6wkuk94r/SzuAubcozPX/+N8w8XxYbRpddtlleh4eY22aRRjN8Vx2\nDdvc1CsvQOlfktdwotoz59kP9Y5P2iQvrxe9pscIriiEiYmI9l58tHQHsFq7mFVrEi8nL7B8BsBL\nBPya1xf5AkBFDMi0iYCRBA3ObC4gYWCDDYuIOhXXYlpv13AS4GJQ4JNUL7GVvccee9R7ecuuw4xA\nDNy87Fy17B59qRgzfuyKqhEm4wB0SPRvSGnXFPGZsZMSw8qw6R0rM3EwmTB5MkazTPPDBqZa1ArC\nl5HM5moVKoZJ6t2KxR9yFZ8C6GaqpQRj6WnLz6woWMnPpJo+phFPU4JcYfWLaEVg8HoIfQ2pGvE9\naPqpXmJ5i7iBnqSvhIK6FiFy1Gt9S12IMNWskrnGdksoGyX7N+2aZp85735A+S2rUicTuT1GR3LE\n+lNPPVXFUsYkn9Lh20L1UF2gQkVsMyPzihOfytKY6DIw0AJjSmwvXz037eQ1fDcFnQr6j7RB1om2\n4UIglpvqXgDAsK1fi1DgoTOoRknAq3Yt53AZYNfPgLHW9Wnna7WJ/2kk+LmsAjVUqXimp92u6WPN\nPnOe/YD+7rjjjnPiB5ZpZtD0A9eoAH6wyYA+kZ1LwC1th6dqNeGypZHyyJEj1YIVORbTd5ZI0qBG\nqmj7tZjpiwGZytoyo5ZM69vekCo3bERsqlJNPNWFHMCpEqvsdhO6NXSVonhVMUcWCaqj7Gs7agZp\nqopIcpKdFnZPmGn5UBhfRcOiVEx4K5zGatXV6vN8dEkYVboNOxhZhmeli2IhcqAHOYB4g9U3TqJs\n4YvC2rG1j+X2kCFDmnripkElvDt76yyX2H7lm8jI9CydsP9gyzZ+OD3kVixHDrSPA0ymeM7zYT/M\nRRBzeD9RXRBxr159ST0tzhVUwhuKeKR2FTwASMhDodhl7x25u2hu7mHbYzlyoBc4gK0Ltk7YBPGF\nUIwsMbZkkl9vvfXU/idpEZ7Hc7cMVMLGoRyVPX9FSUQkjOlQkgIyGNcRkxUDJ3xwIkUORA40zgEm\n7REjRujunXz1QsEER1qME1FyM5lj2JjcEWv8TrX/oy2gkmyGKKQUZEBRtqoAGXYc+Fg7HrkSrk53\nQPA2TjMDTtYXf0cO9DcOoGqQCHqaUDXwUXZ0hoCITda2Q2sBytrFo46ASvLh8A4Wozq1R0DWw5eE\nLV90MGxbs08+bNgwLQ8dOrRrtq+Tzxl/Rw40ygFWILiBoA+RGMOa834wMQMWRN9jlc9kDJjwuxUi\nTSPtLgSoJBuMbw1LOQAGgzASDAVoWLnwOQfABfnQkgQ7chKVPVlV/B050BUcADzQQzLu8TeSYFil\nxLgHQIjVwwRLYjVPKoqtVcjkQoJK2EArAzTiW6LOZjAdJzxyHPk4B9jgBIjIBPNJgA9bZciR7V4C\nWrtjHjkQcoAwGIxZgi9JRD8V/bGipTxq1Ci9FONAJk0c9ixHNdAtOseuAZWwY8IyHzqiQ/A6pnPI\n0dHQadZJ2KNg+QjAkAM+llNGDo0UOZAHB/AAR0HKB9LYfSFRRoRhTJqnOJMgrhxMfkyEiC2WN2Jx\nnEeb866j60GlGkOQO+lIVjjkdK51MCERjNDdsJqhk8lJRILDCIjEfj5uCp2WVa29Me8MB5ikGFMk\nRBUSEfLDFMbGwbCSycsSkxqrZ0R1jvWquN7ToFJt6CGnAjDMKpYYHJTJ0a7jbGbEACB+CtHaiBUb\n5uKqoP4szDAkVj5x18o4V+wc3yI2CphkyAEF4slgcRrmAAnXGjHB0NdMPkxGNiFZGdAoQkxha287\n834LKrWYjOKMqHfMRjYzATQ20MJBB0CFhP5G/KE0OBMAk0ycI4wkCac/yykPHjw4AlLIzDrKrCB4\n4YlKx7ZqmFNGjxEm+pUEkIQBvbgV3uhMEuGkQZlkq1ZyJpgsz/U6mtzTl0RQyaF7kaOZ5cLEgGXg\nhoPZysjVvARhKMqwGSjkAJcwIaJxnDwsc8xCNFjOspsyqysSg9/K9hvgYzVFXq0cngNoaTOKcVJW\nOTyHRzgvLokob1YmZyWITgz+hYljAAUJw8kwJ55vmAB07pdG7IwA2IC4ATtirJWJyGerSytzfaTm\nOBBBpTn+NfXfvBzhzArQhC8MZY7x4oQvVviihS8l5VBka6pxbfxnDB8NCA0YDUAtNyAFKEKwDcu2\n4rM8iqBt7MTgVhFUAmb0QpHVBMBCClcFVma1QGIFwUojmTPrU0eY7Bh6hDCxirHfvMCAQ5hTZpWU\nXCnZaoldOUAkvvy9MPL+/wwRVP7Pi1iKHIgcyIEDqdH0c6g3VhE5EDnQTzkQQaWfdnx87MiBVnEg\ngkqrOBvrjRzopxz4Hwgnjn5CYssyAAAAAElFTkSuQmCC\n", "text/plain": [ "" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import networkx as nx\n", "from nxpd import draw\n", "from nxpd import nxpdParams\n", "nxpdParams['show'] = 'ipynb'\n", "\n", "G = nx.DiGraph()\n", "G.add_node('Inputs (x_1, x_2, ..., x_n)')\n", "draw(G)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "Just one node/bubble (we will get more don't worry). And as you will see in a moment, there is some that we can do by only relying on this assumption, but not much. \n", "\n", "I will call this node the input node, it's basically the dataset. Notice that our dataset contains multiple points (x_1, x_2, ...).\n", "\n", "We start off by assuming we have a series of data points. We will focus on describing these data points today." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## To the data\n", "\n", "So we are not just in the realm of the theoretical, let's \"get\" some actual data." ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": true, "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "data = {\n", " 'quantitative1': [1, 4, 6, 10, 15, 16],\n", " 'quantitative2': [4, 6, 12, 13, 16, 19],\n", " 'qualitative': ['red', 'blue', 'red', 'blue', 'yellow', 'red']\n", "}" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
qualitativequantitative1quantitative2
0red14
1blue46
2red612
3blue1013
4yellow1516
5red1619
\n", "
" ], "text/plain": [ " qualitative quantitative1 quantitative2\n", "0 red 1 4\n", "1 blue 4 6\n", "2 red 6 12\n", "3 blue 10 13\n", "4 yellow 15 16\n", "5 red 16 19" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import pandas as pd\n", "\n", "# and now let's put that data in a DataFrame (python's Excel spreadsheet)\n", "pd.DataFrame(data)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "There we have it. Notice that my data has multiple columns (quant and qual) and rows (6 rows to be exact). The columns can sometimes go by other names: dimensions, features, covariates, m, etc. and the rows can go by other names too: samples, data points, n, etc.\n", "\n", "I will generally call the columns features and the rows data points." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "\n", "\n", "## Quant or Qual?\n", "\n", "And notice that they come in two flavors:\n", "\n", "1. Quantitative: something that you can describe with a number (speed, location, size, etc.)\n", "2. Qualitative: something that is hard to describe with an exact number (language, make of a car, etc.)\n", "\n", "You will also notice very fine lines. The definitions can be crossed pretty easily. But perhaps the best question you can ask about a particular feature to determine if it is a quantitative or a qualitative feature is: can I average the feature?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "In this respect we can see how things that you classically thought were qualitative can become quantitative. For example, this might seem plausible:\n", "\n", "```\n", "ave(depressed , content) = melancholy\n", "```\n", "\n", "Then maybe instead of using qualitative descriptions, you should be using a happiness score/index. But then you can look at this puzzling question:\n", "```\n", "ave(confused , giddy) = ?\n", "```\n", "\n", "Perhaps you should just be using a qualitative feature at this point. " ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "But now we are well on our way to describing data. \n", "\n", "1. We can talk about how many features and data points we have. \n", "2. And we can talk about whether the features are quantitative or qualitative. \n", "\n", "But there is so much more that we can do!" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Describing qualitative features\n", "\n", "The classic way that we can describe features (other than what we alluded to above) is with summary statistics a description of the data using a single number. An example would be the average of a particular column.\n", "\n", "A summary statistic is innately a quantitative one (it is a number). Because qualitative features are non-quantitative (are not numbers themselves), there are only a few summary statistics (numbers describing them) that make sense. \n", "\n", "One of them is the **count** of each term. We can do this using the python library pandas below:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "data": { "text/plain": [ "qualitative\n", "blue 2\n", "red 3\n", "yellow 1\n", "Name: qualitative, dtype: int64" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df = pd.DataFrame(data)\n", "\n", "df.groupby('qualitative').qualitative.count()" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "In such a way we can get a sense of what types of data we have and how many of each. We can easily get the **mode** (or the most common type) from this analysis as well.\n", "\n", "As a final note, some qualitative data can be ordered, and in this case you will be able to find the **median** (the center) of that data as well." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Describing quantiative features\n", "\n", "Now we are getting into the good stuff. We are looking to describe quantitative features with quantitative statistics. Remember that we are doing the impossible. We are taking what could be millions of data points and summarizing them with a single number, so these had better be pretty important numbers! And it turns out they are. They fall into four categories:\n", "\n", "1. **Center statistics**. Things that tell us where the center or majority of the data lives. Eg. the mean, median and mode (plus a ton of other arcane measures, geometric mean, winsorized mean, etc.)\n", "2. **Spread**. Measures how far away from the center is your data. Eg. standard deviation and variance (plus again a ton of other measures like the absolute deviation, range, gini coefficient, etc.)\n", "3. **Dependence**. How related are two factors. Eg. correlation (Pearson's specifically)\n", "4. **Shape**. What is the shape of the distribution of the data (more on this next time)\n" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "These are the most important summary statistics and if you know them about your data, you can generally infer a ton about it. But don't be fooled! You might know less than you think! Just knowing the mean of a data set does not tell you too much about it. The prime example is a bimodal distribution (google it).\n", "\n", "
\n", " A question to ask yourself is: why do we care about any of these measures\n", "
\n", "\n", "Below is how we get the center and spead statistics:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "8.66666666667\n", "6.05530070819\n", "8.0\n" ] } ], "source": [ "print df.quantitative1.mean()\n", "# standard deviation\n", "print df.quantitative1.std()\n", "print df.quantitative1.median()" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
quantitative1quantitative2
count6.0000006.000000
mean8.66666711.666667
std6.0553015.750362
min1.0000004.000000
25%4.5000007.500000
50%8.00000012.500000
75%13.75000015.250000
max16.00000019.000000
\n", "
" ], "text/plain": [ " quantitative1 quantitative2\n", "count 6.000000 6.000000\n", "mean 8.666667 11.666667\n", "std 6.055301 5.750362\n", "min 1.000000 4.000000\n", "25% 4.500000 7.500000\n", "50% 8.000000 12.500000\n", "75% 13.750000 15.250000\n", "max 16.000000 19.000000" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.describe()" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Let me briefly explain what these terms mean:\n", "\n", "1. Count\n", "2. Mean\n", "3. Std - standard deviation. How far is each data point from the mean on average. If you square this you get the variance.\n", "4. Min.\n", "5. 25% - lower quartile. If you ordered the data and removed the bottom 25%, what is the new min.\n", "6. 50% - Median.\n", "7. 75% - upper quartile. If you ordered the data and removed the bottom 75%, what is the new min.\n", "8. Max\n", "\n", "During our next lesson (when we make visualizations) we will get a better feel for these quantities and what they look like.\n", "\n", "For reference, here is the original data. Does it make sense that the mean of the quantitative1 is lower than the mean of quantitative2?" ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
qualitativequantitative1quantitative2
0red14
1blue46
2red612
3blue1013
4yellow1516
5red1619
\n", "
" ], "text/plain": [ " qualitative quantitative1 quantitative2\n", "0 red 1 4\n", "1 blue 4 6\n", "2 red 6 12\n", "3 blue 10 13\n", "4 yellow 15 16\n", "5 red 16 19" ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "And below here is how you would get the dependence summary statistic that we will be using:" ] }, { "cell_type": "code", "execution_count": 27, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
quantitative1quantitative2
quantitative11.0000000.961129
quantitative20.9611291.000000
\n", "
" ], "text/plain": [ " quantitative1 quantitative2\n", "quantitative1 1.000000 0.961129\n", "quantitative2 0.961129 1.000000" ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.corr()" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "The correlation tells us how related two factors are. It lies between -1 and 1. If it is close to 1, the two quantities are very related (when one is positive, the other is also positive), if 0 they seem unrelated and if -1 they are inversely related (when one is positive the other is negative). We will explore this more later in the course." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Wrapping up\n", "\n", "So now you know some stuff. \n", "\n", "* You know it is important to list out your assumptions. \n", "* You know that data comes in two flavors. \n", "* And you know how to describe that data using just a few numbers.\n", "\n", "Well who cares?\n", "\n" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Well once you get more of a feel for what these summary statistics mean (what data that exhibit these statistics looks like) it turns out that you can communicate a lot by just using them. So instead of walking through pages of data during a presentation, you can just go through a couple numbers. \n", "\n", "Then a couple of lectures down the line we will show that we can use them to infer larger truths! \n", "\n", "Because remember, if you only make the above assumption, **we have data**, you can't say too much. \n", "\n", "For example, if you collect a dataset of people's heights and you find the mean is 6 feet, you might be tempted to make assumptions about the height of people in general, **BUT** with the above assumption you can only say: The mean of my data is six feet. Or: the mean heights of the people I sampled is 6 feet. YOU CANNOT SAY ANYTHING IN GENERAL!!\n", "\n", "But don't worry, we will get there in time." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Learning objectives\n", "\n", "We will learn the basic lingo around data and specifically talk about two types of data: quantitative and qualitative.\n", "\n", "We then go over how to succinctly describe both types of data with summary statistics. \n", "\n", "While basic, this lesson will give us a common vocabulary that we will use for the rest of the class." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Comprehension Questions\n", "\n", "1.\tWhat is ‘good’ data? How would you collect data that is good?\n", "2.\tHow many numbers is a single data point?\n", "3.\tIs quantitative data more objective that qualitative data?\n", "4.\tWhat is an example of ordered qualitative data?\n", "5.\tDo highly correlated features cause each other?\n", "6.\tWhat are some common summary statistics that you might hear on the news? What do they communicate? What don’t they communicate?\n", "7.\tIf we have a dataset of all billionaires in the USA, can you say anything about billionaires globally?" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [] } ], "metadata": { "celltoolbar": "Slideshow", "kernelspec": { "display_name": "Python 2", "language": "python", "name": "python2" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 2 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython2", "version": "2.7.13" } }, "nbformat": 4, "nbformat_minor": 2 }