{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Welcome\n", "\n", "This is the Jupyter notebook for Chapter 1 of [*Elements of Data Science*](https://greenteapress.com/wp/elements-of-data-science), by Allen B. Downey.\n", "\n", "\n", "% TODO: Update these links\n", "\n", "If you are not familiar with Jupyter notebooks,\n", "[click here for a short introduction](https://colab.research.google.com/github/AllenDowney/ElementsOfDataScience/blob/master/jupyter_intro.ipynb).\n", "\n", "Then, if you are not already running this notebook on Colab, [click here to run this notebook on Colab](https://colab.research.google.com/github/AllenDowney/ElementsOfDataScience/blob/master/01_variables.ipynb)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Variables and Values" ] }, { "cell_type": "markdown", "metadata": { "tags": [] }, "source": [ "[Run this notebook on Colab](https://colab.research.google.com/github/AllenDowney/ElementsOfDataScience/blob/master/01_variables.ipynb) or \n", "[Download this notebook](https://github.com/AllenDowney/ElementsOfDataScience/raw/master/01_variables.ipynb)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Data science is the use of data to answer questions and guide decision making.\n", "For example, a topic of current debate is whether we should raise the minimum wage in the United States.\n", "Some economists think that raising the minimum wage would raise families out of poverty; others think it would cause more unemployment.\n", "But economic theory can only take us so far.\n", "At some point, we need data.\n", "\n", "A successful data science project requires three elements:\n", "\n", "* A question: For example, what is the relationship between the minimum wage and unemployment?\n", "\n", "* Data: To answer this question, the best data would be results from a well designed experiment. But if we can't get ideal data, we have to work with what we can get. \n", "\n", "* Methods: With the right data, simple methods are often enough to find answers and present them clearly. But sometimes we need more specialized tools.\n", "\n", "In an ideal world, we would pose a question, find data, and choose the appropriate methods, in that order.\n", "More often, the process is iterative. We might start with one question, get stuck, and pivot to a different question.\n", "Or we might explore a new dataset and discover the questions it can answer.\n", "Or we might start with a tool and look for problems it can solve.\n", "Most data science projects require flexibility and persistence." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The goal of this book is to give you the tools you need to execute a data science project from beginning to end, including these steps:\n", "\n", "* Finding questions, data, and methods that go together.\n", "\n", "* Cleaning and validating data.\n", "\n", "* Exploring datasets by visualizing distributions and relationships between variables.\n", "\n", "* Modeling data and generating predictions.\n", "\n", "* Designing data visualizations that tell a compelling story.\n", "\n", "* Communicating results effectively." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We'll start with basic programming concepts and work our way toward data science tools.\n", "\n", "I won't assume that you already know about programming, statistics, or data science. When I use a term, I try to define it immediately, and when I use a programming feature, I try to explain it clearly.\n", "\n", "This book is in the form of Jupyter notebooks. Jupyter is a software development tool you can run in a web browser, so you don't have to install any software. A Jupyter notebook is a document that contains text, Python code, and results. So you can read it like a book, but you can also modify the code, run it, develop new programs, and test them.\n", "\n", "The notebooks contain exercises where you can practice what you learn. I encourage you to do the exercises as you go along." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The topics in this chapter are:\n", "\n", "* Basic programming features in Python: variables and values.\n", "\n", "* Translating formulas from math notation to Python.\n", "\n", "You don't need a lot of math to do data science, but and the end of this chapter I'll review one topic that comes up a lot: logarithms." ] }, { "cell_type": "markdown", "metadata": { "tags": [] }, "source": [ "## Jupyter\n", "\n", "Jupyter is a software development environment, which means you can use it to write and run programs in Python and other programming languages.\n", "\n", "A Jupyter notebook is made up of cells, where each cell contains either text or code you can run." ] }, { "cell_type": "markdown", "metadata": { "tags": [] }, "source": [ "If you are running this notebook on Colab, you should see buttons in the top left that say \"+ Code\" and \"+ Text\". The first one adds a code cell and the second adds a text cell.\n", "\n", "If you want to try them out, select this cell by clicking on it, then press the \"+ Text\" button. A new cell should appear below this one.\n", "\n", "Add some text to the cell. You can use the buttons to format it, or you can mark up the text using [Markdown](https://www.markdownguide.org/basic-syntax/). When you are done, hold down \"Shift\" and press \"Enter\", which will format the text you just typed and then move to the next cell.\n", "\n", "If you select a Code cell, you should see a button on the left with a triangle inside a circle, which is the icon for \"Play\". If you press this button, Jupyter runs the code in the cell and displays the results.\n", "\n", "When you run code in a notebook for the first time, you might get a message warning you about the things a notebook can do. If you are running a notebook from a source you trust -- which I hope includes me -- you can press \"Run Anyway\".\n", "\n", "Instead of clicking the \"Play\" button, you can also run the code in a cell by holding down \"Shift\" and pressing \"Enter\"." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Numbers\n", "\n", "Python provides tools for working with many kinds of data, including numbers, words, dates, times, and locations (latitude and longitude).\n", "\n", "Let's start with numbers. Python can work with several types of numbers, but the two most common are:\n", "\n", "* `int`, which represents integer values like `3`, and\n", "\n", "* `float`, which represents numbers that have a fraction part, like `3.14159`.\n", "\n", "Most often, we use `int` to represent counts and `float` to represent measurements.\n", "Here's an example of an `int`:" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "3" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "When you run a cell that contains a value like this, Jupyter displays the value. Here's an example of a `float`:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "3.14159" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " `float` is short for \"floating-point\", which is the name for the way these numbers are stored." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Floating-point numbers can also be written in scientific notation, like this:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "1.2345e3" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `e` in `1.2345e3` stands for \"exponent\". This way of writing a number is equivalent to $1.2345 \\times 10^{3}$." ] }, { "cell_type": "markdown", "metadata": { "tags": [] }, "source": [ "If you are not familiar with scientific notation, you can read about it at . " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Arithmetic\n", "\n", "Python provides operators that perform arithmetic. The operators that perform addition and subtraction are `+` and `-`:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "3 + 2 - 1" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The operators that perform multiplication and division are `*` and `/`:" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "2 * 3" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "2 / 3" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And the operator for exponentiation is `**`:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "2 ** 3" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Unlike math notation, Python does not allow \"implicit multiplication\". For example, in math notation, if you write $3 (2 + 1)$, that's understood to be the same as $3 \\times (2+ 1)$.\n", "Python does not allow that notation." ] }, { "cell_type": "markdown", "metadata": { "tags": [] }, "source": [ "NOTE: The following cell uses `%%expect`, which is a Jupyter \"magic command\" that means we expect the code in this cell to produce an error. For more on this topic, see the\n", "[Jupyter notebook introduction](https://colab.research.google.com/github/AllenDowney/ThinkPython/blob/v3/chapters/jupyter_intro.ipynb)." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "%%expect TypeError\n", "\n", "3 (2 + 1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this example, the error message is not very helpful, which is why I am warning you now. \n", "\n", "If you want to multiply, you have to use the `*` operator." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The arithmetic operators follow the rules of precedence you might have learned as \"PEMDAS\":\n", "\n", "* Parentheses before\n", "* Exponentiation before\n", "* Multiplication and division before\n", "* Addition and subtraction.\n", "\n", "So in this expression:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "1 + 2 * 3" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The multiplication happens first. If that's not what you want, you can use parentheses to make the order of operations explicit:" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "(1 + 2) * 3" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Exercise:** Write a Python expression that raises `1+2` to the power `3*4`. The answer should be `531441`." ] }, { "cell_type": "markdown", "metadata": { "tags": [] }, "source": [ "Note: in the cell below, it should say\n", "\n", "```\n", "# Solution goes here\n", "```\n", "\n", "Lines like this that begin with `#` are \"comments\"; they provide information, but they have no effect when the program runs.\n", "\n", "When you do this exercise, you should delete the comment and replace it with your solution." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "# Solution goes here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Math Functions\n", "\n", "Python provides functions that compute all the usual mathematical functions, like `sin` and `cos`, `exp` and `log`.\n", "However, they are not part of Python itself; they are in a **library**, which is a collection of functions that supplement the Python language.\n", "\n", "Actually, there are several libraries that provide math functions; the one we'll use is called NumPy, which stands for \"Numerical Python\", and is pronounced \"num pie\".\n", "Before you can use a library, you have to **import** it. Here's how we import NumPy: " ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "import numpy as np" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It is conventional to import `numpy` as `np`, which means we can refer to it by the short name `np` rather than the longer name `numpy`.\n", "Names like this are case-sensitive, which means that `numpy` is not the same as `NumPy`. So even though the name of the library is NumPy, when we import it we have to call it `numpy`. " ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "%%expect ModuleNotFoundError\n", "\n", "import NumPy as np" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "But assuming we import `np` correctly, we can use it to read the value `pi`, which is an approximation of the mathematical constant $\\pi$." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "np.pi" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The result is a `float` with 16 digits. As you might know, we can't represent $\\pi$ with a finite number of digits, so this result is only approximate. \n", "\n", "NumPy provides `log`, which computes the natural logarithm" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "np.log(100)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "NumPy also provides `exp`, which raises the constant `e` to a power." ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "np.exp(1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Exercise:** Use these functions to confirm the mathematical identity $\\log(e^x) = x$, which should be true for any value of $x$.\n", "\n", "With floating-point values, this identity should work for values of $x$ between -700 and 700. What happens when you try it with larger and smaller values?" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [], "source": [ "# Solution goes here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As this example shows, floating-point numbers are finite approximations, which means they don't always behave like math.\n", "\n", "As another example, let's see what happens when you add up `0.1` three times:" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "0.1 + 0.1 + 0.1" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The result is close to `0.3`, but not exact.\n", "We'll see other examples of floating-point approximation later, and learn some ways to deal with it." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Variables\n", "\n", "A **variable** is a name that refers to a value.\n", "The following statement assigns the value `5` to a variable named `x`:" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "x = 5" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The variable we just created has the name `x` and the value `5`." ] }, { "cell_type": "markdown", "metadata": { "tags": [] }, "source": [ "If a variable name appears at the end of a cell, Jupyter displays its value." ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "tags": [] }, "outputs": [], "source": [ "x" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If we use `x` as part of an arithmetic operation, it represents the value `5`:" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [], "source": [ "x + 1" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [], "source": [ "x**2" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can also use `x` with `numpy` functions:" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [], "source": [ "np.exp(x)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Notice that the result from `exp` is a `float`, even though the value of `x` is an `int`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Exercise:** If you have not programmed before, one of the things you have to get used to is that programming languages are picky about details. Natural languages, like English, and semi-formal languages, like math notation, are more forgiving.\n", "\n", "As an example, in math notation, parentheses and square brackets mean the same thing, you can write\n", "\n", "$\\sin (\\omega t)$\n", "\n", "or\n", "\n", "$\\sin [\\omega t]$\n", "\n", "Either one is fine. And you can leave out the parentheses altogether, as long as the meaning is clear:\n", "\n", "$\\sin \\omega t$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In Python, every character counts. For example, the following are all different, and only the first one works:\n", "\n", "```\n", "np.exp(x)\n", "np.Exp(x)\n", "np.exp[x]\n", "np.exp x\n", "```\n", "\n", "While you are learning, I encourage you to make mistakes on purpose to see what goes wrong. Read the error messages carefully. Sometimes they are helpful and tell you exactly what's wrong. Other times they can be misleading. But if you have seen the message before, you might remember some likely causes." ] }, { "cell_type": "markdown", "metadata": { "tags": [] }, "source": [ "In the next cell, try out the different versions of `np.exp(x)` above, and see what error messages you get." ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "tags": [] }, "outputs": [], "source": [ "np.exp(x)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Exercise:** Search the NumPy documentation to find the function that computes square roots, and use it to compute a floating-point approximation of the golden ratio:\n", "\n", "$\\phi = \\frac{1 + \\sqrt{5}}{2}$ \n", "\n", "Hint: The result should be close to `1.618`." ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [], "source": [ "# Solution goes here" ] }, { "cell_type": "markdown", "metadata": { "tags": [] }, "source": [ "## Save your work\n", "\n", "If you are running on Colab and you want to save your work, now is a good time to press the \"Copy to Drive\" button (near the upper left), which saves a copy of this notebook in your Google Drive.\n", "\n", "If you want to change the name of the file, you can click on the name in the upper left.\n", "If you don't use Google Drive, look under the File menu to see other options.\n", "\n", "Once you make a copy, any additional changes you make will be saved automatically, so now you can continue without worrying about losing your work." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Calculating with Variables\n", "\n", "Now we'll use variables to solve a problem involving compound interest.\n", "It might not be the most exciting example, but it uses everything we have done so far, and it reviews exponentiation and logarithms, which we are going to need.\n", "\n", "If we start with a principal sum, $P$, and earn compounded interest, the total accumulated value, $V$, at the end of time $t$ is:\n", "\n", "$V=P\\left(1+{\\frac {r}{n}}\\right)^{nt}$\n", "\n", "where $r$ is the annual interest rate and $n$ is the compounding frequency.\n", "\n", "For example, if you deposit \\$2,100 in a bank paying an annual interest rate of 3.4\\% compounded quarterly, we can compute the balance after 7 years by defining these variables:" ] }, { "cell_type": "code", "execution_count": 40, "metadata": {}, "outputs": [], "source": [ "P = 2100\n", "r = 0.034\n", "n = 4\n", "t = 7" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And computing the total accumulated value like this." ] }, { "cell_type": "code", "execution_count": 41, "metadata": {}, "outputs": [], "source": [ "P * (1 + r/n)**(n*t)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Exercise:** Continuing the previous example, suppose you start with the same principle and the same interest rate, but interest is compounded twice per year, so `n = 2`.\n", "What would the total value be after 7 years? Hint: we expect the answer to be a bit less than the previous answer." ] }, { "cell_type": "code", "execution_count": 42, "metadata": {}, "outputs": [], "source": [ "# Solution goes here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Exercise:** If interest is compounded continuously, the value after time $t$ is given by the formula:\n", "\n", "$V=P~e^{rt}$\n", "\n", "Translate this equation into Python and use it compute the value of the investment in the previous example with continuous compounding. Hint: we expect the answer to be a bit more than the previous answers." ] }, { "cell_type": "code", "execution_count": 43, "metadata": {}, "outputs": [], "source": [ "# Solution goes here" ] }, { "cell_type": "markdown", "metadata": { "tags": [] }, "source": [ "**Exercise** Applying your algebra skills, solve the previous equation for $r$. Now use the formula you just derived to answer this question.\n", "\n", "\"Harvard's tuition in 1970 was \\$4,070 (not including room, board, and fees). \n", "\n", "\"In 2019 it was \\$46,340. What was the annual rate of increase over that period, treating it as if it had compounded continuously?\"" ] }, { "cell_type": "code", "execution_count": 45, "metadata": { "tags": [] }, "outputs": [], "source": [ "# Solution goes here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Summary\n", "\n", "This chapter introduces variables, which are names that refer to values, and two kinds of values, integers and floating-point numbers.\n", "\n", "It presents mathematical operators, like `+` for addition and `*` for multiplication, and mathematical functions like `log` for logarithms and `exp` for raising `e` to a power.\n", "\n", "In the next chapter, we'll see additional data types for representing letters and words, dates and times, and latitude and longitude." ] }, { "cell_type": "markdown", "metadata": { "collapsed": true, "tags": [] }, "source": [ "## A little more Jupyter\n", "\n", "Here are a few tips on using Jupyter to compute and display values.\n", "\n", "Generally, if there is a single expression in a cell, Jupyter computes the value of the expression and displays the result.\n", "For example, we've already seen how to display the value of `np.pi`:" ] }, { "cell_type": "code", "execution_count": 27, "metadata": { "tags": [] }, "outputs": [], "source": [ "np.pi" ] }, { "cell_type": "markdown", "metadata": { "tags": [] }, "source": [ "Here's a more complex example with functions, operators, and numbers:" ] }, { "cell_type": "code", "execution_count": 28, "metadata": { "tags": [] }, "outputs": [], "source": [ "1 / np.sqrt(2 * np.pi) * np.exp(-3**2 / 2)" ] }, { "cell_type": "markdown", "metadata": { "tags": [] }, "source": [ "If you put more than one expression in a cell, Jupyter computes them all, but it only display the result from the last:" ] }, { "cell_type": "code", "execution_count": 29, "metadata": { "tags": [] }, "outputs": [], "source": [ "1\n", "2 + 3\n", "np.exp(1)\n", "(1 + np.sqrt(5)) / 2" ] }, { "cell_type": "markdown", "metadata": { "tags": [] }, "source": [ "If you want to display more than one value, you can separate them with commas:" ] }, { "cell_type": "code", "execution_count": 30, "metadata": { "tags": [] }, "outputs": [], "source": [ "1, 2 + 3, np.exp(1), (1 + np.sqrt(5)) / 2" ] }, { "cell_type": "markdown", "metadata": { "tags": [] }, "source": [ "That result is actually a tuple, which you will learn about in the next chapter.\n", "\n", "Here's one last Jupyter tip: when you assign a value to variable, Jupyter does not display the value:" ] }, { "cell_type": "code", "execution_count": 31, "metadata": { "tags": [] }, "outputs": [], "source": [ "phi = (1 + np.sqrt(5)) / 2" ] }, { "cell_type": "markdown", "metadata": { "tags": [] }, "source": [ "So it is idiomatic to assign a value to a variable and immediately display the result:" ] }, { "cell_type": "code", "execution_count": 32, "metadata": { "tags": [] }, "outputs": [], "source": [ "phi = (1 + np.sqrt(5)) / 2\n", "phi" ] }, { "cell_type": "markdown", "metadata": { "tags": [] }, "source": [ "**Exercise:** Display the value of $\\phi$ and its inverse, $1/\\phi$, on a single line." ] }, { "cell_type": "code", "execution_count": 33, "metadata": { "tags": [] }, "outputs": [], "source": [ "# Solution goes here" ] }, { "cell_type": "markdown", "metadata": { "tags": [] }, "source": [ "*Elements of Data Science*\n", "\n", "Copyright 2021 [Allen B. Downey](https://allendowney.com)\n", "\n", "License: [Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International](https://creativecommons.org/licenses/by-nc-sa/4.0/)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "celltoolbar": "Tags", "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.13" } }, "nbformat": 4, "nbformat_minor": 2 }