{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Data Analysis with Jupyter Notebooks.\n", "\n", "# Tutorial 5\n", "\n", "Benjamin J. Morgan, University of Bath." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Contents\n", "\n", "- [Data analysis and statistics with numpy](#data_analysis)\n", "- [Linear regression](#linear_regression)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Data analysis and statistics with numpy\n", "\n", "`numpy` contains a lot of powerful functions for performing simple statistical analysis on our data. For example, consider the set of numbers 1 to 50:\n", "\n", ">```python\n", "import numpy as np\n", "a = np.arange(1,51)\n", "a\n", "```" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To find the minimum and maximum values we can use `np.min()` and `np.max()`\n", "\n", ">```python\n", "np.min(a)\n", "```" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ ">```python\n", "np.max(a)\n", "```" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To find the **sum** of all these numbers, we can use `np.sum()`\n", "\n", ">```python\n", "np.sum(a)\n", "```" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The **mean** of a set of numbers is defined as \n", "\n", "\\begin{equation}\n", "\\frac{\\sum_i^N x_i}{N}\n", "\\end{equation}\n", "\n", "which we could calculate with\n", "\n", ">```python\n", "np.sum(a) / len(a)\n", "# len(a) returns the length of the array `a`\n", "```" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "or with `np.mean()`\n", "\n", ">```python\n", "np.mean( a )\n", "```" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The **standard deviation**, $\\sigma$ quantifies how much the numbers in our set deviate from the mean.\n", "\n", "\\begin{equation}\n", "\\sigma = \\sqrt{\\frac{1}{N}\\sum_{i=1}^N(x_i-\\mu)^2}\n", "\\end{equation}\n", "\n", "where $\\mu$ is the mean.\n", "\n", "Again, we could write this out in code:\n", "\n", ">```python\n", "import math\n", "sigma = math.sqrt( np.sum( ( a - np.mean(a))**2 ) / len(a) )\n", "sigma\n", "```" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Or use the `np.std()` function\n", "\n", ">```python\n", "np.std(a)\n", "```" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Linear Regression\n", "\n", "Another commonly used data analysis technique is **linear regression**. This is used to calculate the relationship between two data sets, $X$ and $Y$, assuming that this relationship can be described by a straight line\n", "\n", "\\begin{equation}\n", "y_i = m x_i + c.\n", "\\end{equation}\n", "\n", "For any real data set, the data points are unlikely to all fall exactly on the same line. Linear regression is the process of calculating the line that “best fits” the given data." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "Look at the following snippet, and try to work out what the result will be.
\n", "np.random.rand(10) creates an array of 10 random numbers between 0 and 1.\n", "
\n", "\n", ">```python\n", "import matplotlib.pyplot as plt\n", "x = np.arange(1,11)\n", "offset = 2.0\n", "y = x + ( np.random.rand(10) - 0.5 ) * offset\n", "plt.plot( x, y, 'o' )\n", "plt.show()\n", "```" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can see this approximately gives the straight line relationship between $y=x$. We can use linear regression to calculate the “best” straight line that describes these data.\n", "\n", "There are a number of different ways in Python to calculate a line of best-fit. One of the simplest is to use another module, [`scipy.stats`](https://docs.scipy.org/doc/scipy-0.18.1/reference/stats.html). As you might suspect from the name, `scipy.stats` contains an large set of statistical analysis tools. We want [`linregress()`](https://docs.scipy.org/doc/scipy-0.18.1/reference/generated/scipy.stats.linregress.html#scipy.stats.linregress):\n", "\n", ">```python\n", "from scipy.stats import linregress\n", "linregress( x, y )\n", "```" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can see the output is complicated, but includes a list of values that includes the slope and the intercept. In fact you can treat the output like a [list](lists), and use indexing to select a specific result.\n", "\n", ">```python\n", "linregress( x, y )[0] # use indexing to get the slope\n", "```" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Another option is to collect all five of the output values at once\n", "\n", ">```python\n", "slope, intercept, rvalue, pvalue, stderr = linregress( x, y )\n", "print( \"slope =\", slope )\n", "print( \"intercept =\", intercept )\n", "```" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To plot the best-fit line against the original data, we generate a new data set according to $y=mx+c$, where $m$ and $c$ are set to the `slope` and `intercept`, calculated from `linregress`.\n", "\n", ">```python\n", "y_fit = slope * x + intercept \n", "plt.plot( x, y, 'o' )\n", "plt.plot( x, y_fit, '-' )\n", "plt.xlabel( 'x' )\n", "plt.ylabel( 'y' )\n", "plt.title( 'y =', slope, 'x +', intercept )\n", "plt.show()\n", "```" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.1" } }, "nbformat": 4, "nbformat_minor": 2 }