{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# A Python Tour of Data Science: Introduction\n", "\n", "[Michaël Defferrard](http://deff.ch), *PhD student*, [EPFL](http://epfl.ch) [LTS2](http://lts2.epfl.ch)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This short primer is an introduction to the scientific [Python](https://www.python.org) stack for [Data Science](https://en.wikipedia.org/wiki/Data_science). It is designed as a tour around the major Python packages used for the main computational tasks encountered in [the sexiest job of the 21st century](https://hbr.org/2012/10/data-scientist-the-sexiest-job-of-the-21st-century). At the end of this tour, you'll have a broad overview of the available libraries as well as why and how they are used for each task. This notebook aims at answering the following question: **which tool should I use for which task and how**. Before starting, two remarks:\n", "1. There exists better / faster ways to accomplish the presented computations. The goal is to present the packages and get a sense of which problems they solve.\n", "1. It is not meant to teach you (scientific) Python. I however tried to include the main constructions and idioms of the language and packages. A good ressource to learn scientific Python is a [set of lectures](https://github.com/jrjohansson/scientific-python-lectures) from J.R. Janson." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 1 Data Science\n", "\n", "\n", "\n", "This notebook will walk you through a typical Data Science **process**:\n", "1. **Data acquisition**\n", " 1. Importation\n", " 1. Cleaning\n", " 1. Exploration\n", "1. **Data exploitation**\n", " 1. Pre-processing\n", " 1. (Feature extraction)\n", " 1. Modeling\n", " 1. (Algorithm design)\n", " 1. Evaluation\n", "\n", "Our **motivating example**: predict whether a credit card client will default.\n", "* It is a binary classification task: client will default or not ($y=1$ if yes; $y=0$ if no).\n", "* We have data for 30'000 real clients from Taiwan.\n", "* There is 23 numerical & categorical explanatory variables:\n", " 1. $x_1$: amount of the given credit.\n", " 2. $x_2$: gender (1 = male; 2 = female).\n", " 3. $x_3$: education (1 = graduate school; 2 = university; 3 = high school; 4 = others).\n", " 4. $x_4$: marital status (1 = married; 2 = single; 3 = others).\n", " 5. $x_5$: age (year).\n", " 6. $x_6$ to $x_{11}$: history of past payment (monthly from September to April, 2005) (-1 = pay duly; 1 = payment delay for one month; ...; 9 = payment delay for nine months and above).\n", " 7. $x_{12}$ to $x_{17}$: amount of bill statement (monthly from September to April, 2005).\n", " 8. $x_{18}$ to $x_{23}$: amount of previous payment (monthly from September to April, 2005).\n", "* The data comes from the [UCI ML repository](https://archive.ics.uci.edu/ml/datasets/default+of+credit+card+clients)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 2 Python\n", "\n", "Before taking our tour, let's briefly talk about Python.\n", "First thing first, the general characteristics of the language:\n", "* **General purpose**: not built for a particular usage, it works as well for scientific computing as for web and application development. It features high-level data structures and supports multiple paradigms: procedural, object-oriented and functional.\n", "* **Elegant syntax**: easy-to-read and intuitive code, easy-to-learn minimalistic syntax, quick to write (low boilerplate / verbosity), maintainability scales well with size of projects.\n", "* **Expressive language**: fewer lines of code, fewer bugs, easier to maintain.\n", "\n", "Technical details:\n", "* **Dynamically typed**: no need to define the type of variables, function arguments or return types. Everything is an object and can be modified at runtime.\n", "* **Automatic memory management** (garbage collector): no need to explicitly allocate and deallocate memory for variables and data arrays. No memory leak bugs.\n", "* **Interpreted** (JIT is coming): No need to compile the code. The Python interpreter reads and executes the python code directly. It also means that a single Python source runs anywhere a runtime is available, like on Windows, Mac, Linux and in the Cloud.\n", "\n", "From those characteristics emerge the following advantages:\n", "* The main advantage is ease of programming, minimizing the time required to develop, debug and maintain the code.\n", "* The well designed language encourages many good programming practices:\n", " * Modular and object-oriented programming, good system for packaging and re-use of code. This often results in more transparent, maintainable and bug-free code.\n", " * Documentation tightly integrated with the code.\n", "* A large community geared toward open-source, an extensive standard library and a large collection of add-on packages and development tools.\n", "\n", "And the following disadvantages:\n", "* There is two versions of Python in general use: 2 and 3. While Python 3 is around since 2008, there are still libraries which only support Python 2. While you should generally go for Python 3, a specific library or legacy code can hold you on Python 2.\n", "* Due to its interpreted and dynamic nature, the execution of Python code can be slow compared to compiled statically typed programming languages, such as C and Fortran. That is however almost solved, see the available solutions at the end of this notebook.\n", "* There is no compiler to catch your errors. Solutions include unit / integration tests or the use of a [linter](https://en.wikipedia.org/wiki/Lint_%28software%29) such as [pyflakes](https://pypi.python.org/pypi/pyflakes), [Pylint](https://www.pylint.org) or [PyChecker](http://pychecker.sourceforge.net). [Flake8](https://pypi.python.org/pypi/flake8) combines static analysis with style checking." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 3 Why Python for Data Science\n", "\n", "Let's state [why is Python a language of choice for Data Scientists](https://www.quora.com/Why-is-Python-a-language-of-choice-for-data-scientists). Viable alternatives include [matlab](http://ch.mathworks.com/products/matlab), [R](https://www.r-project.org) and [Julia](http://julialang.org), and, for more statistical jobs, the SAS and SPSS statistical packages. The strenghs of Python are:\n", "* Minimal development time.\n", " * Rapid prototyping for data exploration.\n", " * Same language and framework for R&D and production.\n", "* A strong position in scientific computing.\n", " * Large community of users, easy to find help and documentation.\n", " * Extensive ecosystem of open-source scientific libraries and environments.\n", "* Easy integration.\n", " * Many libraries to access data from files, databases or web scraping.\n", " * Many wrappers to legacy code, e.g. C, Fortran or Matlab.\n", "* Available and suitable for High Performance Computing (HPC)\n", " * Close integration with time-tested and highly optimized libraries for fast numerical mathematics like BLAS, LAPACK, ATLAS, OpenBLAS, ARPACK, MKL, etc.\n", " * JIT and AOT compilers.\n", " * Good support for parallel processing with processes and threads, interprocess communication (MPI) and GPU computing (OpenCL and CUDA)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 4 Why Jupyter \n", "\n", "[Jupyter](http://jupyter.org) notebook is an HTML-based notebook which allows you to create and share documents that contain live code, equations, visualizations and explanatory text. It allows a clean presentation of computational results as HTML or PDF reports and is well suited for interactive tasks surch as data cleaning, transformation and exploration, numerical simulation, statistical modeling, machine learning and more. It runs everywhere (Window, Mac, Linux, Cloud) and supports multiple languages through various kernels, e.g. [Python](https://ipython.org), [R](https://irkernel.github.io), [Julia](https://github.com/JuliaLang/IJulia.jl), [Matlab](https://github.com/Calysto/matlab_kernel).\n", "\n", "While Jupyter is itself becoming an Integreted Development Environment (IDE), alternative scientific IDEs include [Spyder](https://pythonhosted.org/spyder) and [Rodeo](http://rodeo.yhat.com). Non-scientific IDEs include [IDLE](https://docs.python.org/3/library/idle.html) and [PyCharm](https://www.jetbrains.com/pycharm). Vim and Emacs lovers (or more recently Atom and Sublime Text) will find full support of Python in their editor of choice. An interactive prompt, useful for experimentations or as a calculator, is offered by Python itself or by [IPython](https://ipython.org), the Jupyter kernel for Python." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 5 Installation\n", "\n", "During this tour, we'll need the packages shown below, which are best installed from [PyPI](https://pypi.python.org) in a [virtual environment](https://docs.python.org/3/library/venv.html). Please see the instructions on the [README](../README.html)." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "numpy\n", "scipy\n", "matplotlib\n", "scikit-learn\n", "\n", "requests\n", "facebook-sdk\n", "tweepy\n", "\n", "pandas\n", "xlrd\n", "xlwt\n", "tables\n", "sqlalchemy\n", "\n", "statsmodels\n", "sympy\n", "autograd\n", "bokeh\n", "numba\n", "Cython\n", "\n", "keras\n", "theano\n", "#tensorflow\n", "\n", "jupyter\n", "ipython\n", "\n", "grip\n" ] } ], "source": [ "%%script sh\n", "cat ../requirements.txt" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# Windows\n", "# !type ..\\requirements.txt" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The statements starting with `%` or `%%` are [built-in magic commands](http://ipython.readthedocs.io/en/stable/interactive/magics.html), i.e. commands interpreted by the IPython kernel. E.g. `%%script sh` tells IPython to run the cell with the shell `sh` (like the `#!` line at the beginning of script)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 6 Environment\n", "\n", "## 6.1 Python\n", "\n", "The Python prompt is what you get when typing `python` in your terminal. It is useful to test commands and check your installation. We however prefer IPython for interactive work, see below.\n", "\n", "Python files, with the extension `.py`, are either scripts or modules. A **Python script** is a file which gets executed with either `python myscript.py` or `./myscript.py` if it has execution permissions as well as a shabang (#!) indicating which interpreter should be used. Below is an example of a typical script. The `!` in front of a command tells IPython to execute the command with the system terminal." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "#!/bin/env python3\r\n", "\r\n", "print('A Network Tour of Data Science: Python installation test')\r\n", "\r\n", "import os\r\n", "import sys\r\n", "major, minor = sys.version_info.major, sys.version_info.minor\r\n", "\r\n", "if major is not 3:\r\n", " raise Exception('please use Python 3, you have Python {}.'.format(major))\r\n", "\r\n", "try:\r\n", " import numpy\r\n", " import scipy\r\n", " import matplotlib\r\n", " import sklearn\r\n", "\r\n", " import requests\r\n", " import facebook\r\n", " import tweepy\r\n", "\r\n", " import pandas\r\n", " import xlrd\r\n", " import xlwt\r\n", " import tables\r\n", " import sqlalchemy\r\n", "\r\n", " import statsmodels\r\n", " import sympy\r\n", " import autograd\r\n", " import bokeh\r\n", " import numba\r\n", " import Cython\r\n", "\r\n", " os.environ['KERAS_BACKEND'] = 'theano' # Easier for Windows users.\r\n", " import keras\r\n", " import theano\r\n", " #import tensorflow\r\n", "\r\n", " import jupyter\r\n", " import IPython, ipykernel\r\n", "\r\n", "except:\r\n", " print('Your installation misses a package.')\r\n", " print('Please look for the package name below and install it with your '\r\n", " 'package manager (conda, brew, apt-get, yum, pacman, etc.) or pip.')\r\n", " raise\r\n", "\r\n", "print('You did successfully install Python {}.{} and '\r\n", " 'most of the Python packages we will use.'.format(major, minor))\r\n", "A Network Tour of Data Science: Python installation test\r\n", "Using Theano backend.\r\n", "You did successfully install Python 3.5 and most of the Python packages we will use.\r\n", "A Network Tour of Data Science: Python installation test\r\n", "Using Theano backend.\r\n", "You did successfully install Python 3.5 and most of the Python packages we will use.\r\n" ] } ], "source": [ "!cat ../check_install.py\n", "!python ../check_install.py\n", "!../check_install.py\n", "\n", "# Windows\n", "# !type ..\\check_install.py\n", "# !python ..\\check_install.py" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A **Python module** is similar to a script, except that it is suposed to be *imported* and used by another script or module. It defines objects like classes or functions which are meant to be exported. Below is an example of a typical module, composed of only one function, `get_data()`. Note that the module itself imports other modules (`pandas`, `urllib` and `os.path`)." ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "cat: ../utils.py: No such file or directory\r\n" ] } ], "source": [ "!cat ../utils.py\n", "# Windows: !type ..\\utils.py" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 6.2 IPython\n", "\n", "The [IPython](https://ipython.org) prompt is that is what you get when running `ipython` in your terminal. It is more convenient than the Python prompt and is useful for interactive work like small experiments or as a **powerful calculator**.\n", "\n", "### 6.3 Jupyter\n", "\n", "The [Jupyter](http://jupyter.org) notebook is the web interface you get when running `jupyter notebook`. It features a file explorer, various kernels (for Python, R, Julia) and can export any notebook to HTML / PDF (via `jupyter nbconvert`). The basic document is a notebook which is composed of *cells* who are either code, results or markdown text / math. The Jupyter notebook is the interface we'll use for most of the course.\n", "\n", "Markdown is a lightweight markup language which is very much used to generate HTML documents (e.g. on GitHub or with static website generators). See this [cheatsheet](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet) as a very short introduction. Or simply edit the cells in this notebook. Markdown can include Latex math such as $y = 2x$.\n", "\n", "### 6.4 Installing and managing packages\n", "\n", "As explained in the [README](../README.html), we prefer to work inside virtual environments. Installing packages, a collection of modules, inside or outside virtual environments is however the same.\n", "\n", "Most of the packages, i.e. reusable pieces of code, are posted on [PyPI](https://pypi.python.org), the Python Package Index, by their authors. The Python package manager, `pip`, is a command-line tool to search and download packages from PyPI.\n", "\n", "> Note that some packages, like NumPy, requires native, i.e. compiled, dependencies. That is why installing with `pip install` may fail, as it only manages Python packages. In that case you need to install those dependencies by hand or with the help of a package manager like `brew` for Mac or whatever your Linux distribution uses.\n", "\n", "Searching for a package goes like this (can be typed in your terminal):" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "OutputLilyPond (1.0.0) - Produce a LilyPond file from a music21 Score.\r\n", "music21 (3.1.0) - A Toolkit for Computer-Aided Musical Analysis and\r\n", " Manipulation.\r\n", "contourviz (0.2.4) - A package that charts musical contours into a web-\r\n", " based interactive using music21 and D3.js.\r\n", "music22 (0.0.2.post1) - A tool for musicological analysis from audio files.\r\n", " For a symbolic analysis, you can use Music21\r\n", " (http://web.mit.edu/music21/). Now it is focused on\r\n", " modal music analysis : Scale analysis, tonic\r\n", " detection\r\n" ] } ], "source": [ "!pip search music21" ] }, { "cell_type": "raw", "metadata": {}, "source": [ "Installing a package goes like this. Note that it warns you if the package is installed already." ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied (use --upgrade to upgrade): numpy in /usr/lib/python3.5/site-packages\r\n" ] } ], "source": [ "!pip install numpy" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can get the list of installed packages with `pip freeze`. These are all the packages that are installed and available on your system. They could have been installed by `pip install packname` (maybe as a dependancy), by `conda install packname` or by your system's package manager." ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "appdirs==1.4.0\r\n", "autograd==1.1.6\r\n", "blinker==1.4\r\n", "bokeh==0.12.3\r\n", "Bottleneck==1.1.0\r\n", "click==6.6\r\n", "cycler==0.10.0\r\n", "Cython==0.24.1\r\n", "decorator==4.0.10\r\n", "docopt==0.6.2\r\n", "docutils==0.12\r\n", "entrypoints==0.2.2\r\n", "facebook-sdk==2.0.0\r\n", "feedgenerator==1.9\r\n", "flake8==3.0.4\r\n", "Flask==0.11.1\r\n", "future==0.15.2\r\n", "gmpy2==2.0.8\r\n", "grip==4.3.2\r\n", "h5py==2.6.0\r\n", "ipykernel==4.5.0\r\n", "ipython==5.1.0\r\n", "ipython-genutils==0.1.0\r\n", "ipywidgets==5.2.2\r\n", "isc==2.0\r\n", "itsdangerous==0.24\r\n", "Jinja2==2.8\r\n", "jsonschema==2.5.1\r\n", "jupyter==1.0.0\r\n", "jupyter-client==4.4.0\r\n", "jupyter-console==5.0.0\r\n", "jupyter-core==4.2.0\r\n", "Keras==1.1.0\r\n", "llvmlite==0.13.0\r\n", "louis==3.0.0\r\n", "Markdown==2.6.6\r\n", "MarkupSafe==0.23\r\n", "matplotlib==1.5.2\r\n", "mccabe==0.5.2\r\n", "mistune==0.7.3\r\n", "mpmath==0.19\r\n", "nbconvert==4.2.0\r\n", "nbformat==4.1.0\r\n", "nose==1.3.7\r\n", "notebook==4.2.3\r\n", "numba==0.28.1\r\n", "numexpr==2.6.1\r\n", "numpy==1.11.2\r\n", "oauthlib==2.0.0\r\n", "packaging==16.7\r\n", "pandas==0.19.0\r\n", "path-and-address==2.0.1\r\n", "path.py==8.2.1\r\n", "patsy==0.4.1\r\n", "pelican==3.6.3\r\n", "pexpect==4.2.1\r\n", "pickleshare==0.7.4\r\n", "prompt-toolkit==1.0.7\r\n", "ptyprocess==0.5.1\r\n", "pycodestyle==2.0.0\r\n", "pyflakes==1.2.3\r\n", "Pygments==2.1.3\r\n", "pygobject==3.20.1\r\n", "pyparsing==2.1.10\r\n", "python-dateutil==2.5.3\r\n", "pytz==2016.6.1\r\n", "pyudev==0.19.0\r\n", "PyYAML==3.12\r\n", "pyzmq==16.0.0\r\n", "qtconsole==4.2.1\r\n", "ranger==1.7.2\r\n", "requests==2.11.1\r\n", "requests-oauthlib==0.7.0\r\n", "scikit-learn==0.18\r\n", "scipy==0.18.1\r\n", "simplegeneric==0.8.1\r\n", "six==1.10.0\r\n", "solaar==0.9.2\r\n", "SQLAlchemy==1.1.0\r\n", "statsmodels==0.6.1\r\n", "sympy==1.0\r\n", "tables==3.3.0\r\n", "terminado==0.6\r\n", "Theano==0.8.2\r\n", "tornado==4.4.2\r\n", "traitlets==4.3.1\r\n", "tweepy==3.5.0\r\n", "Unidecode==0.4.19\r\n", "wcwidth==0.1.7\r\n", "Werkzeug==0.11.11\r\n", "widgetsnbextension==1.2.6\r\n", "xlrd==1.0.0\r\n", "xlwt==1.1.2\r\n" ] } ], "source": [ "!pip freeze" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 6.5 Code versioning\n", "\n", "While mastering git is not a necessity to follow the exercises, it is a good practice to version the code you write and it will definitely be useful to you in the future.\n", "\n", "The commands you need for the exercises is\n", "```sh\n", "git clone https://github.com/mdeff/ntds_2016.git\n", "```\n", "the first time to copy the repository on your computer. Once you have it, you can simply download the updates every Monday morning with\n", "```sh\n", "git pull\n", "```\n", "\n", "Other commands of interest if you want to maintain your own repository are `add`, `commit` and `push`. The basic workfow is\n", "```sh\n", "git clone url\n", "# make your changes\n", "git commit -a -m \"my first commit\"\n", "git push\n", "```" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.5.2" } }, "nbformat": 4, "nbformat_minor": 0 }