![](\"https://upload.wikimedia.org/wikipedia/commons/b/ba/Data_visualization_process_v1.png\")
\n",
"\n",
"This notebook will walk you through a typical Data Science **process**:\n",
"1. **Data acquisition**\n",
" 1. Importation\n",
" 1. Cleaning\n",
" 1. Exploration\n",
"1. **Data exploitation**\n",
" 1. Pre-processing\n",
" 1. (Feature extraction)\n",
" 1. Modeling\n",
" 1. (Algorithm design)\n",
" 1. Evaluation\n",
"\n",
"Our **motivating example**: predict whether a credit card client will default.\n",
"* It is a binary classification task: client will default or not ($y=1$ if yes; $y=0$ if no).\n",
"* We have data for 30'000 real clients from Taiwan.\n",
"* There is 23 numerical & categorical explanatory variables:\n",
" 1. $x_1$: amount of the given credit.\n",
" 2. $x_2$: gender (1 = male; 2 = female).\n",
" 3. $x_3$: education (1 = graduate school; 2 = university; 3 = high school; 4 = others).\n",
" 4. $x_4$: marital status (1 = married; 2 = single; 3 = others).\n",
" 5. $x_5$: age (year).\n",
" 6. $x_6$ to $x_{11}$: history of past payment (monthly from September to April, 2005) (-1 = pay duly; 1 = payment delay for one month; ...; 9 = payment delay for nine months and above).\n",
" 7. $x_{12}$ to $x_{17}$: amount of bill statement (monthly from September to April, 2005).\n",
" 8. $x_{18}$ to $x_{23}$: amount of previous payment (monthly from September to April, 2005).\n",
"* The data comes from the [UCI ML repository](https://archive.ics.uci.edu/ml/datasets/default+of+credit+card+clients)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"During this tour, we'll need the packages shown below, which are best installed from [PyPI](https://pypi.python.org) in a [virtual environment](https://docs.python.org/3/library/venv.html):\n",
"```\n",
"$ pyvenv /path/to/new/virtual/env\n",
"$ . /path/to/new/virtual/env/bin/activate\n",
"$ pip install -r requirements.txt\n",
"$ jupyter-notebook\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"numpy\n",
"scipy\n",
"matplotlib\n",
"scikit-learn\n",
"\n",
"pandas\n",
"xlrd\n",
"xlwt\n",
"tables\n",
"sqlalchemy\n",
"\n",
"statsmodels\n",
"sympy\n",
"autograd\n",
"bokeh\n",
"numba\n",
"Cython\n",
"\n",
"keras\n",
"#theano\n",
"tensorflow\n",
"\n",
"jupyter\n",
"ipython\n"
]
}
],
"source": [
"%%script sh\n",
"cat requirements.txt"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The statements starting with `%` or `%%` are [built-in magic commands](http://ipython.readthedocs.io/en/stable/interactive/magics.html), i.e. commands interpreted by the IPython kernel. E.g. `%%script sh` tells IPython to run the cell with `sh` (like the `#!` line at the beginning of script)."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"# While packages are usually imported at the top, they can\n",
"# be imported wherever you prefer, in whatever scope.\n",
"import numpy as np"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 0 Python\n",
"\n",
"Before taking our tour, let's briefly talk about Python.\n",
"First thing first, the general characteristics of the language:\n",
"* **General purpose**: not built for a particular usage, it works as well for scientific computing as for web and application development. It features high-level data structures and supports multiple paradigms: procedural, object-oriented and functional.\n",
"* **Elegant syntax**: easy-to-read and intuitive code, easy-to-learn minimalistic syntax, quick to write (low boilerplate / verbosity), maintainability scales well with size of projects.\n",
"* **Expressive language**: fewer lines of code, fewer bugs, easier to maintain.\n",
"\n",
"Technical details:\n",
"* **Dynamically typed**: no need to define the type of variables, function arguments or return types. Everything is an object and can be modified at runtime.\n",
"* **Automatic memory management** (garbage collector): no need to explicitly allocate and deallocate memory for variables and data arrays. No memory leak bugs.\n",
"* **Interpreted** (JIT is coming): No need to compile the code. The Python interpreter reads and executes the python code directly. It also means that a single Python source runs anywhere a runtime is available, like on Windows, Mac, Linux and in the Cloud.\n",
"\n",
"From those characteristics emerge the following advantages:\n",
"* The main advantage is ease of programming, minimizing the time required to develop, debug and maintain the code.\n",
"* The well designed language encourages many good programming practices:\n",
" * Modular and object-oriented programming, good system for packaging and re-use of code. This often results in more transparent, maintainable and bug-free code.\n",
" * Documentation tightly integrated with the code.\n",
"* A large community geared toward open-source, an extensive standard library and a large collection of add-on packages and development tools.\n",
"\n",
"And the following disadvantages:\n",
"* There is two versions of Python in general use: 2 and 3. While Python 3 is around since 2008, there are still libraries which only support Python 2. While you should generally go for Python 3, a specific library or legacy code can hold you on Python 2.\n",
"* Due to its interpreted and dynamic nature, the execution of Python code can be slow compared to compiled statically typed programming languages, such as C and Fortran. That is however almost solved, see the available solutions at the end of this notebook.\n",
"* There is no compiler to catch your errors. Solutions include unit / integration tests or the use of a [linter](https://en.wikipedia.org/wiki/Lint_%28software%29) such as [pyflakes](https://pypi.python.org/pypi/pyflakes), [Pylint](https://www.pylint.org) or [PyChecker](http://pychecker.sourceforge.net). [Flake8](https://pypi.python.org/pypi/flake8) combines static analysis with style checking.\n",
"\n",
"### 0.1 Why Python for Data Science\n",
"\n",
"Let's state [why is Python a language of choice for Data Scientists](https://www.quora.com/Why-is-Python-a-language-of-choice-for-data-scientists). Viable alternatives include [matlab](http://ch.mathworks.com/products/matlab), [R](https://www.r-project.org) and [Julia](http://julialang.org), and, for more statistical jobs, the SAS and SPSS statistical packages. The strenghs of Python are:\n",
"* Minimal development time.\n",
" * Rapid prototyping for data exploration.\n",
" * Same language and framework for R&D and production.\n",
"* A strong position in scientific computing.\n",
" * Large community of users, easy to find help and documentation.\n",
" * Extensive ecosystem of open-source scientific libraries and environments.\n",
"* Easy integration.\n",
" * Many libraries to access data from files, databases or web scraping.\n",
" * Many wrappers to legacy code, e.g. C, Fortran or Matlab.\n",
"* Available and suitable for High Performance Computing (HPC)\n",
" * Close integration with time-tested and highly optimized libraries for fast numerical mathematics like BLAS, LAPACK, ATLAS, OpenBLAS, ARPACK, MKL, etc.\n",
" * JIT and AOT compilers.\n",
" * Good support for parallel processing with processes and threads, interprocess communication (MPI) and GPU computing (OpenCL and CUDA).\n",
"\n",
"### 0.2 Why Jupyter \n",
"\n",
"[Jupyter](http://jupyter.org) notebook is an HTML-based notebook which allows you to create and share documents that contain live code, equations, visualizations and explanatory text. It allows a clean presentation of computational results as HTML or PDF reports and is well suited for interactive tasks surch as data cleaning, transformation and exploration, numerical simulation, statistical modeling, machine learning and more. It runs everywhere (Window, Mac, Linux, Cloud) and supports multiple languages through various kernels, e.g. [Python](https://ipython.org), [R](https://irkernel.github.io), [Julia](https://github.com/JuliaLang/IJulia.jl), [Matlab](https://github.com/Calysto/matlab_kernel).\n",
"\n",
"While Jupyter is itself becoming an Integreted Development Environment (IDE), alternative scientific IDEs include [Spyder](https://pythonhosted.org/spyder) and [Rodeo](http://rodeo.yhat.com). Non-scientific IDEs include [IDLE](https://docs.python.org/3/library/idle.html) and [PyCharm](https://www.jetbrains.com/pycharm). Vim and Emacs lovers (or more recently Atom and Sublime Text) will find full support of Python in their editor of choice. An interactive prompt, useful for experimentations or as a calculator, is offered by Python itself or by [IPython](https://ipython.org), the Jupyter kernel for Python."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1 Data Importation\n",
"\n",
"* The world is messy, we got data in CSV, [JSON](http://www.json.org), Excel, [HDF5](https://www.hdfgroup.org/HDF5) files and an SQL database.\n",
"* Could also have been matlab, HTML, XML files or from the web via scraping and APIs (e.g. [Twitter Firhose](https://dev.twitter.com/streaming/firehose)) or noSQL data stores, etc."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"# Download the data.\n",
"import utils\n",
"utils.get_data('data/')"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"bills.hdf5 demographics.csv payments.sqlite\r\n",
"delays.xls original.xls target.json\r\n"
]
}
],
"source": [
"!ls data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 1.1 Importing from an SQL Database\n",
"\n",
"[SQLAlchemy](http://www.sqlalchemy.org/) to the rescue.\n",
"* Abstraction between DBAPIs.\n",
" * Supported databases: SQLite, Postgresql, MySQL, Oracle, MS-SQL, Firebird, Sybase and others.\n",
"* [SQL Expression Language](http://docs.sqlalchemy.org/en/rel_1_0/core/tutorial.html).\n",
"* [Object Relational Mapper (ORM)](http://docs.sqlalchemy.org/en/rel_1_0/orm/tutorial.html)."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2017-12-03 18:24:19,043 INFO sqlalchemy.engine.base.Engine SELECT payments.\"ID\", payments.\"PAY1\", payments.\"PAY2\", payments.\"PAY3\", payments.\"PAY4\", payments.\"PAY5\", payments.\"PAY6\" \n",
"FROM payments\n",
"2017-12-03 18:24:19,043 INFO sqlalchemy.engine.base.Engine ()\n"
]
}
],
"source": [
"import sqlalchemy\n",
"engine = sqlalchemy.create_engine('sqlite:///data/payments.sqlite', echo=False)\n",
"\n",
"# Infer from existing DB.\n",
"metadata = sqlalchemy.MetaData()\n",
"metadata.reflect(engine)\n",
"\n",
"# An SQL SELECT statement.\n",
"table = metadata.tables.get('payments')\n",
"op = sqlalchemy.sql.select([table])\n",
"engine.echo = True\n",
"result = engine.execute(op)\n",
"engine.echo = False"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"ID: 1, payments: (0, 689, 0, 0, 0, 0)\n",
"ID: 2, payments: (0, 1000, 1000, 1000, 0, 2000)\n",
"ID: 3, payments: (1518, 1500, 1000, 1000, 1000, 5000)\n",
"ID: 4, payments: (2000, 2019, 1200, 1100, 1069, 1000)\n",
"ID: 5, payments: (2000, 36681, 10000, 9000, 689, 679)\n",
"ID: 6, payments: (2500, 1815, 657, 1000, 1000, 800)\n",
"ID: 7, payments: (55000, 40000, 38000, 20239, 13750, 13770)\n",
"ID: 8, payments: (380, 601, 0, 581, 1687, 1542)\n",
"ID: 9, payments: (3329, 0, 432, 1000, 1000, 1000)\n",
"ID: 10, payments: (0, 0, 0, 13007, 1122, 0)\n"
]
}
],
"source": [
"# Show some lines, i.e. clients.\n",
"for row in result.fetchmany(size=10):\n",
" print('ID: {:2d}, payments: {}'.format(row[0], row[1:]))\n",
"result.close()"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1299 clients paid 1000 in April 2005\n"
]
}
],
"source": [
"# Execute some raw SQL.\n",
"paid = 1000\n",
"op = sqlalchemy.sql.text('SELECT payments.\"ID\", payments.\"PAY6\" FROM payments WHERE payments.\"PAY6\" = {}'.format(paid))\n",
"result = engine.execute(op).fetchall()\n",
"print('{} clients paid {} in April 2005'.format(len(result), paid))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 1.2 Merging data Sources\n",
"\n",
"Put some [pandas](http://pandas.pydata.org/) in our Python !\n",
"* Import / export data from / to various sources.\n",
"* Data frames manipulations: slicing, dicing, grouping.\n",
"* And many more !"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"def get_data(directory):\n",
" demographics = pd.read_csv(directory + 'demographics.csv', index_col=0)\n",
" delays = pd.read_excel(directory + 'delays.xls', index_col=0)\n",
" bills = pd.read_hdf(directory + 'bills.hdf5', 'bills')\n",
" payments = pd.read_sql('payments', engine, index_col='ID')\n",
" target = pd.read_json(directory + 'target.json')\n",
" return pd.concat([demographics, delays, bills, payments, target], axis=1)\n",
"\n",
"import pandas as pd\n",
"data = get_data('data/')\n",
"attributes = data.columns.tolist()\n",
"\n",
"# Tansform from numerical to categorical variable.\n",
"data['SEX'] = data['SEX'].astype('category')\n",
"data['SEX'].cat.categories = ['MALE', 'FEMALE']\n",
"data['MARRIAGE'] = data['MARRIAGE'].astype('category')\n",
"data['MARRIAGE'].cat.categories = ['UNK', 'MARRIED', 'SINGLE', 'OTHERS']\n",
"data['EDUCATION'] = data['EDUCATION'].astype('category')\n",
"data['EDUCATION'].cat.categories = ['UNK', 'GRAD SCHOOL', 'UNIVERSITY', 'HIGH SCHOOL', 'OTHERS', 'UNK1', 'UNK2']"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 1.3 Looking at the Data"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"| \n", " | LIMIT | \n", "SEX | \n", "EDUCATION | \n", "MARRIAGE | \n", "AGE | \n", "DEFAULT | \n", "
|---|---|---|---|---|---|---|
| 1 | \n", "20000 | \n", "FEMALE | \n", "UNIVERSITY | \n", "MARRIED | \n", "24 | \n", "1 | \n", "
| 2 | \n", "120000 | \n", "FEMALE | \n", "UNIVERSITY | \n", "SINGLE | \n", "26 | \n", "1 | \n", "
| 3 | \n", "90000 | \n", "FEMALE | \n", "UNIVERSITY | \n", "SINGLE | \n", "34 | \n", "0 | \n", "
| 4 | \n", "50000 | \n", "FEMALE | \n", "UNIVERSITY | \n", "MARRIED | \n", "37 | \n", "0 | \n", "
| 5 | \n", "50000 | \n", "MALE | \n", "UNIVERSITY | \n", "MARRIED | \n", "57 | \n", "0 | \n", "
| 6 | \n", "50000 | \n", "MALE | \n", "GRAD SCHOOL | \n", "SINGLE | \n", "37 | \n", "0 | \n", "
| \n", " | AGE | \n", "DELAY1 | \n", "DELAY2 | \n", "DELAY3 | \n", "DELAY4 | \n", "DELAY5 | \n", "
|---|---|---|---|---|---|---|
| 1 | \n", "24 | \n", "2 | \n", "2 | \n", "-1 | \n", "-1 | \n", "-2 | \n", "
| 2 | \n", "26 | \n", "-1 | \n", "2 | \n", "0 | \n", "0 | \n", "0 | \n", "
| 3 | \n", "34 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "
| 4 | \n", "37 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "
| 5 | \n", "57 | \n", "-1 | \n", "0 | \n", "-1 | \n", "0 | \n", "0 | \n", "
| \n", " | BILL1 | \n", "BILL2 | \n", "BILL3 | \n", "BILL4 | \n", "BILL5 | \n", "BILL6 | \n", "PAY1 | \n", "PAY2 | \n", "PAY3 | \n", "PAY4 | \n", "PAY5 | \n", "PAY6 | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | \n", "3913 | \n", "3102 | \n", "689 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "689 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "
| 2 | \n", "2682 | \n", "1725 | \n", "2682 | \n", "3272 | \n", "3455 | \n", "3261 | \n", "0 | \n", "1000 | \n", "1000 | \n", "1000 | \n", "0 | \n", "2000 | \n", "
| 3 | \n", "29239 | \n", "14027 | \n", "13559 | \n", "14331 | \n", "14948 | \n", "15549 | \n", "1518 | \n", "1500 | \n", "1000 | \n", "1000 | \n", "1000 | \n", "5000 | \n", "
| 4 | \n", "46990 | \n", "48233 | \n", "49291 | \n", "28314 | \n", "28959 | \n", "29547 | \n", "2000 | \n", "2019 | \n", "1200 | \n", "1100 | \n", "1069 | \n", "1000 | \n", "
| 5 | \n", "8617 | \n", "5670 | \n", "35835 | \n", "20940 | \n", "19146 | \n", "19131 | \n", "2000 | \n", "36681 | \n", "10000 | \n", "9000 | \n", "689 | \n", "679 | \n", "
| \n", " | LIMIT | \n", "AGE | \n", "BILL1 | \n", "BILL2 | \n", "BILL3 | \n", "BILL4 | \n", "BILL5 | \n", "BILL6 | \n", "PAY1 | \n", "PAY2 | \n", "PAY3 | \n", "PAY4 | \n", "PAY5 | \n", "PAY6 | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| count | \n", "29946 | \n", "29946 | \n", "29946 | \n", "29946 | \n", "29946 | \n", "29946 | \n", "29946 | \n", "29946 | \n", "29946 | \n", "29946 | \n", "29946 | \n", "29946 | \n", "29946 | \n", "29946 | \n", "
| mean | \n", "167546 | \n", "35 | \n", "51278 | \n", "49224 | \n", "47063 | \n", "43306 | \n", "40352 | \n", "38911 | \n", "5659 | \n", "5926 | \n", "5227 | \n", "4829 | \n", "4804 | \n", "5220 | \n", "
| std | \n", "129807 | \n", "9 | \n", "73682 | \n", "71219 | \n", "69393 | \n", "64374 | \n", "60836 | \n", "59592 | \n", "16552 | \n", "23060 | \n", "17618 | \n", "15677 | \n", "15290 | \n", "17791 | \n", "
| min | \n", "10000 | \n", "21 | \n", "-165580 | \n", "-69777 | \n", "-157264 | \n", "-170000 | \n", "-81334 | \n", "-339603 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "
| 25% | \n", "50000 | \n", "28 | \n", "3570 | \n", "2988 | \n", "2684 | \n", "2335 | \n", "1770 | \n", "1261 | \n", "1000 | \n", "836 | \n", "390 | \n", "298 | \n", "255 | \n", "122 | \n", "
| 50% | \n", "140000 | \n", "34 | \n", "22400 | \n", "21221 | \n", "20108 | \n", "19066 | \n", "18121 | \n", "17098 | \n", "2100 | \n", "2010 | \n", "1800 | \n", "1500 | \n", "1500 | \n", "1500 | \n", "
| 75% | \n", "240000 | \n", "41 | \n", "67263 | \n", "64108 | \n", "60240 | \n", "54601 | \n", "50244 | \n", "49248 | \n", "5007 | \n", "5000 | \n", "4511 | \n", "4015 | \n", "4040 | \n", "4000 | \n", "
| max | \n", "1000000 | \n", "79 | \n", "964511 | \n", "983931 | \n", "1664089 | \n", "891586 | \n", "927171 | \n", "961664 | \n", "873552 | \n", "1684259 | \n", "896040 | \n", "621000 | \n", "426529 | \n", "528666 | \n", "
| DEFAULT | \n", "0 | \n", "1 | \n", "All | \n", "
|---|---|---|---|
| SEX | \n", "\n", " | \n", " | \n", " |
| MALE | \n", "9003 | \n", "2871 | \n", "11874 | \n", "
| FEMALE | \n", "14312 | \n", "3760 | \n", "18072 | \n", "
| All | \n", "23315 | \n", "6631 | \n", "29946 | \n", "
\\n\"+\n", " \"BokehJS does not appear to have successfully loaded. If loading BokehJS from CDN, this \\n\"+\n", " \"may be due to a slow or bad network connection. Possible fixes:\\n\"+\n", " \"
\\n\"+\n", " \"\\n\"+\n",
" \"from bokeh.resources import INLINE\\n\"+\n",
" \"output_notebook(resources=INLINE)\\n\"+\n",
" \"\\n\"+\n",
" \"\\n\"+\n \"BokehJS does not appear to have successfully loaded. If loading BokehJS from CDN, this \\n\"+\n \"may be due to a slow or bad network connection. Possible fixes:\\n\"+\n \"
\\n\"+\n \"\\n\"+\n \"from bokeh.resources import INLINE\\n\"+\n \"output_notebook(resources=INLINE)\\n\"+\n \"\\n\"+\n \"