{ "metadata": { "name": "", "signature": "sha256:2d39e721aea6e9d12dd0c22db50ac39a6dcf52de1c53e6c29d980f1c9e338d9c" }, "nbformat": 3, "nbformat_minor": 0, "worksheets": [ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "[![](https://bytebucket.org/davis68/resources/raw/f7c98d2b95e961fae257707e22a58fa1a2c36bec/logos/baseline_cse_wdmk.png?token=be4cc41d4b2afe594f5b1570a3c5aad96a65f0d6)](http://cse.illinois.edu/)" ] }, { "cell_type": "code", "collapsed": false, "input": [ "from __future__ import print_function\n", "\n", "import numpy as np\n", "import matplotlib as mpl\n", "import matplotlib.pyplot as plt\n", "import matplotlib.cm as cm\n", "%matplotlib inline" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "heading", "level": 1, "metadata": {}, "source": [ "Machine Learning" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "_Machine learning_ is the science of getting computers to act without telling them to do so. This is where we have got our self-driving cars, effective web search, speech recognition etc. You use it everyday without even realizing it! " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "So today we will try to go into some of the primary wings of machine learning using Python through examples:\n", "\n", "- Supervised learning : E.g. Classification\n", "\n", "- Unsupervised learning: E.g. Clustering" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "** A representation of the classifiers in scikit-learn **" ] }, { "cell_type": "code", "collapsed": false, "input": [ "print(__doc__)\n", "\n", "\n", "# Code source: Ga\u00ebl Varoquaux\n", "# Andreas M\u00fcller\n", "# Modified for documentation by Jaques Grobler\n", "# License: BSD 3 clause\n", "\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "from matplotlib.colors import ListedColormap\n", "from sklearn.cross_validation import train_test_split\n", "from sklearn.preprocessing import StandardScaler\n", "from sklearn.datasets import make_moons, make_circles, make_classification\n", "from sklearn.neighbors import KNeighborsClassifier\n", "from sklearn.svm import SVC\n", "from sklearn.tree import DecisionTreeClassifier\n", "from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier\n", "from sklearn.naive_bayes import GaussianNB\n", "from sklearn.lda import LDA\n", "from sklearn.qda import QDA\n", "%matplotlib inline\n", "h = .02 # step size in the mesh\n", "\n", "names = [\"Nearest Neighbors\", \"Linear SVM\", \"RBF SVM\", \"Decision Tree\",\n", " \"Random Forest\", \"AdaBoost\", \"Naive Bayes\", \"LDA\", \"QDA\"]\n", "classifiers = [\n", " KNeighborsClassifier(3),\n", " SVC(kernel=\"linear\", C=0.025),\n", " SVC(gamma=2, C=1),\n", " DecisionTreeClassifier(max_depth=5),\n", " RandomForestClassifier(max_depth=5, n_estimators=10, max_features=1),\n", " AdaBoostClassifier(),\n", " GaussianNB(),\n", " LDA(),\n", " QDA()]\n", "\n", "X, y = make_classification(n_features=2, n_redundant=0, n_informative=2,\n", " random_state=1, n_clusters_per_class=1)\n", "rng = np.random.RandomState(2)\n", "X += 2 * rng.uniform(size=X.shape)\n", "linearly_separable = (X, y)\n", "\n", "datasets = [make_moons(noise=0.3, random_state=0),\n", " make_circles(noise=0.2, factor=0.5, random_state=1),\n", " linearly_separable\n", " ]\n", "\n", "figure = plt.figure(figsize=(20, 7))\n", "i = 1\n", "# iterate over datasets\n", "for ds in datasets:\n", " # preprocess dataset, split into training and test part\n", " X, y = ds\n", " X = StandardScaler().fit_transform(X)\n", " X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=.4)\n", "\n", " x_min, x_max = X[:, 0].min() - .5, X[:, 0].max() + .5\n", " y_min, y_max = X[:, 1].min() - .5, X[:, 1].max() + .5\n", " xx, yy = np.meshgrid(np.arange(x_min, x_max, h),\n", " np.arange(y_min, y_max, h))\n", "\n", " # just plot the dataset first\n", " cm = plt.cm.RdBu\n", " cm_bright = ListedColormap(['#FF0000', '#0000FF'])\n", " ax = plt.subplot(len(datasets), len(classifiers) + 1, i)\n", " # Plot the training points\n", " ax.scatter(X_train[:, 0], X_train[:, 1], c=y_train, cmap=cm_bright)\n", " # and testing points\n", " ax.scatter(X_test[:, 0], X_test[:, 1], c=y_test, cmap=cm_bright, alpha=0.6)\n", " ax.set_xlim(xx.min(), xx.max())\n", " ax.set_ylim(yy.min(), yy.max())\n", " ax.set_xticks(())\n", " ax.set_yticks(())\n", " i += 1\n", "\n", " # iterate over classifiers\n", " for name, clf in zip(names, classifiers):\n", " ax = plt.subplot(len(datasets), len(classifiers) + 1, i)\n", " clf.fit(X_train, y_train)\n", " score = clf.score(X_test, y_test)\n", "\n", " # Plot the decision boundary. For that, we will assign a color to each\n", " # point in the mesh [x_min, m_max]x[y_min, y_max].\n", " if hasattr(clf, \"decision_function\"):\n", " Z = clf.decision_function(np.c_[xx.ravel(), yy.ravel()])\n", " else:\n", " Z = clf.predict_proba(np.c_[xx.ravel(), yy.ravel()])[:, 1]\n", "\n", " # Put the result into a color plot\n", " Z = Z.reshape(xx.shape)\n", " ax.contourf(xx, yy, Z, cmap=cm, alpha=.8)\n", "\n", " # Plot also the training points\n", " ax.scatter(X_train[:, 0], X_train[:, 1], c=y_train, cmap=cm_bright)\n", " # and testing points\n", " ax.scatter(X_test[:, 0], X_test[:, 1], c=y_test, cmap=cm_bright,\n", " alpha=0.6)\n", "\n", " ax.set_xlim(xx.min(), xx.max())\n", " ax.set_ylim(yy.min(), yy.max())\n", " ax.set_xticks(())\n", " ax.set_yticks(())\n", " ax.set_title(name)\n", " ax.text(xx.max() - .3, yy.min() + .3, ('%.2f' % score).lstrip('0'),\n", " size=15, horizontalalignment='right')\n", " i += 1\n", "\n", "figure.subplots_adjust(left=.02, right=.98)\n", "plt.show()" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Representing data in scikit-learn**\n", "\n", "Most machine learning algorithms implemented in scikit-learn expect data to be stored in a two-dimensional array or matrix. The arrays can be either numpy arrays, or in some cases scipy.sparse matrices. The size of the array is expected to be [n_samples, n_features]\n", "\n", "- `n_samples`: Each sample is an item to process (e.g. classify). A sample can be a document, a picture, a sound, a video, an astronomical object, a row in database or CSV file, or whatever you can describe with a fixed set of quantitative traits.\n", "\n", "- `n_features`: The number of features or distinct traits that can be used to describe each item in a quantitative manner. Features are generally real-valued, but may be boolean or discrete-valued in some cases. The number of features must be fixed in advance. However it can be very high dimensional (e.g. millions of features) with most of them being zeros for a given sample. This is a case where scipy.sparse matrices can be useful, in that they are much more memory-efficient than numpy arrays." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Loading a dataset**" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Before we begin with machine learning, let's start at its very root by understanding how to load a dataset. We are going to start with the `iris` dataset stored in `scikit-learn`. It can be visualized as follows:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "from IPython.core.display import Image, display\n", "display(Image('http://www.twofrog.com/images/iris38a.jpg'))\n", "print (\"Iris Setosa\\n\")\n", "\n", "display(Image('http://www.gatewaygardens.com/_ccLib/image/plants/DETA-185.jpg'))\n", "print (\"Iris Versicolor\\n\")\n", "\n", "display(Image('http://mirlab.org/jang/books/dcpr/image/Iris-virginica-3_1.jpg'))\n", "print (\"Iris Virginica\")\n", "\n" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Question**\n", "If we want to design an algorithm to recognize iris species, what might the data be?\n", "(Remember: we need a 2D array of size [`n_samples` x `n_features`].)\n", "\n", "- What would the `n_samples` refer to?\n", "- What might the `n_features` refer to?\n", "\n", "Remember that there must be a fixed number of features for each sample, and feature number `i` must be a similar kind of quantity for each sample." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Scikit-learn has a very straightforward set of data on these iris species. The data consists of the following:\n", "\n", "Features in the Iris dataset:\n", "\n", "- sepal length in cm\n", "- sepal width in cm\n", "- petal length in cm\n", "- petal width in cm\n", "\n", "Target classes to predict:\n", "\n", "- Iris Setosa\n", "- Iris Versicolour\n", "- Iris Virginica\n", "\n", "`scikit-learn` embeds a copy of the iris CSV file along with a helper function to load it into numpy arrays:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "from sklearn import datasets\n", "iris = datasets.load_iris() #150 observations of the iris flower with sepal length, sepal width, petal length and petal width" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The data is stored in the `.data` attribute" ] }, { "cell_type": "code", "collapsed": false, "input": [ "#iris.data.shape\n", "#iris.target_names\n", "iris.feature_names" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This data is four dimensional, but we can visualize two of the dimensions at a time using a simple scatter-plot:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "x_index = 0\n", "y_index = 1\n", "\n", "# this formatter will label the colorbar with the correct target names\n", "formatter = plt.FuncFormatter(lambda i, *args: iris.target_names[int(i)])\n", "\n", "plt.scatter(iris.data[:, x_index], iris.data[:, y_index],\n", " c=iris.target)\n", "plt.colorbar(ticks=[0, 1, 2], format=formatter)\n", "plt.xlabel(iris.feature_names[x_index])\n", "plt.ylabel(iris.feature_names[y_index])\n", "plt.show()" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Exercise**\n", "Change x_index and y_index in the above script and find a combination of two parameters which maximally separate the three classes.\n", "This exercise is a preview of dimensionality reduction, which we'll see later." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Other datasets**" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "- Packaged Data: these small datasets are packaged with the scikit-learn installation, and can be downloaded using the tools in sklearn.datasets.load_*\n", "- Downloadable Data: these larger datasets are available for download, and scikit-learn includes tools which streamline this process. These tools can be found in sklearn.datasets.fetch_*\n", "- Generated Data: there are several datasets which are generated from models based on a random seed. These are available in the sklearn.datasets.make_*" ] }, { "cell_type": "code", "collapsed": false, "input": [ "datasets.make_circles" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "import pylab as pl\n", "mpl.rcParams['figure.figsize']=[16,5.4]\n", "\n", "digits = datasets.load_digits()\n", "fig, (ax0, ax1, ax2) = plt.subplots(ncols=3, sharex=True)\n", "ax0.matshow(digits.images[0])\n", "ax1.matshow(digits.images[0], cmap=pl.cm.gray_r) \n", "ax2.imshow(digits.images[2], cmap=pl.cm.gray_r) \n", "fig.show()" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Estimator**" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Every algorithm is exposed in scikit-learn via an ''Estimator'' object. For instance a linear regression is implemented as follows:\n" ] }, { "cell_type": "code", "collapsed": false, "input": [ "from sklearn.linear_model import LinearRegression" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Estimator parameters: All the parameters of an estimator can be set when it is instantiated, and have suitable default values:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "model = LinearRegression(normalize=True)\n", "print (model.normalize)" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "print (model)" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Estimated Model parameters: When data is fit with an estimator, parameters are estimated from the data at hand. All the estimated parameters are attributes of the estimator object ending by an underscore:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "mpl.rcParams['figure.figsize']=[6,3]\n", "\n", "x = np.array([0, 1, 2])\n", "y = np.array([0, 1, 2])\n", "\n", "plt.plot(x, y, 'o')\n", "plt.xlim(-0.5, 2.5)\n", "plt.ylim(-0.5, 2.5)\n", "plt.show()" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "# The input data for sklearn is 2D: (samples == 3 x features == 1)\n", "X = x[:, np.newaxis]\n", "print (X)\n", "print (y)" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "model.fit(X, y) \n", "print (model.coef_)\n", "print (model.intercept_)" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "model.residues_" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The model found a line with a slope 1 and intercept 0" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "# Learning" ] }, { "cell_type": "heading", "level": 2, "metadata": {}, "source": [ "Supervised Learning" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In supervised learning, the computer gets presented with a set of sample data (or \"training\" data as it is called) along with its output. The idea is to develop a rule or a model that maps the sample output to the input. So, when the computer is presented with a new input, it can classify it into a particular category. For example, spam filtering is an instance where the learning algorithm is presented with emails messages labeled beforehand as \"spam\" or \"not spam\", to produce a computer program that labels unseen messages as either spam or not.\n", "\n", "Essentially we have a dataset consisting of both features and labels. The task is to construct an estimator which is able to predict the label of an object given the set of features. \n", "\n", "Supervised learning is further broken down into two categories, **classification** and **regression**. In classification, the label is discrete, while in regression, the label is continuous." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " `scikit-learn` supports a number of supervised learning algorithms, and we will try to elucidate using examples using the following:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- KNN (K Nearest Neighbours)\n", "- Linear Regression\n", "- Support Vector Machines\n" ] }, { "cell_type": "heading", "level": 3, "metadata": {}, "source": [ "Classification" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The figure shows a collection of two-dimensional data, colored according to two different class labels. A classification algorithm may be used to draw a dividing boundary between the two clusters of points:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "![](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAW0AAAD5CAYAAADsgWTDAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XdYk1f7B/BvQthhhDAFBBxMQUAFt+Co4qqK1FFHbetotc6Ot7VVtK3WWux4ta2tra221j2qgtu4F6itG1BApghhJxCSPL8//Jm3NKCEJDwJ3J/r4roIOTnP/dheN4f7POccDsMwDAghhBgFLtsBEEIIaTxK2oQQYkQoaRNCiBGhpE0IIUaEkjYhhBgRStqEEGJEePq+AIfD0fclCCGkRarviexmGWkzDNOkr6VLlzb5s4b2RfdieF8t5T7oXgzzS9v7aAiVRwghxIhQ0iaEECNi0Ek7KiqK7RB0hu7F8LSU+wDoXgyRvu6DwzyreKKLC3A4z6zPEEIIUddQ7jTokTYhhJC69P7IX2tUU1ODtWvXIjUtHeFhoZg+fTq4XPr9SAjRHpVHdEyhUOCFITEolzEI6t4Xl4/uR2RoMDb+/BPboRFCjEhDuZOSto5dunQJL02chE+3HoUJj4dqSRXmDYtE6t27cHV1ZTs8QoiRoJp2M5FIJODbCWDCe1J5Mre0grmFJaRSKcuREUJaAqpp61i3bt1QKX6M/RvXoXOv/ji9fxvaenrCy8uL7dAIIS0AjbR1jM/nQ3TyBEof3MRP8fPAV0hwOCmRJiIJITpBNW1CCDFAVNMmhJAWgJI2IYQYEZqIJE0iFotx/fp1ODg4oHPnzga7b3paWhp+3bQJjJLBxIkTEBQUxHZIhGiFRtpEYykpKfAPDMT89xYjZvhIvDLtVYOct7h58ya69+yJGznFuFVQhj79+iE5OZntsAjRCk1EEo0FBYeg/8SZ6DnkRdRIpVgxYyxWLluC2NhYtkOrY9KUqYCwLYZNmQkAOLZzM4rvJGPv7l0sR0bI89FEJNGZzAcPENorGgBgbmkJ/y49kJ6eznJU6iqrqmDv5KJ6LXByQWVlFYsREaI9StpEY51CQnDqz20AgPKSYlw/cwydO3dmOSp1cWNGY+8Pa5D291U8uP0Xdn27GnGxo9kOixCtUHmEaCw9PR1DYoZCUl2DirJSzJ8/Dx8vX97scZw7dw779x+Ara0NZsyYAUdHR7U23333Hb785r8Aw2DG9NexaOFCg500JeSf9LZhlLe3N2xtbWFiYgJTU1Ncvny5URcmxq22thaZmZkQCAT1Jkt927lzJ2a9OQfRsZMgfpSH9GsXkHLlCiuxEKIPekvaPj4+SElJgYODg0YXJkQb/oFBGDt/KYK69QQA/LBsEWJ6dsG7777LcmSE6IZeJyIpKZPmVllZCaFrG9VrB2c3lFdUsBgRIc1D68U1HA4HAwcOhImJCWbOnInp06ertYmPj1d9HxUV1WIO7iTsGT1qFH77/CNMXBSP4oJcnNr7B97d/yfbYRHSZCKRCCKR6LnttC6P5Ofnw83NDY8fP8agQYPw3//+F3369PnfBag8QvRAJpPh7Xfexd59+2Bja4uVn3yMkSNHsh0WITrTLCfXLFu2DHw+H4sWLXruhQkhhDRMLzVtiUSCiv+vI1ZVVeHIkSMIDg7WpktCCCHPoFVN+9GjRxg9+sliBblcjpdffhkvvPCCTgJrjcRiMdatW4disRhDBg/GkCFD2A6JEGJgaHGNgSgtLUXXbhHw6tQFLl7tcHLnZixb8iFmzJjBdmiEEBbQaez/TywW4/bt23B1dUWHDh3YDkflu+++w+97kzD7s28BAFmpt/H1gmnIz8tlOTJCCBtowygAZ86cga+/P2a+tQAR3Xvgw48+YjskFYlEAhuH/63msxc6QSKRsBgRIcQQtZqRNsMwaOPhgakfrELnnlGoKC1B/JTh2LntD/Ts2ZPt8HDnzh306t0HU99fgTbe7bFj3ecI8vHAxp9/Yjs0QggLWv1IWyqVorioCCE9+gEAbOwF8AvrhtTUVJYjeyIgIAB7du/C2Z0bsf6DNxHRyQ/ffbuO7bAIIQam1Rw3ZmlpCTe3Nrh8PBGRA4eh5HEBbidfQNBH77Edmkq/fv1w5eIFtsMghBiwVlMeAYDk5GQMHzESFtZ8iB8X4sMPF+Pdd95hO6wmUSqVuHnzJqqrqxESEgILCwud9v/gwQN8umIliktKMGJoDF599VW1LU2rq6shl8vB5/N1em1CCD09oiKRSHD//n24uLjA2dmZ7XCapLa2FqPGxOLa9b9gbWMLKGpx8vgxeHh46KT/vLw8hHXpgr6jJsK1bTsc/GUdXps6CYs/+ADAk18Y8xcsxPr134PL4WLAoEHY9scWWFtb1+mnuroaf//9NywsLNCpUydwua2mGkeI1ihptyBffvklftv1J+av+Rk8U1Ps+j4B8scPdXb24VdffYUDpy/jtY8+BwDkZd7H52+Mx6OCfADA+vXr8eW367Hom02wsLTCD/ELEezjgW/XrVX1kZOTg+j+AwCeGaoqyhHWOQR79+yGqampTmIkpKVr9RORLcmdu/cQ3HsAeP+fALtED8Hde/d01r9CoYAJ73/THTxTUyiUCtXrs+cvoM/I8eDb2oNnaoZB417F+QsX6/Qx+625CO0/DMt/T8KqXSIUlFZi7dq1IIRoh5K2EQruFIRrJw9BVlMNhmFw8dA+BHfqpLP+Y2NjkXwyCYe2/ITr507i+w/fwuuvvaZ638vTE/dvpKhGAal/JcPDw71OH3fv3UOX6CfL8HmmpgjuPQB37uruFwshrRWVR4yQXC7Hy5Mm49iJE7C0sobAzhbHjhyGi4vL8z/8/yQSCea8NReJiYmws7fH6lWf1dna9ObNm1gSvwxisRjDhw3FwgULVDXpsrIy9O7bDzC1gCXfBtlpt3FaJELHjh1Vnx/70jjIrIQYP/d91NbUYM2CVzB90njMmTNHLZacnBykpaXBx8cH3t7eTf+HIaQFoZp2C8MwDDIzMyGVSuHr6wseT7OnN1+Z9ipScwsxfv6HeJSdie8/nIsjh5LQpUuXRn1eKpXi2LFjkMlkiIqKglAorPP+o0ePMPCFwSgpK4dUUoWB/ftjy++/wcTEpE67LVu2YPact+DZ3hfZ91OxYsWneGPWLI3uhZCWiJI2qUPo5ISPf0+CwMkVAPDH158isqMHFi9erLNr1NbWIi0tDZaWlvD29lZ7ZLC0tBRe3j5Y/OMOeHbwx6OcLCybOhJ//3Udnp6eOouDEGNEE5GkDhsbWzzOy1G9Ls7PhZ2dnU6vYWpqisDAQPj4+KglbOBJWUTg6ATPDv4AABcPL7j7tEdGRoZO4yCkJWk1KyJJXatWrsDst2ahz8iX8Dg3C8UP0zFlym/NGoOXlxfKSopx79pl+IVFIPPeLeRkpNepjRNC6qLySCt2/vx5HDlyBPb29pg2bZrOR9qNcejQIUx8+WXwbQUoKynGjz+sx0svvQQAKC8vh1gshoeHh8Y1e0KMHdW0idauX7+O48ePQyAQYMKECbC0tNRJv5WVlXj48CHc3d1VvzgS1qzBkiVLYW1jA76VFZISD8LPz0+jfhmGQX5+PqysrGBvb6+TWAlpLlTTZsnWrVvh4uYGcwsLxAwbDrFYzHZITbJ3714MGPQCTly9jXU/b0bvvv0glUp10jefz0dgYKAqYZ8/fx6rE9Zg1c4T+DrxMqJeehUvjZ+gUZ+PHz9GRPce6BTSGe4enpg3fwENHkiLQElbj5KTkzFn7jzM/WIDvj/xN5R8Iaa8Mo3tsAAAmZmZOHDgAG7dutWo9vMWLMScz77DpLeXYdE3m8C1ssXvv/+ul9iuXbuGzr36Q+jiBgCIHj0BN//+C0qlstF9zHpzNlx8Q7D2yFV8dfACEo8ex5YtW/QSLyHNiZK2HolEIkS+MALtAjvDwtIKL81+DyePH2c7LGzduhXhXbpi2edfIqr/QCxb/vFzP1MiFsPd58kEIYfDgat3BxQXF+slPh8fH6Rev4Jq6ZOTe25eOgPPtl4abTiVnJyM/rGTwOFwwLe1R8SgkbiSnKyXeAlpTjS7o0eOjo7Ie5AGhmHA4XCQff8ehI6Oz/+gHkkkEsyYOQuLf9yJth39USYuwocThiBubCwCAwMb/NzAQYOw7b8rMWHBh8jPysD5pD1YMueAXmKMiYnBgD178cFLA9HGyweZqbexZ5dmm2F5+/jg5sUzcPNqB4VcjntXL6B73Ci9xEtIc6KJSD2qqalBVP8BkCo5cPVqj8vHDuLnDT9i1KjmTR4Mw+CHH37AH9u2g8vl4vr161h39Jrq/YS3JmP5B+9g6NChDfZRVlaGqdNexdHDh2EvEGBNwhcYN26cXmO+evUqCgsLER4ertESfeDJ8W39Bw6Ea9t2KC0uQjsvTyQdPAAzMzM9RUyIbun16RGFQoGuXbvCw8MD+/fvb9SFW4uamhrs2LEDxcXFiIqKQufOnZs9hi+/+grffLsecXP+gzJxEX79fAnGzXkXQya8hqzU2/j8zYm4djUFXl5ezR6bPpWUlODSpUuwtrZGz5491ZbQE2LI9Jq016xZg5SUFFRUVODPP/9s1IVJ8wkICsaEdz9Fx5BwAMCu79fg8B8/wdLaGtUSCX78Yb1eR82EEM3p7ZG/nJwcJCYm4vXXX6fkbKBMTEwgl9eqXsvltZg1ayauXLyARwX5lLAJMSJaT0QuWLAAq1evRnl5eYNt4uPjVd9HRUUhKipK28sSDSycPxcfLl2IF6fPR7m4CKf3/YFV5861uHIIIcZMJBJBJBI9t51W5ZEDBw4gKSkJ69atg0gkQkJCAtW0DdTOnTuxfcdOWFtbY9HCBeikw0MTdE0qlcLCwqLeTaYIaS30UtP+4IMPsHnzZvB4PFRXV6O8vByxsbHYtGnTcy9MyL9lZWXhxdFjcPvWTVhYWGL9999hwgTNVkIS0lLofe+RU6dO4YsvvqCRNmmyLt0i4NdjAIa/8iay0+7i8zkv49TJEwb9VwEh+tIse4/Qn7OtV21tLea8NRcOjo5o4+6h8SG+NTU1+Pv6NQyb+gY4HA7a+gYgtFc0Ll26pKeICTFOOkva/fr1U3vcj7QeS+PjcTb5Opb/loR5X/6Cz1avwZ49exr9eTMzM9jY2iHz7k0AgLxWhqx7t+Dm5qavkAkxSrQikuhESGg44hYuQ4fgMADA0e2bwHmcgZ82/NjoPnbu3IkZs95ASI++eJh2F+EhnbB96x8a7TnSVNnZ2UhYswbiklKMHD4MY8eO1fs1CXkW2pqV6JXAQYD8hw9UrwuyH8DBQaBRH2PHjsX5s2fwSuwIfPf1mmZL2AUFBYjo3h0PSqph5uGH+Yve1bi8Q0hzoZE20YmLFy9i6PDhiBw0EtLKctz/+wouX7xoFOWNhIQEJJ5NxutLVgMAMu/dwrp3ZyDnYRbLkZHWjEbaRK+6d++OC+fOYUCXQMS+0A/XUlKMImEDgEwmg4U1X/Xa0pqPWpmMxYgIaRhtzdoKKBQKfLpiBXbt3gsbGz4+XhaP6OhonV/Hz89P4yPBDMGYMWOwOqEXPDsGwMXDC7u+W41JkyexHRYxQE+3Wf63bdu2ITk5GcXFxRCLxSguLsbUqVPx+uuv6zwGStqtwJKlS7En8QjGz1+C4oI8xI6Nw/FjRxEWFsZ2aAbBz88PSQcPYvGSpUguKUHciGH48MPFbIdF9EipVKKsrAzm5uawsrJSe3/9+vU4efIkiouL63z98ssv9U5Sy+VyODo6ws/PD0KhEEKhEB07dtRL7FTTbgW8fNphzhcb4NHOFwCwfd3nCHKzwycfP//EGkIMnUQiUSXVNm3awNnZWa3N6tWrsXv3btUouLS0FHw+Hz/99BNiY2PV2h8/fhyFhYWqBPz0i8/nN9t6lIZyJ420WwELCwtUlZepXksrymHhrf4/NiFsksvlKCkpUSVgsViMgIAAdOjQQa1tfHw8NmzYoDry7mlSXb58OV588UW19kOGDEHPnj3h4OAAoVAIBwcH8HgNp78BAwbo7sZ0jEbaRuLGjRtIT09HYGCgxnXjX3/9Fe9+sBgxk2ZC/CgPl4/sQ8qVK3B3d9dTtKS1k0gkKCwsVCsvREREICIiQq39e++9h4SEBNjZ2dUZ2c6ePRsxMTFq7fPz8yGXyyEUCustb7QEet97RNMLk8ZbufIzrPnqa7QPCkHajWv4bMWnmD59ukZ9JCUlYc/efbCx4WPe3Llo27atnqJtHSoqKvDxJ5/gXmo6wkJD8P5//gNzc3O2w9Kb/Px8pKWl1RkFFxcXIzo6GkOGDFFrv3z5cvz000+qUe3TJBwXF1fvJHh1dTVMTU3pdKF/oKRtpO7fv49ukd3x6R+HYe/ojILsTCydPBxZmRkQCDRbvEJ0Qy6Xo1efvrB0ckdIz2hcOrIPjlZmOLD/T6PZfyc1NRWXL19WS8JDhw7F5MmT1dqvX78emzdvVqvxRkdHIzIykoU7aPmopm2ksrOz4e7dHvaOT2rQrp7esBc6Ij8/n5I2S1JSUlBYJMYna7eCy+WiW/8hmD+8B7KysuDt7c1KTNeuXcOhQ4fUyhGjR4/G22+/rdb+5s2bSExMVCVfPz8/ODg4NPhE0cyZMzFz5kx93wZpBEraBi4gIAA5GWlI+zsFHUO64Pq5k5BWVbCWHMiT595NTHmqUTWXawIulwuFQqFxPzKZDJaWlmrvnT9/Hps2bVKNgJ9+xcXFYc2aNWrtKyoqUFZWBhcXFwQGBqrKEu3atav32mPGjMGYMWM0ipcYBiqPGIHExES8PGkyOFwueDwT7N65E71792Y7rFarpqYGXbpFoG1wV4T2HoDzibshK3mE3zdvAo/Hg6enp9pnRCIRVq9eXScBl5eXY8aMGfj222/V2t+4cQNnzpxRjYSf1oWdnJxa7MQbqYtq2kautrYWjx8/hrOz8zMfVSLaqa2thVgsVo1wraysEB4ertZu165dmD59OiqrqqCQy2Fubg6hUIjx48dj9erVau2zsrJw48aNOglYIBDQxBtpECVt0qowDIPy8nK1Gq9AIMDQoUPV2h85cgRxcXGoqqqCQCBQjXBfeOEFLF26VK29WCxGTk6OKgnXV+IgRBs0EUmMmlQqRXp6ep0ELBaL4ejoWO/+DsePH8eYMWPUHjnr3r17vf337dsXmZmZsLOza9R2sA4ODnBwcND6vgjRFCVtwoqysjKcPXtWLQk7OzsjPj5erf3NmzfxyiuvqNV4G1ogNHDgQJSXlzc6HgsLC1hYWDT1dghpNlQeITohFouxfft2tXKEk5MTNm7cqNY+NTUV8+fPrzMKdnBwQPv27etdAUdIa0PlkRYmLS0NC99+Bzk5uejdqydWfbaySU8VKJXKessBRUVFSEhIqPPImVgshoODA06ePKnWvrq6GteuXVONfkNCQiAUCtGmTZt6r+vr64vExESN4yWktaORthEqKipCcOfOGDj+NXQM6YLDWzZAaMnDb5t+RXl5OTw8PNQ+U1xcjHnz5qmNhB0cHPDgwQO19iUlJVi3bp3aCjgnJyfas4SQZkBPjxgRmUymGtlWVFSoTZ7t2LEDq775DlVVVagoK0VlqRhl4iJYW1ujY8eOuHbtmlqfUqkUO3furJOAHRwcYG9v3yznMBJCNKOX8kh1dTX69euHmpoayGQyvPjii1i5cqU2XbYoTzda/+feDuXl5Rg3bpxa26qqKgQFBaG4uBjV1dV1JtqOHj1ap62ZmRlkNdWInbUINvYO4JpwsXhiDMRiMczMzOqNxdLSst49JQghxkXrkbZEIoGVlRXkcjl69+6NL774os5qvZYy0mYYBrm5uWrLiktKSvDee++pbRQkl8thaWkJKyurOqNbR0dHbN68Wa09wzDIzMyEg4MDbG1tn7nxkEQiQbfI7nDtEIT2weE4tXcLYgZE46sv1Zc3E0KMk94mIp9OfslkMigUCqN6dvXChQsoKipSm2z75ptvYGpqqta+T58+4PP5anVehUKhtkqRx+NBIpHU2099OBwOfHx8GtXWysoK586cxuerv0B21i0snD0LM2bMaNRnCSG6wzCM6uScfw/owsLCGlwXoA2tk7ZSqUR4eDju37+PN954A4GBgbqIq0l2796NgoICtcm27du3g8/nq7VfsmQJzMzM6jx2FhwcXH/xn8NBRkaGRvE0NmE3hb29PVZ8+one+iektZHL5WoDuH/nkvp+xuFw1AZyQqEQ7du310ucWidtLpeL69evo6ysDIMHD4ZIJEJUVFSdNv9cLBEVFaX2fkPWrVuH7OxstX+w48eP13sO3OHDh8HlciEUCuHl5YXw8HAIhcIGk+e/a8WEEOPHMAwqKirUkuvzEnFlZSUEAkGdQdw/v7y8vNR+5uDgoLMNvEQiEUQi0XPb6fTpkY8//hiWlpZ19u/Vpqa9cuVKMAyj9g8VEBCg11EsIcQw1NTU1JtoG0rAT39uYWGhllzrS8T//HljtzBoLnp55K+oqAg8Hg/29vaQSqUYPHgwli5dWudQzJYyEUkIaTqlUonS0tLnJtt//0wmk6mtmq0v+f7zfQcHhxZx9JteJiLz8/MxdepUKJVKKJVKTJ482aBPMSaEaO/pxNuzEvC/3ystLVVN4teXeJ8e3PDv921sbIzmCLfmQotrCGml5HI5SkpKNEq+xcXFqpLls0a9/35PIBBQSVNDtCKSkBaKYRhUVlZqlHiLi4tRUVEBOzs7jeq+QqEQVlZWNPptBpS0CTECtbW1Go98xWIxeDzec5Ptv39ub29PJ+cYMErahDQjhmFUWxhoMvkmlUrrnJzT2CRMe4G3PJS0CWmi6upqjR43e7q9gZWVlUZ1X6FQ+NwtDEjrQUmbtHoKhUL12JkmSVgul2s88ebg4EATb0QrlLRJi8EwDKqqqjSeeCsvL4etra3Go19ra2sa/ZJmR0mbGKTa2lrVY2eaJOCn2xVoUvsVCAQ08UaMBiVtolea7vfw9KuqquqZ+z009HNLS0u2b5kQvaKkTRrt6ck5miTff+73oEn5wdD2eyDEUFDSboX+eXKOJkm4sfs9/LtNS9jvgRBDQUnbyEmlUrUDG56XgP+534Mmiy5ovwdC2EdJ20D8c78HTRZd1LdFbWMm3v59og4hxDhQ0taxp/s9aLrooqKiAvb29hrt9fB0vwdCSOtBSfsZZDJZo48W+ud7ZmZmGpUdnu73QBNvhLRcDMOgvLwcXC4XNjY2Te6nVSTtxu738O/3pVJpozda/+d7NPFGSMtWU1NTJ2cEBATAxcVFrd0777yDAwcOqLYwsLS0xPLlyzF//vwmX9sok3ZeXh6KiooaNQIWi8WN2u+hvqRM+z0Q0rI93cLg3wf39ujRAx06dFBr/8Ybb2Dz5s11nqRydHTEJ598gj59+qi1v3fvXp3tDszMzLSO2SiTdkREBCQSCe33QAip4/Hjx8jNzVUbvA0ZMgRdu3ZVaz9t2jTs3btXLWfMnj0bPXr0UGsvFothamoKPp/P2oDOKJM2IaR1SEtLw507d9T+eo6Li8PAgQPV2i9evBgHDhxQG7yNHTsW4eHhau0ZhjG6v6YpaRNCmk1ycjIuXryoNhKeOnUqJkyYoNZ+3bp1SEpKUvtrul+/fggICGDhDthHSZsQ8lwNjUhFIhGSkpLUkvCMGTMwb948tfY7d+7EyZMn1Sb4g4OD0bZt2+a4FaNHSZuQVuTpFgYcDgf29vZq7ycmJmLTpk1qk/tz587Fp59+qtZeJBLh4sWLaiNhDw+Pevsn2qOkTYiRkkqlquTK5/PRvn17AE+eiHi61eyePXvwxRdfqNqVlJTA2toaCxcuxNKlS9X6/Pvvv3Hz5k21iXx6kspw6CVpZ2dnY8qUKSgsLASHw8GMGTMwd+7cRl2YkNZGoVCo7R3u7OyMyMhItbbbtm3DokWLIBaLoVAoVIl1ypQp6Nq1KyZPHI/8wscIDvDD9t37YGFhgYcPH9bZwoCepDJueknaBQUFKCgoQGhoKCorK9GlSxfs3bu3zsQBJW3SEtXU1KCgoECtxtu2bVuMGDFCrf2WLVswZcoU2NnZ1XniYfjw4XjjjTfU2peWlqK8vFy1hcHT0W9ubi6CA/0xv4sAwc5WOHS/DMcfmyH1QQattG1hGsqdWu0m5OrqCldXVwAAn89HQEAA8vLyWu1sLzE+V69exSuTJiIj6yF8vL0xZ9588Hg8VRJu164dZsyYofa5pKQkzJ07V23NQEP13bi4OIwbN67RJ+fY29vX29fVq1fh68RHqKs1AGBYR3vsSs1Gfn4+3N3dNbhzYqx0tgVcZmYmrl27Vu+feoQ0t4KCAhw/flxtJOzr64tly5YBwJPFGIMGYLKfJSydhPghJR0LFyzA2Lg4ODo6qiba6jNq1CiMGjWq0fHoqlTh7OyMnFIpauS2MOdxUVhVC6lMDoFAoJP+ieHTSdKurKzE2LFj8fXXX4PP56u9Hx8fr/o+KioKUVFRurgsaUWys7Px22+/qS2+8PX1xc8//6zWvrCwEAcPHlSNgn19fSEUClWTeACQkpICT1tz9PO2AwB0a2ONGYfysGzZMnh5eTXbvWkiIiICg4YOx/tHEuHrYIar+RJ8tuoz2gWyBRCJRBCJRM9tp/XTI7W1tRg+fDhiYmLq3RyFatqtl1KpRFVVVb07nWVlZeGzzz5TS8IdO3bE8ePH1dpnZGTg+++/VytHuLu710nEmrhy5QpGxwzCVwNcYGbCRVm1HLOSspGTl2/QI1eGYZCUlISsrCyEhYWhe/fubIdE9EAvE5EMw2Dq1KkQCoX48ssvNbowMR4Mw0AikaC4uBjV1dXw9fVVa5OVlYXZs2fXScJlZWWIjIzEuXPn1NoXFhZix44dantBODo61vvXmj4wDIPxY2Nx4+IpBApMcOWRDFNnzsbyT9SfUyakueklaZ89exZ9+/ZFSEiIanZ75cqVGDJkyHMvTNghl8vr7HQmk8nQv39/tXYPHz7EsGHDVCNgDocDoVCI0NBQHDhwQK19RUUFRCKR2sk5jZ14Y4tSqcS2bdvw4MEDhIeHIyYmhu2QCAFAi2taHIZhUFFRobaiTSaqXokhAAAVnklEQVSTYerUqWrt8/LyEBgYiMrKSggEgjq13l9++UWtfU1NDe7cuUMn5xDCEr088kd0R6FQ1LvLWXV1db0r2sRiMby8vNSWFXt6etbbv4uLCzIyMmBnZ9eo53nNzc0RGhqq9X0RQnSLkraeKBQKHD58WO3QhsrKSvz6669q7WUyGcaNG6dW463vlAwAEAqFqKysbHQ8JiYmBj25RghpHCqPNBLDMFi/fr3aqTklJSU4ffq02uhVqVRi2LBh9R7UMGHCBNrfgRDyTK22PCKXy2FiYlJvkly6dCkKCgrUjiBKTU2FpaVlnbYcDgc3b96EjY0N3Nzc0KlTJ1Uirg+Xy0VSUpJe7ok0LC8vD6dOnQKfz8fgwYN1cuwTIYbEaEbaDMOgqqoKxcXFcHd3B4+n/vtm1qxZePjwYZ2RcEVFBQoLC+Hg4KDWPiEhod4zJT08PGgfByOUnJyMmEEDEehkiWJpLWxcvXDi9Fm1X8CEGAOjfHokNjYW9+7dU42AeTwehEIhLl++rNrz5J927doFCwuLOgnY3t6eEnAr0b1rGHqYPkK0jx0YhsHnl4oRN3exVidiE8IWoyyPvPfee3WSsIWFxTPbx8bGNlNkRFeKioqQkZEBLy8vODs7a9VXfl4+/Lo82UiJw+GgvQ0HudnZugiTEINh0EPQiIgIhISEwN3d/bkJmxifHdu3o2M7b0weNRS+7X3w22+bteqvZ69e+DO9AgolA7FUjlP5MvTq00dH0RJiGAy6PEJaBqlUijUJXyDt7h2ER0Ri9uw5KCsrQzsvT8T3dkY7gQUeltXgo9OPcCftfr2lr8YoKSlB3OgXcfb8BYDDweLFi/HREvVn3BujsLAQ7y1aiLR7dxDWNQIrVn1e7x4qhOiLUZZHiPGTy+UYMrA/FAXp6Czk4efTR3Dy2FH06hsFG3Me2gme/AXV1s4cbQTWePDgQZOTtkAgwDHRaVRVVcHc3LzeyerGqK6uRnSfXvA1LcdgZ3OcOr4LI2L+wskz5+hRTcI6StpEr65cuYLs+/eQEOUMEy4HUd5KTN6diJwbl1BZVYWE87lY2KMNsstlyCupavKOff9kbW2t1eeTk5OhqCzBK30dweFw0MnZCjOS/kZWVha8vb21jo8QbVDSJnpVU1MDK1MeTLhPRqimJhxYmnIxM8QONl0FeONgBt46WoDymlqs+359gytAm5OJiQlqlUowADgAlAwDBcMY/OZXpHWgpE30qlu3bpByzbHtdgnCXCxwOL0UbnwzOFrxwOVwEOLpiAGT3sSsWbPg5OTEdrgAgK5du8KxjRfWpuQizMkUZ/Nl6N2nb4On2BDSnAz66RFi/KytrSE6ex7Stl2w6aEZrj6Wo7eXLbgcDh6W1eBGwZNTjwwlYQNPjgY7JjqNLi9OQbqgM2KmzsbOPftw+PBhxI0eiQlxY3D+/Hm2wyStFD09QprVrVu3MDxmMErEYigYBt+t/wGTJk1mO6zn2r9/P16bPBHj/fiQKRlsv1eJg4ePokePHmyHRlooo1wRSVompVKJoqIi2NvbG83eIIOi+iBc/gC92toCAA6kilHtG41NW7ayHBlpqeiRP2IwuFyu1qsfm5tSoVBNpgKACYcDhULBYkSktaKaNmmy2tpa5OXloba2lu1Q9G7GnHnYeLMc5x6WQ5RZhu2plXh91ptsh0VaIUrapEmOHj0KVydHBPv7ws3ZCSdOnGA7JI1VV1dj48aNSEhIwNWrV5/Zdty4cfhm/U+4xmuPO1Z+2LpzD6Kjo5spUkL+h2raRGNisRgd23nj7a4CBDlb4e9HVfgqpQwPHmbD1taW7fAapbq6Gn17dgcjzoW7NRdncqrw3Y8/49L5c9jw0waYcLmYM3celi3/mFZBElZQTZvoTGpqKlxsLBDk/OSw3xAXawisJEhPT0d4eDjL0TXOH3/8AZTk4sMeQnA4HPRoY4FZr7+KNjZm+GqAG+RKYPWGb+Hu7o6Zs95gO1xCVKg8QjTm4eGBvNIqPK56Ust+VFmLwnIJ3N3dWY6s8YqLi9HGiqsaRXvamUNWLcXYjtYQWpnChW+KF9tZ4tCB/SxHSkhdlLSJxjw8PLB02XK8J3qEz66U4v1Tj/Dpys8MYgl6Y/Xv3x9ncqpwt0iKSpkCG64+At+Mi4dlNao2ORVyODobzz2R1kHrmvarr76KgwcPwtnZGTdu3FC/ANW0W6zbt28jNTUVfn5+CAgIYDscje3YsQML3pqNx8VieNrwMLOrK1acyUEXNz6kCiUeSM1w8UoKvLy82A6VtEIN5U6tR9rTpk3DoUOHtO2GGKHAwECMGjXKKBM2AMTFxSGnoBDLP/kUrvZ8dBRaIGGwN6RyJYrNnHH9xq0mJ+zk5GT06R4BXx8vzJr+GiQSiY6jJ62VTp4eyczMxIgRI2ikTXTi7NmzuHv3LgICAtCrV68m93PixAlcunQJbdu2xfjx4xvcpU8qlWJAvz4Q52bAzsIUDyvkOHn6LPz9/Zt03aysLIR3DsGUAGv4CMyxM7USrp17YfuuvU2+F9L6sPr0SHx8vOr7qKgoREVFNcdliRH6aPEH+Hn9OnRyssKNQglmzpmLpcs+1rif1Z+vwlerViDS1Rx/lCuw9bdN2Hcwqd5Dni0tLXHq3AWcOnUKUqkUvXr1goODQ5Pv4ciRIwhztUS0jx0AYE64GSbtOQilUkmHTJMGiUQiiESi57ajkTYxGFlZWQjtFIj/vtAGtuY8lFbLMfdoHm7dTdXoyZSamhrY29piXYwnHK1MIVcyeEdUiA1/7GqWBTG///47vvloARZ3FwIACqtqseBYLsorJfTMN2k0vdW0CdGVR48ewcXOCrbmT/4AtLfgwcnGEo8ePdKon4qKCpjyuBBaPumHx+XA1dYcYrFY5zHXZ9SoUajg2eKb5CLsuyvGxxeKsPjDjyhhE52gpE0Mhr+/P8RSOS7mVIBhGJx7WI5ymRK+vr4a9SMUCtG+XTtsvV2CihoFLuVU4G5hFbp3766nyOuytrbGhSsp6Df5LVhEjMKa737Cf97/oNGfl8lkrWI/F9JEjJbGjx/PuLm5MWZmZoyHhwfz888/13lfB5cgrcjFixeZtu5ujAmXy3h7ujNXrlxpUj85OTlMVO+eDN/KkvFv78OcPXtWx5HqnkwmY16Z/DJjyuMxpjweM+P1Vxm5XM52WIQlDeVO2nuEGCSZTGY0e23ryrKlS/DnL9/i3QghlAyDzy4V4+U57+Dd9/7DdmiEBVTTJkaltSVsADh98jiGeVvC0pQLazMTxHhb4vSJY2yHRQwMJW1iNBQKBRK+WI1ekV0REuiPN2fNRE5ODtth6Yy7Z1uklcpUr9NLa9HGsy2LERFDROURYjRen/YKLh7ag5h2fNwqlOBCTiXMbexw/e+bcHV11dl1srKyMH3aVNy5cwd+fn74ceOv8PHx0Vn/DcnOzkbPyG7wtAKUDIMCGQ8XLifDzc1N79cmhofOiCRGTSqVQmBnh19f9Ial6ZOVjYuPP4Qpj4cJ8z/EO++8o5Pr1NTUoJO/L3oKatDT3RoXciU4U8zDzbupsLS01Mk1nqWkpASHDx8Gh8PBkCFDYGdnp/drEsNE+2kTo6ZUKgEwdc5pNDXhwIz75EADXblz5w4U0krE9nhyhuUYfzOcEz3G7du30aVLF51dpyECgQDjx4/X+3WI8aKkTQwOwzDYunUrLl+8CJ/27TFz5kxYW1tjxPBhWHX+JEZ0tMOdxxJkllRDaWqBDaNH6+zafD4fFVIZauRKmPO4qJErUSapAZ/P19k1CNEGlUeIwXl7wXzs27oJvVx4uFOmhJWHH46ePAW5XI4lH36A3du3oaKyEt4+Pkj4ei169+6ts2szDIOpkybi6qkjCBea4JpYgeCe/fH71u20opE0K6ppE6NQXl4OV2cn/DjMCzbmJlAoGbx7qhA//rG72TYaUyqV2LRpE27duIHATp0wdepU1UZPEokEa9euRc7DLPTu2w9xcXGUzIleUE2bGAWpVAozngmszZ4kSRMuB/aWZqiqqmq2GLhcLl555RW1n9fU1CC6Ty+Yleagox0X/9mxBTf+uo6PP13RbLERQiNtYlAYhkHPiK5wluRgiI81bhRKsTtDhtv30rTaLlUX9u/fj/ffnIZPezuCw+GgtFqO6QcyUVFZ1SoXAxH9ohWRxChwOBzsTzoMM9/uWHW9GrdMvXDi1BnWEzbwpDQisDBVlUNszEzA5XAgk8me80lCdIdG2oQ0UkFBAUICAzDO1xJ+QgvsT68E4x6IIydEbIdGWiAaaROiJVdXVxw9KcJfHE98eVMG5/D+2L6bjhAjzYtG2oQQYoBopE0IIS0AJW1CCDEilLQJIcSIUNImxEBUVFTg3LlzuHXrFs0DkQbRikhCDMCtW7cwqH8U7M0AcWUNXhg6DL9s/l21fJ6Qp+jpEUIMQER4Z3QzfYzB7e1QI1diybkiLE1Yp7NtWqVSKQoLC+Hm5karN40EPT1CiAFLv/8Ake7WAABzHhfBDiZITU3VSd/bt22Di5MjIkKD4dnGFRcuXNBJv4QdlLQJMQBBAQE4lVUJAKiUKXD1cS2Cg4O17vfhw4eYNeM1LO/jjB9j3DEjyBqjRwynpfdGTOukfejQIfj7+6Njx45YtWqVLmIipNX5dctWiIp4eOtYAd44lINhcRMxatQorfu9desWOjjy0U5gAQDo5s4HRylHbm6u1n0Tdmg1EalQKDBnzhwcO3YM7u7u6NatG0aOHImAgABdxUeIQSopKcHXX32JwoJ8DBoyFKO1PD2nXbt2uJN2H+np6bCzs4O7u7tO4vTy8kJGcRVKq21gb8FDVmkNqmS1cHFx0Un/pPlplbQvX76MDh06wNvbGwAwfvx47Nu3j5I2adEqKirQvWs4vHmV8ORzMW/3dtxPT8fbWh4ubGZmhsDAQB1F+URgYCDmLliERV8moJ3QGmmPK/H9DxtgZWWl0+uQ5qNV0s7NzYWnp6fqtYeHBy5duqTWLj4+XvV9VFRUs51AQog+7Nq1C0JUYXYXRwBAtzYyvLt8GRa9/bZBnmLz0dJ4jBkbh4yMDAQFBcHHx4ftkEg9RCIRRCLRc9tplbQb+z/oP5M2IZpgGAZyuRympqZsh6IikUhga/a/6SA7CxPUGPjEXlBQEIKCgtgOgzzDvwe0y5Ytq7edVhOR7u7uyM7OVr3Ozs6Gh4eHNl0SonLgwAG4ODrA0sIC4SFByMjIYDskAEBMTAyu5ElwMqMMD0qq8d+UEsSOHmWQo2zS8miVtLt27Yq0tDRkZmZCJpNh27ZtGDlypK5iI63Y/fv3MeXl8VgUboudL3VEZ14xRsQMNoiFWj4+Pkg6egwp8MT6NCBscCw2bPyV7bBIK6FVeYTH42Ht2rUYPHgwFAoFXnvtNZqEJDpx+fJlhLjaIMDpyYTZKD97bNuTgbKyMtjb27McHRAREYHTF9TnbwjRN633HomJiUFMTIwuYiFExcXFBVml1ahVKGFqwkVuhQxcLhc2NjZsh0YIq2jDKGKQoqOjEdm3P/5z+iTaC8yRkleJtd9+CxMTE7ZDI4RVtGEUMVgMwyApKQk5OTmIiIhAaGgo2yER0mwayp2UtAkhxADRLn+EENICUNImhBAjQkmbEEKMCCVtQggxIpS0CSHEiFDSJoQQI0JJmxBCjAglbUIIMSKUtAkhxIhQ0iaEECNCSZsQQowIJW1CCDEilLQJIcSIUNImhBAjQkmbEEKMCCVtQggxIpS0CSHEiFDSJoQQI0JJmxBCjEiTk/aOHTsQFBQEExMTXL16VZcxEUIIaUCTk3ZwcDD27NmDvn376jIeQgghz8Br6gf9/f11GQchhJBGoJo2IYQYkWeOtAcNGoSCggK1n69YsQIjRoxo9EXi4+NV30dFRSEqKqrRnyWEkNZAJBJBJBI9tx2HYRhGmwtFR0cjISEB4eHh9V+Aw4GWlyCEkFanodypk/IIJWVCCGkeTU7ae/bsgaenJy5evIhhw4YhJiZGl3ERQgiph9blkedegMojhBCiMb2WRwghhDQPg07ajZlJNRZ0L4anpdwHQPdiiPR1H5S0mwndi+FpKfcB0L0YolaZtAkhhNRFSZsQQoxIszw9QgghRHP1pecmbxilzUUJIYQ0DZVHCCHEiFDSJoQQI2LwSdvYT8g5dOgQ/P390bFjR6xatYrtcJrs1VdfhYuLC4KDg9kORWvZ2dmIjo5GUFAQOnXqhG+++YbtkJqsuroakZGRCA0NRWBgIN5//322Q9KKQqFAWFiYRruIGiJvb2+EhIQgLCwMERERuu2cMXB37txh7t27x0RFRTEpKSlsh6MRuVzOtG/fnsnIyGBkMhnTuXNn5vbt22yH1SSnT59mrl69ynTq1IntULSWn5/PXLt2jWEYhqmoqGB8fX2N9r8LwzBMVVUVwzAMU1tby0RGRjJnzpxhOaKmS0hIYCZOnMiMGDGC7VC04u3tzRQXF+ulb4Mfafv7+8PX15ftMJrk8uXL6NChA7y9vWFqaorx48dj3759bIfVJH369IFAIGA7DJ1wdXVFaGgoAIDP5yMgIAB5eXksR9V0VlZWAACZTAaFQgEHBweWI2qanJwcJCYm4vXXX28RDzDo6x4MPmkbs9zcXHh6eqpee3h4IDc3l8WIyL9lZmbi2rVriIyMZDuUJlMqlQgNDYWLiwuio6MRGBjIdkhNsmDBAqxevRpcrvGnJQ6Hg4EDB6Jr16748ccfddq33h/5awxdnZBjaOgZdcNWWVmJsWPH4uuvvwafz2c7nCbjcrm4fv06ysrKMHjwYIhEIqM7HerAgQNwdnZGWFhYi1jGfu7cObi5ueHx48cYNGgQ/P390adPH530bRBJ++jRo2yHoBfu7u7Izs5Wvc7OzoaHhweLEZGnamtrERsbi0mTJmHUqFFsh6MTdnZ2GDZsGJKTk40uaZ8/fx5//vknEhMTUV1djfLyckyZMgWbNm1iO7QmcXNzAwA4OTlh9OjRuHz5ss6StlH9HWJsda6uXbsiLS0NmZmZkMlk2LZtG0aOHMl2WK0ewzB47bXXEBgYiPnz57MdjlaKiopQWloKAJBKpTh69CjCwsJYjkpzK1asQHZ2NjIyMrB161b079/faBO2RCJBRUUFAKCqqgpHjhzR6VNXBp+0jfmEHB6Ph7Vr12Lw4MEIDAzEuHHjEBAQwHZYTTJhwgT07NkTqamp8PT0xMaNG9kOqcnOnTuH3377DSdPnkRYWBjCwsJw6NAhtsNqkvz8fPTv3x+hoaGIjIzEiBEjMGDAALbD0poxlxYfPXqEPn36qP6bDB8+HC+88ILO+tf73iOEEEJ0x+BH2oQQQv6HkjYhhBgRStqEEGJEKGkTQogRoaRNCCFGhJI2IYQYkf8DIEt6N5m7yKkAAAAASUVORK5CYII=)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**KNN Classification**" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "K nearest neighbors (kNN) is one of the simplest learning strategies: given a new, unknown observation, look up in your reference database which ones have the closest features and assign the predominant class." ] }, { "cell_type": "code", "collapsed": false, "input": [ "from sklearn import neighbors, datasets\n", "\n", "iris = datasets.load_iris()\n", "X, y = iris.data, iris.target\n", "\n", "# create the model\n", "knn = neighbors.KNeighborsClassifier(n_neighbors=2)\n", "\n", "# fit the model\n", "knn.fit(X, y)\n", "\n", "# What kind of iris has 3cm x 5cm sepal and 4cm x 2cm petal?\n", "# call the \"predict\" method:\n", "result = knn.predict([[3, 5, 4, 2],])\n", "\n", "print(iris.target_names[result])" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Ordinary Least Squares(OLS)**" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "OLS fits a linear model by evaluating the coefficients $\\alpha= (\\alpha_1,\\alpha_2,...\\alpha_n)$ to minimize the residual sum of squares between the observed responses in the dataset, and the responses predicted by the linear approximation. Mathematically it solves a problem of the form:\n", "$$\\min_{\\alpha} {\\|{X\\alpha - y}} \\| \\text{.}$$" ] }, { "cell_type": "code", "collapsed": false, "input": [ "from sklearn import linear_model\n", "clf = linear_model.LinearRegression()\n", "clf.fit ([[0, 0], [1, 1], [2, 2]], [0, 1, 2])" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "clf.coef_" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "# Create some simple data\n", "import numpy as np\n", "np.random.seed(0)\n", "X = np.random.random(size=(20, 1))\n", "y = 3 * X.squeeze() + 2 + np.random.normal(size=20)\n", "\n", "# Fit a linear regression to it\n", "model = LinearRegression(fit_intercept=True)\n", "model.fit(X, y)\n", "print (\"Model coefficient: %.5f, and intercept: %.5f\"\n", " % (model.coef_, model.intercept_))\n", "\n", "# Plot the data and the model prediction\n", "X_test = np.linspace(0, 1, 100)[:, np.newaxis]\n", "y_test = model.predict(X_test)\n", "\n", "plt.plot(X.squeeze(), y, 'x')\n", "plt.plot(X_test.squeeze(), y_test);\n" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The model has been learned from the training data, and can be used to predict the result of test data: here, we might be given an x-value, and the model would allow us to predict the y value." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Support Vector Machines (SVM)**" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Support vector machines (SVMs) are a set of supervised learning methods used for classification, regression and outliers detection.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "They are particularly advantageous in the following situations :\n", "\n", "- High dimensional spaces\n", "- Number of dimensions greater than the number of samples\n", "- Memory efficient\n", "- Versatile\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The support vector machines in `scikit-learn` support both dense (`numpy.ndarray`) and sparse (any `scipy.sparse`) sample vectors as input. However, to use an SVM to make predictions for sparse data, it must have been fit on such data. " ] }, { "cell_type": "code", "collapsed": false, "input": [ "from sklearn import svm\n", "clf = svm.LinearSVC()\n", "clf.fit(iris.data, iris.target) #learn from the data" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "clf.predict([[ 5.0, 3.6, 1.3, 0.25]])" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "clf.coef_ #Access the coefficients" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "from sklearn import svm\n", "svc = svm.SVC(kernel='linear')\n", "svc=svc.fit(iris.data, iris.target)\n" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "svc.coef_" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "print(__doc__)\n", "\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "from sklearn import svm, datasets\n", "\n", "# import some data to play with\n", "iris = datasets.load_iris()\n", "X = iris.data[:, :2] # we only take the first two features. We could\n", " # avoid this ugly slicing by using a two-dim dataset\n", "y = iris.target\n", "\n", "h = .02 # step size in the mesh\n", "\n", "# we create an instance of SVM and fit out data. We do not scale our\n", "# data since we want to plot the support vectors\n", "C = 1.0 # SVM regularization parameter\n", "svc = svm.SVC(kernel='linear', C=C).fit(X, y)\n", "rbf_svc = svm.SVC(kernel='rbf', gamma=0.7, C=C).fit(X, y)\n", "poly_svc = svm.SVC(kernel='poly', degree=3, C=C).fit(X, y)\n", "lin_svc = svm.LinearSVC(C=C).fit(X, y)\n", "\n", "# create a mesh to plot in\n", "x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1\n", "y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1\n", "xx, yy = np.meshgrid(np.arange(x_min, x_max, h),\n", " np.arange(y_min, y_max, h))\n", "\n", "# title for the plots\n", "titles = ['SVC with linear kernel',\n", " 'LinearSVC (linear kernel)',\n", " 'SVC with RBF kernel',\n", " 'SVC with polynomial (degree 3) kernel']\n", "\n", "mpl.rcParams['figure.figsize']=[12,12]\n", "for i, clf in enumerate((svc, lin_svc, rbf_svc, poly_svc)):\n", " # Plot the decision boundary. For that, we will assign a color to each\n", " # point in the mesh [x_min, m_max]x[y_min, y_max].\n", " plt.subplot(2, 2, i + 1)\n", " plt.subplots_adjust(wspace=0.4, hspace=0.4)\n", "\n", " Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])\n", "\n", " # Put the result into a color plot\n", " Z = Z.reshape(xx.shape)\n", " plt.contourf(xx, yy, Z, cmap=plt.cm.Paired, alpha=0.8)\n", "\n", " # Plot also the training points\n", " plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.Paired)\n", " plt.xlabel('Sepal length')\n", " plt.ylabel('Sepal width')\n", " plt.xlim(xx.min(), xx.max())\n", " plt.ylim(yy.min(), yy.max())\n", " plt.xticks(())\n", " plt.yticks(())\n", " plt.title(titles[i])\n", "\n", "plt.show()" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "heading", "level": 2, "metadata": {}, "source": [ "Unsupervised learning" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In the case of unsupervised learning, the idea is to find a hidden pattern in the data. In a sense, you can think of unsupervised learning as a means of discovering labels from the data itself. Unsupervised learning comprises tasks such as dimensionality reduction, clustering, and density estimation. \n", "Approaches to unsupervised learning are supported in `scikit-learn` such as:\n", "\n", "- K-means clustering\n", "- Principal Component Analysis\n", "- Neural networks" ] }, { "cell_type": "heading", "level": 2, "metadata": {}, "source": [ "Clustering" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It is the task of grouping a set of objects together such as they are similar to one another in some way." ] }, { "cell_type": "heading", "level": 2, "metadata": {}, "source": [ "Principal Component Analysis(PCA)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "PCA is extensively used in reducing dimensionality of data. PCA finds the directions in which the data is not flat and it can reduce the dimensionality of the data by projecting on a subspace. It finds the combination of variables that exhibit the maximum variance" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "![](http://shapeofdata.files.wordpress.com/2013/02/pca22.png)" ] }, { "cell_type": "code", "collapsed": false, "input": [ "mpl.rcParams['figure.figsize']=[12,5]\n", "np.random.seed(1)\n", "X = np.dot(np.random.random(size=(2, 2)), np.random.normal(size=(2, 200))).T\n", "plt.plot(X[:, 0], X[:, 1], 'og')\n", "plt.axis('equal')" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "from sklearn.decomposition import PCA\n", "pca = PCA(n_components=2)\n", "pca.fit(X)\n", "print(pca.explained_variance_)\n", "print(pca.components_)" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "mpl.rcParams['figure.figsize']=[12,5]\n", "plt.plot(X[:, 0], X[:, 1], 'og', alpha=0.3)\n", "plt.axis('equal')\n", "for length, vector in zip(pca.explained_variance_, pca.components_):\n", " v = vector * 3 * np.sqrt(length)\n", " plt.plot([0, v[0]], [0, v[1]], '-k', lw=3)" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "clf = PCA(0.95) # if we only keep 95% of the variance\n", "X_trans = clf.fit_transform(X)\n", "print(X.shape)\n", "print(X_trans.shape)" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "X_new = clf.inverse_transform(X_trans)\n", "plt.plot(X[:, 0], X[:, 1], 'og', alpha=0.2)\n", "plt.plot(X_new[:, 0], X_new[:, 1], 'og', alpha=0.8)\n", "plt.axis('equal');" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "X, y = iris.data, iris.target\n", "from sklearn.decomposition import PCA\n", "pca = PCA(n_components=2)\n", "pca.fit(X)\n", "X_reduced = pca.transform(X)\n", "print (\"Reduced dataset shape:\", X_reduced.shape)\n", "\n", "import pylab as pl\n", "mpl.rcParams['figure.figsize']=[12,3]\n", "pl.scatter(X_reduced[:, 0], X_reduced[:, 1], c=y)\n", "\n", "print (\"Meaning of the 2 components:\")\n", "for component in pca.components_:\n", " print(\" + \".join(\"%.3f x %s\" % (value, name)\n", " for value, name in zip(component,\n", " iris.feature_names)))" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "heading", "level": 2, "metadata": {}, "source": [ "K-means" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The simplest clustering algorithm is k-means. This divides a set into k clusters, assigning each observation to a cluster so as to minimize the distance of that observation (in n-dimensional space) to the cluster\u2019s mean; the means are then recomputed. This operation is run iteratively until the clusters converge, for a maximum for max_iter rounds." ] }, { "cell_type": "code", "collapsed": false, "input": [ "X, y = iris.data, iris.target\n", "from sklearn.cluster import KMeans\n", "k_means = KMeans(n_clusters=5, random_state=0) # Fixing the RNG in kmeans\n", "k_means.fit(X)\n", "y_pred = k_means.predict(X)\n", "\n", "pl.scatter(X_reduced[:, 0], X_reduced[:, 1], c=y_pred);\n" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "print(__doc__)\n", "\n", "\n", "# Code source: Ga\u00ebl Varoquaux\n", "# Modified for documentation by Jaques Grobler\n", "# License: BSD 3 clause\n", "\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "from mpl_toolkits.mplot3d import Axes3D\n", "\n", "\n", "from sklearn.cluster import KMeans\n", "from sklearn import datasets\n", "\n", "np.random.seed(5)\n", "\n", "centers = [[1, 1], [-1, -1], [1, -1]]\n", "iris = datasets.load_iris()\n", "X = iris.data\n", "y = iris.target\n", "\n", "estimators = {'k_means_iris_3': KMeans(n_clusters=3),\n", " 'k_means_iris_8': KMeans(n_clusters=8),\n", " 'k_means_iris_bad_init': KMeans(n_clusters=3, n_init=1,\n", " init='random')}\n", "\n", "\n", "fignum = 1\n", "for name, est in estimators.items():\n", " fig = plt.figure(fignum, figsize=(4, 3))\n", " plt.clf()\n", " ax = Axes3D(fig, rect=[0, 0, .95, 1], elev=48, azim=134)\n", "\n", " plt.cla()\n", " est.fit(X)\n", " labels = est.labels_\n", "\n", " ax.scatter(X[:, 3], X[:, 0], X[:, 2], c=labels.astype(np.float))\n", "\n", " ax.w_xaxis.set_ticklabels([])\n", " ax.w_yaxis.set_ticklabels([])\n", " ax.w_zaxis.set_ticklabels([])\n", " ax.set_xlabel('Petal width')\n", " ax.set_ylabel('Sepal length')\n", " ax.set_zlabel('Petal length')\n", " fignum = fignum + 1\n", "\n", "# Plot the ground truth\n", "fig = plt.figure(fignum, figsize=(4, 3))\n", "plt.clf()\n", "ax = Axes3D(fig, rect=[0, 0, .95, 1], elev=48, azim=134)\n", "\n", "plt.cla()\n", "\n", "for name, label in [('Setosa', 0),\n", " ('Versicolour', 1),\n", " ('Virginica', 2)]:\n", " ax.text3D(X[y == label, 3].mean(),\n", " X[y == label, 0].mean() + 1.5,\n", " X[y == label, 2].mean(), name,\n", " horizontalalignment='center',\n", " bbox=dict(alpha=.5, edgecolor='w', facecolor='w'))\n", "# Reorder the labels to have colors matching the cluster results\n", "y = np.choose(y, [1, 2, 0]).astype(np.float)\n", "ax.scatter(X[:, 3], X[:, 0], X[:, 2], c=y)\n", "\n", "ax.w_xaxis.set_ticklabels([])\n", "ax.w_yaxis.set_ticklabels([])\n", "ax.w_zaxis.set_ticklabels([])\n", "ax.set_xlabel('Petal width')\n", "ax.set_ylabel('Sepal length')\n", "ax.set_zlabel('Petal length')\n", "plt.show()" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Neural Networks**\n", "\n", "In general Neural Networks or Artificial Neural Networks(ANNs) as they are commonly known, are used both in machine learning and pattern recognition.\n", "\n", "For example, a neural network for handwriting recognition is defined by a set of input neurons which may be activated by the pixels of an input image. After being weighted and transformed by a function (determined by the network's designer), the activations of these neurons are then passed on to other neurons. This process is repeated until finally, an output neuron is activated. This determines which character was read.\n", "\n", "Typically, the ANN is represented by three layers. The hidden layer's job is to transform the inputs into something that the output layer can use. The output of a neuron is a function of the weighted sum of the inputs plus a bias.\n", "\n", "![](http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/img/example_network.svg)\n", "\n", "A restricted Boltzmann machine (RBM) is a generative stochastic artificial neural network that can learn a probability distribution over its set of inputs. This is the example of unsupervised learning which is supported by `scikit-learn`." ] }, { "cell_type": "heading", "level": 3, "metadata": {}, "source": [ "Some Real World Examples" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Image compression using clustering**" ] }, { "cell_type": "code", "collapsed": false, "input": [ "from sklearn.datasets import load_sample_image\n", "c1= load_sample_image(\"china.jpg\")\n", "plt.imshow(c1);" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "import scipy.misc as misc\n", "from sklearn.datasets import load_sample_image\n", "c = load_sample_image(\"china.jpg\")\n", "#plt.imshow(c)\n", "c = c.astype(np.float32)\n", "#plt.imshow(c)\n", "X = c.reshape((-1, 1)) # Reshaping it into an array\n", "k_means = KMeans(n_clusters=5)\n", "k_means.fit(X)" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "values = k_means.cluster_centers_.squeeze()\n", "labels = k_means.labels_\n", "c_compressed = np.choose(labels, values)\n", "c_compressed.shape = c.shape" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "mpl.rcParams['figure.figsize']=[12,5.5]\n", "fig, (ax0, ax1) = plt.subplots(ncols=2, sharex=True)\n", "ax0.imshow(c)\n", "ax1.imshow(c_compressed) \n", "fig.show()" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**K-means clustering on PCA reduced Data**" ] }, { "cell_type": "code", "collapsed": false, "input": [ "print(__doc__)\n", "\n", "from time import time\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "\n", "from sklearn import metrics\n", "from sklearn.cluster import KMeans\n", "from sklearn.datasets import load_digits\n", "from sklearn.decomposition import PCA\n", "from sklearn.preprocessing import scale\n", "\n", "np.random.seed(42)\n", "\n", "digits = load_digits()\n", "data = scale(digits.data)\n", "\n", "n_samples, n_features = data.shape\n", "n_digits = len(np.unique(digits.target))\n", "labels = digits.target\n", "\n", "sample_size = 300\n", "\n", "print(\"n_digits: %d, \\t n_samples %d, \\t n_features %d\"\n", " % (n_digits, n_samples, n_features))\n", "\n", "\n", "print(79 * '_')\n", "print('% 9s' % 'init'\n", " ' time inertia homo compl v-meas ARI AMI silhouette')\n", "\n", "\n", "def bench_k_means(estimator, name, data):\n", " t0 = time()\n", " estimator.fit(data)\n", " print('% 9s %.2fs %i %.3f %.3f %.3f %.3f %.3f %.3f'\n", " % (name, (time() - t0), estimator.inertia_,\n", " metrics.homogeneity_score(labels, estimator.labels_),\n", " metrics.completeness_score(labels, estimator.labels_),\n", " metrics.v_measure_score(labels, estimator.labels_),\n", " metrics.adjusted_rand_score(labels, estimator.labels_),\n", " metrics.adjusted_mutual_info_score(labels, estimator.labels_),\n", " metrics.silhouette_score(data, estimator.labels_,\n", " metric='euclidean',\n", " sample_size=sample_size)))\n", "\n", "bench_k_means(KMeans(init='k-means++', n_clusters=n_digits, n_init=10),\n", " name=\"k-means++\", data=data)\n", "\n", "bench_k_means(KMeans(init='random', n_clusters=n_digits, n_init=10),\n", " name=\"random\", data=data)\n", "\n", "# in this case the seeding of the centers is deterministic, hence we run the\n", "# kmeans algorithm only once with n_init=1\n", "pca = PCA(n_components=n_digits).fit(data)\n", "bench_k_means(KMeans(init=pca.components_, n_clusters=n_digits, n_init=1),\n", " name=\"PCA-based\",\n", " data=data)\n", "print(79 * '_')\n", "\n", "###############################################################################\n", "# Visualize the results on PCA-reduced data\n", "\n", "reduced_data = PCA(n_components=2).fit_transform(data)\n", "kmeans = KMeans(init='k-means++', n_clusters=n_digits, n_init=10)\n", "kmeans.fit(reduced_data)\n", "\n", "# Step size of the mesh. Decrease to increase the quality of the VQ.\n", "h = .02 # point in the mesh [x_min, m_max]x[y_min, y_max].\n", "\n", "# Plot the decision boundary. For that, we will assign a color to each\n", "x_min, x_max = reduced_data[:, 0].min() + 1, reduced_data[:, 0].max() - 1\n", "y_min, y_max = reduced_data[:, 1].min() + 1, reduced_data[:, 1].max() - 1\n", "xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))\n", "\n", "# Obtain labels for each point in mesh. Use last trained model.\n", "Z = kmeans.predict(np.c_[xx.ravel(), yy.ravel()])\n", "\n", "# Put the result into a color plot\n", "Z = Z.reshape(xx.shape)\n", "plt.figure(1)\n", "plt.clf()\n", "plt.imshow(Z, interpolation='nearest',\n", " extent=(xx.min(), xx.max(), yy.min(), yy.max()),\n", " cmap=plt.cm.Paired,\n", " aspect='auto', origin='lower')\n", "\n", "plt.plot(reduced_data[:, 0], reduced_data[:, 1], 'k.', markersize=2)\n", "# Plot the centroids as a white X\n", "centroids = kmeans.cluster_centers_\n", "plt.scatter(centroids[:, 0], centroids[:, 1],\n", " marker='x', s=169, linewidths=3,\n", " color='w', zorder=10)\n", "plt.title('K-means clustering on the digits dataset (PCA-reduced data)\\n'\n", " 'Centroids are marked with white cross')\n", "plt.xlim(x_min, x_max)\n", "plt.ylim(y_min, y_max)\n", "plt.xticks(())\n", "plt.yticks(())\n", "plt.show()\n" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "heading", "level": 4, "metadata": {}, "source": [ "A clustering example from the real world" ] }, { "cell_type": "code", "collapsed": false, "input": [ "print(__doc__)\n", "\n", "# Author: Gael Varoquaux gael.varoquaux@normalesup.org\n", "# License: BSD 3 clause\n", "\n", "import datetime\n", "\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "from matplotlib import finance\n", "from matplotlib.collections import LineCollection\n", "\n", "from sklearn import cluster, covariance, manifold\n", "\n", "###############################################################################\n", "# Retrieve the data from Internet\n", "\n", "# Choose a time period reasonably calm (not too long ago so that we get\n", "# high-tech firms, and before the 2008 crash)\n", "d1 = datetime.datetime(2003, 1, 1)\n", "d2 = datetime.datetime(2010, 1, 1)\n", "\n", "# kraft symbol has now changed from KFT to MDLZ in yahoo\n", "symbol_dict = {\n", " 'TOT': 'Total',\n", " 'XOM': 'Exxon',\n", " 'CVX': 'Chevron',\n", " 'COP': 'ConocoPhillips',\n", " 'VLO': 'Valero Energy',\n", " 'MSFT': 'Microsoft',\n", " 'IBM': 'IBM',\n", " 'TWX': 'Time Warner',\n", " 'CMCSA': 'Comcast',\n", " 'CVC': 'Cablevision',\n", " 'YHOO': 'Yahoo',\n", " 'DELL': 'Dell',\n", " 'HPQ': 'HP',\n", " 'AMZN': 'Amazon',\n", " 'TM': 'Toyota',\n", " 'CAJ': 'Canon',\n", " 'MTU': 'Mitsubishi',\n", " 'SNE': 'Sony',\n", " 'F': 'Ford',\n", " 'HMC': 'Honda',\n", " 'NAV': 'Navistar',\n", " 'NOC': 'Northrop Grumman',\n", " 'BA': 'Boeing',\n", " 'KO': 'Coca Cola',\n", " 'MMM': '3M',\n", " 'MCD': 'Mc Donalds',\n", " 'PEP': 'Pepsi',\n", " 'MDLZ': 'Kraft Foods',\n", " 'K': 'Kellogg',\n", " 'UN': 'Unilever',\n", " 'MAR': 'Marriott',\n", " 'PG': 'Procter Gamble',\n", " 'CL': 'Colgate-Palmolive',\n", " 'GE': 'General Electrics',\n", " 'WFC': 'Wells Fargo',\n", " 'JPM': 'JPMorgan Chase',\n", " 'AIG': 'AIG',\n", " 'AXP': 'American express',\n", " 'BAC': 'Bank of America',\n", " 'GS': 'Goldman Sachs',\n", " 'AAPL': 'Apple',\n", " 'SAP': 'SAP',\n", " 'CSCO': 'Cisco',\n", " 'TXN': 'Texas Instruments',\n", " 'XRX': 'Xerox',\n", " 'LMT': 'Lockheed Martin',\n", " 'WMT': 'Wal-Mart',\n", " 'WAG': 'Walgreen',\n", " 'HD': 'Home Depot',\n", " 'GSK': 'GlaxoSmithKline',\n", " 'PFE': 'Pfizer',\n", " 'SNY': 'Sanofi-Aventis',\n", " 'NVS': 'Novartis',\n", " 'KMB': 'Kimberly-Clark',\n", " 'R': 'Ryder',\n", " 'GD': 'General Dynamics',\n", " 'RTN': 'Raytheon',\n", " 'CVS': 'CVS',\n", " 'CAT': 'Caterpillar',\n", " 'DD': 'DuPont de Nemours'}\n", "\n", "symbols, names = np.array(list(symbol_dict.items())).T\n", "\n", "quotes = [finance.quotes_historical_yahoo(symbol, d1, d2, asobject=True)\n", " for symbol in symbols]\n", "\n", "open = np.array([q.open for q in quotes]).astype(np.float)\n", "close = np.array([q.close for q in quotes]).astype(np.float)\n", "\n", "# The daily variations of the quotes are what carry most information\n", "variation = close - open\n", "\n", "###############################################################################\n", "# Learn a graphical structure from the correlations\n", "edge_model = covariance.GraphLassoCV()\n", "\n", "# standardize the time series: using correlations rather than covariance\n", "# is more efficient for structure recovery\n", "X = variation.copy().T\n", "X /= X.std(axis=0)\n", "edge_model.fit(X)\n", "\n", "###############################################################################\n", "# Cluster using affinity propagation\n", "\n", "_, labels = cluster.affinity_propagation(edge_model.covariance_)\n", "n_labels = labels.max()\n", "\n", "for i in range(n_labels + 1):\n", " print('Cluster %i: %s' % ((i + 1), ', '.join(names[labels == i])))\n", "\n", "###############################################################################\n", "# Find a low-dimension embedding for visualization: find the best position of\n", "# the nodes (the stocks) on a 2D plane\n", "\n", "# We use a dense eigen_solver to achieve reproducibility (arpack is\n", "# initiated with random vectors that we don't control). In addition, we\n", "# use a large number of neighbors to capture the large-scale structure.\n", "node_position_model = manifold.LocallyLinearEmbedding(\n", " n_components=2, eigen_solver='dense', n_neighbors=6)\n", "\n", "embedding = node_position_model.fit_transform(X.T).T\n", "\n", "###############################################################################\n", "# Visualization\n", "plt.figure(1, facecolor='w', figsize=(10, 8))\n", "plt.clf()\n", "ax = plt.axes([0., 0., 1., 1.])\n", "plt.axis('off')\n", "\n", "# Display a graph of the partial correlations\n", "partial_correlations = edge_model.precision_.copy()\n", "d = 1 / np.sqrt(np.diag(partial_correlations))\n", "partial_correlations *= d\n", "partial_correlations *= d[:, np.newaxis]\n", "non_zero = (np.abs(np.triu(partial_correlations, k=1)) > 0.02)\n", "\n", "# Plot the nodes using the coordinates of our embedding\n", "plt.scatter(embedding[0], embedding[1], s=100 * d ** 2, c=labels,\n", " cmap=plt.cm.spectral)\n", "\n", "# Plot the edges\n", "start_idx, end_idx = np.where(non_zero)\n", "#a sequence of (*line0*, *line1*, *line2*), where::\n", "# linen = (x0, y0), (x1, y1), ... (xm, ym)\n", "segments = [[embedding[:, start], embedding[:, stop]]\n", " for start, stop in zip(start_idx, end_idx)]\n", "values = np.abs(partial_correlations[non_zero])\n", "lc = LineCollection(segments,\n", " zorder=0, cmap=plt.cm.hot_r,\n", " norm=plt.Normalize(0, .7 * values.max()))\n", "lc.set_array(values)\n", "lc.set_linewidths(15 * values)\n", "ax.add_collection(lc)\n", "\n", "# Add a label to each node. The challenge here is that we want to\n", "# position the labels to avoid overlap with other labels\n", "for index, (name, label, (x, y)) in enumerate(\n", " zip(names, labels, embedding.T)):\n", "\n", " dx = x - embedding[0]\n", " dx[index] = 1\n", " dy = y - embedding[1]\n", " dy[index] = 1\n", " this_dx = dx[np.argmin(np.abs(dy))]\n", " this_dy = dy[np.argmin(np.abs(dx))]\n", " if this_dx > 0:\n", " horizontalalignment = 'left'\n", " x = x + .002\n", " else:\n", " horizontalalignment = 'right'\n", " x = x - .002\n", " if this_dy > 0:\n", " verticalalignment = 'bottom'\n", " y = y + .002\n", " else:\n", " verticalalignment = 'top'\n", " y = y - .002\n", " plt.text(x, y, name, size=10,\n", " horizontalalignment=horizontalalignment,\n", " verticalalignment=verticalalignment,\n", " bbox=dict(facecolor='w',\n", " edgecolor=plt.cm.spectral(label / float(n_labels)),\n", " alpha=.6))\n", "\n", "plt.xlim(embedding[0].min() - .15 * embedding[0].ptp(),\n", " embedding[0].max() + .10 * embedding[0].ptp(),)\n", "plt.ylim(embedding[1].min() - .03 * embedding[1].ptp(),\n", " embedding[1].max() + .03 * embedding[1].ptp())\n", "\n", "plt.show()" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Digit Classification using Neural Networks**" ] }, { "cell_type": "code", "collapsed": false, "input": [ "from __future__ import print_function\n", "\n", "print(__doc__)\n", "\n", "# Authors: Yann N. Dauphin, Vlad Niculae, Gabriel Synnaeve\n", "# License: BSD\n", "\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "\n", "from scipy.ndimage import convolve\n", "from sklearn import linear_model, datasets, metrics\n", "from sklearn.cross_validation import train_test_split\n", "from sklearn.neural_network import BernoulliRBM\n", "from sklearn.pipeline import Pipeline\n", "\n", "\n", "###############################################################################\n", "# Setting up\n", "\n", "def nudge_dataset(X, Y):\n", " \"\"\"\n", " This produces a dataset 5 times bigger than the original one,\n", " by moving the 8x8 images in X around by 1px to left, right, down, up\n", " \"\"\"\n", " direction_vectors = [\n", " [[0, 1, 0],\n", " [0, 0, 0],\n", " [0, 0, 0]],\n", "\n", " [[0, 0, 0],\n", " [1, 0, 0],\n", " [0, 0, 0]],\n", "\n", " [[0, 0, 0],\n", " [0, 0, 1],\n", " [0, 0, 0]],\n", "\n", " [[0, 0, 0],\n", " [0, 0, 0],\n", " [0, 1, 0]]]\n", "\n", " shift = lambda x, w: convolve(x.reshape((8, 8)), mode='constant',\n", " weights=w).ravel()\n", " X = np.concatenate([X] +\n", " [np.apply_along_axis(shift, 1, X, vector)\n", " for vector in direction_vectors])\n", " Y = np.concatenate([Y for _ in range(5)], axis=0)\n", " return X, Y\n", "\n", "# Load Data\n", "digits = datasets.load_digits()\n", "X = np.asarray(digits.data, 'float32')\n", "X, Y = nudge_dataset(X, digits.target)\n", "X = (X - np.min(X, 0)) / (np.max(X, 0) + 0.0001) # 0-1 scaling\n", "\n", "X_train, X_test, Y_train, Y_test = train_test_split(X, Y,\n", " test_size=0.2,\n", " random_state=0)\n", "\n", "# Models we will use\n", "logistic = linear_model.LogisticRegression()\n", "rbm = BernoulliRBM(random_state=0, verbose=True)\n", "\n", "classifier = Pipeline(steps=[('rbm', rbm), ('logistic', logistic)])\n", "\n", "###############################################################################\n", "# Training\n", "\n", "# Hyper-parameters. These were set by cross-validation,\n", "# using a GridSearchCV. Here we are not performing cross-validation to\n", "# save time.\n", "rbm.learning_rate = 0.06\n", "rbm.n_iter = 20\n", "# More components tend to give better prediction performance, but larger\n", "# fitting time\n", "rbm.n_components = 100\n", "logistic.C = 6000.0\n", "\n", "# Training RBM-Logistic Pipeline\n", "classifier.fit(X_train, Y_train)\n", "\n", "# Training Logistic regression\n", "logistic_classifier = linear_model.LogisticRegression(C=100.0)\n", "logistic_classifier.fit(X_train, Y_train)\n", "\n", "###############################################################################\n", "# Evaluation\n", "\n", "print()\n", "print(\"Logistic regression using RBM features:\\n%s\\n\" % (\n", " metrics.classification_report(\n", " Y_test,\n", " classifier.predict(X_test))))\n", "\n", "print(\"Logistic regression using raw pixel features:\\n%s\\n\" % (\n", " metrics.classification_report(\n", " Y_test,\n", " logistic_classifier.predict(X_test))))\n", "\n", "###############################################################################\n", "# Plotting\n", "\n", "plt.figure(figsize=(4.2, 4))\n", "for i, comp in enumerate(rbm.components_):\n", " plt.subplot(10, 10, i + 1)\n", " plt.imshow(comp.reshape((8, 8)), cmap=plt.cm.gray_r,\n", " interpolation='nearest')\n", " plt.xticks(())\n", " plt.yticks(())\n", "plt.suptitle('100 components extracted by RBM', fontsize=16)\n", "plt.subplots_adjust(0.08, 0.02, 0.92, 0.85, 0.08, 0.23)\n", "\n", "plt.show()" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**References**\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- [scikit-learn Documentation](http://scikit-learn.org/stable/)\n", "- [Machine Learning using Astronomical Data](http://www.astroml.org/)\n", "- [scikit-learn tutorial](http://www.astroml.org/sklearn_tutorial/)\n", "- [Stock Market Example](http://scikit-learn.org/stable/auto_examples/applications/plot_stock_market.html#example-applications-plot-stock-market-py)\n", "- [PyCon14](http://nbviewer.ipython.org/github/jakevdp/sklearn_pycon2014/blob/master/notebooks/04_supervised_in_depth.ipynb)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Credits\n", "\n", "Lakshmi Rao, Abhishek Sharma, and Neal Davis developed these materials for [Computational Science and Engineering](http://cse.illinois.edu/) at the University of Illinois at Urbana\u2013Champaign.\n", "\n", "\n", "This content is available under a [Creative Commons Attribution 3.0 Unported License](https://creativecommons.org/licenses/by/3.0/).\n", "\n", "[![](https://bytebucket.org/davis68/resources/raw/f7c98d2b95e961fae257707e22a58fa1a2c36bec/logos/baseline_cse_wdmk.png?token=be4cc41d4b2afe594f5b1570a3c5aad96a65f0d6)](http://cse.illinois.edu/)" ] } ], "metadata": {} } ] }