{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Adult Census Income - A classification problem\n", "Using income data of the adult US population, we want to classify who earns more or less than 50k USD" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Prerequirements" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "General imports needed" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import time\n", "import pandas as pd\n", "import logging as logging\n", "import pprint\n", "import plotly\n", "plotly.offline.init_notebook_mode(connected=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Pailab specific imports" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "from pailab import MLRepo, MeasureConfiguration, FIRST_VERSION, LAST_VERSION" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Basic configuration" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "logging.basicConfig(level=logging.FATAL)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Read the data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this notebook we use the adult data from the census 1994 it can be found in [UCI Machine Learning Repository](http://archive.ics.uci.edu/ml/index.php) or from [Kaggle](https://www.kaggle.com/uciml/adult-census-income). The data set consists of 48842 observations each with 14 attributes. The table consists of qualitative (e.g. workclass, education) and quantitative attributes." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "data = pd.read_csv('adult.csv')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Some basic adjustments, which are not part of the preprocessing" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# change the output format\n", "data['income']=data['income'].map({'<=50K': 0, '>50K': 1, '<=50K.': 0, '>50K.': 1})" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "convert categorical features into dummy variable" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "categorical_columns = ['workclass', 'education', 'marital.status', 'occupation', 'relationship', 'race', 'sex', 'native.country']\n", "data = pd.get_dummies(data, columns=categorical_columns)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Create a new repository\n", "We first create a new repository for our task. The repository is the central key around all functionality is built. Similar to a repository used for source control in classical software development it contains all data and algorithms needed for the machine learning task. The repository needs storages for \n", "- scripts containing the machine learning algorithms and interfaces,\n", "- numerical objects such as arrays and matrices representing data, e.g. input data, data from the valuation of the models,\n", "- json documents representing parameters, e.g. training parameter, model parameter.\n", "\n", "To keep things simple, we just start using in memory storages. Note that the used memory interfaces are except for testing and playing around not be the first choice, since when ending the session, everything will be lost...\n", "\n", "In addition to the storages the repository needs a reference to a JobRunner which the platform can use to execute machine learning jobs. For this example we use the most simple one, executing everything sequential in the same thread, the repository runs in." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# setting up the repository\n", "config = None\n", "\n", "# set this to True if you would like to test the git_handler together with a \n", "# hdf handler for the big data parts instead of the memory handler\n", "if False: \n", " config = {'user': 'test_user',\n", " 'workspace': 'c:/temp',\n", " 'repo_store': \n", " {\n", " 'type': 'git_handler', \n", " 'config': {\n", " 'folder': 'c:/temp', \n", " 'file_format': 'pck'\n", " }\n", " },\n", " 'numpy_store':\n", " {\n", " 'type': 'hdf_handler',\n", " 'config':{\n", " 'folder': 'c:/temp/hdf',\n", " 'version_files': True\n", " }\n", " }\n", " }\n", "\n", "ml_repo = MLRepo( user = 'test_user', config=config)\n", "job_runner = ml_repo._job_runner" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Add a tree\n", "To navigate in a simple way over all objects, one can add a so-called tree to the repository. The tree allows one to use auto completion to acces objcts and respectiv methods." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "from pailab.tools.tree import MLTree\n", "MLTree.add_tree(ml_repo)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Adding data\n", "The data in the repository is handled by two different data objects:\n", "- RawData is the object containing real data.\n", "- DataSet is the object conaining the logical data, i.e. a reference to a RawData object together with a specification, which data from the RawData will be used. Here, one can specify a fixed version of the underlying RawData object (then changes to the RawData will not affect the derived DataSet) or a fixed or floating subset of the RawData by defining start and endindex cutting the derived data just out of the original data.\n", "\n", "Normally one will add RawData and then define DataSets which are used to train or test a model which is exactly the way shown in the following. \n", "The target variable is 'income'. By default all other variables are used for as input_variables." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ml_repo.tree.raw_data.add('adult_census_data', data, target_variables=['income'])" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Add RawData. A convenient way to add RawData is simply to use the method add on the raw_data collection.\n", "# This method just takes a pandas dataframe and the specification, which columns belong to the input \n", "#and which to the targets.\n", "ml_repo.tree.raw_data.add('adult_census_data', data, target_variables = ['income'])\n", "# based on the raw data we now define training and test sets\n", "ml_repo.tree.training_data.add('training_sample', ml_repo.tree.raw_data.adult_census_data(), 0, 1000)\n", "ml_repo.tree.test_data.add('test_sample', ml_repo.tree.raw_data.adult_census_data(), 1001, None)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ml_repo.tree.training_data.training_sample.history()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "When creating the DataSet we have to set two important informations for the repository, given as a dictionary:\n", "- The object name. Each object in the repository needs to have a unique name in the repository.\n", "- The object type which gives. In our example here we say that we specify that the DataSet are training and test data. Note that on can have only one training data object pre repository while the repository can obtain many different test data sets.\n", "\n", "Some may wonder what is now stored in *version_list*.\n", "** Adding an object (independent if it is a data object or some other object such as a parameter), the object gets a version number and no object will be removed, adding just adds a new version.** The add method returns a dictionary of the object names together with their version number." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Adding a model\n", "The next step to do machine learning would be to define a model which will be used in the repository. A model consists of the following pieces\n", "- a skript where the code for the model valuation is defined together with the function name of the evaluation method\n", "- a skript where the code for the model training is defined together with th function nam of the training method\n", "- a model parameter object defining the model parameter and which must have implemented the correct interface so that it can be used within the repository (see the documentation on integrating new objects, normally there is not more to do then just simply add *@repo_object_init()* to the line above your *__init__* method)\n", "- a training parameter object defining training parameters (such as number of optimization steps etc.), if necessary for your algorithms (this oen is optional)\n", "\n", "** SKLearn models as an example**\n", "\n", "We do not have to define the pieces defined above, if we use the sklearn module. Instead we can use the pailab.externals.sklearn module interfacing \n", "the sklearn package so that this can be used within the repository. This interface provides a simple method (add_model) to add an arbitrary sklearn model as a model which can be handled by the repository. This method adds a bunch of repo objects to the repository (according to the pieces described above):\n", "- An object defining the function to be called to evaluate the model\n", "- An object defining the function to be called to train the model\n", "- An object defining the model\n", "- An object defining the model parameter\n", "For the following we just use a DecisionTree as our model." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### First add the preprocessing" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# import pailab.externals.pandas_interface as pandas_interface\n", "\n", "# categorical_columns = ['workclass', 'education', 'marital.status', 'occupation', 'relationship', 'race', 'sex', 'native.country']\n", "# pandas_interface.add_preprocessor(ml_repo, pd.get_dummies, preprocessor_name='Pandas_Get_Dummies', preprocessor_param={columns=categorical_columns})" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import pailab.externals.numpy_interface as numpy_interface\n", "select_columns = ['age', 'fnlwgt', 'education.num', 'capital.gain', 'capital.loss', 'hours.per.week', 'workclass_?','workclass_Local-gov', 'workclass_Private', 'workclass_Self-emp-not-inc', 'education_HS-grad', 'education_Some-college', 'education_Bachelors', 'education_Masters',\n", "'marital.status_Divorced', 'marital.status_Married-civ-spouse', 'marital.status_Never-married', 'marital.status_Separated', \n", "'occupation_Prof-specialty', 'occupation_Craft-repair', 'occupation_Exec-managerial', 'occupation_Adm-clerical', 'relationship_Husband', 'relationship_Not-in-family', 'relationship_Own-child', 'relationship_Unmarried', 'race_Asian-Pac-Islander', 'race_Black', 'race_White', 'sex_Female', \n", "'native.country_United-States', 'native.country_Mexico', 'native.country_?', 'native.country_Philippines']\n", "numpy_interface.add_preprocessor_select_columns(ml_repo, preprocessor_name='Numpy_Select_Columns', \n", " preprocessor_param={'columns':select_columns})" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import pailab.externals.sklearn_interface as sklearn_interface\n", "from sklearn.preprocessing import StandardScaler\n", "\n", "sklearn_interface.add_preprocessor(ml_repo, StandardScaler(), preprocessor_name='SKLStandardScaler')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Add the model" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from sklearn.linear_model import LogisticRegression\n", "sklearn_interface.add_model(ml_repo, LogisticRegression(), preprocessors=['Numpy_Select_Columns', 'SKLStandardScaler'], model_param={'max_iter': 1000})" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Train the model\n", "Now, model taining is very simple, since you have defined training and testing data as well as methods to value and fit your model and the model parameter.\n", "So, you can just call *run_training* on the repository, and the training is perfomred automatically.\n", "The training job is executed via the JobRunner you specified setting up the repository. All method of the repository involving jobs return the job id when adding the job to the JobRunner so that you can control the status of the task and see if it sucessfully finished." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "job_id = ml_repo.run_training() \n", "print(job_id)\n", "job_info = job_runner.get_info(job_id[0], job_id[1])" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "## Run evaluation\n", "To measure errors and to provide plots the model must be evaluated on all test and training datasets." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ml_repo.tree.models.LogisticRegression.model.load()\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ml_repo.tree.models.LogisticRegression.model.obj.to_dict()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "job_id = ml_repo.run_evaluation()\n", "# print information about the job\n", "info = job_runner.get_info(job_id[0][0], job_id[0][1]) \n", "print(str(info))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Add and compute measures" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ml_repo.add_measure(MeasureConfiguration.MAX)\n", "ml_repo.add_measure(MeasureConfiguration.R2)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "job_ids = ml_repo.run_measures()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ml_repo.tree.models.LogisticRegression.measures.test_sample.r2.load()\n", "print(str(ml_repo.tree.models.LogisticRegression.measures.test_sample.r2.obj.value))" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "# Changing the preprocessor\n", "Introducing a new version of the preprocessor to the repository. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "sklearn_interface.add_preprocessor(\n", " ml_repo, StandardScaler(), preprocessor_name='SKLStandardScaler', preprocessor_param={'with_mean': False})" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "After we have changed the model parameter, we use the *checker* submodules *run* method for open tasks/inconsistencies. This method is called for the repository. If nothing is specified, all labeled models will be checked.\n", "Applying the method to the latest model we see that the output shows that the models last version has been calibrated using a different model parameter version then the current version. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import pailab.tools.checker as checker\n", "results = checker.run(ml_repo)\n", "pprint.pprint(results)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": false }, "outputs": [], "source": [ "job_id = ml_repo.run_training(run_descendants=True)# we use run_descendants so that the issues with th measures are resolved too\n", "print(job_id)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "results = checker.run(ml_repo)\n", "pprint.pprint(results)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [] } ], "metadata": { "hide_input": false, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.1" }, "toc": { "base_numbering": 1, "colors": { "hover_highlight": "#DAA520", "navigate_num": "#000000", "navigate_text": "#333333", "running_highlight": "#FF0000", "selected_highlight": "#FFD700", "sidebar_border": "#EEEEEE", "wrapper_background": "#FFFFFF" }, "moveMenuLeft": true, "nav_menu": { "height": "264px", "width": "252px" }, "navigate_menu": true, "number_sections": true, "sideBar": true, "skip_h1_title": false, "threshold": 4, "title_cell": "Table of Contents", "title_sidebar": "Contents", "toc_cell": false, "toc_position": {}, "toc_section_display": "block", "toc_window_display": false, "widenNotebook": false }, "varInspector": { "cols": { "lenName": 16, "lenType": 16, "lenVar": 40 }, "kernels_config": { "python": { "delete_cmd_postfix": "", "delete_cmd_prefix": "del ", "library": "var_list.py", "varRefreshCmd": "print(var_dic_list())" }, "r": { "delete_cmd_postfix": ") ", "delete_cmd_prefix": "rm(", "library": "var_list.r", "varRefreshCmd": "cat(var_dic_list()) " } }, "types_to_exclude": [ "module", "function", "builtin_function_or_method", "instance", "_Feature" ], "window_display": false } }, "nbformat": 4, "nbformat_minor": 2 }