{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Scikit Learn Machine Learning Library\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"http://scikit-learn.org/stable/index.html"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Preprocessing\n",
"### Feature extraction and normalization.\n",
"\n",
"* Application: Transforming input data such as text for use with machine learning algorithms.\n",
"* Modules: preprocessing, feature extraction."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Model selection\n",
"### Comparing, validating and choosing parameters and models.\n",
"\n",
"* Goal: Improved accuracy via parameter tuning\n",
"* Modules: grid search, cross validation, metrics."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Regression\n",
"### Predicting a continuous-valued attribute associated with an object.\n",
"\n",
"* Applications: Drug response, Stock prices.\n",
"* Algorithms: SVR, ridge regression, Lasso"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Classification\n",
"### Identifying to which category an object belongs to.\n",
"\n",
"* Applications: Spam detection, Image recognition.\n",
"* Algorithms: SVM, nearest neighbors, random forest"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Clustering\n",
"### Automatic grouping of similar objects into sets.\n",
"\n",
"* Applications: Customer segmentation, Grouping experiment outcomes\n",
"* Algorithms: k-Means, spectral clustering, mean-shift"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Dimensionality reduction\n",
"*** Reducing the number of random variables to consider.\n",
"\n",
"* Applications: Visualization, Increased efficiency\n",
"* Algorithms: PCA, feature selection, non-negative matrix factorization"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"from sklearn import datasets"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"iris = datasets.load_iris()"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"sklearn.utils.Bunch"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"type(iris)"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[[5.1 3.5 1.4 0.2]\n",
" [4.9 3. 1.4 0.2]\n",
" [4.7 3.2 1.3 0.2]\n",
" [4.6 3.1 1.5 0.2]\n",
" [5. 3.6 1.4 0.2]\n",
" [5.4 3.9 1.7 0.4]\n",
" [4.6 3.4 1.4 0.3]\n",
" [5. 3.4 1.5 0.2]\n",
" [4.4 2.9 1.4 0.2]\n",
" [4.9 3.1 1.5 0.1]\n",
" [5.4 3.7 1.5 0.2]\n",
" [4.8 3.4 1.6 0.2]\n",
" [4.8 3. 1.4 0.1]\n",
" [4.3 3. 1.1 0.1]\n",
" [5.8 4. 1.2 0.2]\n",
" [5.7 4.4 1.5 0.4]\n",
" [5.4 3.9 1.3 0.4]\n",
" [5.1 3.5 1.4 0.3]\n",
" [5.7 3.8 1.7 0.3]\n",
" [5.1 3.8 1.5 0.3]\n",
" [5.4 3.4 1.7 0.2]\n",
" [5.1 3.7 1.5 0.4]\n",
" [4.6 3.6 1. 0.2]\n",
" [5.1 3.3 1.7 0.5]\n",
" [4.8 3.4 1.9 0.2]\n",
" [5. 3. 1.6 0.2]\n",
" [5. 3.4 1.6 0.4]\n",
" [5.2 3.5 1.5 0.2]\n",
" [5.2 3.4 1.4 0.2]\n",
" [4.7 3.2 1.6 0.2]\n",
" [4.8 3.1 1.6 0.2]\n",
" [5.4 3.4 1.5 0.4]\n",
" [5.2 4.1 1.5 0.1]\n",
" [5.5 4.2 1.4 0.2]\n",
" [4.9 3.1 1.5 0.1]\n",
" [5. 3.2 1.2 0.2]\n",
" [5.5 3.5 1.3 0.2]\n",
" [4.9 3.1 1.5 0.1]\n",
" [4.4 3. 1.3 0.2]\n",
" [5.1 3.4 1.5 0.2]\n",
" [5. 3.5 1.3 0.3]\n",
" [4.5 2.3 1.3 0.3]\n",
" [4.4 3.2 1.3 0.2]\n",
" [5. 3.5 1.6 0.6]\n",
" [5.1 3.8 1.9 0.4]\n",
" [4.8 3. 1.4 0.3]\n",
" [5.1 3.8 1.6 0.2]\n",
" [4.6 3.2 1.4 0.2]\n",
" [5.3 3.7 1.5 0.2]\n",
" [5. 3.3 1.4 0.2]\n",
" [7. 3.2 4.7 1.4]\n",
" [6.4 3.2 4.5 1.5]\n",
" [6.9 3.1 4.9 1.5]\n",
" [5.5 2.3 4. 1.3]\n",
" [6.5 2.8 4.6 1.5]\n",
" [5.7 2.8 4.5 1.3]\n",
" [6.3 3.3 4.7 1.6]\n",
" [4.9 2.4 3.3 1. ]\n",
" [6.6 2.9 4.6 1.3]\n",
" [5.2 2.7 3.9 1.4]\n",
" [5. 2. 3.5 1. ]\n",
" [5.9 3. 4.2 1.5]\n",
" [6. 2.2 4. 1. ]\n",
" [6.1 2.9 4.7 1.4]\n",
" [5.6 2.9 3.6 1.3]\n",
" [6.7 3.1 4.4 1.4]\n",
" [5.6 3. 4.5 1.5]\n",
" [5.8 2.7 4.1 1. ]\n",
" [6.2 2.2 4.5 1.5]\n",
" [5.6 2.5 3.9 1.1]\n",
" [5.9 3.2 4.8 1.8]\n",
" [6.1 2.8 4. 1.3]\n",
" [6.3 2.5 4.9 1.5]\n",
" [6.1 2.8 4.7 1.2]\n",
" [6.4 2.9 4.3 1.3]\n",
" [6.6 3. 4.4 1.4]\n",
" [6.8 2.8 4.8 1.4]\n",
" [6.7 3. 5. 1.7]\n",
" [6. 2.9 4.5 1.5]\n",
" [5.7 2.6 3.5 1. ]\n",
" [5.5 2.4 3.8 1.1]\n",
" [5.5 2.4 3.7 1. ]\n",
" [5.8 2.7 3.9 1.2]\n",
" [6. 2.7 5.1 1.6]\n",
" [5.4 3. 4.5 1.5]\n",
" [6. 3.4 4.5 1.6]\n",
" [6.7 3.1 4.7 1.5]\n",
" [6.3 2.3 4.4 1.3]\n",
" [5.6 3. 4.1 1.3]\n",
" [5.5 2.5 4. 1.3]\n",
" [5.5 2.6 4.4 1.2]\n",
" [6.1 3. 4.6 1.4]\n",
" [5.8 2.6 4. 1.2]\n",
" [5. 2.3 3.3 1. ]\n",
" [5.6 2.7 4.2 1.3]\n",
" [5.7 3. 4.2 1.2]\n",
" [5.7 2.9 4.2 1.3]\n",
" [6.2 2.9 4.3 1.3]\n",
" [5.1 2.5 3. 1.1]\n",
" [5.7 2.8 4.1 1.3]\n",
" [6.3 3.3 6. 2.5]\n",
" [5.8 2.7 5.1 1.9]\n",
" [7.1 3. 5.9 2.1]\n",
" [6.3 2.9 5.6 1.8]\n",
" [6.5 3. 5.8 2.2]\n",
" [7.6 3. 6.6 2.1]\n",
" [4.9 2.5 4.5 1.7]\n",
" [7.3 2.9 6.3 1.8]\n",
" [6.7 2.5 5.8 1.8]\n",
" [7.2 3.6 6.1 2.5]\n",
" [6.5 3.2 5.1 2. ]\n",
" [6.4 2.7 5.3 1.9]\n",
" [6.8 3. 5.5 2.1]\n",
" [5.7 2.5 5. 2. ]\n",
" [5.8 2.8 5.1 2.4]\n",
" [6.4 3.2 5.3 2.3]\n",
" [6.5 3. 5.5 1.8]\n",
" [7.7 3.8 6.7 2.2]\n",
" [7.7 2.6 6.9 2.3]\n",
" [6. 2.2 5. 1.5]\n",
" [6.9 3.2 5.7 2.3]\n",
" [5.6 2.8 4.9 2. ]\n",
" [7.7 2.8 6.7 2. ]\n",
" [6.3 2.7 4.9 1.8]\n",
" [6.7 3.3 5.7 2.1]\n",
" [7.2 3.2 6. 1.8]\n",
" [6.2 2.8 4.8 1.8]\n",
" [6.1 3. 4.9 1.8]\n",
" [6.4 2.8 5.6 2.1]\n",
" [7.2 3. 5.8 1.6]\n",
" [7.4 2.8 6.1 1.9]\n",
" [7.9 3.8 6.4 2. ]\n",
" [6.4 2.8 5.6 2.2]\n",
" [6.3 2.8 5.1 1.5]\n",
" [6.1 2.6 5.6 1.4]\n",
" [7.7 3. 6.1 2.3]\n",
" [6.3 3.4 5.6 2.4]\n",
" [6.4 3.1 5.5 1.8]\n",
" [6. 3. 4.8 1.8]\n",
" [6.9 3.1 5.4 2.1]\n",
" [6.7 3.1 5.6 2.4]\n",
" [6.9 3.1 5.1 2.3]\n",
" [5.8 2.7 5.1 1.9]\n",
" [6.8 3.2 5.9 2.3]\n",
" [6.7 3.3 5.7 2.5]\n",
" [6.7 3. 5.2 2.3]\n",
" [6.3 2.5 5. 1.9]\n",
" [6.5 3. 5.2 2. ]\n",
" [6.2 3.4 5.4 2.3]\n",
" [5.9 3. 5.1 1.8]]\n"
]
}
],
"source": [
"print(iris.data)"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"numpy.ndarray"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"type(iris.data)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Loading from external datasets\n",
"http://scikit-learn.org/stable/datasets/index.html#external-datasets"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"\n"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"from sklearn import svm\n",
"clf = svm.SVC()"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,\n",
" decision_function_shape='ovr', degree=3, gamma='auto', kernel='rbf',\n",
" max_iter=-1, probability=False, random_state=None, shrinking=True,\n",
" tol=0.001, verbose=False)"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"X, y = iris.data, iris.target\n",
"clf.fit(X, y) "
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"150"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"len(iris.data)"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([0])"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Persistent Models using pickle\n",
"import pickle\n",
"s = pickle.dumps(clf)\n",
"clf2 = pickle.loads(s)\n",
"clf2.predict(X[0:1])"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"sklearn.svm.classes.SVC"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"type(clf2)"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"y[0]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Big data pickle\n",
"In the specific case of the scikit, it may be more interesting to use joblib’s replacement of pickle (joblib.dump & joblib.load), which is more efficient on big data, but can only pickle to the disk and not to a string:\n",
"\n",
"\n",
"` from sklearn.externals import joblib`\n",
"\n",
"` joblib.dump(clf, 'filename.pkl') `"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Full Scikit learn process"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"1. Define Problem.\n",
"2. Prepare Data.\n",
"3. Evaluate Algorithms.\n",
"4. Improve Results.\n",
"5. Present Results.\n",
"\n",
"from https://machinelearningmastery.com/machine-learning-in-python-step-by-step/"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [],
"source": [
"# Library setup/import\n",
"import pandas as pd\n",
"from pandas.plotting import scatter_matrix\n",
"import matplotlib.pyplot as plt\n",
"from sklearn import model_selection\n",
"from sklearn.metrics import classification_report\n",
"from sklearn.metrics import confusion_matrix\n",
"from sklearn.metrics import accuracy_score\n",
"from sklearn.linear_model import LogisticRegression\n",
"from sklearn.tree import DecisionTreeClassifier\n",
"from sklearn.neighbors import KNeighborsClassifier\n",
"from sklearn.discriminant_analysis import LinearDiscriminantAnalysis\n",
"from sklearn.naive_bayes import GaussianNB\n",
"from sklearn.svm import SVC"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [],
"source": [
"# Load dataset\n",
"url = \"https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data\" #UCI ml archive\n",
"names = ['sepal-length', 'sepal-width', 'petal-length', 'petal-width', 'class']\n",
"dataset = pd.read_csv(url, names=names)"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"(150, 5)\n"
]
}
],
"source": [
"# Summarizing data\n",
"\n",
"# shape\n",
"print(dataset.shape)"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" sepal-length | \n",
" sepal-width | \n",
" petal-length | \n",
" petal-width | \n",
" class | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" 5.1 | \n",
" 3.5 | \n",
" 1.4 | \n",
" 0.2 | \n",
" Iris-setosa | \n",
"
\n",
" \n",
" | 1 | \n",
" 4.9 | \n",
" 3.0 | \n",
" 1.4 | \n",
" 0.2 | \n",
" Iris-setosa | \n",
"
\n",
" \n",
" | 2 | \n",
" 4.7 | \n",
" 3.2 | \n",
" 1.3 | \n",
" 0.2 | \n",
" Iris-setosa | \n",
"
\n",
" \n",
" | 3 | \n",
" 4.6 | \n",
" 3.1 | \n",
" 1.5 | \n",
" 0.2 | \n",
" Iris-setosa | \n",
"
\n",
" \n",
" | 4 | \n",
" 5.0 | \n",
" 3.6 | \n",
" 1.4 | \n",
" 0.2 | \n",
" Iris-setosa | \n",
"
\n",
" \n",
"
\n",
"
\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" sepal-length | \n",
" sepal-width | \n",
" petal-length | \n",
" petal-width | \n",
"
\n",
" \n",
" \n",
" \n",
" | count | \n",
" 150.000000 | \n",
" 150.000000 | \n",
" 150.000000 | \n",
" 150.000000 | \n",
"
\n",
" \n",
" | mean | \n",
" 5.843333 | \n",
" 3.054000 | \n",
" 3.758667 | \n",
" 1.198667 | \n",
"
\n",
" \n",
" | std | \n",
" 0.828066 | \n",
" 0.433594 | \n",
" 1.764420 | \n",
" 0.763161 | \n",
"
\n",
" \n",
" | min | \n",
" 4.300000 | \n",
" 2.000000 | \n",
" 1.000000 | \n",
" 0.100000 | \n",
"
\n",
" \n",
" | 25% | \n",
" 5.100000 | \n",
" 2.800000 | \n",
" 1.600000 | \n",
" 0.300000 | \n",
"
\n",
" \n",
" | 50% | \n",
" 5.800000 | \n",
" 3.000000 | \n",
" 4.350000 | \n",
" 1.300000 | \n",
"
\n",
" \n",
" | 75% | \n",
" 6.400000 | \n",
" 3.300000 | \n",
" 5.100000 | \n",
" 1.800000 | \n",
"
\n",
" \n",
" | max | \n",
" 7.900000 | \n",
" 4.400000 | \n",
" 6.900000 | \n",
" 2.500000 | \n",
"
\n",
" \n",
"
\n",
"