{
"metadata": {
"name": "",
"signature": "sha256:f47cdb88b392068632307f333229ec83e8305108b83c91e0efc2ff0062f3b3c2"
},
"nbformat": 3,
"nbformat_minor": 0,
"worksheets": [
{
"cells": [
{
"cell_type": "heading",
"level": 1,
"metadata": {},
"source": [
"Introduction"
]
},
{
"cell_type": "heading",
"level": 1,
"metadata": {},
"source": [
"Sections"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- [Cross Validation](Cross-validation)\n",
"- [Bootstrapping](#Bootstrapping)\n",
"- [Jackknifing](#Jackknifing)\n",
"- [A K-fold Cross Validation Example](#A-K-fold-Cross-Validation-Example)\n",
" - [Reading in the Wine dataset](#Reading-in-the-Wine-dataset)\n",
" - [Resampling into Test and Training datasets](#Resampling-into-Test-and-Training-datasets)\n",
" - [Cross Validation](#Cross-Validation)\n",
" - [Training the classifier](#Training-the-classifier)\n",
" - [Standardization](#Standardization)\n",
" - [Dimensionality Reduction: Linear Discriminant Analysis (LDA)](Dimensionality-Reduction:-Linear-Discriminant-Analysis-(LDA))\n",
" - [Naive Bayes Classifier](#Naive-Bayes-Classifier)\n",
" - [Evaluation](#Evaluation)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"
\n",
"
"
]
},
{
"cell_type": "heading",
"level": 1,
"metadata": {},
"source": [
"Cross Validation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[[back to top](#Sections)]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"
\n",
"
"
]
},
{
"cell_type": "heading",
"level": 1,
"metadata": {},
"source": [
"Bootstrapping"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[[back to top](#Sections)]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"
\n",
"
"
]
},
{
"cell_type": "heading",
"level": 1,
"metadata": {},
"source": [
"Jackknifing"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[[back to top](#Sections)]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"
\n",
"
"
]
},
{
"cell_type": "heading",
"level": 1,
"metadata": {},
"source": [
"A K-fold Cross Validation Example"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[[back to top](#Sections)]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"
\n",
"
"
]
},
{
"cell_type": "heading",
"level": 2,
"metadata": {},
"source": [
"Reading in the Wine dataset"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[[back to top](#Sections)]"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"import pandas as pd\n",
"\n",
"df = pd.io.parsers.read_csv(\n",
" filepath_or_buffer='https://raw.githubusercontent.com/rasbt/pattern_classification/master/data/wine_data.csv', \n",
" header=None, \n",
" sep=',', \n",
" )\n",
"df.tail()"
],
"language": "python",
"metadata": {},
"outputs": [
{
"html": [
"\n",
"
\n",
" \n",
" \n",
" | \n",
" 0 | \n",
" 1 | \n",
" 2 | \n",
" 3 | \n",
" 4 | \n",
" 5 | \n",
" 6 | \n",
" 7 | \n",
" 8 | \n",
" 9 | \n",
" 10 | \n",
" 11 | \n",
" 12 | \n",
" 13 | \n",
"
\n",
" \n",
" \n",
" \n",
" | 173 | \n",
" 3 | \n",
" 13.71 | \n",
" 5.65 | \n",
" 2.45 | \n",
" 20.5 | \n",
" 95 | \n",
" 1.68 | \n",
" 0.61 | \n",
" 0.52 | \n",
" 1.06 | \n",
" 7.7 | \n",
" 0.64 | \n",
" 1.74 | \n",
" 740 | \n",
"
\n",
" \n",
" | 174 | \n",
" 3 | \n",
" 13.40 | \n",
" 3.91 | \n",
" 2.48 | \n",
" 23.0 | \n",
" 102 | \n",
" 1.80 | \n",
" 0.75 | \n",
" 0.43 | \n",
" 1.41 | \n",
" 7.3 | \n",
" 0.70 | \n",
" 1.56 | \n",
" 750 | \n",
"
\n",
" \n",
" | 175 | \n",
" 3 | \n",
" 13.27 | \n",
" 4.28 | \n",
" 2.26 | \n",
" 20.0 | \n",
" 120 | \n",
" 1.59 | \n",
" 0.69 | \n",
" 0.43 | \n",
" 1.35 | \n",
" 10.2 | \n",
" 0.59 | \n",
" 1.56 | \n",
" 835 | \n",
"
\n",
" \n",
" | 176 | \n",
" 3 | \n",
" 13.17 | \n",
" 2.59 | \n",
" 2.37 | \n",
" 20.0 | \n",
" 120 | \n",
" 1.65 | \n",
" 0.68 | \n",
" 0.53 | \n",
" 1.46 | \n",
" 9.3 | \n",
" 0.60 | \n",
" 1.62 | \n",
" 840 | \n",
"
\n",
" \n",
" | 177 | \n",
" 3 | \n",
" 14.13 | \n",
" 4.10 | \n",
" 2.74 | \n",
" 24.5 | \n",
" 96 | \n",
" 2.05 | \n",
" 0.76 | \n",
" 0.56 | \n",
" 1.35 | \n",
" 9.2 | \n",
" 0.61 | \n",
" 1.60 | \n",
" 560 | \n",
"
\n",
" \n",
"
\n",
"
\n",
"
"
]
},
{
"cell_type": "heading",
"level": 2,
"metadata": {},
"source": [
"Resampling into Test and Training datasets"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[[back to top](#Sections)]"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"from sklearn.cross_validation import train_test_split\n",
"\n",
"X_train, X_test, y_train, y_test = train_test_split(df.values[:,1:], df.values[:,0],\n",
" test_size=0.30, random_state=123)"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 40
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"
\n",
"
"
]
},
{
"cell_type": "heading",
"level": 2,
"metadata": {},
"source": [
"Cross Validation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[[back to top](#Sections)]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Accuracy is calculated as the number of correct predictions divided by the number of total predictions"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"X_train[:,0:1].shape"
],
"language": "python",
"metadata": {},
"outputs": [
{
"metadata": {},
"output_type": "pyout",
"prompt_number": 66,
"text": [
"(124, 1)"
]
}
],
"prompt_number": 66
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"np.concatenate((X_train[:,0].reshape(X_train.shape[0],1), X_train[:,0].reshape(X_train.shape[0],1)), axis=1).shape"
],
"language": "python",
"metadata": {},
"outputs": [
{
"metadata": {},
"output_type": "pyout",
"prompt_number": 63,
"text": [
"(124, 2)"
]
}
],
"prompt_number": 63
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"class ColumnExtractor(object):\n",
" \n",
" def __init__(self, cols):\n",
" self.cols = cols\n",
" def transform(self, X):\n",
" col_list = []\n",
" for c in self.cols:\n",
" col_list.append(X[:, c:c+1])\n",
" return np.concatenate(col_list, axis=1)\n",
"\n",
" def fit(self, X, y=None):\n",
" return self\n",
" \n",
"#ColumnExtractor(cols=(1,6)).transform(X_train)"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 82
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"from sklearn.cross_validation import cross_val_score, KFold\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.naive_bayes import GaussianNB\n",
"from sklearn.neighbors import KNeighborsClassifier\n",
"from sklearn.preprocessing import StandardScaler\n",
"from sklearn.lda import LDA\n",
"from sklearn.decomposition import PCA\n",
"\n",
"clf_all = Pipeline(steps=[\n",
" ('scaler', StandardScaler()),\n",
" ('reduce_dim', ColumnExtractor(cols=(0,5))), \n",
" ('classification', GaussianNB()) \n",
" ])\n",
"\n",
"clf_pca = Pipeline(steps=[\n",
" ('scaler', StandardScaler()), \n",
" ('reduce_dim', PCA(n_components=2)),\n",
" ('classification', GaussianNB()) \n",
" ])\n",
"\n",
"clf_lda = Pipeline(steps=[\n",
" ('scaler', StandardScaler()), \n",
" ('reduce_dim', LDA(n_components=2)),\n",
" ('classification', GaussianNB()) \n",
" ])\n",
"\n",
"# Constructing the k-fold cross validation iterator (k=10) \n",
"\n",
"cv = KFold(n=X_train.shape[0], # total number of samples\n",
" n_folds=10, # number of folds the dataset is divided into\n",
" shuffle=True,\n",
" random_state=123)\n",
"\n",
"scores = [\n",
" cross_val_score(clf, X_train, y_train, cv=cv, scoring='accuracy')\n",
" for clf in [clf_all, clf_pca, clf_lda]\n",
" ]"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 79
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"print('Scores (all samples):', scores[0])\n",
"print(\"Accuracy: {:.2%} (+/- {:.2%})\".format(scores[0].mean(), scores[0].std()))"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"Scores (all samples): [ 0.84615385 0.84615385 1. 0.92307692 0.66666667 0.75\n",
" 0.83333333 0.83333333 0.91666667 0.91666667]\n",
"Accuracy: 85.32% (+/- 8.99%)\n"
]
}
],
"prompt_number": 80
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"for score,label in zip(scores, \n",
" ['all samples', \n",
" 'PCA dim. red. (n=2)', \n",
" 'LDA dim. red. (n=2)', \n",
" ]\n",
" ):\n",
" print(\"Accuracy: {:.2%} (+/- {:.2%}), {:}\".format(score.mean(), score.std(), label))"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"Accuracy: 85.32% (+/- 8.99%), all samples\n",
"Accuracy: 95.90% (+/- 5.55%), PCA dim. red. (n=2)\n",
"Accuracy: 97.50% (+/- 3.82%), LDA dim. red. (n=2)\n"
]
}
],
"prompt_number": 81
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"
\n",
"
"
]
},
{
"cell_type": "heading",
"level": 2,
"metadata": {},
"source": [
"Training the classifier"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[[back to top](#Sections)]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"
\n",
"
"
]
},
{
"cell_type": "heading",
"level": 3,
"metadata": {},
"source": [
"Standardization"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[[back to top](#Sections)]"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"std_scale = StandardScaler().fit(X_train)\n",
"X_train = std_scale.transform(X_train)\n",
"X_test = std_scale.transform(X_test)"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 18
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"
\n",
"
"
]
},
{
"cell_type": "heading",
"level": 3,
"metadata": {},
"source": [
"Dimensionality Reduction: Linear Discriminant Analysis (LDA)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[[back to top](#Sections)]"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"sklearn_lda = LDA(n_components=2).fit(X_train, y_train)\n",
"X_train = sklearn_lda.transform(X_train)\n",
"X_test = sklearn_lda.transform(X_test)"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 19
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"
\n",
"
"
]
},
{
"cell_type": "heading",
"level": 3,
"metadata": {},
"source": [
"Naive Bayes Classifier"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[[back to top](#Sections)]"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"gnb_clf = GaussianNB()\n",
"gnb_clf.fit(X_train, y_train)\n",
"\n",
"pred_test = gnb_clf.predict(X_test)"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 20
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"
\n",
"
"
]
},
{
"cell_type": "heading",
"level": 3,
"metadata": {},
"source": [
"Evaluation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[[back to top](#Sections)]"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"from sklearn import metrics\n",
"\n",
"pred_test = gnb_clf.predict(X_test)\n",
"\n",
"print('Prediction accuracy for the test dataset')\n",
"print('{:.2%}'.format(metrics.accuracy_score(y_test, pred_test)))\n",
"\n",
"print('Confusion Matrix of the GNB-classifier')\n",
"print(metrics.confusion_matrix(y_test, gnb_clf.predict(X_test)))"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"Prediction accuracy for the test dataset\n",
"98.15%\n",
"Confusion Matrix of the GNB-classifier\n",
"[[14 0 0]\n",
" [ 0 18 0]\n",
" [ 0 1 21]]\n"
]
}
],
"prompt_number": 21
}
],
"metadata": {}
}
]
}