{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "## 内容概要\n", "- 模型评估的目的及一般评估流程\n", "- 分类准确率的用处及其限制\n", "- 混淆矩阵(confusion matrix)是如何表示一个分类器的性能\n", "- 混淆矩阵中的度量是如何计算的\n", "- 通过改变分类阈值来调整分类器性能\n", "- ROC曲线的用处\n", "- 曲线下面积(Area Under the Curve, AUC)与分类准确率的不同" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. 回顾\n", "模型评估可以用于在不同的模型类型、调节参数、特征组合中选择适合的模型,所以我们需要一个模型评估的流程来估计训练得到的模型对于非样本数据的泛化能力,并且还需要恰当的模型评估度量手段来衡量模型的性能表现。\n", "\n", "对于模型评估流程而言,之前介绍了K折交叉验证的方法,针对模型评估度量方法,回归问题可以采用平均绝对误差(Mean Absolute Error)、均方误差(Mean Squared Error)、均方根误差(Root Mean Squared Error),而分类问题可以采用分类准确率和这篇文章中介绍的度量方法。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. 分类准确率(Classification accuracy)\n", "这里我们使用Pima Indians Diabetes dataset,其中包含健康数据和糖尿病状态数据,一共有768个病人的数据。" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# read the data into a Pandas DataFrame\n", "import pandas as pd\n", "url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/pima-indians-diabetes/pima-indians-diabetes.data'\n", "col_names = ['pregnant', 'glucose', 'bp', 'skin', 'insulin', 'bmi', 'pedigree', 'age', 'label']\n", "pima = pd.read_csv(url, header=None, names=col_names)" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", " | pregnant | \n", "glucose | \n", "bp | \n", "skin | \n", "insulin | \n", "bmi | \n", "pedigree | \n", "age | \n", "label | \n", "
---|---|---|---|---|---|---|---|---|---|
0 | \n", "6 | \n", "148 | \n", "72 | \n", "35 | \n", "0 | \n", "33.6 | \n", "0.627 | \n", "50 | \n", "1 | \n", "
1 | \n", "1 | \n", "85 | \n", "66 | \n", "29 | \n", "0 | \n", "26.6 | \n", "0.351 | \n", "31 | \n", "0 | \n", "
2 | \n", "8 | \n", "183 | \n", "64 | \n", "0 | \n", "0 | \n", "23.3 | \n", "0.672 | \n", "32 | \n", "1 | \n", "
3 | \n", "1 | \n", "89 | \n", "66 | \n", "23 | \n", "94 | \n", "28.1 | \n", "0.167 | \n", "21 | \n", "0 | \n", "
4 | \n", "0 | \n", "137 | \n", "40 | \n", "35 | \n", "168 | \n", "43.1 | \n", "2.288 | \n", "33 | \n", "1 | \n", "