{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# PYCM"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Version : 0.5\n",
"-----"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"from pycm import *"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"y_actu = [2, 0, 2, 2, 0, 1, 1, 2, 2, 0, 1, 2]\n",
"y_pred = [0, 0, 2, 1, 0, 2, 1, 0, 2, 0, 2, 2]"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"cm = ConfusionMatrix(y_actu, y_pred)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"pycm.ConfusionMatrix(classes: [0, 1, 2])"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[2, 0, 2, 2, 0, 1, 1, 2, 2, 0, 1, 2]"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.actual_vector"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[0, 0, 2, 1, 0, 2, 1, 0, 2, 0, 2, 2]"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.predict_vector"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[0, 1, 2]"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.classes"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'ACC': {0: 0.8333333333333334, 1: 0.75, 2: 0.5833333333333334},\n",
" 'BM': {0: 0.7777777777777777, 1: 0.2222222222222221, 2: 0.16666666666666652},\n",
" 'DOR': {0: 'None', 1: 3.999999999999998, 2: 1.9999999999999998},\n",
" 'ERR': {0: 0.16666666666666663, 1: 0.25, 2: 0.41666666666666663},\n",
" 'F0.5': {0: 0.6521739130434783,\n",
" 1: 0.45454545454545453,\n",
" 2: 0.5769230769230769},\n",
" 'F1': {0: 0.75, 1: 0.4, 2: 0.5454545454545454},\n",
" 'F2': {0: 0.8823529411764706, 1: 0.35714285714285715, 2: 0.5172413793103449},\n",
" 'FDR': {0: 0.4, 1: 0.5, 2: 0.4},\n",
" 'FN': {0: 0, 1: 2, 2: 3},\n",
" 'FNR': {0: 0.0, 1: 0.6666666666666667, 2: 0.5},\n",
" 'FOR': {0: 0.0, 1: 0.19999999999999996, 2: 0.4285714285714286},\n",
" 'FP': {0: 2, 1: 1, 2: 2},\n",
" 'FPR': {0: 0.2222222222222222,\n",
" 1: 0.11111111111111116,\n",
" 2: 0.33333333333333337},\n",
" 'G': {0: 0.7745966692414834, 1: 0.408248290463863, 2: 0.5477225575051661},\n",
" 'LR+': {0: 4.5, 1: 2.9999999999999987, 2: 1.4999999999999998},\n",
" 'LR-': {0: 0.0, 1: 0.7500000000000001, 2: 0.75},\n",
" 'MCC': {0: 0.6831300510639732, 1: 0.25819888974716115, 2: 0.1690308509457033},\n",
" 'MK': {0: 0.6000000000000001, 1: 0.30000000000000004, 2: 0.17142857142857126},\n",
" 'N': {0: 9, 1: 9, 2: 6},\n",
" 'NPV': {0: 1.0, 1: 0.8, 2: 0.5714285714285714},\n",
" 'P': {0: 3, 1: 3, 2: 6},\n",
" 'POP': {0: 12, 1: 12, 2: 12},\n",
" 'PPV': {0: 0.6, 1: 0.5, 2: 0.6},\n",
" 'PRE': {0: 0.25, 1: 0.25, 2: 0.5},\n",
" 'RACC': {0: 0.10416666666666667,\n",
" 1: 0.041666666666666664,\n",
" 2: 0.20833333333333334},\n",
" 'TN': {0: 7, 1: 8, 2: 4},\n",
" 'TNR': {0: 0.7777777777777778, 1: 0.8888888888888888, 2: 0.6666666666666666},\n",
" 'TON': {0: 7, 1: 10, 2: 7},\n",
" 'TOP': {0: 5, 1: 2, 2: 5},\n",
" 'TP': {0: 3, 1: 1, 2: 3},\n",
" 'TPR': {0: 1.0, 1: 0.3333333333333333, 2: 0.5}}"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.class_stat"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* Notice : cm.statistic_result in prev versions (<0.2)"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'Bennett_S': 0.37500000000000006,\n",
" 'Gwet_AC1': 0.3893129770992367,\n",
" 'Kappa': 0.35483870967741943,\n",
" 'Overall_ACC': 0.5833333333333334,\n",
" 'Overall_RACC': 0.3541666666666667,\n",
" 'PPV_Macro': 0.5666666666666668,\n",
" 'PPV_Micro': 0.5833333333333334,\n",
" 'Scott_PI': 0.34426229508196726,\n",
" 'Strength_Of_Agreement(Altman)': 'Fair',\n",
" 'Strength_Of_Agreement(Fleiss)': 'Poor',\n",
" 'Strength_Of_Agreement(Landis and Koch)': 'Fair',\n",
" 'TPR_Macro': 0.611111111111111,\n",
" 'TPR_Micro': 0.5833333333333334}"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.overall_stat"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* Notice : new in version 0.3"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{0: {0: 3, 1: 0, 2: 0}, 1: {0: 0, 1: 1, 2: 2}, 2: {0: 2, 1: 1, 2: 3}}"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.table"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Basic Parameters"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### TP (True positive / hit)"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{0: 3, 1: 1, 2: 3}"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.TP"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### TN (True negative/correct rejection)"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{0: 7, 1: 8, 2: 4}"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.TN"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### FP (False positive/false alarm/Type I error)"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{0: 2, 1: 1, 2: 2}"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.FP"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### FN (False negative/miss/Type II error)"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{0: 0, 1: 2, 2: 3}"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.FN"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### P (Condition positive)"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{0: 3, 1: 3, 2: 6}"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.P"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### N (Condition negative)"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{0: 9, 1: 9, 2: 6}"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.N"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### TOP (Test outcome positive)"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{0: 5, 1: 2, 2: 5}"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.TOP"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### TON (Test outcome negative)"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{0: 7, 1: 10, 2: 7}"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.TON"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### POP (Population)"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{0: 12, 1: 12, 2: 12}"
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.POP"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Class Statistics"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### TPR (sensitivity, recall, hit rate, or true positive rate)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$TPR=\\frac{TP}{P}=\\frac{TP}{TP+FN}$$"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{0: 1.0, 1: 0.3333333333333333, 2: 0.5}"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.TPR"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### TNR (specificity or true negative rate)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$TNR=\\frac{TN}{N}=\\frac{TN}{TN+FP}$$"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{0: 0.7777777777777778, 1: 0.8888888888888888, 2: 0.6666666666666666}"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.TNR"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### PPV (precision or positive predictive value)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$PPV=\\frac{TP}{TP+FP}$$"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{0: 0.6, 1: 0.5, 2: 0.6}"
]
},
"execution_count": 22,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.PPV"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### NPV (negative predictive value)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$NPV=\\frac{TN}{TN+FN}$$"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{0: 1.0, 1: 0.8, 2: 0.5714285714285714}"
]
},
"execution_count": 23,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.NPV"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### FNR (miss rate or false negative rate)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$FNR=\\frac{FN}{P}=\\frac{FN}{FN+TP}=1-TPR$$"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{0: 0.0, 1: 0.6666666666666667, 2: 0.5}"
]
},
"execution_count": 24,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.FNR"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### FPR (fall-out or false positive rate)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$FPR=\\frac{FP}{N}=\\frac{FP}{FP+TN}=1-TNR$$"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{0: 0.2222222222222222, 1: 0.11111111111111116, 2: 0.33333333333333337}"
]
},
"execution_count": 25,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.FPR"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### FDR (false discovery rate)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$FDR=\\frac{FP}{FP+TP}=1-PPV$$"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{0: 0.6, 1: 0.5, 2: 0.6}"
]
},
"execution_count": 26,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.PPV"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### FOR (false omission rate)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$FOR=\\frac{FN}{FN+TN}=1-NPV$$"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{0: 0.0, 1: 0.19999999999999996, 2: 0.4285714285714286}"
]
},
"execution_count": 27,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.FOR"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### ACC (accuracy)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$ACC=\\frac{TP+TN}{P+N}=\\frac{TP+TN}{TP+TN+FP+FN}$$"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{0: 0.8333333333333334, 1: 0.75, 2: 0.5833333333333334}"
]
},
"execution_count": 28,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.ACC"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### ERR(Error rate)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$ERR=\\frac{FP+FN}{P+N}=\\frac{FP+FN}{TP+TN+FP+FN}=1-ACC$$"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{0: 0.16666666666666663, 1: 0.25, 2: 0.41666666666666663}"
]
},
"execution_count": 29,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.ERR"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* Notice : new in version 0.4"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### FBeta-Score"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$F_{\\beta}=(1+\\beta^2).\\frac{PPV.TPR}{(\\beta^2.PPV)+TPR}=\\frac{(1+\\beta^2).TP}{(1+\\beta^2).TP+FP+\\beta^2.FN}$$"
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{0: 0.75, 1: 0.4, 2: 0.5454545454545454}"
]
},
"execution_count": 30,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.F1"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{0: 0.6521739130434783, 1: 0.45454545454545453, 2: 0.5769230769230769}"
]
},
"execution_count": 31,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.F05"
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{0: 0.8823529411764706, 1: 0.35714285714285715, 2: 0.5172413793103449}"
]
},
"execution_count": 32,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.F2"
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{0: 0.9622641509433962, 1: 0.34, 2: 0.504950495049505}"
]
},
"execution_count": 33,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.F_beta(Beta=4)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* Notice : new in version 0.4"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### MCC (Matthews correlation coefficient)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$MCC=\\frac{TP \\times TN-FP \\times FN}{\\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)}}$$"
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{0: 0.6831300510639732, 1: 0.25819888974716115, 2: 0.1690308509457033}"
]
},
"execution_count": 34,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.MCC"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### BM (Informedness or Bookmaker Informedness)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$BM=TPR+TNR-1$$"
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{0: 0.7777777777777777, 1: 0.2222222222222221, 2: 0.16666666666666652}"
]
},
"execution_count": 35,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.BM"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### MK (Markedness)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$MK=PPV+NPV-1$$"
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{0: 0.6000000000000001, 1: 0.30000000000000004, 2: 0.17142857142857126}"
]
},
"execution_count": 36,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.MK"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### PLR (Positive likelihood ratio)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$(LR+)=\\frac{TPR}{FPR}$$"
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{0: 4.5, 1: 2.9999999999999987, 2: 1.4999999999999998}"
]
},
"execution_count": 37,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.PLR"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### NLR (Negative likelihood ratio)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$(LR-)=\\frac{FNR}{TNR}$$"
]
},
{
"cell_type": "code",
"execution_count": 38,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{0: 0.0, 1: 0.7500000000000001, 2: 0.75}"
]
},
"execution_count": 38,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.NLR"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### DOR (Diagnostic odds ratio)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$DOR=\\frac{LR+}{LR-}$$"
]
},
{
"cell_type": "code",
"execution_count": 39,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{0: 'None', 1: 3.999999999999998, 2: 1.9999999999999998}"
]
},
"execution_count": 39,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.DOR"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### PRE (Prevalence)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$Prevalence=\\frac{P}{Population}$$"
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{0: 0.25, 1: 0.25, 2: 0.5}"
]
},
"execution_count": 40,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.PRE"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### G (G-measure geometric mean of precision and sensitivity)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$G=\\sqrt{PPV.TPR}$$"
]
},
{
"cell_type": "code",
"execution_count": 41,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{0: 0.7745966692414834, 1: 0.408248290463863, 2: 0.5477225575051661}"
]
},
"execution_count": 41,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.G"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### RACC(Random accuracy)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$RACC=\\frac{TOP\\times P}{Population^2}$$"
]
},
{
"cell_type": "code",
"execution_count": 42,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{0: 0.10416666666666667, 1: 0.041666666666666664, 2: 0.20833333333333334}"
]
},
"execution_count": 42,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.RACC"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* Notice : new in version 0.3"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Overall Statistics"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Kappa (Nominal)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$Kappa=\\frac{ACC_{Overall}-RACC_{Overall}}{1-RACC_{Overall}}$$"
]
},
{
"cell_type": "code",
"execution_count": 43,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.35483870967741943"
]
},
"execution_count": 43,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.Kappa"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* Notice : new in version 0.3"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Bennett et al.'s S score (Nominal)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$p_c=\\frac{1}{|C|}$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$S=\\frac{ACC_{Overall}-p_c}{1-p_c}$$"
]
},
{
"cell_type": "code",
"execution_count": 44,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.37500000000000006"
]
},
"execution_count": 44,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.S"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* Notice : new in version 0.5"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Scott's pi (Nominal)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$p_c=\\sum_{i=1}^{|C|}(\\frac{TOP_i + P_i}{2\\times Population})^2$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\\pi=\\frac{ACC_{Overall}-p_c}{1-p_c}$$"
]
},
{
"cell_type": "code",
"execution_count": 45,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.34426229508196726"
]
},
"execution_count": 45,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.PI"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* Notice : new in version 0.5"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Gwet's AC1"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\\pi=\\frac{TOP_i + P_i}{2\\times Population}$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$p_c=\\frac{1}{|C|-1}\\sum_{i=1}^{|C|}(\\pi_i\\times (1-\\pi_i))$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$AC1=\\frac{ACC_{Overall}-p_c}{1-p_c}$$"
]
},
{
"cell_type": "code",
"execution_count": 46,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.3893129770992367"
]
},
"execution_count": 46,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.AC1"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* Notice : new in version 0.5"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### SOA1 (Strength of Agreement, Landis and Koch benchmark)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"
\n",
" \n",
" Kappa | \n",
" Strength of Agreement | \n",
"
\n",
" \n",
" 0 > | \n",
" Poor | \n",
"
\n",
" \n",
" 0 - 0.20 | \n",
" Slight | \n",
"
\n",
" \n",
" 0.21 – 0.40 | \n",
" Fair | \n",
"
\n",
" \n",
" 0.41 – 0.60 | \n",
" Moderate | \n",
"
\n",
" \n",
" 0.61 – 0.80 | \n",
" Substantial | \n",
"
\n",
" \n",
" 0.81 – 1.00 | \n",
" Almost perfect | \n",
"
\n",
" \n",
" \n",
"
"
]
},
{
"cell_type": "code",
"execution_count": 47,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'Fair'"
]
},
"execution_count": 47,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.SOA1"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* Notice : new in version 0.3"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### SOA2 (Strength of Agreement, : Fleiss’ benchmark)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
" \n",
" Kappa | \n",
" Strength of Agreement | \n",
"
\n",
" \n",
" 0.40 > | \n",
" Poor | \n",
"
\n",
" \n",
" 0.4 - 0.75 | \n",
" Intermediate to Good | \n",
"
\n",
" \n",
" More than 0.75 | \n",
" Excellent | \n",
"
\n",
" \n",
" \n",
"
"
]
},
{
"cell_type": "code",
"execution_count": 48,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'Poor'"
]
},
"execution_count": 48,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.SOA2"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* Notice : new in version 0.4"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### SOA3 (Strength of Agreement, Altman’s benchmark)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
" \n",
" Kappa | \n",
" Strength of Agreement | \n",
"
\n",
" \n",
" 0.2 > | \n",
" Poor | \n",
"
\n",
" \n",
" 0.21 – 0.40 | \n",
" Fair | \n",
"
\n",
" \n",
" 0.41 – 0.60 | \n",
" Moderate | \n",
"
\n",
" \n",
" 0.61 – 0.80 | \n",
" Good | \n",
"
\n",
" \n",
" 0.81 – 1.00 | \n",
" Very Good | \n",
"
\n",
" \n",
" \n",
"
"
]
},
{
"cell_type": "code",
"execution_count": 49,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'Fair'"
]
},
"execution_count": 49,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.SOA3"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* Notice : new in version 0.4"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Overall_ACC"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$ACC_{Overall}=\\frac{\\sum_{i=1}^{|C|}TP_i}{Population}$$"
]
},
{
"cell_type": "code",
"execution_count": 50,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.5833333333333334"
]
},
"execution_count": 50,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.Overall_ACC"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* Notice : new in version 0.4"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Overall_RACC"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$RACC_{Overall}=\\sum_{i=1}^{|C|}RACC_i$$"
]
},
{
"cell_type": "code",
"execution_count": 51,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.3541666666666667"
]
},
"execution_count": 51,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.Overall_RACC"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### PPV_Micro"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$PPV_{Micro}=\\frac{\\sum_{i=1}^{|C|}TP_i}{\\sum_{i=1}^{|C|}TP_i+FP_i}$$"
]
},
{
"cell_type": "code",
"execution_count": 52,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.5833333333333334"
]
},
"execution_count": 52,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.PPV_Micro"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* Notice : new in version 0.4"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### TPR_Micro"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$TPR_{Micro}=\\frac{\\sum_{i=1}^{|C|}TP_i}{\\sum_{i=1}^{|C|}TP_i+FN_i}$$"
]
},
{
"cell_type": "code",
"execution_count": 53,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.5833333333333334"
]
},
"execution_count": 53,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.TPR_Micro"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* Notice : new in version 0.4"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### PPV_Macro"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$PPV_{Macro}=\\frac{1}{|C|}\\sum_{i=1}^{|C|}\\frac{TP_i}{TP_i+FP_i}$$"
]
},
{
"cell_type": "code",
"execution_count": 54,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.5666666666666668"
]
},
"execution_count": 54,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.PPV_Macro"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* Notice : new in version 0.4"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### TPR_Micro"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$TPR_{Macro}=\\frac{1}{|C|}\\sum_{i=1}^{|C|}\\frac{TP_i}{TP_i+FN_i}$$"
]
},
{
"cell_type": "code",
"execution_count": 55,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.611111111111111"
]
},
"execution_count": 55,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.TPR_Macro"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* Notice : new in version 0.4"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Print"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Full"
]
},
{
"cell_type": "code",
"execution_count": 56,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Predict 0 1 2 \n",
"Actual\n",
"0 3 0 0 \n",
"1 0 1 2 \n",
"2 2 1 3 \n",
"\n",
"\n",
"\n",
"\n",
"Overall Statistics : \n",
"\n",
"Bennett_S 0.375\n",
"Gwet_AC1 0.38931\n",
"Kappa 0.35484\n",
"Overall_ACC 0.58333\n",
"Overall_RACC 0.35417\n",
"PPV_Macro 0.56667\n",
"PPV_Micro 0.58333\n",
"Scott_PI 0.34426\n",
"Strength_Of_Agreement(Altman) Fair\n",
"Strength_Of_Agreement(Fleiss) Poor\n",
"Strength_Of_Agreement(Landis and Koch) Fair\n",
"TPR_Macro 0.61111\n",
"TPR_Micro 0.58333\n",
"\n",
"Class Statistics :\n",
"\n",
"Classes 0 1 2 \n",
"ACC(accuracy) 0.83333 0.75 0.58333 \n",
"BM(Informedness or Bookmaker Informedness) 0.77778 0.22222 0.16667 \n",
"DOR(Diagnostic odds ratio) None 4.0 2.0 \n",
"ERR(Error Rate) 0.16667 0.25 0.41667 \n",
"F0.5(F0.5 Score) 0.65217 0.45455 0.57692 \n",
"F1(F1 Score - harmonic mean of precision and sensitivity) 0.75 0.4 0.54545 \n",
"F2(F2 Score) 0.88235 0.35714 0.51724 \n",
"FDR(false discovery rate) 0.4 0.5 0.4 \n",
"FN(false negative/miss/Type II error) 0 2 3 \n",
"FNR(miss rate or false negative rate) 0.0 0.66667 0.5 \n",
"FOR(false omission rate) 0.0 0.2 0.42857 \n",
"FP(false positive/Type I error/false alarm) 2 1 2 \n",
"FPR(fall-out or false positive rate) 0.22222 0.11111 0.33333 \n",
"G(G-measure geometric mean of precision and sensitivity) 0.7746 0.40825 0.54772 \n",
"LR+(Positive likelihood ratio) 4.5 3.0 1.5 \n",
"LR-(Negative likelihood ratio) 0.0 0.75 0.75 \n",
"MCC(Matthews correlation coefficient) 0.68313 0.2582 0.16903 \n",
"MK(Markedness) 0.6 0.3 0.17143 \n",
"N(Condition negative) 9 9 6 \n",
"NPV(negative predictive value) 1.0 0.8 0.57143 \n",
"P(Condition positive) 3 3 6 \n",
"POP(Population) 12 12 12 \n",
"PPV(precision or positive predictive value) 0.6 0.5 0.6 \n",
"PRE(Prevalence) 0.25 0.25 0.5 \n",
"RACC(Random Accuracy) 0.10417 0.04167 0.20833 \n",
"TN(true negative/correct rejection) 7 8 4 \n",
"TNR(specificity or true negative rate) 0.77778 0.88889 0.66667 \n",
"TON(Test outcome negative) 7 10 7 \n",
"TOP(Test outcome positive) 5 2 5 \n",
"TP(true positive/hit) 3 1 3 \n",
"TPR(sensitivity, recall, hit rate, or true positive rate) 1.0 0.33333 0.5 \n",
"\n"
]
}
],
"source": [
"print(cm)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Matrix"
]
},
{
"cell_type": "code",
"execution_count": 57,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Predict 0 1 2 \n",
"Actual\n",
"0 3 0 0 \n",
"1 0 1 2 \n",
"2 2 1 3 \n",
"\n"
]
}
],
"source": [
"cm.matrix()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Normalized Matrix"
]
},
{
"cell_type": "code",
"execution_count": 58,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Predict 0 1 2 \n",
"Actual\n",
"0 1.0 0.0 0.0 \n",
"1 0.0 0.33333 0.66667 \n",
"2 0.33333 0.16667 0.5 \n",
"\n"
]
}
],
"source": [
"cm.normalized_matrix()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Stat"
]
},
{
"cell_type": "code",
"execution_count": 59,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Overall Statistics : \n",
"\n",
"Bennett_S 0.375\n",
"Gwet_AC1 0.38931\n",
"Kappa 0.35484\n",
"Overall_ACC 0.58333\n",
"Overall_RACC 0.35417\n",
"PPV_Macro 0.56667\n",
"PPV_Micro 0.58333\n",
"Scott_PI 0.34426\n",
"Strength_Of_Agreement(Altman) Fair\n",
"Strength_Of_Agreement(Fleiss) Poor\n",
"Strength_Of_Agreement(Landis and Koch) Fair\n",
"TPR_Macro 0.61111\n",
"TPR_Micro 0.58333\n",
"\n",
"Class Statistics :\n",
"\n",
"Classes 0 1 2 \n",
"ACC(accuracy) 0.83333 0.75 0.58333 \n",
"BM(Informedness or Bookmaker Informedness) 0.77778 0.22222 0.16667 \n",
"DOR(Diagnostic odds ratio) None 4.0 2.0 \n",
"ERR(Error Rate) 0.16667 0.25 0.41667 \n",
"F0.5(F0.5 Score) 0.65217 0.45455 0.57692 \n",
"F1(F1 Score - harmonic mean of precision and sensitivity) 0.75 0.4 0.54545 \n",
"F2(F2 Score) 0.88235 0.35714 0.51724 \n",
"FDR(false discovery rate) 0.4 0.5 0.4 \n",
"FN(false negative/miss/Type II error) 0 2 3 \n",
"FNR(miss rate or false negative rate) 0.0 0.66667 0.5 \n",
"FOR(false omission rate) 0.0 0.2 0.42857 \n",
"FP(false positive/Type I error/false alarm) 2 1 2 \n",
"FPR(fall-out or false positive rate) 0.22222 0.11111 0.33333 \n",
"G(G-measure geometric mean of precision and sensitivity) 0.7746 0.40825 0.54772 \n",
"LR+(Positive likelihood ratio) 4.5 3.0 1.5 \n",
"LR-(Negative likelihood ratio) 0.0 0.75 0.75 \n",
"MCC(Matthews correlation coefficient) 0.68313 0.2582 0.16903 \n",
"MK(Markedness) 0.6 0.3 0.17143 \n",
"N(Condition negative) 9 9 6 \n",
"NPV(negative predictive value) 1.0 0.8 0.57143 \n",
"P(Condition positive) 3 3 6 \n",
"POP(Population) 12 12 12 \n",
"PPV(precision or positive predictive value) 0.6 0.5 0.6 \n",
"PRE(Prevalence) 0.25 0.25 0.5 \n",
"RACC(Random Accuracy) 0.10417 0.04167 0.20833 \n",
"TN(true negative/correct rejection) 7 8 4 \n",
"TNR(specificity or true negative rate) 0.77778 0.88889 0.66667 \n",
"TON(Test outcome negative) 7 10 7 \n",
"TOP(Test outcome positive) 5 2 5 \n",
"TP(true positive/hit) 3 1 3 \n",
"TPR(sensitivity, recall, hit rate, or true positive rate) 1.0 0.33333 0.5 \n",
"\n"
]
}
],
"source": [
"cm.stat()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* Notice : cm.params() in prev versions (<0.2)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Save"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### .pcym file"
]
},
{
"cell_type": "code",
"execution_count": 60,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'Message': 'C:\\\\Users\\\\Sepkjaer\\\\Desktop\\\\Desktop\\\\JupyterNotebooks\\\\cm1.pycm',\n",
" 'Status': True}"
]
},
"execution_count": 60,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.save_stat(\"cm1\")"
]
},
{
"cell_type": "code",
"execution_count": 61,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'Message': \"[Errno 2] No such file or directory: 'cm1asdasd/.pycm'\",\n",
" 'Status': False}"
]
},
"execution_count": 61,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.save_stat(\"cm1asdasd/\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* Notice : new in version 0.4"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### HTML"
]
},
{
"cell_type": "code",
"execution_count": 62,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'Message': 'C:\\\\Users\\\\Sepkjaer\\\\Desktop\\\\Desktop\\\\JupyterNotebooks\\\\cm1.html',\n",
" 'Status': True}"
]
},
"execution_count": 62,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.save_html(\"cm1\")"
]
},
{
"cell_type": "code",
"execution_count": 63,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'Message': \"[Errno 2] No such file or directory: 'cm1asdasd/.html'\",\n",
" 'Status': False}"
]
},
"execution_count": 63,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cm.save_html(\"cm1asdasd/\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* Notice : new in version 0.5"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Input Errors"
]
},
{
"cell_type": "code",
"execution_count": 64,
"metadata": {},
"outputs": [
{
"ename": "pycmError",
"evalue": "Input Vectors Must Be List",
"output_type": "error",
"traceback": [
"\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[1;31mpycmError\u001b[0m Traceback (most recent call last)",
"\u001b[1;32m\u001b[0m in \u001b[0;36m\u001b[1;34m()\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mcm2\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mConfusionMatrix\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0my_actu\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;36m2\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[1;32m~\\AppData\\Local\\Programs\\Python\\Python35-32\\lib\\site-packages\\pycm\\pycm_obj.py\u001b[0m in \u001b[0;36m__init__\u001b[1;34m(self, actual_vector, predict_vector)\u001b[0m\n\u001b[0;32m 12\u001b[0m \u001b[1;32mdef\u001b[0m \u001b[0m__init__\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mactual_vector\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mpredict_vector\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 13\u001b[0m \u001b[1;32mif\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[0misinstance\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mactual_vector\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mlist\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mor\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[0misinstance\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mpredict_vector\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mlist\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 14\u001b[1;33m \u001b[1;32mraise\u001b[0m \u001b[0mpycmError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"Input Vectors Must Be List\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 15\u001b[0m \u001b[1;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mactual_vector\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m!=\u001b[0m\u001b[0mlen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mpredict_vector\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 16\u001b[0m \u001b[1;32mraise\u001b[0m \u001b[0mpycmError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"Input Vectors Must Be The Same Length\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
"\u001b[1;31mpycmError\u001b[0m: Input Vectors Must Be List"
]
}
],
"source": [
"cm2=ConfusionMatrix(y_actu, 2)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"cm3=ConfusionMatrix(y_actu, [1,2,3])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## References"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"1- Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977; 33:159–174
\n",
"\n",
"2- Powers, D. M. W. (2011). Evaluation: from precision, recall and f-measure to roc, informedness, markedness & correlation. Journal of Machine Learning Technologies.
\n",
"\n",
"\n",
"3- C. Sammut, G. Webb, Encyclopedia of Machine Learning. Springer, 2011. Springer reference.
\n",
"\n",
"4- Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378-382.\n",
"http://dx.doi.org/10.1037/h0031619
\n",
"\n",
"5- Altman D.G. 1991. Practical Statistics for Medical Research.\n",
"Chapman and Hall, London.
\n",
"\n",
"6- Gwet, K. L. (2008). Computing inter-rater reliability and its variance in the presence of high agreement. The British Journal of Mathematical and Statistical Psychology, 61(1), 29–48.
\n",
"\n",
"7- Scott, W. A. (1955). Reliability of content analysis: The case of nominal scaling. Public Opinion Quarterly, 19(3), 321–325.
\n",
"\n",
"8- Bennett, E. M., Alpert, R., & Goldstein, A. C. (1954). Communication through limited response questioning. The Public Opinion Quarterly, 18(3), 303–308.
\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.5.2"
}
},
"nbformat": 4,
"nbformat_minor": 2
}