{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PYCM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Version : 0.6\n",
    "-----"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pycm import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_actu = [2, 0, 2, 2, 0, 1, 1, 2, 2, 0, 1, 2]\n",
    "y_pred = [0, 0, 2, 1, 0, 2, 1, 0, 2, 0, 2, 2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "cm = ConfusionMatrix(y_actu, y_pred,digit=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  digit (the number of digits to the right of the decimal point in a number) is new in version 0.6 (default valaue : 5)\n",
    "* Only for print and save"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pycm.ConfusionMatrix(classes: [0, 1, 2])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[2, 0, 2, 2, 0, 1, 1, 2, 2, 0, 1, 2]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.actual_vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 0, 2, 1, 0, 2, 1, 0, 2, 0, 2, 2]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.predict_vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 1, 2]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'ACC': {0: 0.8333333333333334, 1: 0.75, 2: 0.5833333333333334},\n",
       " 'BM': {0: 0.7777777777777777, 1: 0.2222222222222221, 2: 0.16666666666666652},\n",
       " 'DOR': {0: 'None', 1: 3.999999999999998, 2: 1.9999999999999998},\n",
       " 'ERR': {0: 0.16666666666666663, 1: 0.25, 2: 0.41666666666666663},\n",
       " 'F0.5': {0: 0.6521739130434783,\n",
       "  1: 0.45454545454545453,\n",
       "  2: 0.5769230769230769},\n",
       " 'F1': {0: 0.75, 1: 0.4, 2: 0.5454545454545454},\n",
       " 'F2': {0: 0.8823529411764706, 1: 0.35714285714285715, 2: 0.5172413793103449},\n",
       " 'FDR': {0: 0.4, 1: 0.5, 2: 0.4},\n",
       " 'FN': {0: 0, 1: 2, 2: 3},\n",
       " 'FNR': {0: 0.0, 1: 0.6666666666666667, 2: 0.5},\n",
       " 'FOR': {0: 0.0, 1: 0.19999999999999996, 2: 0.4285714285714286},\n",
       " 'FP': {0: 2, 1: 1, 2: 2},\n",
       " 'FPR': {0: 0.2222222222222222,\n",
       "  1: 0.11111111111111116,\n",
       "  2: 0.33333333333333337},\n",
       " 'G': {0: 0.7745966692414834, 1: 0.408248290463863, 2: 0.5477225575051661},\n",
       " 'LR+': {0: 4.5, 1: 2.9999999999999987, 2: 1.4999999999999998},\n",
       " 'LR-': {0: 0.0, 1: 0.7500000000000001, 2: 0.75},\n",
       " 'MCC': {0: 0.6831300510639732, 1: 0.25819888974716115, 2: 0.1690308509457033},\n",
       " 'MK': {0: 0.6000000000000001, 1: 0.30000000000000004, 2: 0.17142857142857126},\n",
       " 'N': {0: 9, 1: 9, 2: 6},\n",
       " 'NPV': {0: 1.0, 1: 0.8, 2: 0.5714285714285714},\n",
       " 'P': {0: 3, 1: 3, 2: 6},\n",
       " 'POP': {0: 12, 1: 12, 2: 12},\n",
       " 'PPV': {0: 0.6, 1: 0.5, 2: 0.6},\n",
       " 'PRE': {0: 0.25, 1: 0.25, 2: 0.5},\n",
       " 'RACC': {0: 0.10416666666666667,\n",
       "  1: 0.041666666666666664,\n",
       "  2: 0.20833333333333334},\n",
       " 'TN': {0: 7, 1: 8, 2: 4},\n",
       " 'TNR': {0: 0.7777777777777778, 1: 0.8888888888888888, 2: 0.6666666666666666},\n",
       " 'TON': {0: 7, 1: 10, 2: 7},\n",
       " 'TOP': {0: 5, 1: 2, 2: 5},\n",
       " 'TP': {0: 3, 1: 1, 2: 3},\n",
       " 'TPR': {0: 1.0, 1: 0.3333333333333333, 2: 0.5}}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.class_stat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  cm.statistic_result in prev versions (<0.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Bennett_S': 0.37500000000000006,\n",
       " 'Gwet_AC1': 0.3893129770992367,\n",
       " 'Kappa': 0.35483870967741943,\n",
       " 'Overall_ACC': 0.5833333333333334,\n",
       " 'Overall_RACC': 0.3541666666666667,\n",
       " 'PPV_Macro': 0.5666666666666668,\n",
       " 'PPV_Micro': 0.5833333333333334,\n",
       " 'Scott_PI': 0.34426229508196726,\n",
       " 'Strength_Of_Agreement(Altman)': 'Fair',\n",
       " 'Strength_Of_Agreement(Fleiss)': 'Poor',\n",
       " 'Strength_Of_Agreement(Landis and Koch)': 'Fair',\n",
       " 'TPR_Macro': 0.611111111111111,\n",
       " 'TPR_Micro': 0.5833333333333334}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.overall_stat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: {0: 3, 1: 0, 2: 0}, 1: {0: 0, 1: 1, 2: 2}, 2: {0: 2, 1: 1, 2: 3}}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic Parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TP (True positive / hit)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 3, 1: 1, 2: 3}"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.TP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TN (True negative/correct rejection)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 7, 1: 8, 2: 4}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.TN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### FP (False positive/false alarm/Type I error)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 2, 1: 1, 2: 2}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.FP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### FN (False negative/miss/Type II error)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0, 1: 2, 2: 3}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.FN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### P (Condition positive)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 3, 1: 3, 2: 6}"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.P"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### N (Condition negative)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 9, 1: 9, 2: 6}"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.N"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TOP (Test outcome positive)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 5, 1: 2, 2: 5}"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.TOP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TON (Test outcome negative)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 7, 1: 10, 2: 7}"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.TON"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### POP (Population)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 12, 1: 12, 2: 12}"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.POP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Class Statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TPR  (sensitivity, recall, hit rate, or true positive rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$TPR=\\frac{TP}{P}=\\frac{TP}{TP+FN}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 1.0, 1: 0.3333333333333333, 2: 0.5}"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.TPR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TNR (specificity or true negative rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$TNR=\\frac{TN}{N}=\\frac{TN}{TN+FP}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0.7777777777777778, 1: 0.8888888888888888, 2: 0.6666666666666666}"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.TNR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PPV (precision or positive predictive value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$PPV=\\frac{TP}{TP+FP}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0.6, 1: 0.5, 2: 0.6}"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.PPV"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NPV (negative predictive value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$NPV=\\frac{TN}{TN+FN}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 1.0, 1: 0.8, 2: 0.5714285714285714}"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.NPV"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### FNR (miss rate or false negative rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$FNR=\\frac{FN}{P}=\\frac{FN}{FN+TP}=1-TPR$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0.0, 1: 0.6666666666666667, 2: 0.5}"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.FNR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### FPR (fall-out or false positive rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$FPR=\\frac{FP}{N}=\\frac{FP}{FP+TN}=1-TNR$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0.2222222222222222, 1: 0.11111111111111116, 2: 0.33333333333333337}"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.FPR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### FDR (false discovery rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$FDR=\\frac{FP}{FP+TP}=1-PPV$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0.6, 1: 0.5, 2: 0.6}"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.PPV"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### FOR (false omission rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$FOR=\\frac{FN}{FN+TN}=1-NPV$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0.0, 1: 0.19999999999999996, 2: 0.4285714285714286}"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.FOR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ACC (accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ACC=\\frac{TP+TN}{P+N}=\\frac{TP+TN}{TP+TN+FP+FN}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0.8333333333333334, 1: 0.75, 2: 0.5833333333333334}"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.ACC"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ERR(Error rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ERR=\\frac{FP+FN}{P+N}=\\frac{FP+FN}{TP+TN+FP+FN}=1-ACC$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0.16666666666666663, 1: 0.25, 2: 0.41666666666666663}"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.ERR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### FBeta-Score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$F_{\\beta}=(1+\\beta^2).\\frac{PPV.TPR}{(\\beta^2.PPV)+TPR}=\\frac{(1+\\beta^2).TP}{(1+\\beta^2).TP+FP+\\beta^2.FN}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0.75, 1: 0.4, 2: 0.5454545454545454}"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.F1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0.6521739130434783, 1: 0.45454545454545453, 2: 0.5769230769230769}"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.F05"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0.8823529411764706, 1: 0.35714285714285715, 2: 0.5172413793103449}"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.F2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0.9622641509433962, 1: 0.34, 2: 0.504950495049505}"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.F_beta(Beta=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### MCC (Matthews correlation coefficient)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$MCC=\\frac{TP \\times TN-FP \\times FN}{\\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)}}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0.6831300510639732, 1: 0.25819888974716115, 2: 0.1690308509457033}"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.MCC"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### BM (Informedness or Bookmaker Informedness)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$BM=TPR+TNR-1$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0.7777777777777777, 1: 0.2222222222222221, 2: 0.16666666666666652}"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.BM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### MK (Markedness)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$MK=PPV+NPV-1$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0.6000000000000001, 1: 0.30000000000000004, 2: 0.17142857142857126}"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.MK"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PLR (Positive likelihood ratio)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$(LR+)=\\frac{TPR}{FPR}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 4.5, 1: 2.9999999999999987, 2: 1.4999999999999998}"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.PLR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NLR (Negative likelihood ratio)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$(LR-)=\\frac{FNR}{TNR}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0.0, 1: 0.7500000000000001, 2: 0.75}"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.NLR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DOR (Diagnostic odds ratio)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$DOR=\\frac{LR+}{LR-}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 'None', 1: 3.999999999999998, 2: 1.9999999999999998}"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.DOR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PRE (Prevalence)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$Prevalence=\\frac{P}{Population}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0.25, 1: 0.25, 2: 0.5}"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.PRE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### G (G-measure geometric mean of precision and sensitivity)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$G=\\sqrt{PPV.TPR}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0.7745966692414834, 1: 0.408248290463863, 2: 0.5477225575051661}"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.G"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### RACC(Random accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$RACC=\\frac{TOP\\times P}{Population^2}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0.10416666666666667, 1: 0.041666666666666664, 2: 0.20833333333333334}"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.RACC"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overall Statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Kappa (Nominal)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$Kappa=\\frac{ACC_{Overall}-RACC_{Overall}}{1-RACC_{Overall}}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.35483870967741943"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.Kappa"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bennett et al.'s S score (Nominal)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$p_c=\\frac{1}{|C|}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$S=\\frac{ACC_{Overall}-p_c}{1-p_c}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.37500000000000006"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.S"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scott's pi (Nominal)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$p_c=\\sum_{i=1}^{|C|}(\\frac{TOP_i + P_i}{2\\times Population})^2$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\pi=\\frac{ACC_{Overall}-p_c}{1-p_c}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.34426229508196726"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.PI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Gwet's AC1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\pi=\\frac{TOP_i + P_i}{2\\times Population}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$p_c=\\frac{1}{|C|-1}\\sum_{i=1}^{|C|}(\\pi_i\\times (1-\\pi_i))$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$AC1=\\frac{ACC_{Overall}-p_c}{1-p_c}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.3893129770992367"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.AC1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SOA1 (Strength of Agreement, Landis and Koch benchmark)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table>\n",
    "       <tr>\n",
    "           <td style=\"text-align:center\">Kappa</td>\n",
    "           <td style=\"text-align:center\">Strength of Agreement</td>\n",
    "       </tr>\n",
    "        <tr>\n",
    "           <td style=\"text-align:center\"> 0 ></td>\n",
    "           <td style=\"text-align:center\">Poor</td>\n",
    "       </tr>\n",
    "       <tr>\n",
    "           <td style=\"text-align:center\">0 - 0.20</td>\n",
    "           <td style=\"text-align:center\">Slight</td>\n",
    "       </tr>\n",
    "        <tr>\n",
    "           <td style=\"text-align:center\">0.21 – 0.40</td>\n",
    "           <td style=\"text-align:center\">Fair</td>\n",
    "       </tr>\n",
    "        <tr>\n",
    "           <td style=\"text-align:center\">0.41 – 0.60</td>\n",
    "           <td style=\"text-align:center\">Moderate</td>\n",
    "       </tr>\n",
    "       <tr>\n",
    "           <td style=\"text-align:center\">0.61 – 0.80</td>\n",
    "           <td style=\"text-align:center\">Substantial</td>\n",
    "       </tr>\n",
    "       <tr>\n",
    "           <td style=\"text-align:center\">0.81 – 1.00</td>\n",
    "           <td style=\"text-align:center\">Almost perfect</td>\n",
    "       </tr>\n",
    "    \n",
    "    \n",
    "</table>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Fair'"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.SOA1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SOA2 (Strength of Agreement, : Fleiss’ benchmark)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table>\n",
    "       <tr>\n",
    "           <td style=\"text-align:center\">Kappa</td>\n",
    "           <td style=\"text-align:center\">Strength of Agreement</td>\n",
    "       </tr>\n",
    "        <tr>\n",
    "           <td style=\"text-align:center\"> 0.40 ></td>\n",
    "           <td style=\"text-align:center\">Poor</td>\n",
    "       </tr>\n",
    "       <tr>\n",
    "           <td style=\"text-align:center\">0.4 - 0.75</td>\n",
    "           <td style=\"text-align:center\">Intermediate to Good</td>\n",
    "       </tr>\n",
    "       <tr>\n",
    "           <td style=\"text-align:center\">More than 0.75</td>\n",
    "           <td style=\"text-align:center\">Excellent</td>\n",
    "       </tr>\n",
    "    \n",
    "    \n",
    "</table>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Poor'"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.SOA2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SOA3 (Strength of Agreement, Altman’s benchmark)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table>\n",
    "       <tr>\n",
    "           <td style=\"text-align:center\">Kappa</td>\n",
    "           <td style=\"text-align:center\">Strength of Agreement</td>\n",
    "       </tr>\n",
    "        <tr>\n",
    "           <td style=\"text-align:center\"> 0.2 ></td>\n",
    "           <td style=\"text-align:center\">Poor</td>\n",
    "       </tr>\n",
    "        <tr>\n",
    "           <td style=\"text-align:center\">0.21 – 0.40</td>\n",
    "           <td style=\"text-align:center\">Fair</td>\n",
    "       </tr>\n",
    "        <tr>\n",
    "           <td style=\"text-align:center\">0.41 – 0.60</td>\n",
    "           <td style=\"text-align:center\">Moderate</td>\n",
    "       </tr>\n",
    "       <tr>\n",
    "           <td style=\"text-align:center\">0.61 – 0.80</td>\n",
    "           <td style=\"text-align:center\">Good</td>\n",
    "       </tr>\n",
    "       <tr>\n",
    "           <td style=\"text-align:center\">0.81 – 1.00</td>\n",
    "           <td style=\"text-align:center\">Very Good</td>\n",
    "       </tr>\n",
    "    \n",
    "    \n",
    "</table>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Fair'"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.SOA3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Overall_ACC"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ACC_{Overall}=\\frac{\\sum_{i=1}^{|C|}TP_i}{Population}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5833333333333334"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.Overall_ACC"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Overall_RACC"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$RACC_{Overall}=\\sum_{i=1}^{|C|}RACC_i$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.3541666666666667"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.Overall_RACC"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PPV_Micro"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$PPV_{Micro}=\\frac{\\sum_{i=1}^{|C|}TP_i}{\\sum_{i=1}^{|C|}TP_i+FP_i}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5833333333333334"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.PPV_Micro"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TPR_Micro"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$TPR_{Micro}=\\frac{\\sum_{i=1}^{|C|}TP_i}{\\sum_{i=1}^{|C|}TP_i+FN_i}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5833333333333334"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.TPR_Micro"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PPV_Macro"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$PPV_{Macro}=\\frac{1}{|C|}\\sum_{i=1}^{|C|}\\frac{TP_i}{TP_i+FP_i}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5666666666666668"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.PPV_Macro"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TPR_Macro"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$TPR_{Macro}=\\frac{1}{|C|}\\sum_{i=1}^{|C|}\\frac{TP_i}{TP_i+FN_i}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.611111111111111"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.TPR_Macro"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Print"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Full"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predict          0        1        2        \n",
      "Actual\n",
      "0                3        0        0        \n",
      "1                0        1        2        \n",
      "2                2        1        3        \n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Overall Statistics : \n",
      "\n",
      "Bennett_S                                                        0.375\n",
      "Gwet_AC1                                                         0.38931\n",
      "Kappa                                                            0.35484\n",
      "Overall_ACC                                                      0.58333\n",
      "Overall_RACC                                                     0.35417\n",
      "PPV_Macro                                                        0.56667\n",
      "PPV_Micro                                                        0.58333\n",
      "Scott_PI                                                         0.34426\n",
      "Strength_Of_Agreement(Altman)                                    Fair\n",
      "Strength_Of_Agreement(Fleiss)                                    Poor\n",
      "Strength_Of_Agreement(Landis and Koch)                           Fair\n",
      "TPR_Macro                                                        0.61111\n",
      "TPR_Micro                                                        0.58333\n",
      "\n",
      "Class Statistics :\n",
      "\n",
      "Classes                                                          0                       1                       2                       \n",
      "ACC(Accuracy)                                                    0.83333                 0.75                    0.58333                 \n",
      "BM(Informedness or bookmaker informedness)                       0.77778                 0.22222                 0.16667                 \n",
      "DOR(Diagnostic odds ratio)                                       None                    4.0                     2.0                     \n",
      "ERR(Error rate)                                                  0.16667                 0.25                    0.41667                 \n",
      "F0.5(F0.5 score)                                                 0.65217                 0.45455                 0.57692                 \n",
      "F1(F1 score - harmonic mean of precision and sensitivity)        0.75                    0.4                     0.54545                 \n",
      "F2(F2 score)                                                     0.88235                 0.35714                 0.51724                 \n",
      "FDR(False discovery rate)                                        0.4                     0.5                     0.4                     \n",
      "FN(False negative/miss/type 2 error)                             0                       2                       3                       \n",
      "FNR(Miss rate or false negative rate)                            0.0                     0.66667                 0.5                     \n",
      "FOR(False omission rate)                                         0.0                     0.2                     0.42857                 \n",
      "FP(False positive/type 1 error/false alarm)                      2                       1                       2                       \n",
      "FPR(Fall-out or false positive rate)                             0.22222                 0.11111                 0.33333                 \n",
      "G(G-measure geometric mean of precision and sensitivity)         0.7746                  0.40825                 0.54772                 \n",
      "LR+(Positive likelihood ratio)                                   4.5                     3.0                     1.5                     \n",
      "LR-(Negative likelihood ratio)                                   0.0                     0.75                    0.75                    \n",
      "MCC(Matthews correlation coefficient)                            0.68313                 0.2582                  0.16903                 \n",
      "MK(Markedness)                                                   0.6                     0.3                     0.17143                 \n",
      "N(Condition negative)                                            9                       9                       6                       \n",
      "NPV(Negative predictive value)                                   1.0                     0.8                     0.57143                 \n",
      "P(Condition positive)                                            3                       3                       6                       \n",
      "POP(Population)                                                  12                      12                      12                      \n",
      "PPV(Precision or positive predictive value)                      0.6                     0.5                     0.6                     \n",
      "PRE(Prevalence)                                                  0.25                    0.25                    0.5                     \n",
      "RACC(Random accuracy)                                            0.10417                 0.04167                 0.20833                 \n",
      "TN(True negative/correct rejection)                              7                       8                       4                       \n",
      "TNR(Specificity or true negative rate)                           0.77778                 0.88889                 0.66667                 \n",
      "TON(Test outcome negative)                                       7                       10                      7                       \n",
      "TOP(Test outcome positive)                                       5                       2                       5                       \n",
      "TP(True positive/hit)                                            3                       1                       3                       \n",
      "TPR(Sensitivity, recall, hit rate, or true positive rate)        1.0                     0.33333                 0.5                     \n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(cm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predict          0        1        2        \n",
      "Actual\n",
      "0                3        0        0        \n",
      "1                0        1        2        \n",
      "2                2        1        3        \n",
      "\n"
     ]
    }
   ],
   "source": [
    "cm.matrix()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Normalized Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predict          0              1              2              \n",
      "Actual\n",
      "0                1.0            0.0            0.0            \n",
      "1                0.0            0.33333        0.66667        \n",
      "2                0.33333        0.16667        0.5            \n",
      "\n"
     ]
    }
   ],
   "source": [
    "cm.normalized_matrix()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overall Statistics : \n",
      "\n",
      "Bennett_S                                                        0.375\n",
      "Gwet_AC1                                                         0.38931\n",
      "Kappa                                                            0.35484\n",
      "Overall_ACC                                                      0.58333\n",
      "Overall_RACC                                                     0.35417\n",
      "PPV_Macro                                                        0.56667\n",
      "PPV_Micro                                                        0.58333\n",
      "Scott_PI                                                         0.34426\n",
      "Strength_Of_Agreement(Altman)                                    Fair\n",
      "Strength_Of_Agreement(Fleiss)                                    Poor\n",
      "Strength_Of_Agreement(Landis and Koch)                           Fair\n",
      "TPR_Macro                                                        0.61111\n",
      "TPR_Micro                                                        0.58333\n",
      "\n",
      "Class Statistics :\n",
      "\n",
      "Classes                                                          0                       1                       2                       \n",
      "ACC(Accuracy)                                                    0.83333                 0.75                    0.58333                 \n",
      "BM(Informedness or bookmaker informedness)                       0.77778                 0.22222                 0.16667                 \n",
      "DOR(Diagnostic odds ratio)                                       None                    4.0                     2.0                     \n",
      "ERR(Error rate)                                                  0.16667                 0.25                    0.41667                 \n",
      "F0.5(F0.5 score)                                                 0.65217                 0.45455                 0.57692                 \n",
      "F1(F1 score - harmonic mean of precision and sensitivity)        0.75                    0.4                     0.54545                 \n",
      "F2(F2 score)                                                     0.88235                 0.35714                 0.51724                 \n",
      "FDR(False discovery rate)                                        0.4                     0.5                     0.4                     \n",
      "FN(False negative/miss/type 2 error)                             0                       2                       3                       \n",
      "FNR(Miss rate or false negative rate)                            0.0                     0.66667                 0.5                     \n",
      "FOR(False omission rate)                                         0.0                     0.2                     0.42857                 \n",
      "FP(False positive/type 1 error/false alarm)                      2                       1                       2                       \n",
      "FPR(Fall-out or false positive rate)                             0.22222                 0.11111                 0.33333                 \n",
      "G(G-measure geometric mean of precision and sensitivity)         0.7746                  0.40825                 0.54772                 \n",
      "LR+(Positive likelihood ratio)                                   4.5                     3.0                     1.5                     \n",
      "LR-(Negative likelihood ratio)                                   0.0                     0.75                    0.75                    \n",
      "MCC(Matthews correlation coefficient)                            0.68313                 0.2582                  0.16903                 \n",
      "MK(Markedness)                                                   0.6                     0.3                     0.17143                 \n",
      "N(Condition negative)                                            9                       9                       6                       \n",
      "NPV(Negative predictive value)                                   1.0                     0.8                     0.57143                 \n",
      "P(Condition positive)                                            3                       3                       6                       \n",
      "POP(Population)                                                  12                      12                      12                      \n",
      "PPV(Precision or positive predictive value)                      0.6                     0.5                     0.6                     \n",
      "PRE(Prevalence)                                                  0.25                    0.25                    0.5                     \n",
      "RACC(Random accuracy)                                            0.10417                 0.04167                 0.20833                 \n",
      "TN(True negative/correct rejection)                              7                       8                       4                       \n",
      "TNR(Specificity or true negative rate)                           0.77778                 0.88889                 0.66667                 \n",
      "TON(Test outcome negative)                                       7                       10                      7                       \n",
      "TOP(Test outcome positive)                                       5                       2                       5                       \n",
      "TP(True positive/hit)                                            3                       1                       3                       \n",
      "TPR(Sensitivity, recall, hit rate, or true positive rate)        1.0                     0.33333                 0.5                     \n",
      "\n"
     ]
    }
   ],
   "source": [
    "cm.stat()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  cm.params() in prev versions (<0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### .pcym file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Message': 'C:\\\\Users\\\\Sepkjaer\\\\Desktop\\\\Desktop\\\\JupyterNotebooks\\\\cm1.pycm',\n",
       " 'Status': True}"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.save_stat(\"cm1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Message': \"[Errno 2] No such file or directory: 'cm1asdasd/.pycm'\",\n",
       " 'Status': False}"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.save_stat(\"cm1asdasd/\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### HTML"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Message': 'C:\\\\Users\\\\Sepkjaer\\\\Desktop\\\\Desktop\\\\JupyterNotebooks\\\\cm1.html',\n",
       " 'Status': True}"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.save_html(\"cm1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Message': \"[Errno 2] No such file or directory: 'cm1asdasd/.html'\",\n",
       " 'Status': False}"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.save_html(\"cm1asdasd/\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### CSV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Message': 'C:\\\\Users\\\\Sepkjaer\\\\Desktop\\\\Desktop\\\\JupyterNotebooks\\\\cm1.csv',\n",
       " 'Status': True}"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.save_csv(\"cm1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Message': \"[Errno 2] No such file or directory: 'cm1asdasd/.csv'\",\n",
       " 'Status': False}"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm.save_csv(\"cm1asdasd/\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Notice :  new in version 0.6"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Input Errors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "ename": "pycmError",
     "evalue": "Input Vectors Must Be List",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mpycmError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-66-572bf15e689d>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mcm2\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mConfusionMatrix\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0my_actu\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;36m2\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;32m~\\AppData\\Local\\Programs\\Python\\Python35-32\\lib\\site-packages\\pycm\\pycm_obj.py\u001b[0m in \u001b[0;36m__init__\u001b[1;34m(self, actual_vector, predict_vector, digit)\u001b[0m\n\u001b[0;32m     12\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0m__init__\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mactual_vector\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mpredict_vector\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mdigit\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;36m5\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     13\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[0misinstance\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mactual_vector\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mlist\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mor\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[0misinstance\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mpredict_vector\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mlist\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 14\u001b[1;33m             \u001b[1;32mraise\u001b[0m \u001b[0mpycmError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"Input Vectors Must Be List\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     15\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mactual_vector\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m!=\u001b[0m\u001b[0mlen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mpredict_vector\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     16\u001b[0m             \u001b[1;32mraise\u001b[0m \u001b[0mpycmError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"Input Vectors Must Be The Same Length\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mpycmError\u001b[0m: Input Vectors Must Be List"
     ]
    }
   ],
   "source": [
    "cm2=ConfusionMatrix(y_actu, 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "ename": "pycmError",
     "evalue": "Input Vectors Must Be The Same Length",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mpycmError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-67-fe0a030b981a>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mcm3\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mConfusionMatrix\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0my_actu\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m[\u001b[0m\u001b[1;36m1\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;36m2\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;36m3\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;32m~\\AppData\\Local\\Programs\\Python\\Python35-32\\lib\\site-packages\\pycm\\pycm_obj.py\u001b[0m in \u001b[0;36m__init__\u001b[1;34m(self, actual_vector, predict_vector, digit)\u001b[0m\n\u001b[0;32m     14\u001b[0m             \u001b[1;32mraise\u001b[0m \u001b[0mpycmError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"Input Vectors Must Be List\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     15\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mactual_vector\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m!=\u001b[0m\u001b[0mlen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mpredict_vector\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 16\u001b[1;33m             \u001b[1;32mraise\u001b[0m \u001b[0mpycmError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"Input Vectors Must Be The Same Length\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     17\u001b[0m         \u001b[0mmatrix_param\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mmatrix_params_calc\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mactual_vector\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mpredict_vector\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     18\u001b[0m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdigit\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mdigit\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mpycmError\u001b[0m: Input Vectors Must Be The Same Length"
     ]
    }
   ],
   "source": [
    "cm3=ConfusionMatrix(y_actu, [1,2,3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<blockquote>1- Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977; 33:159–174 </blockquote>\n",
    "\n",
    "<blockquote>2- Powers, D. M. W. (2011). Evaluation: from precision, recall and f-measure to roc, informedness, markedness & correlation. Journal of Machine Learning Technologies.</blockquote>\n",
    "\n",
    "\n",
    "<blockquote>3-  C. Sammut, G. Webb, Encyclopedia of Machine Learning. Springer, 2011. Springer reference.</blockquote>\n",
    "\n",
    "<blockquote>4- Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378-382.\n",
    "http://dx.doi.org/10.1037/h0031619</blockquote>\n",
    "\n",
    "<blockquote>5- Altman D.G. 1991. Practical Statistics for Medical Research.\n",
    "Chapman and Hall, London.</blockquote>\n",
    "\n",
    "<blockquote>6- Gwet, K. L. (2008). Computing inter-rater reliability and its variance in the presence of high agreement. The British Journal of Mathematical and Statistical Psychology, 61(1), 29–48.</blockquote>\n",
    "\n",
    "<blockquote>7- Scott, W. A. (1955). Reliability of content analysis: The case of nominal scaling. Public Opinion Quarterly, 19(3), 321–325.</blockquote>\n",
    "\n",
    "<blockquote>8- Bennett, E. M., Alpert, R., & Goldstein, A. C. (1954). Communication through limited response questioning. The Public Opinion Quarterly, 18(3), 303–308.</blockquote>\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}