{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import pandas as pd\n",
"import seaborn as sns\n",
"import matplotlib.pyplot as plt\n",
"import sys\n",
"import warnings\n",
"if not sys.warnoptions:\n",
" warnings.simplefilter(\"ignore\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Machine learning topics"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- Academics tend to use SVM\n",
"- Industry tends to use Logistic regressions"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 1. Linear classifiers\n",
"\n",
"Seperate object classes\n",
"\n",
"$$x^T w =0$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1.1 Python example: Linear classifiers"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 1.1.1 Importing dataset from sklearn "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### 1.1.1.1 User function: convert sklearn data $\\rightarrow$ dataframe"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"def to_df(dataset):\n",
" df = pd.DataFrame(dataset.data, columns= dataset.feature_names)\n",
" df['target'] = pd.Series(dataset.target)\n",
" df['target_cat'] = pd.Categorical.from_codes(dataset.target, dataset.target_names)\n",
" return df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### 1.1.1.2 Import breast cancer data set with user created fucntion"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" mean radius | \n",
" mean texture | \n",
" mean perimeter | \n",
" mean area | \n",
" mean smoothness | \n",
" mean compactness | \n",
" mean concavity | \n",
" mean concave points | \n",
" mean symmetry | \n",
" mean fractal dimension | \n",
" ... | \n",
" worst perimeter | \n",
" worst area | \n",
" worst smoothness | \n",
" worst compactness | \n",
" worst concavity | \n",
" worst concave points | \n",
" worst symmetry | \n",
" worst fractal dimension | \n",
" target | \n",
" target_cat | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" 17.99 | \n",
" 10.38 | \n",
" 122.8 | \n",
" 1001.0 | \n",
" 0.11840 | \n",
" 0.27760 | \n",
" 0.3001 | \n",
" 0.14710 | \n",
" 0.2419 | \n",
" 0.07871 | \n",
" ... | \n",
" 184.6 | \n",
" 2019.0 | \n",
" 0.1622 | \n",
" 0.6656 | \n",
" 0.7119 | \n",
" 0.2654 | \n",
" 0.4601 | \n",
" 0.11890 | \n",
" 0 | \n",
" malignant | \n",
"
\n",
" \n",
" | 1 | \n",
" 20.57 | \n",
" 17.77 | \n",
" 132.9 | \n",
" 1326.0 | \n",
" 0.08474 | \n",
" 0.07864 | \n",
" 0.0869 | \n",
" 0.07017 | \n",
" 0.1812 | \n",
" 0.05667 | \n",
" ... | \n",
" 158.8 | \n",
" 1956.0 | \n",
" 0.1238 | \n",
" 0.1866 | \n",
" 0.2416 | \n",
" 0.1860 | \n",
" 0.2750 | \n",
" 0.08902 | \n",
" 0 | \n",
" malignant | \n",
"
\n",
" \n",
"
\n",
"
2 rows × 32 columns
\n",
"
"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"from sklearn.svm import LinearSVC\n",
"from sklearn.linear_model import LogisticRegression\n",
"from sklearn.neighbors import KNeighborsClassifier\n",
"\n",
"fig, ax = plt.subplots(3, figsize=(14,12))\n",
"classifiers = ['LinearSVC', 'LogisticRegression', 'KNeighborsClassifier']\n",
"for idx, linear_class in enumerate(classifiers):\n",
" fig.add_subplot(3, 1, idx+1)\n",
" fig.subplots_adjust(hspace=.5)\n",
" LinearClass(X_twofeatures, y, linear_class)\n",
" plt.title('Linear classifier: ' + str(linear_class))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1.1 Support vector machine\n",
"\n",
"Chooses the line which maximizes the \"margin\"\n",
"\n",
"Margin width = $\\frac{1}{||w||}$\n",
"\n",
"Upper margin: $x^T w =1$\n",
"Lower margin: $x^T w =-1$\n",
"- Changes the y int\n",
"\n",
"- The margin width depends on the norm of w. \n",
"- If w is tiny then you need to change x a lot to get $x^T$ from 0 to 1\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 1.1.1 Python example\n",
"Code refrence: [Pythonprogramming](https://pythonprogramming.net/support-vector-machine-intro-machine-learning-tutorial/)"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
"#Import oop module\n",
"sys.path.append('/home/user/Modules')\n",
"from L3_SVM import *"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Optimized a step.\n",
"Optimized a step.\n",
"Optimized a step.\n",
"[1 7] : 1.271999999999435\n",
"[2 8] : 1.271999999999435\n",
"[3 8] : 1.0399999999995864\n",
"[5 1] : 1.0479999999990506\n",
"[ 6 -1] : 1.7439999999985962\n",
"[7 3] : 1.0479999999990506\n"
]
}
],
"source": [
"data_dict = {-1:np.array([[1,7],\n",
" [2,8],\n",
" [3,8],]),\n",
" \n",
" 1:np.array([[5,1],\n",
" [6,-1],\n",
" [7,3],])}\n",
"\n",
"svm = Support_Vector_Machine(visualization=False)\n",
"svm.fit(data=data_dict)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 1.1.1 Hinge loss\n",
"\n",
"$$\\sum_i h(x_i^T w)$$\n",
"\n",
"For class 1:\n",
"$\\sum_i h(x_i^T w)$\n",
"\n",
"For class -1:\n",
"$\\sum_i h(-x_i^T w)$\n",
"\n",
"In general:\n",
"$\\sum_i h (y_i * x_i^T w)$\n",
"\n",
"Combined objective:\n",
"\n",
"$$min \\frac{1}{2} ||w||^2 + h(\\hat{X} w)$$\n"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.3454905619336187"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from sklearn.metrics import hinge_loss\n",
"from sklearn import svm\n",
"X = cancer_df[['mean radius','mean texture']]\n",
"y = cancer_df.target.values\n",
"est= svm.LinearSVC()\n",
"est.fit(X,y)\n",
"predict = est.decision_function(X)\n",
"hinge_loss(y, predict)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1.2 Logistic regression\n",
"- Industry prefers more than academics\n",
"- Similar to hinge but more smooth\n",
"\n",
"$$f(z) =\\frac{e^z}{1+e^z}$$\n",
"\n",
"$$p[y=1]= \\frac{e^{x^T w}}{1+e^{x^T w}} \\\\\n",
"p[y=-1]= 1-\\frac{e^{x^T w}}{1+e^{x^T w}}= \\frac{1}{1+e^{x^T w}}$$\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 1.2.1 Python example: Logistic regression by hand\n",
"User created\n",
"#### 1.2.1.1 Fit model"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Iteration 0\n",
" Gradient: [-0.56 -1.6 ], Loss: 0.693, Theta: [0.06 0.16]\n",
"\n",
"Iteration 250\n",
" Gradient: [-5.81 -7.5 ], Loss: 3.908, Theta: [-1.8 2.09]\n",
"\n",
"Iteration 500\n",
" Gradient: [-5.8 -7.5], Loss: 3.909, Theta: [-1.8 2.09]\n",
"\n"
]
}
],
"source": [
"#Import oop module\n",
"sys.path.append('/home/user/Modules')\n",
"from L3_Linear import *\n",
"\n",
"#Import data\n",
"from sklearn.datasets import load_breast_cancer\n",
"cancer = load_breast_cancer()\n",
"X_cancer = cancer.data[:,:2] #First two columns\n",
"y_cancer = cancer.target\n",
"\n",
"#Fit model from imported module\n",
"model = LogisticRegression(verbose=True)\n",
"model.fit(X_cancer, y_cancer)"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([-1.80453427, 2.08825091])"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"Model_weights = model.theta\n",
"Model_weights "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### 1.2.1.2 Predict label\n",
"\n",
"$$p(y_i=1|x_i)=\\sigma(x_i w)=\\frac{e^{x_i w}}{1+e^{e^x_i w}}$$"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" mean radius | \n",
" mean texture | \n",
" target | \n",
" Probability | \n",
" Prediction | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" 17.99 | \n",
" 10.38 | \n",
" 0 | \n",
" 0.000021 | \n",
" 0 | \n",
"
\n",
" \n",
" | 1 | \n",
" 20.57 | \n",
" 17.77 | \n",
" 0 | \n",
" 0.497237 | \n",
" 0 | \n",
"
\n",
" \n",
" | 2 | \n",
" 19.69 | \n",
" 21.25 | \n",
" 0 | \n",
" 0.999856 | \n",
" 1 | \n",
"
\n",
" \n",
" | 3 | \n",
" 11.42 | \n",
" 20.38 | \n",
" 0 | \n",
" 1.000000 | \n",
" 1 | \n",
"
\n",
" \n",
"
\n",
"
"
]
},
"metadata": {
"image/png": {
"height": 357,
"width": 592
}
},
"output_type": "display_data"
}
],
"source": [
"plt.figure(figsize=(10, 6))\n",
"tn = (df_cancer.target==0) & (df_cancer.Prediction==0)\n",
"fn = (df_cancer.target==1) & (df_cancer.Prediction==0)\n",
"tp = (df_cancer.target==1) & (df_cancer.Prediction==1) \n",
"fp = (df_cancer.target==0) & (df_cancer.Prediction==1)\n",
"for data, name in zip([tn,fn,tp,fp], ['tn','fn','tp', 'fp']):\n",
" plt.scatter(df_cancer.loc[data, 'mean radius'], df_cancer.loc[data, 'mean texture'], label=str(name)+': n='+str(len(df_cancer.loc[data, 'mean texture'])))\n",
" plt.legend()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 1.2.1 Likelihood functions\n",
"\n",
"Probability of observing label given data\n",
"\n",
"Observed y=1\n",
"$$NLL(x) = log(1+e^{x^T w})-x^T w$$\n",
"\n",
"\n",
"Observed y=-1\n",
"$$NLL(x) = log(1+e^{x^T w})$$\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### 1.2.1.1 Python example: Likelihood function"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" Probability | \n",
" Prediction | \n",
" Prediction_sklearn | \n",
" nll_pos | \n",
" nll_neg | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" 0.000021 | \n",
" 0 | \n",
" 0 | \n",
" 10.7875 | \n",
" 0.0000 | \n",
"
\n",
" \n",
" | 1 | \n",
" 0.497237 | \n",
" 0 | \n",
" 0 | \n",
" 0.6987 | \n",
" 0.6876 | \n",
"
\n",
" \n",
" | 2 | \n",
" 0.999856 | \n",
" 1 | \n",
" 0 | \n",
" 0.0001 | \n",
" 8.8442 | \n",
"
\n",
" \n",
" | 3 | \n",
" 1.000000 | \n",
" 1 | \n",
" 1 | \n",
" 0.0000 | \n",
" 21.9508 | \n",
"
\n",
" \n",
" | 4 | \n",
" 0.001269 | \n",
" 0 | \n",
" 0 | \n",
" 6.6698 | \n",
" 0.0013 | \n",
"
\n",
" \n",
"
\n",
"