{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# # FOR EVALUATION USE BY GRAB" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "___\n", "# Background:\n", "According to the World Health Organization, approximately 1.35 million people die as a result of a road traffic crash every year. Between 20 and 50 million more suffer non-fatal injuries, with many incurring a disability as a result of their injury.\n", "\n", "Road traffic injuries cause considerable economic losses to individuals, their families, and to nations as a whole. These losses arise from the cost of treatment as well as lost productivity for those killed or disabled by their injuries, and for family members who need to take time off work or school to care for the injured. Road traffic crashes cost most countries 3% of their gross domestic product.\n", "\n", "In my project, I will derive a classification model primarily through feature engineering to detect dangerous driving trips using time-series telemetry data from Grab. This model will be evaluated based on the AUC-ROC curve.\n", "\n", "##### Dataset Obtained from (https://s3-ap-southeast-1.amazonaws.com/grab-aiforsea-dataset/safety.zip)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "_______\n", "# Dataset characteristics:\n", "\n", "Dataset is a time-series of the following features for every \"bookingID\"\n", "\n", "|Datapoint: | Description: | \n", "| ------------- |:-------------: |\n", "|bookingID |trip id |\n", "|Label |1 indicates dangerous driving & 0 indicates safe driving |\n", "\n", "|Features (time-series): | Description: |\n", "| ------------- |:-------------: |\n", "|Accuracy |accuracy inferred by GPS in meters | \n", "|Bearing |GPS bearing in degree |\n", "|acceleration_x |accelerometer reading at x axis (m/s2) |\n", "|acceleration_y |accelerometer reading at y axis (m/s2) |\n", "|acceleration_z |accelerometer reading at z axis (m/s2) |\n", "|gyro_x |gyroscope reading in x axis (rad/s) |\n", "|gyro_y |gyroscope reading in y axis (rad/s) |\n", "|gyro_z |gyroscope reading in z axis (rad/s) |\n", "|second |time of the record by number of seconds|\n", "|Speed |speed measured by GPS in m/s |" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "____\n", "# Engineered Features:\n", "Features are decided upon primarily through domain knowledge and analysis of correlation with label using a pairplot.\n", "\n", "|Engineered Features (Datapoint):|Description: |\n", "| ------------- |---|\n", "|Accuracy |Median Accuracy value with respect to range of timeseries data. If Median Accuracy is not available, Min Accuracy value is used instead \n", "|Bearing_delta |Standard Deviation of the difference between each timeseries datapoint\n", "|acceleration_x |Number of threshold crossovers with threshold set at 0\n", "|acceleration_y |Number of threshold crossovers with threshold set at Mean of acceleration_y\n", "|acceleration_z |Number of threshold crossovers with threshold set at Mean of acceleration_z\n", "|gyro_x |Max gyro_x value - Min gyro_z value\n", "|gyro_y |Max gyro_y value - Min gyro_z value\n", "|gyro_z |Max gyro_z value - Min gyro_z value\n", "|Speed |Mean of Speed in timeseries data\n", "|missing_rng |Number of missing values in the range of timeseries data" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "import pandas as pd\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "import seaborn as sns\n", "import math\n", "import matplotlib.pyplot as plt\n", "from pprint import pprint\n", "sns.set()\n", "from pathlib import Path" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Using TensorFlow backend.\n" ] } ], "source": [ "import tensorflow as tf\n", "from sklearn.metrics import roc_auc_score\n", "from keras.models import Sequential\n", "from keras.layers import Dense\n", "from keras.wrappers.scikit_learn import KerasClassifier\n", "\n", "from sklearn.model_selection import train_test_split\n", "from keras.models import load_model" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Helper Functions" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "def threshold_crosssovers(df, metric, threshold):\n", " f = np.array(list(df[['second', metric]].itertuples(index=False, name=None)))\n", " g = [(0, int(threshold))] * len(f)\n", " idx = np.argwhere(np.diff(np.sign(f - g))).flatten()\n", " return int(len(idx)/2)\n", "\n", "def missing_range(df):\n", " return int(len(set(df['second']) ^ set(range(0, int(df['second'].iloc[-1])))))\n", "\n", "def difference(dataset, interval=1):\n", " diff = []\n", " for i in range(interval, len(dataset)):\n", " value = dataset[i] - dataset[i - interval]\n", " diff.append(value)\n", " return diff\n", "\n", "def match_ID_to_label(ID):\n", " return df_labels[df_labels['bookingID']==ID]['label'].iloc[0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Function to convert TimeSeries to Datapoint" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "def convert_timeseries_to_datapoint(ID):\n", " ID_df = df_features[df_features['bookingID']==ID]\n", " ID_df_stats = ID_df.describe()\n", " \n", " missing_rng = missing_range(ID_df)\n", " if missing_rng==0:\n", " Accuracy = ID_df_stats['Accuracy']['50%']\n", " else:\n", " median_idx = int(ID_df['second'].iloc[-1]/2)\n", " if median_idx in set(ID_df['second']):\n", " Accuracy = ID_df[ID_df['second']==median_idx]['Accuracy'].iloc[0]\n", " else:\n", " Accuracy = ID_df_stats['Accuracy']['min']\n", " Bearing_delta = np.std(difference(ID_df.loc[:, ['Bearing', 'second']].values))\n", " acceleration_x = threshold_crosssovers(ID_df, 'acceleration_x', 0) \n", " acceleration_y = threshold_crosssovers(ID_df, 'acceleration_y', ID_df_stats['acceleration_y']['mean']) \n", " acceleration_z = threshold_crosssovers(ID_df, 'acceleration_z', ID_df_stats['acceleration_z']['mean'])\n", " gyro_x = ID_df_stats['gyro_x']['max'] - ID_df_stats['gyro_x']['min']\n", " gyro_y = ID_df_stats['gyro_y']['max'] - ID_df_stats['gyro_y']['min']\n", " gyro_z = ID_df_stats['gyro_z']['max'] - ID_df_stats['gyro_z']['min']\n", " Speed = ID_df_stats['Speed']['mean']\n", " \n", " return [ID, Accuracy, Bearing_delta, acceleration_x, acceleration_y, acceleration_z, gyro_x, gyro_y, gyro_z, Speed, missing_rng]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Read Datasets & sort them according to BookingID & seconds(if any)" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "# Read cleaned features .csv file to ipynb\n", "df_features = pd.read_csv('safety/data_cleaned_features.csv')\n", "df_features = df_features.sort_values(['bookingID', 'second']).reset_index(drop=True)\n", "# Read cleaned labels .csv file to ipynb\n", "df_labels = pd.read_csv('safety/data_cleaned_labels.csv')\n", "df_labels = df_labels.sort_values('bookingID').reset_index(drop=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Find unique_BookingIDs " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "unique_BookingIDs = df_labels['bookingID'].unique().tolist()\n", "print(unique_BookingIDs)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Convert timeseries to Datapoint" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "col=['BookingID', 'Accuracy', 'Bearing', \n", " 'acceleration_x', 'acceleration_y', 'acceleration_z', \n", " 'gyro_x', 'gyro_y', 'gyro_z', \n", " 'Speed']\n", "\n", "eng_features_col = ['ID', 'Accuracy', 'Bearing_delta', \n", " 'acceleration_x', 'acceleration_y', 'acceleration_z',\n", " 'gyro_x', 'gyro_y', 'gyro_z',\n", " 'Speed', 'missing_rng']\n", "\n", "df_datapoints = pd.DataFrame()\n", "\n", "for ID in unique_BookingIDs:\n", " datapoint = pd.DataFrame(convert_timeseries_to_datapoint(ID), index=eng_features_col).transpose()\n", " df_datapoints = pd.concat([df_datapoints, datapoint],ignore_index=True)\n", " print(unique_BookingIDs.index(ID))\n", " \n", "df_datapoints['ID'] = df_datapoints['ID'].astype('int64')\n", "print(\"df_datapoints shape is \", df_datapoints.shape)\n", "df_datapoints.head()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df_datapoints.to_csv(r'safety/engineered_features.csv', index=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Get df_combined" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df_combined = df_datapoints.copy()\n", "df_combined['label'] = df_combined['bookingID'].apply(match_ID_to_label).sort_values('bookingID').reset_index(drop=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Load Model" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "loaded_model = load_model(\"train_model_ver1.h5\", custom_objects={'auroc':auroc, 'precision':precision, 'recall':recall})\n", "print(\"Model loaded from disk\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Get X & y to input into model for prediction" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "from sklearn import preprocessing\n", "from sklearn.preprocessing import MinMaxScaler, StandardScaler\n", "\n", "print(df_combined.shape)\n", "X = df_combined[eng_features_col].values\n", "sc = StandardScaler()\n", "X = sc.fit_transform(X)\n", "X" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(df_labels.shape)\n", "y = list(df_labels['label'])\n", "y = np.array(y)\n", "print(y)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Get AUC-ROC score & Confusion Matrix" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from sklearn.metrics import roc_curve\n", "from sklearn.metrics import auc\n", "\n", "# Set arbitrary_threshold for CM\n", "arbitrary_threshold = 0.5 \n", "\n", "y_pred_raw = loaded_model.predict(X)\n", "y_pred = y_train_pred_raw > arbitrary_threshold\n", "\n", "fpr_model, tpr_model, thresholds_model = roc_curve(y, y_pred_raw.ravel())\n", "auc_model = auc(fpr_model, tpr_model)\n", "print(\"AUC-ROC score is %.4f\" % mean_auc_score)\n", "\n", "\n", "from sklearn.metrics import confusion_matrix\n", "\n", "plt.figure(figsize = (25,50))\n", "f, axes = plt.subplots(1, 1, figsize=(12, 4))\n", "sns.heatmap(confusion_matrix(y, y_pred),\n", " annot = True, fmt=\".0f\", annot_kws={\"size\": 18}, ax = axes)\n", "\n", "axes[0].set_title(\"Confusion Matrix Results\", fontsize=15)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "\n", "(Credits: https://cdn-images-1.medium.com/max/1600/1*Z54JgbS4DUwWSknhDCvNTQ.png)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Plot AUC-ROC graph" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plt.figure(1)\n", "plt.plot([0, 1], [0, 1], 'k--')\n", "plt.plot(fpr_model, tpr_model, label='Model (area = {:.3f})'.format(auc_model))\n", "plt.xlabel('False positive rate')\n", "plt.ylabel('True positive rate (Sensitivity/Recall)')\n", "plt.title('ROC curve')\n", "plt.legend(loc='best')\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Zoom in view of the upper left corner.\n", "plt.figure(2)\n", "plt.xlim(0, 0.5)\n", "plt.ylim(0.5, 1)\n", "plt.plot([0, 1], [0, 1], 'k--')\n", "plt.plot(fpr_model, tpr_model, label='Model (area = {:.3f})'.format(auc_model))\n", "plt.xlabel('False positive rate')\n", "plt.ylabel('True positive rate (Sensitivity/Recall)')\n", "plt.title('ROC curve (zoomed in at top left)')\n", "plt.legend(loc='best')\n", "plt.show()" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.8" } }, "nbformat": 4, "nbformat_minor": 2 }