{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Project: Artificial Financial Advisor on Peer-to-Peer lending -- Part 2\n", "\n", "--by Lu Tang\n", "\n", "## Part 2 Choosing the best model for investors\n", "\n", "### Table of Contents\n", " " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Import general library and make settings**" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# import library\n", "import pandas as pd\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "%matplotlib inline\n", "import seaborn as sns\n", "\n", "from sklearn.metrics import confusion_matrix\n", "from sklearn.metrics import classification_report\n", "from sklearn.metrics import precision_recall_curve\n", "\n", "# some other library are imported along with the code\n", "from sklearn.linear_model import LogisticRegression\n", "from sklearn.ensemble import RandomForestClassifier\n", "\n", "sns.despine(left=True, bottom=True)\n", "\n", "# to avoid warnings\n", "import warnings\n", "warnings.filterwarnings('ignore')\n", "\n", "# Pandas options\n", "pd.set_option('display.max_colwidth', 1000, 'display.max_rows', None, 'display.max_columns', None)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Read the data we have cleaned in Part 1 *loans_2007_2019_cleaned.csv*, this is our target data split as train data for features column**" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
loan_amnttermint_rateinstallmentgradeemp_lengthhome_ownershipannual_incverification_statuspurposeaddr_statedtidelinq_2yrsinq_last_6mthsopen_accpub_recrevol_balrevol_utiltotal_accinitial_list_statustot_cur_baltotal_rev_hi_limacc_open_past_24mthsavg_cur_balbc_open_to_buybc_utilmo_sin_old_il_acctmo_sin_old_rev_tl_opmo_sin_rcnt_rev_tl_opmo_sin_rcnt_tlmort_accmths_since_recent_bcmths_since_recent_inqnum_accts_ever_120_pdnum_actv_bc_tlnum_actv_rev_tlnum_bc_satsnum_bc_tlnum_il_tlnum_op_rev_tlnum_rev_acctsnum_rev_tl_bal_gt_0num_satsnum_tl_op_past_12mpct_tl_nvr_dlqpercent_bc_gt_75pub_rec_bankruptciestot_hi_cred_limtotal_bal_ex_morttotal_bc_limittotal_il_high_credit_limitloan_status
016000.036 months11.14524.89B8.0RENT65000.0Not Verifiedcredit_cardVA15.760.01.09.00.013484.037.614.0f39639.035900.03.04404.020256.038.1130.0155.07.03.00.07.03.00.03.04.05.06.05.06.08.04.09.02.0100.020.00.063549.039639.032700.025728.01
\n", "
" ], "text/plain": [ " loan_amnt term int_rate installment grade emp_length \\\n", "0 16000.0 36 months 11.14 524.89 B 8.0 \n", "\n", " home_ownership annual_inc verification_status purpose addr_state \\\n", "0 RENT 65000.0 Not Verified credit_card VA \n", "\n", " dti delinq_2yrs inq_last_6mths open_acc pub_rec revol_bal \\\n", "0 15.76 0.0 1.0 9.0 0.0 13484.0 \n", "\n", " revol_util total_acc initial_list_status tot_cur_bal total_rev_hi_lim \\\n", "0 37.6 14.0 f 39639.0 35900.0 \n", "\n", " acc_open_past_24mths avg_cur_bal bc_open_to_buy bc_util \\\n", "0 3.0 4404.0 20256.0 38.1 \n", "\n", " mo_sin_old_il_acct mo_sin_old_rev_tl_op mo_sin_rcnt_rev_tl_op \\\n", "0 130.0 155.0 7.0 \n", "\n", " mo_sin_rcnt_tl mort_acc mths_since_recent_bc mths_since_recent_inq \\\n", "0 3.0 0.0 7.0 3.0 \n", "\n", " num_accts_ever_120_pd num_actv_bc_tl num_actv_rev_tl num_bc_sats \\\n", "0 0.0 3.0 4.0 5.0 \n", "\n", " num_bc_tl num_il_tl num_op_rev_tl num_rev_accts num_rev_tl_bal_gt_0 \\\n", "0 6.0 5.0 6.0 8.0 4.0 \n", "\n", " num_sats num_tl_op_past_12m pct_tl_nvr_dlq percent_bc_gt_75 \\\n", "0 9.0 2.0 100.0 20.0 \n", "\n", " pub_rec_bankruptcies tot_hi_cred_lim total_bal_ex_mort total_bc_limit \\\n", "0 0.0 63549.0 39639.0 32700.0 \n", "\n", " total_il_high_credit_limit loan_status \n", "0 25728.0 1 " ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "The total number of rows and columns in our cleaned 2007 to 2019 loan data is: (1023834, 52)\n" ] } ], "source": [ "# load the data we saved from the first notebook part 1 \n", "loans_2007_2019 = pd.read_csv('loans_2007_2019_cleaned.csv')\n", "\n", "# view 1th rows\n", "display(loans_2007_2019.head(1))\n", "\n", "print('The total number of rows and columns in our cleaned 2007 to 2019 loan data is: ',loans_2007_2019.shape)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "## 1. Review Part 1 and clarify our goal in modelling" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "From part 1, we found the Charged Off rate for all the loans in average is 19.57%. This means **if an investor randomly choose loans to invest, he/she could have about 19% of Charged-Off rate**. \n", "\n", "I digged deeper and found **if a super conservative investor randomly choose ONLY A loans to invest, he/she could have about 5.9% of Charged-Off rate**.\n", "\n", "The following table is the one I saved from Part 1 after analyzing the loan grade and returns. Let's do further analysis" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Loan GradeAverage Interest RateUnnamed: 2Averafe Charged-Off Rateloan_grade
0A7.10%NaN5.95%A
1B10.67%NaN13.28%B
2C14.04%NaN22.14%C
3D17.81%NaN29.85%D
4E21.27%NaN37.87%E
5F25.03%NaN44.78%F
6G27.82%NaN49.48%G
\n", "
" ], "text/plain": [ " Loan Grade Average Interest Rate Unnamed: 2 Averafe Charged-Off Rate \\\n", "0 A 7.10% NaN 5.95% \n", "1 B 10.67% NaN 13.28% \n", "2 C 14.04% NaN 22.14% \n", "3 D 17.81% NaN 29.85% \n", "4 E 21.27% NaN 37.87% \n", "5 F 25.03% NaN 44.78% \n", "6 G 27.82% NaN 49.48% \n", "\n", " loan_grade \n", "0 A \n", "1 B \n", "2 C \n", "3 D \n", "4 E \n", "5 F \n", "6 G " ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# read investment data made from part 1.\n", "investment = pd.read_csv('investment.csv')\n", "# add loan grade column\n", "investment['loan_grade']=['A','B','C','D','E','F','G'] # since I saved the data as index=False, so i need to add it back.\n", "# view the table\n", "investment" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note: From part 1, we have found the proportion of the loans between different grade are not the same, and I also mentioned that 'Charged-Off' rate does NOT mean the when the loans are charged off, investor will loss all the principal and interest, because loan payment schedul is a mortgage-like arrangement and a loan maybe charged off during the loan term. This can explain why the charged-off rate is even higher than interest rate except for A loans.\n", "\n", "From first row, we can see if an investor put 25 dollar per loan and invest in 1000 loan, and ONLY choose A loans to invest, the average expected return is **7.1%**, but if he/she lose all the money for the loans that are charged-off, the net return is: 0.071004-0.059498 = **1.5%**. We can also calculate as following:" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0.011505999999999997" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# calculate the worst case for expected return \n", "((0.071004-0.059498)*25*1000)/(25*1000)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Our goal in buidling machine learning model**\n", "\n", "1. Our model should perform better than if an investor randomly pick up all the loans to invest, which is **to reduce the risk at least lower than 19.57%**. \n", "\n", "2. Our model should perform better than if a conservative investor randomly choose only A loans to invest. **Our model that have expected return higher than 7.1%**, and **for the worst case, still have expected return higher than 1.15%**" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "## 2. Define error metric" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Before diving in and selecting an algorithm to apply to the data, we should select an error metric, and it is not always an accuracy score or AUC-ROC, it depends on the problems we are trying to solve with our model.\n", "\n", "We're using a machine learning model to predict whether or not we should fund a loan on the Lending Club platform. Our objective in this is to make money -- we want to fund enough loans that are paid off on time to offset our losses from loans that aren't paid off. An error metric will help us determine if our algorithm will make us money or lose us money.\n", "\n", "In this case, we're primarily concerned with **false positives** and **false negatives**. Since we're viewing this problem from the standpoint of a **conservative investor**, we need to treat false positives differently than false negatives. A conservative investor would want to **minimize risk**, and **avoid false positives as much as possible**. They'd be more okay with funding a safer loan (false positives) than missing out on opportunities (false negatives).\n", "\n", "From part 1, we can see there is a **significant class imbalance** in the loan_status column. This causes a major issue when we use accuracy as a metric. This is because due to the class imbalance, a classifier can predict 1 for every row, and still have high accuracy. Read more on [beyond-accuracy-precision-and-recall](https://towardsdatascience.com/beyond-accuracy-precision-and-recall-3da06bea9f6c)\n", "\n", "This means that we should **optimize** for:\n", "- **Low fpr (false positive rate)** \n", "\n", "fpr = fp / (fp + tn), i.e. \"among the loans that ACTUALLLY 'Charged-Off', what percentage the model is wrong.\" \n", "\n", "- **High Precision (similar with Low fpr)**\n", "\n", "precision = tp/(tp+fp), i.e.\"among the loans that the model PREDICTED as 'will Fully-Paid', what percentage the model is right.\n", "\n", "- **High tpr (true positive rate/Recall/Sensitivity) (this comes with a cost when we optimize the fpr and Precision)**\n", "\n", "tpr = tp / (tp + fn), i.e.\"among the loans that ACTUALLLY 'Fully-Paid', what percentage of loans that should be funded that I would fund\". \n", "\n", "Generally, if we want to reduce false positive rate, true positive rate will go down. This is because if we want to reduce the risk of false positives, we wouldn't think about funding riskier loans in the first place. In other words, our model will be very 'strict' on picking up the loans that it thinks will much more likely to pay off; it basically discard a lot of bad and good loans, and select only those very good ones." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "I wrote my own **model metric function** to print out only the metrics we need. (The classification report and other popular metric library is not useful here, and only makes readers more confusing). \n", "\n", "I also wrote a function for **precision-recall curve**. I choose precision-recall curve to visualize the trade-off between the precision and recall **because this is imbalanced dataset and one class is more important than another one**. Although ROC-AUC is popular, it is not useful to solve our problem. (Read more [roc-and-precision-recall-with-imbalanced-datasets](https://classeval.wordpress.com/simulation-analysis/roc-and-precision-recall-with-imbalanced-datasets/). Also [here](https://machinelearningmastery.com/roc-curves-and-precision-recall-curves-for-classification-in-python/) And [Interpreting ROC Curves, Precision-Recall Curves, and AUCs](https://www.datascienceblog.net/post/machine-learning/interpreting-roc-curves-auc/)). Also for f1_score, I did not use F1_score, because if I have a classifier which F1 score is low, we can’t tell whether it has problems with false positives or false negatives. In this case, the best way to “debug” such a classifier is to use confusion matrix to diagnose the problem and then look at the problematic cases in the validation or test dataset. (Read more [here](https://www.mikulskibartosz.name/f1-score-explained/) and read [here](https://stats.stackexchange.com/questions/49226/how-to-interpret-f-measure-values)). (At the first I actually calculate F1_score and AUC for ROC, it's high but it does not give more informative insights.)\n", "\n", "Lastly, I wrote my own **investment portfolio return function** , i.e. if an investor invested on the loans selected selected by my model, how much investment return (%) he/she can expect. If we only choose model based on those metric score or a confusion metrix, it is still very confusing, because for our purpose, it does not mean a model with lowesr false positive rate is the best model. With this function, we can choose a model that perform better than if the investor randomly pick up loans to invest without my model. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**1. Define my metric function**" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "def my_metric(model, X_test, y_test):\n", " \n", " # getting predicted values\n", " y_predict = model.predict(X_test)\n", " \n", " # False positives\n", " fp_filter = (y_predict == 1) & (y_test == 0)\n", " fp = len(y_predict[fp_filter])\n", " \n", " # True positives.\n", " tp_filter = (y_predict == 1) & (y_test == 1)\n", " tp = len(y_predict[tp_filter])\n", " \n", " # False negatives.\n", " fn_filter = (y_predict == 0) & (y_test == 1)\n", " fn = len(y_predict[fn_filter])\n", " \n", " # True negatives\n", " tn_filter = (y_predict == 0) & (y_test == 0)\n", " tn = len(y_predict[tn_filter])\n", " \n", " # Rates\n", " tpr = tp / (tp + fn)\n", " fpr = fp / (fp + tn)\n", " precision = tp/(tp+fp)\n", " \n", " print('Confusion matrix: \\n', confusion_matrix(y_test, y_predict))\n", " # print(classification_report(y_test,y_predict)) # we don't need this\n", " print('The True Positive Rate (Recall/Sensitivity)is tpr = tp/(tp+fn): ', tpr)\n", " print('The False Positive Rate (1-specification)is fpr = fp/(fp+tn): ', fpr)\n", " print('The Precision (tp/(tp+fp)) is: ', precision)\n", " print('\\n')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**2. Define precision-recall curves**" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "def precision_recall_curves(model, X_test, y_test):\n", " \n", " # plot Precision Recall Curve\n", " plt.figure(figsize=(8,6))\n", " \n", " # predict probabilities\n", " probs = model.predict_proba(X_test)\n", " # keep probabilities for the positive outcome only\n", " probs = probs[:, 1]\n", " \n", " # get predicted class value\n", " y_predict = model.predict(X_test)\n", " \n", " # calculate precision-recall curve\n", " precision, recall, thresholds = precision_recall_curve(y_test, probs)\n", " \n", " # plot no skill\n", " # I set 0.8 because our imbalanced data is roughly 80% as majority class\n", " plt.plot([0, 1], [0.8, 0.8], linestyle='--') \n", " # plot the precision-recall curve for the model\n", " plt.plot(recall, precision, color='darkorange', marker='.')\n", " plt.xlabel('Recall (sensitivity, or True Positive Rate)')\n", " plt.ylabel('Precision')\n", " plt.title('Precision Recall Curve')\n", " \n", " return plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**3. Define investment return with my model**" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "def investment_return_with_my_model(model, X_test, y_test):\n", " \n", " # get predicted class value\n", " y_predict = model.predict(X_test)\n", " # reshape\n", " y_predict = y_predict.reshape((y_predict.shape[0], 1))\n", " # put it into a dataframe\n", " y_predict = pd.DataFrame(y_predict, index=range(0,len(y_predict)), columns=['loan_status_predicted'])\n", " \n", " # Join X_test and y_test using 'join' since they have the same index\n", " loans_test = X_test.join(y_test)\n", " \n", " # Then join the test dataframe with y_predict; Since it is different index, I created a 'Join' column and then use 'merge'\n", " loans_test['Join']=list(range(0,len(y_predict)))\n", " y_predict['Join']=list(range(0,len(y_predict)))\n", " # Merge test data with predicted data\n", " loans_test_with_predict = pd.merge(loans_test, y_predict, on='Join')\n", " \n", " # filter the rows that are predicted as 1\n", " predict_should_invest = loans_test_with_predict[loans_test_with_predict['loan_status_predicted']==1]\n", " \n", " # Among the loans predicted as 1, filter the rows thar are actually as 1\n", " actual_should_invest = predict_should_invest[predict_should_invest['loan_status']==1]\n", " \n", " # calculate the mistake rate, this is the same as fpr\n", " mistake_rate = (predict_should_invest.shape[0]-actual_should_invest.shape[0])/predict_should_invest.shape[0]\n", " \n", " # add a new column, indicating for each loan we invest 25 dollars\n", " predict_should_invest['invest_amount'] = 25\n", " \n", " # add a new column, calculating interest earned from this loan by multiplying interest rate per loan with the amount invested per loan\n", " predict_should_invest['interest_earned']=(predict_should_invest['int_rate']/100)*predict_should_invest['invest_amount']\n", " \n", " # sum the interest earned for all the loans we invested\n", " total_interest = predict_should_invest['interest_earned'].sum()\n", " \n", " # calculate our investment return\n", " investment_return = total_interest/(25*len(predict_should_invest))\n", " print('The investment return with this model is: ', '{:.2%}'.format(investment_return))\n", " \n", " # calculate the return for the worse case, meaning if we lost all the money from the loans that are charged-off\n", " #(including full amount of principal and interest, which is less likely), how much return we will get.\n", " investment_return_with_maxlosss = (total_interest-(mistake_rate*(25*len(predict_should_invest))))/(25*len(predict_should_invest))\n", " print('The investment return with this model for the worse case is: ', '{:.2%}'.format(investment_return_with_maxlosss))\n", " print('\\n')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "## 3. Preprocessing the data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Encode object columns as dummy variables**" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
termgradehome_ownershipverification_statuspurposeaddr_stateinitial_list_status
036 monthsBRENTNot Verifiedcredit_cardVAf
\n", "
" ], "text/plain": [ " term grade home_ownership verification_status purpose \\\n", "0 36 months B RENT Not Verified credit_card \n", "\n", " addr_state initial_list_status \n", "0 VA f " ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# select object columns\n", "object_columns = loans_2007_2019.select_dtypes(include=[\"object\"])\n", "\n", "object_columns.head(1)" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
loan_amntint_rateinstallmentemp_lengthannual_incdtidelinq_2yrsinq_last_6mthsopen_accpub_recrevol_balrevol_utiltotal_acctot_cur_baltotal_rev_hi_limacc_open_past_24mthsavg_cur_balbc_open_to_buybc_utilmo_sin_old_il_acctmo_sin_old_rev_tl_opmo_sin_rcnt_rev_tl_opmo_sin_rcnt_tlmort_accmths_since_recent_bcmths_since_recent_inqnum_accts_ever_120_pdnum_actv_bc_tlnum_actv_rev_tlnum_bc_satsnum_bc_tlnum_il_tlnum_op_rev_tlnum_rev_acctsnum_rev_tl_bal_gt_0num_satsnum_tl_op_past_12mpct_tl_nvr_dlqpercent_bc_gt_75pub_rec_bankruptciestot_hi_cred_limtotal_bal_ex_morttotal_bc_limittotal_il_high_credit_limitloan_statusterm_ 60 monthsgrade_Bgrade_Cgrade_Dgrade_Egrade_Fgrade_Ghome_ownership_MORTGAGEhome_ownership_NONEhome_ownership_OTHERhome_ownership_OWNhome_ownership_RENTverification_status_Source Verifiedverification_status_Verifiedpurpose_credit_cardpurpose_debt_consolidationpurpose_educationalpurpose_home_improvementpurpose_housepurpose_major_purchasepurpose_medicalpurpose_movingpurpose_otherpurpose_renewable_energypurpose_small_businesspurpose_vacationpurpose_weddingaddr_state_ALaddr_state_ARaddr_state_AZaddr_state_CAaddr_state_COaddr_state_CTaddr_state_DCaddr_state_DEaddr_state_FLaddr_state_GAaddr_state_HIaddr_state_IAaddr_state_IDaddr_state_ILaddr_state_INaddr_state_KSaddr_state_KYaddr_state_LAaddr_state_MAaddr_state_MDaddr_state_MEaddr_state_MIaddr_state_MNaddr_state_MOaddr_state_MSaddr_state_MTaddr_state_NCaddr_state_NDaddr_state_NEaddr_state_NHaddr_state_NJaddr_state_NMaddr_state_NVaddr_state_NYaddr_state_OHaddr_state_OKaddr_state_ORaddr_state_PAaddr_state_RIaddr_state_SCaddr_state_SDaddr_state_TNaddr_state_TXaddr_state_UTaddr_state_VAaddr_state_VTaddr_state_WAaddr_state_WIaddr_state_WVaddr_state_WYinitial_list_status_w
016000.011.14524.898.065000.015.760.01.09.00.013484.037.614.039639.035900.03.04404.020256.038.1130.0155.07.03.00.07.03.00.03.04.05.06.05.06.08.04.09.02.0100.020.00.063549.039639.032700.025728.01010000000001001000000000000000000000000000000000000000000000000000000001000000
\n", "
" ], "text/plain": [ " loan_amnt int_rate installment emp_length annual_inc dti \\\n", "0 16000.0 11.14 524.89 8.0 65000.0 15.76 \n", "\n", " delinq_2yrs inq_last_6mths open_acc pub_rec revol_bal revol_util \\\n", "0 0.0 1.0 9.0 0.0 13484.0 37.6 \n", "\n", " total_acc tot_cur_bal total_rev_hi_lim acc_open_past_24mths \\\n", "0 14.0 39639.0 35900.0 3.0 \n", "\n", " avg_cur_bal bc_open_to_buy bc_util mo_sin_old_il_acct \\\n", "0 4404.0 20256.0 38.1 130.0 \n", "\n", " mo_sin_old_rev_tl_op mo_sin_rcnt_rev_tl_op mo_sin_rcnt_tl mort_acc \\\n", "0 155.0 7.0 3.0 0.0 \n", "\n", " mths_since_recent_bc mths_since_recent_inq num_accts_ever_120_pd \\\n", "0 7.0 3.0 0.0 \n", "\n", " num_actv_bc_tl num_actv_rev_tl num_bc_sats num_bc_tl num_il_tl \\\n", "0 3.0 4.0 5.0 6.0 5.0 \n", "\n", " num_op_rev_tl num_rev_accts num_rev_tl_bal_gt_0 num_sats \\\n", "0 6.0 8.0 4.0 9.0 \n", "\n", " num_tl_op_past_12m pct_tl_nvr_dlq percent_bc_gt_75 pub_rec_bankruptcies \\\n", "0 2.0 100.0 20.0 0.0 \n", "\n", " tot_hi_cred_lim total_bal_ex_mort total_bc_limit \\\n", "0 63549.0 39639.0 32700.0 \n", "\n", " total_il_high_credit_limit loan_status term_ 60 months grade_B grade_C \\\n", "0 25728.0 1 0 1 0 \n", "\n", " grade_D grade_E grade_F grade_G home_ownership_MORTGAGE \\\n", "0 0 0 0 0 0 \n", "\n", " home_ownership_NONE home_ownership_OTHER home_ownership_OWN \\\n", "0 0 0 0 \n", "\n", " home_ownership_RENT verification_status_Source Verified \\\n", "0 1 0 \n", "\n", " verification_status_Verified purpose_credit_card \\\n", "0 0 1 \n", "\n", " purpose_debt_consolidation purpose_educational purpose_home_improvement \\\n", "0 0 0 0 \n", "\n", " purpose_house purpose_major_purchase purpose_medical purpose_moving \\\n", "0 0 0 0 0 \n", "\n", " purpose_other purpose_renewable_energy purpose_small_business \\\n", "0 0 0 0 \n", "\n", " purpose_vacation purpose_wedding addr_state_AL addr_state_AR \\\n", "0 0 0 0 0 \n", "\n", " addr_state_AZ addr_state_CA addr_state_CO addr_state_CT addr_state_DC \\\n", "0 0 0 0 0 0 \n", "\n", " addr_state_DE addr_state_FL addr_state_GA addr_state_HI addr_state_IA \\\n", "0 0 0 0 0 0 \n", "\n", " addr_state_ID addr_state_IL addr_state_IN addr_state_KS addr_state_KY \\\n", "0 0 0 0 0 0 \n", "\n", " addr_state_LA addr_state_MA addr_state_MD addr_state_ME addr_state_MI \\\n", "0 0 0 0 0 0 \n", "\n", " addr_state_MN addr_state_MO addr_state_MS addr_state_MT addr_state_NC \\\n", "0 0 0 0 0 0 \n", "\n", " addr_state_ND addr_state_NE addr_state_NH addr_state_NJ addr_state_NM \\\n", "0 0 0 0 0 0 \n", "\n", " addr_state_NV addr_state_NY addr_state_OH addr_state_OK addr_state_OR \\\n", "0 0 0 0 0 0 \n", "\n", " addr_state_PA addr_state_RI addr_state_SC addr_state_SD addr_state_TN \\\n", "0 0 0 0 0 0 \n", "\n", " addr_state_TX addr_state_UT addr_state_VA addr_state_VT addr_state_WA \\\n", "0 0 0 1 0 0 \n", "\n", " addr_state_WI addr_state_WV addr_state_WY initial_list_status_w \n", "0 0 0 0 0 " ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "The number of rows and columns in our machine learning model is : (1023834, 123)\n" ] } ], "source": [ "# Encode object columns to integer values and return a Dataframe containing the dummy columns.\n", "dummy_df = pd.get_dummies(loans_2007_2019[object_columns.columns], drop_first=True)\n", "\n", "# combine dummy column dataframe with original dataframe as column\n", "loans_2007_2019 = pd.concat([loans_2007_2019, dummy_df], axis=1)\n", "\n", "# drop original object columns\n", "loans_2007_2019 = loans_2007_2019.drop(object_columns.columns, axis=1)\n", "\n", "# review the result with 3 rows\n", "display(loans_2007_2019.head(1))\n", "\n", "print('The number of rows and columns in our machine learning model is : ', loans_2007_2019.shape)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Not scale on our data*\n", "\n", "Scale data is not very neccessary for this project, I have tested with the same machines learning model with standard scaled data, the results are not too much different. However, scaled data makes interpretation more difficult, I need the original test data to with interest column to calculate interest on investment." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Train Test split**\n", "\n", "We take 70% of the data to use as trainning data that will be fit to a model, the rest 30% of the data will be used as test data to get predicted value; we then compare our predicted value with actual value to calculate our error metric fpr and tpr." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "features = loans_2007_2019.drop('loan_status', axis=1)\n", "\n", "target = loans_2007_2019['loan_status']\n", "\n", "from sklearn.model_selection import train_test_split\n", "\n", "X_train, X_test, y_train, y_test = train_test_split(features, target, test_size=0.3, random_state=42)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "## 4. Fit the data in different models and analyze the results" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 4.1 Try Logistic Regression with default papameters\n", "\n", "A good first algorithm to apply to binary classification problems is logistic regression, for the following reasons:\n", "\n", "- it's quick to train and we can iterate more quickly,\n", "- it's less prone to overfitting than more complex models like decision trees,\n", "- it's easy to interpret." ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Confusion matrix: \n", " [[ 325 58998]\n", " [ 307 247521]]\n", "The True Positive Rate (Recall/Sensitivity)is tpr = tp/(tp+fn): 0.9987612376325516\n", "The False Positive Rate (1-specification)is fpr = fp/(fp+tn): 0.9945215177924245\n", "The Precision (tp/(tp+fp)) is: 0.8075225353077623\n", "\n", "\n", "The investment return with this model is: 13.19%\n", "The investment return with this model for the worse case is: -6.06%\n", "\n", "\n" ] } ], "source": [ "# creat our model\n", "lr_mymodel_default = LogisticRegression().fit(X_train, y_train)\n", "# use my function\n", "my_metric(lr_mymodel_default, X_test, y_test)\n", "investment_return_with_my_model(lr_mymodel_default, X_test, y_test)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Analyze the result**\n", "\n", "Our tpr and fpr are all very high, this means our model predict 99% of the loans that are Fully-paid that are actually Fully-paid (which is good), but also predict 99% of the loans as Fully-paid when it is asctuall Charged-off. If we check what is our prediction, we will see it's almost all as 1.\n", "\n", "Precision is 80%, This seems high, but our target columns is a highly imbalanced class, 80% of the values are 1 and 20% of the values are 0. So 80% of precision does not mean much.\n", "\n", "Lastly, if we use this model, we could lost 6.06% of our money.\n", "\n", "**My stragegy to fix the problem** is to tell the classifier to penalize certain rows more by setting the class_weight parameter to balanced. This tells scikit-learn to penalize the misclassification of the minority class during the training process. The penalty means that the logistic regression classifier pays more attention to correctly classifying rows where loan_status is 0. This lowers accuracy when loan_status is 1, but raises accuracy when loan_status is 0. By **setting the class_weight parameter to balanced**, the penalty is set to be inversely proportional to the class frequencies. This would mean that for the classifier, correctly classifying a row where loan_status is 0 is 4 times more important than correctly classifying a row where loan_status is 1. (4 times is not default value, it is based on the proportionate of our loan status data)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 4.2 Try Logistic Regression with class_weight = 'balanced'" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Confusion matrix: \n", " [[ 38777 20546]\n", " [ 87184 160644]]\n", "The True Positive Rate (Recall/Sensitivity)is tpr = tp/(tp+fn): 0.6482076278709428\n", "The False Positive Rate (1-specification)is fpr = fp/(fp+tn): 0.3463412167287561\n", "The Precision (tp/(tp+fp)) is: 0.8866052210386887\n", "\n", "\n", "The investment return with this model is: 10.48%\n", "The investment return with this model for the worse case is: -0.86%\n", "\n", "\n" ] } ], "source": [ "# creat model, set model parameter with balanced\n", "lr_mymodel_balanced = LogisticRegression(class_weight='balanced').fit(X_train, y_train)\n", "# use my function\n", "my_metric(lr_mymodel_balanced, X_test, y_test)\n", "investment_return_with_my_model(lr_mymodel_balanced, X_test, y_test)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Analyze the result**\n", "\n", "We significantly improved false positive rate by balancing the classes, which also reduced true positive rate. Our true positive rate is now around 65%, and our false positive rate is around 35%. From a conservative investor's standpoint, it's reassuring that the false positive rate is lower because it means that we'll be able to do a better job at avoiding bad loans than if we funded everything. However, investor can still lose 0.86% of investment." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Using precision_recall_curves to visulize the trade off**" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfUAAAGDCAYAAAAyM4nNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xm4JHV97/H3hwEEREAdNAIDI5sKiAojytUYjMagN0JUIhg3jEpiVIJLXO7NVYNJjCaoiUsM7jsgiWQ0KFECogYigyAwIDoiOOMYnREBFZRlvvePqgM9Z87M6bP0OX2q36/n6efU1lXfru6ZT/+qfl2VqkKSJC18W813AZIkaXYY6pIkdYShLklSRxjqkiR1hKEuSVJHGOqSJHWEoS4NiSQrkxwxyTJ7JvlFkkVzVNZAJTk+ydd6xivJvvNZk7SQGerSJJJcl+TWNkx/nOTDSXac7e1U1YFVdf4ky/ygqnasqjtne/ttoP6yfZ0/TPL2YfvykOR3k1yQ5OdJ1iX5SpKj5rsuaVgY6lJ/nlpVOwKHAI8E/mL8Amks9H9TD2tf528BxwJ/NM/13CXJMcBngI8BewD3B94APHUa6+rCeyVtwg+1NAVV9UPgC8BBAEnOT/LXSb4O3ALsnWTnJB9M8qO2xftXvS3eJC9OcnXb2rwqySHt9OuSPLEdPizJiiQ3t0cH3t5OX9q2qLdux3dLsjzJDUlWJXlxz3belOSMJB9rt7UyybI+X+cq4OvAw3vWN93X9bok3+uZ/rSp7vckAd4OvLmqPlBVN1XVhqr6SlW9uOf1fqLnOeP31fj36v8kWTFuO69IsrwdvkeSv0/yg/Y9eF+S7adauzSXDHVpCpIsAZ4CXNoz+bnACcC9gOuBjwJ3APsCjwCeBLyoff4fAG8CngfsBBwF/HSCTf0D8A9VtROwD3DGZkr6NLAG2A04BvibJE/omX8UcBqwC7AceHefr/PBwG8Cq3omT/d1fa9d187AXwKfSPKAfuro8SBgCXDmFJ83Xu979S7gQUn265n/h8Cn2uG3AvvTfLHZF9id5siANLQMdak/ZyW5Efga8BXgb3rmfaSqVlbVHcB9gCcDJ1XVL6vqJ8A7gOPaZV8EvK2qLq7Gqqq6foLt3Q7sm2RxVf2iqi4av0D7BeOxwGur6ldVdRnwAZrgGvO1qjq7PQf/ceBhk7zObyb5JXA1cD7w3nZb95/u66qqz1TV2rZlfTrwXeCwSeoY777t3x9N8Xnj3fVeVdVNwL8BzwJow/3BwPL2yMCLgVdU1Q1V9XOa9/y4za1YGgaGutSf36+qXapqr6r606q6tWfe6p7hvYBtgB8lubH9IvDPwP3a+UtoWq6TeSFNK/HbSS5O8nsTLLMbMBY4Y66naVGO+Z+e4VuA7cYOR2/GIcCONOfTHwXcc6avK8nzklzW87yDgMVbqGEiY63+qbbwx1s9bvxTtKFO00o/q6puAXYFdgAu6an7i+10aWgZ6tLM9d7qcDXwa2Bx+yVgl6raqaoO7Jm/z6QrrPpuVT2LJjTfCpyZ5J7jFlsL3CfJvXqm7Qn8cLovpN12VdUZwIXcfbh5Wq8ryV7A+4GXAfetql2AK4FMsaxr2m08YwvL/JImiMf8xgTLjL8t5X8Ai5M8nCbcxw69rwduBQ7seb07t50IpaFlqEuzqKp+RBMUpyTZKclWSfZJ8lvtIh8AXp3k0LYH9r5t8G0kyXOS7FpVG4Ab28kb/YytqlYD/wW8Jcl2SQ6maeF/cpZezt8CJyT5jRm8rnvSBOm69nW9gLaT4VRUc4/oVwL/L8kLemp4bJJT28UuAx6X5rf8OwOv72O9d9Ccp/87mlMnX2qnb6D5MvKOJPdra989ye9OtXZpLhnq0ux7HrAtcBXwM5rQeAA055eBv6ZpEf4cOIsmTMY7EliZ5Bc0neaOq6pfTbDcs4ClNK32zwJvrKovzcaLqKoraPoP/Pl0X1dVXQWcQtPq/zHwUJpe9dOp50zu/pnd2nZ9f0VzXpz2dZ8OXA5cAny+z1V/Cngi8Jk25Me8lqaj4EVJbga+TNNhTxpaab4AS5Kkhc6WuiRJHWGoS5LUEYa6JEkdYahLktQRhrokSR2xpStLDaXFixfX0qVL57sMSZLmxCWXXLK+qvq6muGCC/WlS5eyYsWKyReUJKkDkkx0f4gJefhdkqSOMNQlSeoIQ12SpI4w1CVJ6ghDXZKkjjDUJUnqCENdkqSOMNQlSeoIQ12SpI4YWKgn+VCSnyS5cjPzk+Qfk6xKcnmSQwZViyRJo2CQLfWPAEduYf6Tgf3axwnAPw2wloldfiqc+bvNX0mSFriBXfu9qi5IsnQLixwNfKyqCrgoyS5JHlBVPxpUTRu5/FT40h83w9f/R/P34BPmZNOSJA3CfJ5T3x1Y3TO+pp22iSQnJFmRZMW6detmZ+vf+Zctj0uStMDMZ6hngmk10YJVdWpVLauqZbvu2tfd5ya3/zO2PC5J0gIzn6G+BljSM74HsHbOtt57qP2gF3roXZK04M1nqC8Hntf2gn80cNOcnU8fb8/Hz8tmJUmaTQPrKJfk08ARwOIka4A3AtsAVNX7gLOBpwCrgFuAFwyqFkmSRsEge78/a5L5Bbx0UNuXJGnUeEU5gJqwf54kSQuKoS5JUkcY6pIkdYShDmzm5/GSJC0ohjrAF54PHz5gvquQJGlGDHUACm642mCXJC1ohnqvG66e7wokSZo2Q328U9I8JElaYAZ28ZkFb3ywP+TZ8JRPzE8tkiT1wVDv19WfbB5jXmWPeUnScPHw+3R5iF6SNGQM9Znw/LskaYh4+H02jA92D81LkubB6Ib65acObt29IW/AS5LmyOiG+hUf3PL88WE83cPstuIlSXNkdEP9jts2P2+i4B2bNtNz6Ia8JGlARjfUN/x6es+brXAfs7n1GPaSpCka3VDffldggsvC9humvcsNogf8ROs06CVJWzC6ob7dfWZvXbN1/n0yBr0kaQtGN9QHqTdo331f+PUNg9uWPe0lSS1DfdBe9tNNp81VS96Ql6SRMrqhPp8XgpvPw/Wbq0GStOCNbqivWznfFdxtSwHr+XlJUp9GN9Rv/v58V9CfQfey72XLXpIWtNENdSYKqUVzXsWUTBSsc3VDGX9PL0lDb3RDfat7wJ23bDwtC/COa3N1fn5zDHtJGhqjG+p1x6bT0oE70W4uTOc77A15SRq4EQ71CUJmmx3mvo65Mt9h7/l6SRq40Q31rbaBO2/feNrBJ8xPLfNpPnreT7YdQ16SpmWEQ30ruLN3fBt43FvnrZyhNF+te1v1kjQtoxvq2+wIt//i7vHt7jt/tSw083ko39/XS9JmjW6oZ+stj2vq5qsn/kTbuddecMJ1c7N9SRoSo5tkv1q/5XHN3Hyer//59bbqJY2c0Q31rbaBO3+18bjmzjC16g16SR0xuqG+zY5w+883Htf8mc9WvRfQkdQRoxvqi7bd8riGx3xdHtde+JIWmNEN9Xvcuznv2juuhWMYL6Zj0EuaZ6Mb6rf8eMvjWpi86Y2kETa6oX7bz7c8ru6Yz6Df3LYMekkDMLqhPv7mLV24mYv65+F7SR00uqHOhknGNZLshS9pARvdUB9/mVh/0qbJDFMvfINe0gRGN9R3+A245X82HpemaqJwveC1cPHbBrtdg17SBEY31HsDfaJxaboe99aJ7/jn4XtJAza6od576H2icWm2efhe0oCNbqh7Tl3DYJjuWW/QSwve6Ia659Q1zOajVT/R+h/5molPJUgaSqMb6p5T10IzH0F/8dsm7vRnq14aSqMb6rVhy+PSQuDhe0k9RjfUl/4OXP3JjcelrhiWw/ebq0XSQAw01JMcCfwDsAj4QFX97bj5ewEfAnYFbgCeU1VrBlnTXba515bHpa6x973UeQML9SSLgPcAvwOsAS5OsryqrupZ7O+Bj1XVR5P8NvAW4LmDqknSOAa91CmDbKkfBqyqqmsBkpwGHA30hvoBwCva4fOAswZYz8YOfB5c/r5mOIuacUmep5cWsEGG+u7A6p7xNcCjxi3zLeAZNIfonwbcK8l9q+qnA6yrR4DyDm1SP4blPL1BL23WIEN9on/t4/81vhp4d5LjgQuAHwJ3bLKi5ATgBIA999xzdqpbff7d5dSdzfhuh8/OuqVRMR+t+vHrNuSluwwy1NcAS3rG9wDW9i5QVWuBpwMk2RF4RlXdNH5FVXUqcCrAsmXLZudf8JIjgK2ADbDVNu24pFkxl616W/PSXQYZ6hcD+yV5IE0L/DjgD3sXSLIYuKGqNgCvp+kJPzd2OxwWHwjrr4Aj3mkrXRo0g14auIGFelXdkeRlwDk0P2n7UFWtTHIysKKqlgNHAG9JUjSH3186qHomtO1Ozd/FB83pZiW1DHppVg30d+pVdTZw9rhpb+gZPhM4c5A1bNFtNzd/118Jezx23sqQ1GM+g96Q1wI3uleUW3shrF/ZDJ9/EtzvYR6Cl4bV+LA15KUJjW6orz4faK/3vuF2e79LC8lcteY9ZK8FZnRD3d7vUrfMV9Ab8hoioxvq9n6Xum8ugt7WvIbI6IY62PtdGkVzcX7eoNc8Ge1QlyQ74alDDHVJ6jVf5+azDbzyttnfjkaKoS5Jk5mL1nzdbmteMzbaoe7FZyRNhz3tNaRGN9S9+Iyk2WRPew2B0Q11Lz4jadDmo6e9IT/SRjfUvfiMpLk2H615Q36kjG6oe/EZScNg0K15Q36kjG6ogxefkTR85jLkDfjOGe1Ql6RhN8hD9rbiO8dQl6SFZlCteUN+wTPUJWmhM+TVMtQlqWsM+ZE12qHuFeUkjYLe8J3NjneG/NAZ3VD3inKSRtEge9fbs37ejW6oe0U5SfJQfceMbqh7RTlJ2tRchLwBPzCjG+peUU6SJjeIkLcVPzCjG+rgFeUkaaoGHfIG/IyMdqhLkmZmtnvWG/AzYqhLkmbHbLfiDfgpM9QlSYMxm614A74vhrokafBmsxVvwG+WoS5Jmnuz1Yofe67hDhjqkqT5NhsBb+sdMNQlScNkNgN+BMPdUJckDaeZBvwItt5HO9S9S5skLQyzFfAdD/fRDXXv0iZJC9NYMNt638Tohrp3aZOkhc3W+yZGN9S9S5skdcdYMJ+6FH5+/dSe26HW++iGundpk6TuOeG6u4dHsPW+1XwXMK+8S5skdderavrhfEpm717yc2h0W+qSpNEwk3PvC6zlPtotdUnSaJlu632BtNxtqUuSRs90W+9D3nK3pS5JGm3Tab0PaavdUJckCaYe7qcE3n3fwdUzDYa6JEm9phLuv75hqFrthrokSROZSrgPSbAb6pIkbUm/4T4EPeQNdUmS+rEAWu2GuiRJ/ZpKq30eGOqSJE3VkAa7oS5J0nQM4QVoDHVJkqZrJjeNGYCBhnqSI5Nck2RVktdNMH/PJOcluTTJ5UmeMsh6JEkaiCEJ9oGFepJFwHuAJwMHAM9KcsC4xf4COKOqHgEcB7x3UPVM6Labm7/rr5zTzUqSOmh8sM9D0A/yhi6HAauq6lqAJKcBRwNX9SxTQHtTc3YG1g6wno2tvRDWrwTgtnNP5OT/2orvLnooAL938AN47uFLufW2Ozn+w9/Y5KnHHLoHf7BsCTf88jZe8olLNpn/nEfvxVMfthtrb7yVV5x+2SbzX/ybe/PEA+7P99b9gv/zr1dsMv/lv70fj91vMSvX3sTJn7tqk/mvOfJBHLrXfbjk+ht42xev2WT+G556AAfutjNf++563vWf391k/t88/aHss+uOfPmqH/P+r167yfx3HPtwdttlez73rbV84qLrN5n/T885lPvcc1s+s2I1Z16yZpP5H3nBYWy/7SI+fuF1fP7yH20y//Q/PhyAUy/4Hude/ZON5m23zSI++keHAfCP536Xr69av9H8e++wLe977qEAvPWL3+ab1/9so/kP2Hk73nncIwD4y8+t5Kq1N280f+9d78lbnn4wAK//18u5dt0vN5p/wG478canHgjASaddyo9u+tVG8w/Z69689sgHA/AnH7+En91y20bzH7PvYk58wn4APP9D3+BXt9+50fwnPOR+nPC4fQA49p8vZDw/e372wM/egv7s/a/v89zDl24yf64M8vD77sDqnvE17bRebwKek2QNcDbw8olWlOSEJCuSrFi3bt3sVLf6fGADAIu4kwPvvHR21itJ0jxJ1WAODyT5A+B3q+pF7fhzgcOq6uU9y7yyreGUJIcDHwQOqqoNm1vvsmXLasWKFTMvcO2F8OnHAhtg0T3gmefBbofPfL2SJM2iJJdU1bJ+lh1kS30NsKRnfA82Pbz+QuAMgKq6ENgOWDzAmu622+GwuDnUxRHvNNAlSQveIEP9YmC/JA9Msi1NR7jl45b5AfAEgCQPoQn1WTq+3odt29P5iw+as01KkjQoAwv1qroDeBlwDnA1TS/3lUlOTnJUu9irgBcn+RbwaeD4GtT5AEmSOm6Qvd+pqrNpOsD1TntDz/BVwGMGWYMkSaPCK8pJktQRhrokSR1hqEuS1BGGuiRJHdF3R7kkuwN79T6nqi4YRFGSJGnq+gr1JG8FjqW5bvvYBYULMNQlSRoS/bbUfx94UFX9epDFSJKk6ev3nPq1wDaDLESSJM1Mvy31W4DLkpwL3NVar6oTB1KVJEmasn5DfTmbXrddkiQNkb5Cvao+2t6UZf920jVVdfvgypIkSVPVb+/3I4CPAtcBAZYkeb4/aZMkaXj0e/j9FOBJVXUNQJL9ae6qduigCpMkSVPTb+/3bcYCHaCqvoO94SVJGir9ttRXJPkg8PF2/NnAJYMpSZIkTUe/of4S4KXAiTTn1C8A3juooiRJ0tT12/v918Db24ckSRpCWwz1JGdU1TOTXEFzrfeNVNXBA6tMkiRNyWQt9T9r//7eoAuRJEkzs8Xe71X1o3ZwPbC6qq4H7gE8DFg74NokSdIU9PuTtguA7dp7qp8LvAD4yKCKkiRJU9dvqKeqbgGeDryrqp4GHDC4siRJ0lT1HepJDqf5ffq/t9P6/Tnc8Lrt5ubv+ivntw5JkmZBv6F+EvB64LNVtTLJ3sB5gytrDqy9ENavbIbPP6kZlyRpAev3d+pfAb7SM34tzYVoFq7V5wMbmuENtzfjux0+jwVJkjQzk/1O/Z1VdVKSzzHx79SPGlhlg7bkCJoDFRtgq23acUmSFq7JWupj13r/+0EXMud2OxwWHwjrr4Aj3mkrXZK04G0x1Ktq7KYtK4Bbq2oDQJJFNL9XX9i23an5u/ig+a1DkqRZ0G9HuXOBHXrGtwe+PPvlSJKk6eo31Lerql+MjbTDO2xheUmSNMf6DfVfJjlkbCTJocCtgylJkiRNR78XkDkJ+EySseu9PwA4djAlSZKk6ej3d+oXJ3kw8CAgwLer6vaBViZJkqakr8PvSXYAXgv8WVVdASxN4u1YJUkaIv2eU/8wcBsw9mPuNcBfDaQiSZI0Lf2G+j5V9TbgdoCqupXmMLwkSRoS/Yb6bUm2p71UbJJ9gF8PrCpJkjRl/fZ+fyPwRWBJkk8CjwGOH1RRkiRp6iYN9SQBvg08HXg0zWH3P6uq9QOuTZIkTcGkoV5VleSsqjoU+Pc5qEmSJE1Dv+fUL0ryyIFWIkmSZqTfc+qPB/4kyXXAL2kOwVdVHTyowiRJ0tT0G+pPHmgVkiRpxrYY6km2A/4E2Be4AvhgVd0xF4VJkqSpmeyc+keBZTSB/mTglIFXJEmSpmWyw+8HVNVDAZJ8EPjG4EuSJEnTMVlL/a47sXnYXZKk4TZZS/1hSW5uhwNs346P9X7faaDVSZKkvm0x1Ktq0VwVIkmSZqbfi89IkqQhN9BQT3JkkmuSrEryugnmvyPJZe3jO0luHGQ9kiR1Wb8Xn5myJIuA9wC/A6wBLk6yvKquGlumql7Rs/zLgUcMqh5JkrpukC31w4BVVXVtVd0GnAYcvYXlnwV8eoD1SJLUaYMM9d2B1T3ja9ppm0iyF/BA4D83M/+EJCuSrFi3bt2sFypJUhcMMtQzwbTazLLHAWdW1Z0TzayqU6tqWVUt23XXXWetQEmSumSQob4GWNIzvgewdjPLHoeH3iVJmpFBhvrFwH5JHphkW5rgXj5+oSQPAu4NXDjAWiRJ6ryBhXp7WdmXAecAVwNnVNXKJCcnOapn0WcBp1XV5g7NS5KkPgzsJ20AVXU2cPa4aW8YN/6mQdYgSdKo8IpykiR1hKEuSVJHGOqSJHWEoS5JUkcY6pIkdYShLklSRxjqkiR1hKEuSVJHGOqSJHWEoS5JUkcY6pIkdYShLklSR4x2qN92c/N3/ZXzW4ckSbNgdEN97YWwfmUzfP5JzbgkSQvY6Ib66vOBDc3whtvbcUmSFq7RDfUlR3DXy99qm3ZckqSFa3RDfbfDYfGBzfAR72zGJUlawEY31AG23an5u/ig+a1DkqRZMNqhLklShxjqkiR1hKEuSVJHGOqSJHWEoS5JUkcY6pIkdYShLklSRxjqkiR1hKEuSVJHGOqSJHWEoS5JUkcY6pIkdYShLklSRxjqkiR1hKEuSVJHGOqSJHWEoS5JUkcY6pIkdYShLklSRxjqkiR1hKEuSVJHGOqSJHWEoS5JUkcY6pIkdYShLklSRxjqkiR1hKEuSVJHGOqSJHWEoS5JUkcY6pIkdYShLklSRxjqkiR1xEBDPcmRSa5JsirJ6zazzDOTXJVkZZJPDbIeSZK6bOtBrTjJIuA9wO8Aa4CLkyyvqqt6ltkPeD3wmKr6WZL7DaoeSZK6bpAt9cOAVVV1bVXdBpwGHD1umRcD76mqnwFU1U8GWI8kSZ02yFDfHVjdM76mndZrf2D/JF9PclGSIwdYjyRJnTaww+9AJphWE2x/P+AIYA/gq0kOqqobN1pRcgJwAsCee+45+5VKktQBg2yprwGW9IzvAaydYJl/q6rbq+r7wDU0Ib+Rqjq1qpZV1bJdd911YAVLkrSQDTLULwb2S/LAJNsCxwHLxy1zFvB4gCSLaQ7HXzvAmiRJ6qyBhXpV3QG8DDgHuBo4o6pWJjk5yVHtYucAP01yFXAe8OdV9dNB1SRJUpcN8pw6VXU2cPa4aW/oGS7gle1DkiTNgFeUkySpIwx1SZI6wlCXJKkjDHVJkjrCUJckqSMMdUmSOsJQlySpIwx1SZI6wlCXJKkjDHVJkjrCUJckqSMMdUmSOsJQlySpIwx1SZI6wlCXJKkjDHVJkjrCUJckqSMMdUmSOsJQlySpIwx1SZI6wlCXJKkjDHVJkjrCUJckqSMMdUmSOsJQlySpIwx1SZI6wlCXJKkjDHVJkjrCUJckqSMMdUmSOsJQlySpIwx1SZI6wlCXJKkjDHVJkjrCUJckqSMMdUmSOsJQlySpIwx1SZI6wlCXJKkjDHVJkjrCUJckqSMMdUmSOsJQlySpIwx1SZI6wlCXJKkjDHVJkjrCUJckqSMMdUmSOsJQlySpIwx1SZI6wlCXJKkjBhrqSY5Mck2SVUleN8H845OsS3JZ+3jRIOuRJKnLBhbqSRYB7wGeDBwAPCvJARMsenpVPbx9fGBQ9Uzotpubv+uvnNPNSpI0CINsqR8GrKqqa6vqNuA04OgBbm9q1l4IP13ZDJ9/UjMuSdICNshQ3x1Y3TO+pp023jOSXJ7kzCRLJlpRkhOSrEiyYt26dbNT3erzgTTDG+5oxyVJWrgGGeqZYFqNG/8csLSqDga+DHx0ohVV1alVtayqlu26666zU92SI2DRtpBFzd8lR8zOeiVJmidbD3Dda4DelvcewNreBarqpz2j7wfeOsB6Nrbb4fAH5zYt9CVHNOOSJC1ggwz1i4H9kjwQ+CFwHPCHvQskeUBV/agdPQq4eoD1bGq3ww1zSVJnDCzUq+qOJC8DzgEWAR+qqpVJTgZWVNVy4MQkRwF3ADcAxw+qHkmSui5V409zD7dly5bVihUr5rsMSZLmRJJLqmpZP8t6RTlJkjrCUJckqSMMdUmSOsJQlySpIwx1SZI6wlCXJKkjDHVJkjrCUJckqSMMdUmSOmLBXVEuyTrg+llc5WJg/Syub1S5H2fOfThz7sOZcx/O3Gzvw72qqq9blC64UJ9tSVb0e/k9bZ77cebchzPnPpw59+HMzec+9PC7JEkdYahLktQRhjqcOt8FdIT7cebchzPnPpw59+HMzds+HPlz6pIkdYUtdUmSOmJkQj3JkUmuSbIqyesmmH+PJKe38/87ydK5r3K49bEPX5nkqiSXJzk3yV7zUecwm2wf9ix3TJJKYi/kCfSzH5M8s/08rkzyqbmucdj18e95zyTnJbm0/Tf9lPmoc1gl+VCSnyS5cjPzk+Qf2/17eZJD5qSwqur8A1gEfA/YG9gW+BZwwLhl/hR4Xzt8HHD6fNc9TI8+9+HjgR3a4Ze4D6e+D9vl7gVcAFwELJvvuoft0edncT/gUuDe7fj95rvuYXr0uQ9PBV7SDh8AXDffdQ/TA3gccAhw5WbmPwX4AhDg0cB/z0Vdo9JSPwxYVVXXVtVtwGnA0eOWORr4aDt8JvCEJJnDGofdpPuwqs6rqlva0YuAPea4xmHXz+cQ4M3A24BfzWVxC0g/+/HFwHuq6mcAVfWTOa5x2PWzDwvYqR3eGVg7h/UNvaq6ALhhC4scDXysGhcBuyR5wKDrGpVQ3x1Y3TO+pp024TJVdQdwE3DfOaluYehnH/Z6Ic23VN1t0n2Y5BHAkqr6/FwWtsD081ncH9g/ydeTXJTkyDmrbmHoZx++CXhOkjXA2cDL56a0zpjq/5mzYutBb2BITNTiHt/tv59lRlnf+yfJc4BlwG8NtKKFZ4v7MMlWwDuA4+eqoAWqn8/i1jSH4I+gOWL01SQHVdWNA65toehnHz4L+EhVnZLkcODj7T7cMPjyOmFeMmVUWuprgCU943uw6aGku5ZJsjXN4aYtHVoZNf3sQ5I8Efi/wFFV9es5qm2hmGwf3gs4CDg/yXU05+GW21luE/3+e/63qrq9qr4PXEMT8mr0sw9fCJwBUFUXAtvRXNNc/enr/8zZNiqhfjGwX5IHJtmWpiPc8nHLLAee3w4fA/xntb0dBPSxD9tDx/9ME+iew9zUFvdhVd1UVYuramlVLaXpl3BUVa1PndtwAAAIpklEQVSYn3KHVj//ns+i6bhJksU0h+OvndMqh1s/+/AHwBMAkjyEJtTXzWmVC9ty4HltL/hHAzdV1Y8GvdGROPxeVXckeRlwDk2vzw9V1cokJwMrqmo58EGaw0uraFrox81fxcOnz334d8COwGfaPoY/qKqj5q3oIdPnPtQk+tyP5wBPSnIVcCfw51X10/mrerj0uQ9fBbw/yStoDhsfb0Pnbkk+TXN6Z3Hb7+CNwDYAVfU+mn4ITwFWAbcAL5iTunyPJEnqhlE5/C5JUucZ6pIkdYShLklSRxjqkiR1hKEuSVJHGOoaWknuTHJZkiuTfC7JLrO8/uOTvLsdflOSV29muZOSPG82t72Z7ZzcXrxnbJs79Mw7e0uvP8luSc5shx8+l3fUSnLf9n26LMn/JPlhz/i2s7ytJya5qV331Un+7zTWsSjJV9vhvZMc1zPvUUneMct1fjvJ3/bxnEP6uZxt+/5+YKY1qpsMdQ2zW6vq4VV1EM21A1461wW0Vxf8I2Dgt+6sqjdU1Zfb0ZOAHXrmPWVLlzitqrVVdUw7+nCa38cOTLtfxrb90/Z9ejjwPuAdY+PtzULGnpP2UrgzdV67rUcCL0zysKk8uarurKrfbEf3pueaFFX131X1ilmosbfOQ4BnJHnUJMsfAkwa6lV1GbBPkoFfR1wLj6GuheJCem6GkOTPk1zc3qf4L3umP6+d9q0kH2+nPTXJf6e5L/SXk9x/Ctv9beCb7U1+SHJi7r5n/GnttHumubfyxe02jm6nH5/kX5N8Mcl3k7ytnb4oyUfaIxBXtBf3oJ12TJITgd2A85Kc1867LsniJG9N8qc9r/dNSV6VZGm7vm2Bk4Fj21bise22d22X3yrN/Z03e7nPJNsl+XBb26VJxq7MdnySzyT5HPAf/ey8JPu2db0P+CawJMmNPfOPG2t1Jrl/u79WJPlGmqtwbVZV/aJd5z5Jtk/y0bbmbyZ5XLvOh7bvy2Xte7Z3kq17avhb4PHt/BPbFvZZ7Xt0fZKd2vUkybXtezDVOm+hubXp7u26Hp3kwnbffj3Jfkm2B94APLut5ZgkO7afiW+0yz61Z7WfB47t5z3QiJmL+7v68DGdB/CL9u8i4DPAke34k2ju9RyaL6afp7m38YE01/he3C53n/bvvbn7QksvAk5ph48H3t0Ovwl49QQ1/CXw8p7xtcA92uFd2r9/AzxnbBrwHeCe7fqvpbmPwHbA9TTXgj4U+FLPOsfW8xHgmHb4urHX0TsOPAL4Ss/0q4A9gaW093XufV3t+BuBk3r23b9Mst9fBXy4HX4wzeVCt2vXu2Zsv27muRvtR2BfYAPwyHZ8a+DGnvnHAR9oh08HHt0O3/V6xq3/icBZ7fCubW0PAl4LvL+dfmC7r7cF/gk4tp1+j/Z13FVD7/omWP97gOe2w48BvjjNOu9D8+Xjfu34zsCidvhI4PSez+Y7e9bxNuC4ns/wd4Dt2vHfAj473/9GfQzfYyQuE6sFa/skl9H8x3kJ8KV2+pPax6Xt+I40N+t4GHBmVa0HqKqxG/LsAZye5l7G2wLfn0INDwCu7hm/HPhkkrNori8+Vs9Rufuc/HY0QQtwblXdBJDmkqV7ASuBvZO8C/h3+mz1tq/p0iT3S7IbTaj9rKp+kGTpFp72IeDfgHfSnEr48CSbeSzwrnZ7305yPc2106H5MjLVGx19r6ou7mO5JwIPSu66udW9k2xfVbeOW+7xSS6l+bLw5qq6JsljaS5TTDWXO11L84Xiv4C/SLIX8K9VtSo9pw4mcTrwGuDjNF8+Tp9GnZfTfDF6c919P4RdgI8l2WeS7T8JeHKS17XjY5+r7wA/oTmaI23Ew+8aZrdWc05yL5owHjunHuAtdfd5232r6oPt9Imue/wumpbrQ4E/pvnPse8axi3/v2lacIcCl7QBEeAZPfXsWVVjXwR671R3J7B1Vf2M5gvI+e1rmmqnpzNpbjp0LHDaZAtX1Wrgx0l+G3gUk9/nfqJbRo75Zb9FbuY5G8atv3ffBjisZz/uPkFQQnOu+hFVdWhVvX9LNVfVx4Gn0bwPXxo7LN+nrwIHJLkvcBTw2WnUeTBwMHBikoe20/8aOKeaviK/z+Y/jwF+f9zn6jvtvO1oPpvSRgx1Db22pXsi8Ook29DchOKPkuwIkGT3JPcDzgWe2f4nTJL7tKvYGfhhO/x8puZqmhbf2P3Ol1TVeTQtuF1ojhKcA7w8bdMtzd3qNqs9n71VVf0L8P9oOkiN93OaW7FO5DSaluMxNAHfz3M/AHwCOKOq7mzreFqSt0zw/AuAZ7fL7E/TOrxmS6+pX9Xci/tn7XnkrWgCd8yX6ekMmeThU1h1b80PoTnCsirJ3lW1qqr+geaoyMHjnrfZ/VxVxd1HOL5Vd3dUnFKdVfVtmkPpr2kn9X4ej99CLefQfO7HttP7udofuHJL29VoMtS1IFTVpTSdjY6rqv+g6Y1+YZIraILtXlW1kqYV9JUk3wLe3j79TTR3jvsqsH6Km/4Czfl6aM7tf6Ld5qU0vbxvBN5Mc3emy5Nc2Y5vye4090y/jOY8+usnWOZU4AtpO8r1al/nvYAf1sS3cjyPpoV5WZKxzlTLab6A9B563we4eYLnvxdY1L7O02nuzvXrCZabrtcCX6T5EramZ/pLgce0HdquAl48hXW+i+Z0zRXAJ4HnVdPz/g+TrGz39d40X2x6XUrzWr+VpoPieKcDz+HuQ+/TrfO9wBOS7Am8Ffi7JF8ft8x/Ag9rO8UdQ9OfY4e2899Kms/xmMfTfEmRNuJd2qRJJPks8Jqq+u581zJdSZbRfAn5zZ5pnwBeUVXeI3sBaXvKnwc8ZuyoizTGUJcmkeRBwP2r6oL5rmU62o5WLwGeXVVfm+96NDML/fOowTLUJUnqCM+pS5LUEYa6JEkdYahLktQRhrokSR1hqEuS1BGGuiRJHfH/AV/p5q3OVAPEAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "precision_recall_curves(lr_mymodel_balanced, X_test, y_test)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "When recall is close to 1.0, precision is about 80%, this is similar in the logistic regression without balanced class in our first model. When recall is roughly 0.65, precision is about 0.89 as shown in balanced logistic regression.\n", "\n", "**As recall reduces, precision can increase. But since there is 'missing good investment opportunity' vs 'being safer', to which point we shoud choose a model that can give best investment return, this is where my investmet portfolio function can help.** " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 4.3 Try Random Forest Classifier with class_weight as balanced\n", "\n", "Random forests are able to work with nonlinear data, and learn complex conditionals. Logistic regressions are only able to work with linear data. Training a random forest algorithm may enable us to get more accuracy due to columns that correlate nonlinearly with loan_status." ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Confusion matrix: \n", " [[ 9346 49977]\n", " [ 14450 233378]]\n", "The True Positive Rate (Recall/Sensitivity)is tpr = tp/(tp+fn): 0.9416934325419243\n", "The False Positive Rate (1-specification)is fpr = fp/(fp+tn): 0.8424557085784603\n", "The Precision (tp/(tp+fp)) is: 0.8236240758059677\n", "\n", "\n", "The investment return with this model is: 12.72%\n", "The investment return with this model for the worse case is: -4.91%\n", "\n", "\n" ] } ], "source": [ "# create model, with balanced \n", "rf_balanced = RandomForestClassifier(class_weight=\"balanced\", random_state=1).fit(X_train, y_train)\n", "# use my function\n", "my_metric(rf_balanced, X_test, y_test)\n", "investment_return_with_my_model(rf_balanced, X_test, y_test)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Analysis the result**\n", "\n", "Unfortunately, using a random forest classifier didn't improve our false positive rate. The model is likely weighting too heavily on the 1 class, and still mostly predicting 1s. Compared with Logistic Regression Balanced class, the false positive rate with Random Forest is still very high." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "**My strategy**\n", "\n", "Choose logistic regression and try to lower the false positive rate further by **assigning a harsher penalty for misclassifying the negative class**. While setting class_weight to balanced will automatically set a penalty based on the number of 1s and 0s in the column, we can also set a manual penalty. (The default penalty scikit-learn imposed for misclassifying a 0 would have been around 4, since there are 4 times as many 1s as 0s)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 4.4 Logistic Regression with class_weight with different penalty in a for loop" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Confusion matrix: \n", " [[ 48188 11135]\n", " [132224 115604]]\n", "The True Positive Rate (Recall/Sensitivity)is tpr = tp/(tp+fn): 0.4664686798908921\n", "The False Positive Rate (1-specification)is fpr = fp/(fp+tn): 0.18770122886570134\n", "The Precision (tp/(tp+fp)) is: 0.912142276647283\n", "\n", "\n", "The investment return with this model is: 9.42%\n", "The investment return with this model for the worse case is: 0.63%\n", "\n", "\n", "Confusion matrix: \n", " [[ 50930 8393]\n", " [148812 99016]]\n", "The True Positive Rate (Recall/Sensitivity)is tpr = tp/(tp+fn): 0.399535161482964\n", "The False Positive Rate (1-specification)is fpr = fp/(fp+tn): 0.1414796959020953\n", "The Precision (tp/(tp+fp)) is: 0.9218594344980402\n", "\n", "\n", "The investment return with this model is: 8.93%\n", "The investment return with this model for the worse case is: 1.12%\n", "\n", "\n", "Confusion matrix: \n", " [[ 53789 5534]\n", " [171675 76153]]\n", "The True Positive Rate (Recall/Sensitivity)is tpr = tp/(tp+fn): 0.3072816630889165\n", "The False Positive Rate (1-specification)is fpr = fp/(fp+tn): 0.09328590934376212\n", "The Precision (tp/(tp+fp)) is: 0.932253602164359\n", "\n", "\n", "The investment return with this model is: 8.61%\n", "The investment return with this model for the worse case is: 1.84%\n", "\n", "\n", "Confusion matrix: \n", " [[ 54774 4549]\n", " [180205 67623]]\n", "The True Positive Rate (Recall/Sensitivity)is tpr = tp/(tp+fn): 0.2728626305340801\n", "The False Positive Rate (1-specification)is fpr = fp/(fp+tn): 0.07668189403772567\n", "The Precision (tp/(tp+fp)) is: 0.9369700160727151\n", "\n", "\n", "The investment return with this model is: 8.30%\n", "The investment return with this model for the worse case is: 2.00%\n", "\n", "\n", "Confusion matrix: \n", " [[ 56074 3249]\n", " [193410 54418]]\n", "The True Positive Rate (Recall/Sensitivity)is tpr = tp/(tp+fn): 0.21957970850751327\n", "The False Positive Rate (1-specification)is fpr = fp/(fp+tn): 0.05476796520742377\n", "The Precision (tp/(tp+fp)) is: 0.9436592852064439\n", "\n", "\n", "The investment return with this model is: 8.04%\n", "The investment return with this model for the worse case is: 2.41%\n", "\n", "\n" ] } ], "source": [ "# set model parameter with different penalty\n", "for i in [6, 7, 8, 9, 10]: \n", " # creat our model\n", " penalty = {0: i, 1: 1} \n", " lrf_mymodel_penalty = LogisticRegression(class_weight=penalty).fit(X_train, y_train)\n", " # use my function\n", " my_metric(lrf_mymodel_penalty, X_test, y_test) \n", " investment_return_with_my_model(lrf_mymodel_penalty, X_test, y_test)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "# Summary" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Different model returns different results, which one to choose depends on the investor. As explained by the precision recall curve, there is a trade off between 'missng good investment opportunity' vs 'being safer'. For this project, we **consider from a more conservative perspective and help investor reduce risk and still get decent return**. **Remember our goal for is: Choose a model that has lower than 19.57% charged off rate, and have expected return higher than 7.1% (i.e. average interest for only A loans), and for the worst case, still have expected return higher than 1.15%**.\n", "\n", "Keeping this in mind, **I would recommend the Logistic Regression model with class_weight set as 10, because in this model:**\n", "\n", "- **False posistive rate is: 5.4% (In plain English, this is the Charged-off rate, similar as default rate, the percentage of the loans invested that not pay you back). This is much lower than 19% loss if the investor randomly choose loans.**\n", "- **Investing in this portfolio will have expected return is the range of of 8.04%, and the expected return for the worse case is: 2.41%.**\n", "\n", "This gives 1% higher return and TWICE safer than if the investor only randomly choose A loans. Well, 1% higher maybe not too significant, but we all know the bank gives us interest rate like 0.1%; If there is a way we can invest more safely and get higher return, it is something worth doing. And if you invest a lot, 1% increase can give you a lot more interest. \n", "\n", "**If you are willing to take higher risk**, I would recomend the the Logistic Regression model with class_weight set as 6, and **your expected return will be in the range of 0.63% to 9.42%**. Not too bad, at least for the worst case, you still keep all of your principle safe with 0.63% of return." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Other things to consider:\n", "\n", "1. I did not use other popular advanced model like SVM, Gridient Boosting and Neural Network for this data. First reason is: with Logistic Regression, I already got good results, even better than Random forest, and Logistic Regression is easier to interpret and quicker to interpret. Second reason is: I tried with other model, it never runs even if I took only a sample, the running time is very slow. \n", "2. I think Artificial Financial Advisor can be a trend for the future, it doesn't mean to replace finance manager, but another options. I plan to do more project about data science application on finance. Thank you for reading." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.0" } }, "nbformat": 4, "nbformat_minor": 2 }