{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "\n", " \n", "## [mlcourse.ai](mlcourse.ai) – Open Machine Learning Course \n", "###
Author: Pavel Taiga, ptaiga\n", " \n", "##
Telco Customer Churn Prediction" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Dataset: https://www.kaggle.com/blastchar/telco-customer-churn\n", "\n", "The individual project was сompleted during the fall2018-session on December 16, 2018" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "__Content__:\n", "+ 1 Feature and data explanation\n", "+ 2 Basic data analysis\n", " - 2.1 Data loading and inspection\n", " - 2.2 Data preprocessing\n", " - 2.3 Initial data analysis\n", " - 2.4 New features\n", "+ 3 Visual data analysis\n", "+ 4 Cross-validation and adjustment of model hyperparameters\n", "+ 5 Conclusions" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1 Feature and data explanation" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The raw data contains 7043 rows (customers) and 21 columns (features). Each row represents a customer, each column contains customer’s attributes.\n", "\n", "**The “Churn” column is our target.**" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "_Columns_:\n", "\n", " - **customerID**: Customer ID\n", " - **gender**: Customer gender (female, male)\n", " - **SeniorCitizen**: Whether the customer is a senior citizen or not (1, 0)\n", " - **Partner**: Whether the customer has a partner or not (Yes, No)\n", " - **Dependents**: Whether the customer has dependents or not (Yes, No)\n", " - **tenure**: Number of months the customer has stayed with the company\n", " - **PhoneService**: Whether the customer has a phone service or not (Yes, No)\n", " - **MultipleLines**: Whether the customer has multiple lines or not (Yes, No, No phone service)\n", " - **InternetService**: Customer’s internet service provider (DSL, Fiber optic, No)\n", " - **OnlineSecurity**: Whether the customer has online security or not (Yes, No, No internet service)\n", " - **OnlineBackup**: Whether the customer has online backup or not (Yes, No, No internet service)\n", " - **DeviceProtection**: Whether the customer has device protection or not (Yes, No, No internet service)\n", " - **TechSupport**: Whether the customer has tech support or not (Yes, No, No internet service)\n", " - **StreamingTV**: Whether the customer has streaming TV or not (Yes, No, No internet service)\n", " - **StreamingMovies**: Whether the customer has streaming movies or not (Yes, No, No internet service)\n", " - **Contract**: The contract term of the customer (Month-to-month, One year, Two year)\n", " - **PaperlessBilling**: Whether the customer has paperless billing or not (Yes, No)\n", " - **PaymentMethod**: The customer’s payment method (Electronic check, Mailed check, Bank transfer (automatic), Credit card (automatic))\n", " - **MonthlyCharges**: The amount charged to the customer monthly\n", " - **TotalCharges**: The total amount charged to the customer\n", " - **Churn**: Whether the customer churned or not (Yes or No)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We will solve a typical customer churn prediction problem. When the market is \"saturated\" and it is difficult to find new customers (аs in the case of telecommunications companies), the important task of the company is to retain existing customers, that is, to minimize churn." ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "To begin with, we will inspect the data, then convert them into the format necessary for further use. Next, we will try to train several models and conduct a visual analysis." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2 Basic data analysis" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's start by loading, inspecting and preprocessing the data (https://www.kaggle.com/blastchar/telco-customer-churn). And also perform a basic analysis." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 2.1 Data loading and inspection " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "import numpy as np" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "data = pd.read_csv('telco-customer-churn.zip')\n", "data.shape" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "data.info()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "So we see that most of our data is non-numeric. Let's try to find out whether this is true, and perform type conversion if necessary." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 2.2 Data preprocessing" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Select and convert binary features, as well as the target column \"Churn\"." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "yes_no_cols = ['Partner', 'Dependents', 'PhoneService', 'PaperlessBilling', 'Churn']" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "for col in yes_no_cols:\n", " data[col] = data[col].map({'Yes': 1, 'No': 0})" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "data['Male'] = data['gender'].map({'Male': 1, 'Female': 0})\n", "data.drop('gender', axis=1, inplace=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The column \"TotalCharges\" in addition to numerical values contains a string containing a space. This check can be done for example with the following code:\n", "\n", "try:\n", " for val in data['TotalCharges']:\n", " float(val)\n", "except ValueError:\n", " print(f\"'{val}'\")\n", "\n", "\n", "Perform the processing and replace the spaces with zeros. After that, we convert this attribute into a numeric format." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print('Row indexes to be replaced:')\n", "for i in range(len(data['TotalCharges'])):\n", " if data.loc[i, 'TotalCharges'] == ' ':\n", " data.loc[i, 'TotalCharges'] = 0\n", " print(i, end=', ')\n", "data['TotalCharges'] = data['TotalCharges'].astype('float')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now let's see what features are of type 'object'." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def object_features(data, drop=[]):\n", " '''Selects columns of type 'object' in the data and deletes the specified in the drop list.'''\n", " obj_cols = []\n", " for col in data:\n", " if data[col].dtype == 'object':\n", " obj_cols.append(col)\n", " for col in drop:\n", " obj_cols.remove(col)\n", " return obj_cols\n", "\n", "print(object_features(data))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And what values does each of these features, except for the _'customerID'_ column, which has no special application for this task, since it is only an identifier." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "for col in object_features(data, drop=['customerID']):\n", " print(col, ':', data[col].unique())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It is not difficult to notice that some featuress have an intersection. For example, those customers who do not have _'PhoneService'_ have _'No phone services'_ in the _'MuptipleLines'_ column." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "sum(pd.get_dummies(data['MultipleLines'], prefix='MultipleLines')['MultipleLines_No phone service'] \n", " - data['PhoneService'].map({1:0, 0:1}))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Thus, from the feature _'MultipleLines'_ we used dummy (one-hot) encoding making two new columns and delete the third responsible for the existence of 'PhoneServices'." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "data = pd.concat([data, pd.get_dummies(data['MultipleLines'], prefix='MultipleLines')\n", " .drop('MultipleLines_No phone service', axis=1)], axis=1) \\\n", " .drop('MultipleLines', axis=1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now let's work with the features associated with the presence/absence of the Internet from the customers. Start with the 'InternetServices' column and convert it to three using dummy encoding." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "i_s = pd.get_dummies(data['InternetService'])\n", "data['DSL'] = i_s['DSL']\n", "data['FiberOptic'] = i_s['Fiber optic']\n", "data['InternetService'] = i_s['No'].map({1:0, 0:1})\n", "data.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Take the features associated with Internet services." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "i_s_cols = ['OnlineSecurity', 'OnlineBackup', 'DeviceProtection', \n", " 'TechSupport', 'StreamingTV', 'StreamingMovies']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Perform dummy encoding for these features. And by analogy with the telephone services will remove the value _'No internet service'_, so this information is already contained in the column _'InternetService'_, if we perform the inversion." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "for col in i_s_cols:\n", " dum = pd.get_dummies(data[col], prefix=col)\n", " diff = dum[col+'_No internet service'] \\\n", " - data['InternetService'].map({0:1, 1:0})\n", " if diff.isnull().sum():\n", " print(col, 'contains NaN values!')\n", " else:\n", " print(col + \":\", diff.sum())\n", " data = pd.concat([data.drop(col, axis=1), dum.drop(col+'_No internet service', axis=1)], \n", " axis=1)" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "And at the end of data preprocessing we will perform the transformation of the remaining two categorical features.\n", "\n", "In the case of _'Contract'_, you can try a method where the _'Month-to-month'_ is coded at _0_, _'One year'_ at _1_, and _'Two years'_ at _2_, as well as dummy encoding. But since the first method showed the worst quality, so let's focus on the second method." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# data['Contract'] = data['Contract'].map({'Month-to-month': 0, 'One year': 1, 'Two year': 2}) \\\n", "# .astype(np.uint8)\n", "\n", "contract = pd.get_dummies(data['Contract'])\n", "for i, col in enumerate(contract):\n", " data['Contract_'+str(i)] = contract[col]\n", "data.drop('Contract', axis=1, inplace=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Perform a similar operation for the _'PaymentMethod'_." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "pm = pd.get_dummies(data['PaymentMethod'])\n", "for i, col in enumerate(pm):\n", " data['PM_'+str(i+1)] = pm[col]\n", "data.drop('PaymentMethod', axis=1, inplace=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we have all of the features numerical. And we can do basic data analysis." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 2.3 Initial data analysis" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Necessary libraries for further work." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from sklearn.tree import DecisionTreeClassifier\n", "from sklearn.ensemble import RandomForestClassifier\n", "from sklearn.linear_model import LogisticRegression\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.metrics import roc_auc_score\n", "from sklearn.pipeline import Pipeline\n", "from sklearn.preprocessing import StandardScaler\n", "import warnings" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Separate the feature matrix and the target variable." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "y = data['Churn']\n", "X = data.drop(['Churn'] + object_features(data), axis=1)\n", "y.value_counts(normalize=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this task, there is a small imbalance of classes of the target variable: the fraction of churn (class \"1\") is less than its absence (class \"0\"). Therefore, it is important for us not to miss the presence of a possible churn. If some client is not going to leave, but we predict the churn, then there is nothing wrong. But we must not lose a client who is about to leave. Therefore, our metric should be able to account for True Negative errors. Therefore, it is advisable to choose the __ROC AUC__ metric for this task.\n", "\n", "Also, the advantage of this metric is that we operate on the probabilities of predictions, which allows us to determine how confident the algorithm is in a particular answer." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "X.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Since there are only 33 features in our dataset, it is possible that a Random Forest or even a Decision Tree will cope well with it. But at the same time, we will certainly try to apply Logistic Regression as well." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Separate our data into the main and test sets in the ratio 70/30." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "First, let's check the quality (ROC AUC) of the _DecisionTreeClassifier_, _RandomForestClassifier_ and _LogisticRegression_ on our data without setting the models parameters (out of the box)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "tree = DecisionTreeClassifier(random_state=42)\n", "tree.fit(X_train, y_train)\n", "print(roc_auc_score(y_test, tree.predict_proba(X_test)[:,1]))" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "forest = RandomForestClassifier(n_estimators=100, random_state=42)\n", "forest.fit(X_train, y_train)\n", "forest_score = roc_auc_score(y_test, forest.predict_proba(X_test)[:,1])\n", "print(forest_score)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "logit = LogisticRegression(random_state=42, solver='liblinear')\n", "logit.fit(X_train, y_train)\n", "logit_score = roc_auc_score(y_test, logit.predict_proba(X_test)[:,1])\n", "print(logit_score)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Before logistic regression training, it is good practice to normalize features, for example, using StandardScaler. If we do this, we will see that the quality of the model predictions increased slightly." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "warnings.filterwarnings('ignore')\n", "logit_pipe = Pipeline([\n", " ('scaler', StandardScaler()),\n", " ('logit', LogisticRegression(random_state=42, solver='liblinear'))\n", "])\n", "logit_pipe.fit(X_train, y_train)\n", "print(roc_auc_score(y_test, logit_pipe.predict_proba(X_test)[:,1]))\n", "warnings.filterwarnings('default')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "models_scores = pd.DataFrame([logit_score, forest_score], \n", " columns=['out-of-box'], index=['logit', 'forest'])" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 2.4 New features" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this section, we will create new features." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from sklearn.preprocessing import PolynomialFeatures" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "First, try to create a binary attribute *long_term_client*, which will be equal to _1_ if the client has a contract for one or two years." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "long_term_client = (data['Contract_0'] != 1).astype(np.uint8)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It is possible that there are some non-linear dependencies in our data. Сreate polynomial features based on the existing ones with the help of _PolynomialFeatures_. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "poly = PolynomialFeatures()\n", "poly_feat = poly.fit_transform(X)\n", "X_poly = np.hstack([X.values, poly_feat])\n", "X_poly.shape" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If we add these features to the existing ones and check the quality of the models, we will notice a deterioration in quality of models, especially in the case of logistic regression. Therefore, we will not leave them and will continue to work without them, since it makes no sense to add 628 new features that do not particularly affect the result." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "pd.concat([models_scores, pd.Series([0.8192703982177666, 0.83254715379234], \n", " name='poly', index=['logit', 'forest'])], axis=1)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3. Visual data analysis" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Import libraries needed for visual analysis." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from matplotlib import pyplot as plt\n", "import seaborn as sns; sns.set()\n", "%matplotlib inline" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We will add features one by one and see how the quality metric changes." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def features_score(model, X, y, seed=42, test_size=0.3):\n", " X_train, X_test, y_train, y_test = \\\n", " train_test_split(X, y, test_size=test_size, random_state=seed, stratify=y)\n", " features, scores, drop = [], [], []\n", " for feat in X:\n", " features.append(feat)\n", " model.fit(X_train[features], y_train)\n", " scores.append(roc_auc_score(y_test, model.predict_proba(X_test[features])[:,1]))\n", " if len(scores) > 1 and scores[-1] < scores[-2]:\n", " drop.append(feat)\n", " # print(features[-2:], scores[-2:])\n", " print(drop)\n", " return pd.Series(scores, index=features)\n", "\n", "print('The features that reduce the quality metric of \\'forest\\':')\n", "s1 = features_score(forest, X, y) \n", "print('\\nThe features that reduce the quality metric of \\'logit\\':')\n", "s2 = features_score(logit, X, y)\n", "pd.DataFrame([s1, s2], index=['Forest score', 'Logit score']).T \\\n", " .plot(kind='bar', rot=90, figsize=(10,5));" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We see that when we add (one by one) features, the quality increases with small fluctuations. \n", "\n", "If we remove the features that reduce the quality in such an experiment, the quality of LogisticRegrassion will slightly improve, but the quality of RandomForest will fall. Thus, it makes sense to leave all the indicative space." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now look at the importance of each of the features according to the random forest. Show the top-5 most important and build a plot." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "feat_imp = pd.Series(forest.feature_importances_, index=X.columns, name='Feature importances') \\\n", " .sort_values(ascending=False)\n", "feat_imp.plot(kind='bar', figsize=(12,4), title='Feature importances')\n", "plt.xlabel('Features'), plt.ylabel('Importance');\n", "feat_imp.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Plot the distribution of the most important features." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "data[feat_imp.index[:4]].hist(figsize=(12,6));" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Also look at the pairwise correlation of features and include the _Churn_ as well." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "_, axes = plt.subplots(figsize=(12, 10))\n", "sns.heatmap(data.corr(), ax=axes);" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The heatmap shows that there are no features that strongly correlate with each other except _'TotalCharges'_ to _'MonthlyCharges'_ and _'TotalCharges'_ to _'tenure'_. But these features are very significant from the point of view of a random forest. Let's look at their distribution relative to each other." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "sns.pairplot(data[feat_imp.index[:3]]);" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Build the distribution _'MonthlyCharges'_ and _'tenure'_ relative to _'TotalyCharges'_ and highlight the churn using orange color." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "_, axes = plt.subplots(figsize=(12, 6), nrows=1, ncols=2)\n", "plt.suptitle('Distribution \\'MonthlyCharges\\' and \\'tenure\\' relative to \\'TotalyCharges\\'')\n", "axes[0].scatter(data['tenure'], data['TotalCharges'], c=y.map({1:'orange', 0:'blue'}), alpha=0.5)\n", "axes[0].set_xlabel('tenure'), axes[0].set_ylabel('TotalyCharges')\n", "axes[1].scatter(data['MonthlyCharges'], data['TotalCharges'], c=y.map({1:'orange', 0:'blue'}), alpha=0.5)\n", "axes[1].set_xlabel('MonthlyCharges'), axes[1].set_ylabel('TotalyCharges');" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "According to these two graphs, there are two types of clients that are prone to churn.\n", "\n", "On the left graph we see that customers who are prone to churn tend to have a high level of _'TotalCharges'_ steadily increasing over time. Apparently it is loyal customers who suddenly something was not satisfied.\n", "\n", "On the right graph, we see that churn-prone customers have high monthly charges with low total charges. Most likely these are those clients who came just to try.\n", "\n", "Therefore, it is logical to assume that the model comparing these three features with each other can assume the probability of churn. And that's why these features are so important according to random forest." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 4 Cross-validation and adjustment of model hyperparameters" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's start by importing the libraries we need." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from sklearn.model_selection import StratifiedKFold, GridSearchCV" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Create cross-validation object that returns stratified folds: the folds are made by preserving the percentage of samples for each class. Since we have little data, we will make a split into 3 folds. And also we will shuffle the original sample, since there is no reference to the time stamps." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "skf = StratifiedKFold(n_splits=3, shuffle=True, random_state=42)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For logistic regression, we will configure the _'C'_: \n", " - this parameter is the inverse of regularization strength; \n", " - must be a positive float;\n", " - like in support vector machines, smaller values specify stronger\n", " regularization;" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%time\n", "gs_logit = GridSearchCV(logit, param_grid={'C': np.logspace(-3, 0, 20)}, \n", " cv=skf, scoring='roc_auc', return_train_score=True)\n", "gs_logit.fit(X_train, y_train)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plt.plot(np.logspace(-3, 0, 20), gs_logit.cv_results_['mean_train_score'])\n", "plt.plot(np.logspace(-3, 0, 20), gs_logit.cv_results_['mean_test_score'])\n", "plt.xscale('log')\n", "plt.xlabel('C')\n", "plt.ylabel('ROC AUC')\n", "plt.title('Learning curve for LogisticRegression')\n", "print('Best parameter:', gs_logit.best_params_) \n", "print('Best cv-score:', gs_logit.best_score_)\n", "logit_score = roc_auc_score(y_test, gs_logit.best_estimator_.predict_proba(X_test)[:,1])\n", "print('Score for test set:', logit_score)\n", "models_scores['cv'] = pd.Series(name='cv', index=['logit', 'forest'])\n", "models_scores['test'] = pd.Series(name='test', index=['logit', 'forest'])\n", "models_scores['cv']['logit'] = gs_logit.best_score_\n", "models_scores['test']['logit'] = logit_score" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To simplify the search for optimal parameters and plot learning curves, we write a function." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def learning_and_plot(model, params, axes, cv=3):\n", " '''Training 'model' consistently with 'params' and plot learning curves in a given 'axes' '''\n", " best_params = {}\n", " i = 0\n", " for param in param_grid:\n", " p = best_params\n", " p[param] = param_grid[param]\n", " gs = GridSearchCV(model, p, cv=cv, scoring='roc_auc', return_train_score=True)\n", " gs.fit(X_train, y_train)\n", " for best in gs.best_params_:\n", " best_params[best] = [gs.best_params_[best]]\n", "\n", " axes[i].plot(param_grid[param], gs.cv_results_['mean_train_score'])\n", " axes[i].plot(param_grid[param], gs.cv_results_['mean_test_score'])\n", " axes[i].set_xlabel(param), axes[i].set_ylabel('ROC AUC') \n", " i += 1\n", " print('Best parameters:', gs.best_params_) \n", " print('Best cv-score:', gs.best_score_)\n", " return gs.best_estimator_, gs.best_score_" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And try this function first on a _RandomForestClassifier_. \n", "\n", "We will configure two parameters *'n_estimators'* and *'max_depth'*:\n", "\n", " - *'n_estimators'* - the number of trees in the forest;\n", " - *'max_depth'* - the maximum depth of the tree." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "forest = RandomForestClassifier(random_state=42)\n", "\n", "_, axes = plt.subplots(figsize=(12, 6), nrows=1, ncols=2)\n", "plt.suptitle('Learning curves for Random Forest Classifier')\n", "\n", "param_grid = {\n", " 'n_estimators': [50, 100, 150, 200],\n", " 'max_depth': [1,2,3,4,5,6,7,8,9,10,11,12],\n", "}\n", "\n", "best_model, cv_score = learning_and_plot(forest, param_grid, axes, skf)\n", "forest_score = roc_auc_score(y_test, best_model.predict_proba(X_test)[:,1])\n", "print('Score for test set:', forest_score)\n", "models_scores['cv']['forest'] = cv_score\n", "models_scores['test']['forest'] = forest_score" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And also do the same for the _LGBMClassifier_." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from lightgbm import LGBMClassifier" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's see the quality of the model without setting parameters." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "lgbm = LGBMClassifier(n_estimators=200, max_depth=10, random_state=42)\n", "lgbm.fit(X_train, y_train)\n", "lgbm_score = roc_auc_score(y_test, lgbm.predict_proba(X_test)[:,1])\n", "models_scores.loc['lgbm','out-of-box'] = lgbm_score\n", "print(lgbm_score)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We will configure the following parameters:\n", " - *max_depth* - maximum tree depth for base learners;\n", " - *learning_rate* - boosting learning rate;\n", " - *n_estimators* - number of boosted trees to fit." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "_, axes = plt.subplots(figsize=(12, 6), nrows=1, ncols=3)\n", "plt.suptitle('Learning curves for LightGBM Classifier')\n", "\n", "param_grid = {\n", " 'n_estimators': [50, 100, 150, 200, 250, 300, 400],\n", " 'max_depth': [1,2,3,4,5,6,7,8,9,10,11,12],\n", " 'learning_rate': [0.01, 0.05, 0.1, 0.2, 0.5, 1]\n", "}\n", "\n", "best_model, cv_score = learning_and_plot(lgbm, param_grid, axes, skf)\n", "lgbm_score = roc_auc_score(y_test, best_model.predict_proba(X_test)[:,1])\n", "print('Score for test set:', lgbm_score)\n", "models_scores['cv']['lgbm'] = cv_score\n", "models_scores['test']['lgbm'] = lgbm_score" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It can be seen that the results of the models on the test (hold-out) sample do not contradict the results on cross-validation. An interesting finding is that logistic regression works for this task as well as a random forest, and gradient boosting." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "models_scores" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can also see that gradient boosting is easily retrained on the available data and that the simplest model is enough to give a good quality on the test sample. Most likely this is due to the fact that only a small part of the features works and gives a good prediction. That is not contrary to the visual analysis." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 5 Conclusions" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this problem, we obtained a sufficiently high quality of prediction using simple models, which indicates the presence of simple dependencies on the target variable.\n", "\n", "It seems clear that improving quality requires both more data and additional features to provide more information about users." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It is reasonable to take logistic regression for this task. The training time of random forest significantly more that will be particularly noticeable with increase in volume of the dataset. Gradient boosting is easy retrain on such features, therefore, for the optimal settings of the algorithm needs more data." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.3" }, "toc": { "base_numbering": 1, "nav_menu": {}, "number_sections": true, "sideBar": true, "skip_h1_title": false, "title_cell": "Table of Contents", "title_sidebar": "Contents", "toc_cell": true, "toc_position": {}, "toc_section_display": true, "toc_window_display": false } }, "nbformat": 4, "nbformat_minor": 2 }