{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "\n", " \n", "## [mlcourse.ai](mlcourse.ai) – Open Machine Learning Course \n", "###
Author: Korgun Dmitry, @tbb\n", " \n", "##
Individual data analysis project" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import warnings\n", "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "\n", "import seaborn as sns\n", "sns.set_palette('Set3')\n", "\n", "%matplotlib inline\n", "warnings.filterwarnings('ignore')\n", "\n", "from sklearn.ensemble import GradientBoostingRegressor\n", "\n", "from sklearn.model_selection import train_test_split, cross_val_score, GridSearchCV\n", "from sklearn.model_selection import validation_curve, learning_curve\n", "from sklearn.metrics import mean_squared_error" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Dataset and features description \n", "### [Kaggle link](https://www.kaggle.com/c/elo-merchant-category-recommendation)\n", "\n", "Elo - one of the largest payment brands in Brazil. In the dataset we can see clients who use Elo and their transactions. We need to predict the loyalty score for each card_id.\n", "\n", "The description of the files are\n", "\n", "* train.csv - the training set\n", "* test.csv - the test set\n", "* sample_submission.csv - a sample submission file in the correct format - contains all card_ids you are expected to predict for.\n", "* historical_transactions.csv - up to 3 months' worth of historical transactions for each card_id\n", "* merchants.csv - additional information about all merchants / merchant_ids in the dataset.\n", "* new_merchant_transactions.csv - two months' worth of data for each card_id containing ALL purchases that card_id made at merchant_ids that were not visited in the historical data.\n", "\n", "The *historical_transactions.csv* and *new_merchant_transactions.csv* files contain information about each card's transactions. *historical_transactions.csv* contains up to 3 months' worth of transactions for every card at any of the provided merchant_ids. *new_merchant_transactions.csv* contains the transactions at new merchants (merchant_ids that this particular card_id has not yet visited) over a period of two months.\n", "\n", "*merchants.csv* contains aggregate information for each merchant_id represented in the data set.\n", "\n", "#### Main dataset:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "train = pd.read_csv('../../data/ELO/train.csv', parse_dates=['first_active_month'])\n", "test = pd.read_csv('../../data/ELO/test.csv', parse_dates=['first_active_month'])\n", "\n", "train.head()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# columns description\n", "pd.read_excel('../../data/ELO/Data_Dictionary.xlsx', sheet_name='train', header=2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Historical Transactions:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "hist = pd.read_csv('../../data/ELO/historical_transactions.csv')\n", "hist.head()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# columns description\n", "pd.read_excel('../../data/ELO/Data_Dictionary.xlsx', sheet_name='history', header=2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### New merchant transactions" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "transaction = pd.read_csv('../../data/ELO/new_merchant_transactions.csv')\n", "transaction.head()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# columns description\n", "pd.read_excel('../../data/ELO/Data_Dictionary.xlsx', sheet_name='new_merchant_period', header=2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Little bit preprocessing" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "train.info()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As features are categorical we can change type to free some memory." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "train['feature_1'] = train['feature_1'].astype('category')\n", "train['feature_2'] = train['feature_2'].astype('category')\n", "train['feature_3'] = train['feature_3'].astype('category')\n", "\n", "test['feature_1'] = test['feature_1'].astype('category')\n", "test['feature_2'] = test['feature_2'].astype('category')\n", "test['feature_3'] = test['feature_3'].astype('category')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "train.info()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Exploratory data analysis and feature engineering" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Check missed data" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "train.isna().sum()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "test.isna().sum()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Target column\n", "Let start analys with target value" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fig, ax = plt.subplots(figsize = (16, 6))\n", "plt.suptitle('Target value distribution', fontsize=24)\n", "sns.distplot(train['target'], bins=50, ax=ax);" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can see that some of the loyalty values are far apart (less than -30) compared to others." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "(train['target'] < -30).sum(), round((train['target'] < -30).sum() / train['target'].count(), 2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "So, there is 2207 rows (about 1% of the data), which has values different from the rest. Since the metric RMSE these rows might play an important role. So beware of them." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### First Active Month\n", "In this section, let see if there are any distribution change between train and test sets with respect to first active month of the card." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fig, ax = plt.subplots(figsize = (14, 6))\n", "\n", "first_month_count_train = train['first_active_month'].dt.date.value_counts().sort_index()\n", "sns.barplot(first_month_count_train.index,\n", " first_month_count_train.values,\n", " alpha=0.8, ax=ax, color='#96CAC0')\n", "\n", "first_month_count_test = test['first_active_month'].dt.date.value_counts().sort_index()\n", "sns.barplot(first_month_count_test.index,\n", " first_month_count_test.values,\n", " alpha=0.8, ax=ax, color='#F6F6BC')\n", "\n", "plt.xticks(rotation='vertical')\n", "plt.xlabel('First active month', fontsize=12)\n", "plt.ylabel('Number of cards', fontsize=12)\n", "plt.title('First active month count')\n", "\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Looks like the distribution is kind of similar between train and test set. So we need not really have to do time based split I think." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Anonymous features\n", "In this section, let see if the other variables in the train dataset has good predictive power in finding the loyalty score." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fig, ax = plt.subplots(1, 3, figsize = (16, 6))\n", "plt.suptitle('Counts of categiories for features', fontsize=24)\n", "sns.countplot(data=train, x='feature_1', ax=ax[0])\n", "sns.countplot(data=train, x='feature_2', ax=ax[1]).set(ylabel=None)\n", "sns.countplot(data=train, x='feature_3', ax=ax[2]).set(ylabel=None);" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fig, ax = plt.subplots(1, 3, figsize=(16, 6))\n", "plt.suptitle('Violineplots for features and target', fontsize=24)\n", "sns.violinplot(x='feature_1', y='target', data=train, ax=ax[0], title='feature_1', palette='Set3')\n", "sns.violinplot(x='feature_2', y='target', data=train, ax=ax[1], title='feature_2', palette='Set3')\n", "sns.violinplot(x='feature_3', y='target', data=train, ax=ax[2], title='feature_3', palette='Set3');" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To the naked eyes, the distribution of the different categories in all three features look kind of similar. May be the models are able to find something here." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now let us make some features based on the historical transactions and merge them with train and test set.\n", "\n", "#### Number of Historical Transactions for the card\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "history_purchase_amount = hist.groupby('card_id')['purchase_amount'].size().reset_index()\n", "history_purchase_amount.columns = ['card_id', 'history_purchase_amount']\n", "train = pd.merge(train, history_purchase_amount, on='card_id', how='left')\n", "test = pd.merge(test, history_purchase_amount, on='card_id', how='left')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "history_purchase_amount = train.groupby('history_purchase_amount')['target'].mean().sort_index()[:-50]\n", "fig, ax = plt.subplots(figsize=(16, 6))\n", "plt.suptitle('Loyalty score by Number of historical transactions', fontsize=24)\n", "sns.lineplot(history_purchase_amount.index[::-1],\n", " history_purchase_amount.values[::-1],\n", " ax=ax);" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now let bin the count of historical transactions and then do some box plots to see the plots better." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "bins = [0] + [2 ** p for p in range(4, 13)]\n", "train['binned_history_purchase_amount'] = pd.cut(train['history_purchase_amount'], bins)\n", "\n", "plt.figure(figsize=(16, 6))\n", "sns.boxplot(x='binned_history_purchase_amount', y='target', data=train, showfliers=False)\n", "plt.xticks(rotation='vertical')\n", "plt.xlabel('binned_num_hist_transactions', fontsize=12)\n", "plt.ylabel('Loyalty score', fontsize=12)\n", "plt.title('Distribution of binned history purchase amount', fontsize=24)\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Value of Historical Transactions\n", "Check the value of the historical transactions for the cards and check the loyalty score distribution based on that." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "gdf = hist.groupby('card_id')['purchase_amount'].agg(['sum', 'mean', 'std', 'min', 'max']).reset_index()\n", "gdf.columns = ['card_id', \n", " 'sum_history_purchase_amount', \n", " 'mean_history_purchase_amount', \n", " 'std_history_purchase_amount', \n", " 'min_history_purchase_amount', \n", " 'max_history_purchase_amount']\n", "train = pd.merge(train, gdf, on='card_id', how='left')\n", "test = pd.merge(test, gdf, on='card_id', how='left')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "bins = np.percentile(train['sum_history_purchase_amount'], range(0,101,10))\n", "train['binned_sum_history_purchase_amount'] = pd.cut(train['sum_history_purchase_amount'], bins)\n", "\n", "plt.figure(figsize=(16, 6))\n", "sns.boxplot(x='binned_sum_history_purchase_amount', y='target', data=train, showfliers=False)\n", "plt.xticks(rotation='vertical')\n", "plt.xlabel('Binned sum history purchase amount', fontsize=12)\n", "plt.ylabel('Loyalty score', fontsize=12)\n", "plt.title('Sum of historical transaction value (binned) distribution', fontsize=24)\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As we could see, the loyalty score seem to increase with the `sum of historical transaction value`. This is expected. Now we can do the same plot with `Mean value of historical transaction`." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "bins = np.percentile(train['mean_history_purchase_amount'], range(0,101,10))\n", "train['binned_mean_history_purchase_amount'] = pd.cut(train['mean_history_purchase_amount'], bins)\n", "\n", "plt.figure(figsize=(16, 6))\n", "sns.boxplot(x='binned_mean_history_purchase_amount', y='target', data=train, showfliers=False)\n", "plt.xticks(rotation='vertical')\n", "plt.xlabel('Binned Mean Historical Purchase Amount', fontsize=12)\n", "plt.ylabel('Loyalty score', fontsize=12)\n", "plt.title('Mean of historical transaction value (binned) distribution', fontsize=24)\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### New Merchant Transactions\n", "In this section, let look at the new merchant transactions data and do some analysis" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "gdf = transaction.groupby('card_id')['purchase_amount'].size().reset_index()\n", "gdf.columns = ['card_id', 'transactions_count']\n", "train = pd.merge(train, gdf, on='card_id', how='left')\n", "test = pd.merge(test, gdf, on='card_id', how='left')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "bins = [0, 10, 20, 30, 40, 50, 75, 10000]\n", "train['binned_transactions_count'] = pd.cut(train['transactions_count'], bins)\n", "\n", "plt.figure(figsize=(16, 6))\n", "sns.boxplot(x='binned_transactions_count', y='target', data=train, showfliers=False)\n", "plt.xticks(rotation='vertical')\n", "plt.xlabel('Binned transactions count', fontsize=12)\n", "plt.ylabel('Loyalty score', fontsize=12)\n", "plt.title('Number of new merchants transaction (binned) distribution', fontsize=24)\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Loyalty score seem to decrease as the number of new merchant transactions increases except for the last bin." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "gdf = transaction.groupby('card_id')['purchase_amount'].agg(['sum', 'mean', 'std', 'min', 'max']).reset_index()\n", "gdf.columns = ['card_id', \n", " 'sum_transactions_count', \n", " 'mean_transactions_count', \n", " 'std_transactions_count', \n", " 'min_transactions_count', \n", " 'max_transactions_count']\n", "train = pd.merge(train, gdf, on='card_id', how='left')\n", "test = pd.merge(test, gdf, on='card_id', how='left')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "bins = np.nanpercentile(train['sum_transactions_count'], range(0,101,10))\n", "train['binned_sum_transactions_count'] = pd.cut(train['sum_transactions_count'], bins)\n", "\n", "plt.figure(figsize=(16, 6))\n", "sns.boxplot(x='binned_sum_transactions_count', y='target', data=train, showfliers=False)\n", "plt.xticks(rotation='vertical')\n", "plt.xlabel('binned sum of new merchant transactions', fontsize=12)\n", "plt.ylabel('Loyalty score', fontsize=12)\n", "plt.title('Sum of new merchants transaction value (binned) distribution', fontsize=24)\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Loyalty scores seem to increase with the increase in the sum of new merchant transaction values but for the last bin." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "bins = np.nanpercentile(train['mean_transactions_count'], range(0,101,10))\n", "train['binned_mean_transactions_count'] = pd.cut(train['mean_transactions_count'], bins)\n", "\n", "plt.figure(figsize=(16, 6))\n", "sns.boxplot(x='binned_mean_transactions_count', y='target', data=train, showfliers=False)\n", "plt.xticks(rotation='vertical')\n", "plt.xlabel('binned mean of new merchant transactions', fontsize=12)\n", "plt.ylabel('Loyalty score', fontsize=12)\n", "plt.title('Mean of New merchants transaction value (binned) distribution', fontsize=24)\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Patterns, insights, pecularities of data" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "So, according to the results of the data analysis, the following conclusions can be drawn:\n", "* There are no gaps in the train/tets data, but detailed information provided only for the last 3 month, so we have some missed data in generated features.\n", "* There are outliers in the target variable that require additional analysis. This could be fraud blocking, or, for example, badly filled gaps.\n", "* Judging by the dependence of loyalty on the number of purchases, loyalty grows with a sufficiently large number of purchases (> 75), and before that usually falls. This is expected, since those who stopped at a small number of purchases, as a rule, are not satisfied with the service.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Data preprocessing" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "1 row in test data have missed `first_active_month`, so lets fix it." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "test.loc[test['first_active_month'].isna(), 'first_active_month'] = test.loc[\n", " (test['feature_1'] == 5) & (test['feature_2'] == 2) & (test['feature_3'] == 1),\n", " 'first_active_month'].min()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Fill in the data on `card_id` that do not have transactions over the past three months." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "cols_to_fill = [\n", " 'transactions_count', 'sum_transactions_count', \n", " 'mean_transactions_count', 'std_transactions_count',\n", " 'min_transactions_count', 'max_transactions_count', \n", "]\n", "\n", "train[cols_to_fill] = train[cols_to_fill].fillna(0)\n", "test[cols_to_fill] = test[cols_to_fill].fillna(0)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Add another several features" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here we add common date features." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "max_date = train['first_active_month'].dt.date.max()\n", "def process_main(df):\n", " date_parts = ['year', 'weekday', 'month']\n", " for part in date_parts:\n", " part_col = 'first_' + part\n", " df[part_col] = getattr(df['first_active_month'].dt, part).astype(int)\n", " \n", " df['elapsed_time'] = (max_date - df['first_active_month'].dt.date).dt.days\n", " \n", " return df" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "train = process_main(train)\n", "test = process_main(test)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Cross-validation, hyperparameter tuning" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Baseline Model\n", "Let build a baseline model using the features created so far. First of all we have to split data to train and validation sets." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "cols_to_use = [\n", " 'feature_1', 'feature_2', 'feature_3',\n", " 'first_year', 'first_month', 'first_weekday', 'elapsed_time',\n", " 'history_purchase_amount', 'sum_history_purchase_amount',\n", " 'mean_history_purchase_amount', 'std_history_purchase_amount', \n", " 'min_history_purchase_amount', 'max_history_purchase_amount',\n", " 'transactions_count', 'sum_transactions_count', \n", " 'mean_transactions_count', 'std_transactions_count',\n", " 'min_transactions_count', 'max_transactions_count',\n", "]\n", "\n", "X_train, X_holdout, y_train, y_holdout = train_test_split(train[cols_to_use],\n", " train['target'],\n", " test_size=0.2)\n", "X_test = test[cols_to_use]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now that we have prepared data, we can delete raw data." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "del train, test, hist, transaction" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "params = {\n", " 'learning_rate': 0.1,\n", " 'n_estimators': 100,\n", " 'subsample': 1.0,\n", " 'max_depth': 3,\n", " 'max_features': 'sqrt',\n", " 'n_iter_no_change': 5,\n", " 'validation_fraction': 0.2,\n", " 'tol': 0.00001,\n", " 'random_state': 11,\n", "}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Fit baseline model" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%time\n", "model = GradientBoostingRegressor(**params)\n", "model.fit(X_train[cols_to_use], y_train)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "score = mean_squared_error(y_holdout, model.predict(X_holdout))\n", "print(f'Baseline model score: {np.sqrt(score)}')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fi = list(zip(cols_to_use, model.feature_importances_))\n", "fi = pd.DataFrame(sorted(fi, key=lambda x: x[1], reverse=True), columns=['Feature', 'Importance'])" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plt.figure(figsize=(16, 6))\n", "sns.barplot(x='Importance', y='Feature', data=fi, orient='h')\n", "plt.title('Features importance', fontsize=24);" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Validation and learning curves" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Change params and tune `n_estimators` with validation curve." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "params = {\n", " 'learning_rate': 0.1,\n", " 'n_estimators': 100,\n", " 'subsample': 0.8,\n", " 'max_depth': 7,\n", " 'max_features': 'sqrt',\n", " 'n_iter_no_change': 5,\n", " 'validation_fraction': 0.2,\n", " 'tol': 0.00001,\n", " 'random_state': 11,\n", "}" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def plot_validation_curve(model, X_train, y_train,\n", " param, param_range, cv=3,\n", " scoring='neg_mean_squared_error'):\n", " train_scores, test_scores = validation_curve(\n", " model, X_train, y_train, cv=cv,\n", " param_name=param, param_range=param_range,\n", " scoring=scoring, n_jobs=-1\n", " )\n", " train_scores_mean = np.mean(train_scores, axis=1)\n", " train_scores_std = np.std(train_scores, axis=1)\n", " test_scores_mean = np.mean(test_scores, axis=1)\n", " test_scores_std = np.std(test_scores, axis=1)\n", " \n", " plt.figure(figsize=(16, 6))\n", " plt.title('Validation Curve')\n", " plt.xlabel('n_estimators')\n", " plt.ylabel('Score')\n", "\n", " plt.semilogx(param_range, train_scores_mean, label='Training score',\n", " color='darkorange', lw=2)\n", " plt.fill_between(param_range, train_scores_mean - train_scores_std,\n", " train_scores_mean + train_scores_std, alpha=0.2,\n", " color='darkorange', lw=2)\n", " plt.semilogx(param_range, test_scores_mean, label='Cross-validation score',\n", " color='navy', lw=2)\n", " plt.fill_between(param_range, test_scores_mean - test_scores_std,\n", " test_scores_mean + test_scores_std, alpha=0.2,\n", " color='navy', lw=2)\n", " plt.legend(loc='best')\n", " plt.show()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%time\n", "plot_validation_curve(GradientBoostingRegressor(**params),\n", " X_train[cols_to_use], y_train,\n", " param='n_estimators',\n", " param_range=[10 ** x for x in range(1, 6)])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This validation curve poses two possibilities: first, that we do not have the correct param_range to find the best `n_estimators` and need to expand our search to larger values. The second is that other hyperparameters (such as `learning_rate` or `max_depth`, or even `subsample`) may have more influence on the default model than `n_estimators` by itself does. Although validation curves can give us some intuition about the performance of a model to a single hyperparameter, grid search is required to understand the performance of a model with respect to multiple hyperparameters." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def plot_learning_curve(model, X_train, y_train, cv=3,\n", " train_sizes=None, scoring='neg_mean_squared_error',\n", " random_state=11):\n", " if not train_sizes:\n", " train_sizes = np.linspace(.1, 1.0, 8)\n", " \n", " train_sizes, train_scores, test_scores = learning_curve(\n", " model, X_train, y_train, cv=cv,\n", " train_sizes=train_sizes,\n", " scoring=scoring,\n", " random_state=random_state,\n", " n_jobs=-1\n", " )\n", "\n", " train_scores_mean = np.mean(train_scores, axis=1)\n", " train_scores_std = np.std(train_scores, axis=1)\n", " test_scores_mean = np.mean(test_scores, axis=1)\n", " test_scores_std = np.std(test_scores, axis=1)\n", "\n", " \n", " plt.figure(figsize=(16, 6))\n", " plt.title('Learning curve')\n", " plt.xlabel('Training examples')\n", " plt.ylabel('Score') \n", " plt.grid()\n", " plt.fill_between(train_sizes, train_scores_mean - train_scores_std,\n", " train_scores_mean + train_scores_std, alpha=0.1,\n", " color='r')\n", " plt.fill_between(train_sizes, test_scores_mean - test_scores_std,\n", " test_scores_mean + test_scores_std, alpha=0.1, color='g')\n", " plt.plot(train_sizes, train_scores_mean, 'o-', color='r',\n", " label='Training score')\n", " plt.plot(train_sizes, test_scores_mean, 'o-', color='g',\n", " label='Cross-validation score')\n", "\n", " plt.legend(loc='best')\n", " plt.show()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%time\n", "gbm = GradientBoostingRegressor(**params)\n", "plot_learning_curve(gbm, X_train[cols_to_use], y_train)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This learning curve shows high test variability and a low score. We can see that the training and test scores have not yet converged, so potentially this model would benefit from more training data. Finally, this model does not suffer from error due to variance (the CV scores for the test data are more variable than for training data) so it is possible that the model is underfitting." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Prediction for hold-out and test samples " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%time\n", "new_params = params\n", "new_params['n_iter_no_change'] = None\n", "new_params['n_estimators'] = 100\n", "model = GradientBoostingRegressor(**new_params)\n", "model.fit(X_train[cols_to_use], y_train)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "score = mean_squared_error(y_holdout, model.predict(X_holdout))\n", "print(f'Final model score: {np.sqrt(score)}')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "submission = pd.read_csv('../../data/ELO/sample_submission.csv')\n", "submission['target'] = model.predict(X_test)\n", "submission.to_csv('submit.csv', index=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Metrics description" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Predictions are scored on the root mean squared error. RMSE defined as:\n", "$$ RMSE = \\sqrt{ \\frac{1}{n} \\sum_{i=1}^n (y_i - \\hat{y}_i) ^ 2 }$$\n", "where $\\hat{y}$ is the predicted loaylty score for each `card_id`, and $y$ is the actual loyalty score assigned to a `card_id`.\n", "\n", "The RMSE is the square root of the variance of the residuals. It indicates the absolute fit of the model to the data–how close the observed data points are to the model’s predicted values. RMSE is a good measure of how accurately the model predicts the response, and it is the most important criterion for fit if the main purpose of the model is prediction." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Model evaluation\n", "\n", "The result - a fairly accurate model (the middle of the leaderboard) and having a small variance." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Conclusions" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "To sum up, we have the model far away from perfect predictions and there is plenty room for improvement here. \n", "* First, it would be much better fine tune parameters (frankly, I stoped wait the end of GridSearch after second night). \n", "* Second, creating more informative features and try other models (e.g. xgboost, LightGBM, CatBoost).\n", "\n", "So, the time has finish. Thank you for your attentions!" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.3" } }, "nbformat": 4, "nbformat_minor": 2 }