{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 📓 Ensemble Models: Decision Trees, Random Forests, & XGBoost\n",
    "\n",
    "This notebook is the hands-on companion to **Part 3** of our Ensemble Models series. We will build a complete machine learning pipeline to train, tune, and evaluate:\n",
    "1. **Decision Trees** (Baseline model)\n",
    "2. **Random Forests** (Bagging ensemble)\n",
    "3. **XGBoost** (Gradient Boosting ensemble)\n",
    "\n",
    "We will use a synthetic customer churn dataset to demonstrate data preprocessing, hyperparameter search using cross-validation, and model interpretation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---  \n",
    "## 🛠️ 1. Setup and Library Imports\n",
    "\n",
    "First, we import the standard data science and machine learning libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "# Scikit-Learn tools\n",
    "from sklearn.datasets import make_classification\n",
    "from sklearn.model_selection import train_test_split, RandomizedSearchCV, KFold\n",
    "from sklearn.preprocessing import OrdinalEncoder\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.metrics import (\n",
    "    classification_report, \n",
    "    accuracy_score, \n",
    "    roc_auc_score, \n",
    "    roc_curve, \n",
    "    confusion_matrix\n",
    ")\n",
    "from sklearn.tree import DecisionTreeClassifier, plot_tree\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "# XGBoost\n",
    "from xgboost import XGBClassifier\n",
    "\n",
    "# Formatting\n",
    "sns.set_theme(style=\"whitegrid\")\n",
    "plt.rcParams[\"figure.figsize\"] = (10, 6)\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---  \n",
    "## 📊 2. Synthetic Dataset Generation\n",
    "\n",
    "To simulate a real-world customer churn scenario, we will generate a dataset containing both continuous and categorical features. We will also inject some missing values (`NaN`) to show how different models handle them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1. Create a classification dataset with 20 features\n",
    "X_raw, y_raw = make_classification(\n",
    "    n_samples=2000, \n",
    "    n_features=12, \n",
    "    n_informative=8, \n",
    "    n_redundant=4, \n",
    "    random_state=42\n",
    ")\n",
    "\n",
    "# 2. Convert to a DataFrame and add descriptive column names\n",
    "feature_names = [\n",
    "    \"age\", \"tenure\", \"monthly_charges\", \"total_charges\",\n",
    "    \"support_tickets\", \"login_frequency\", \"usage_metric_a\", \"usage_metric_b\",\n",
    "    \"usage_metric_c\", \"usage_metric_d\", \"usage_metric_e\", \"usage_metric_f\"\n",
    "]\n",
    "df = pd.DataFrame(X_raw, columns=feature_names)\n",
    "df[\"churn\"] = y_raw\n",
    "\n",
    "# 3. Convert some numeric columns to categories\n",
    "# E.g., splitting a feature into 'contract_type' and 'payment_method'\n",
    "df[\"contract_type\"] = pd.qcut(df[\"usage_metric_e\"], q=3, labels=[\"Month-to-month\", \"One year\", \"Two year\"])\n",
    "df[\"payment_method\"] = pd.qcut(df[\"usage_metric_f\"], q=3, labels=[\"Electronic check\", \"Credit card\", \"Bank transfer\"])\n",
    "\n",
    "# Drop the original columns used for binning\n",
    "df = df.drop(columns=[\"usage_metric_e\", \"usage_metric_f\"])\n",
    "\n",
    "# 4. Inject missing values (NaN) into 'monthly_charges' and 'tenure'\n",
    "# E.g., simulating incomplete records\n",
    "mask_charges = np.random.rand(*df[\"monthly_charges\"].shape) < 0.08\n",
    "df.loc[mask_charges, \"monthly_charges\"] = np.nan\n",
    "\n",
    "mask_tenure = np.random.rand(*df[\"tenure\"].shape) < 0.05\n",
    "df.loc[mask_tenure, \"tenure\"] = np.nan\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---  \n",
    "## 🧪 3. Data Splitting & Pipeline Preprocessing\n",
    "\n",
    "We separate our target column `churn` and split our dataset into **80% Training** and **20% Testing**.\n",
    "\n",
    "Since tree models are scale-invariant, we **do not need to scale our features**. We only need to:\n",
    "1. **Impute missing values** (for Scikit-Learn trees).\n",
    "2. **Encode categorical values** into numbers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split target and features\n",
    "X = df.drop(columns=[\"churn\"])\n",
    "y = df[\"churn\"]\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.2, stratify=y, random_state=42\n",
    ")\n",
    "\n",
    "print(f\"Training shape: {X_train.shape}\")\n",
    "print(f\"Testing shape: {X_test.shape}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating the Preprocessing Pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Identify numeric and categorical columns\n",
    "numeric_features = X.select_dtypes(include=[np.number]).columns.tolist()\n",
    "categorical_features = X.select_dtypes(exclude=[np.number]).columns.tolist()\n",
    "\n",
    "# 1. Numeric pipeline: Impute missing values with the median\n",
    "numeric_transformer = Pipeline(steps=[\n",
    "    ('imputer', SimpleImputer(strategy='median'))\n",
    "])\n",
    "\n",
    "# 2. Categorical pipeline: Ordinal Encode categorical columns (assigning integer labels)\n",
    "categorical_transformer = Pipeline(steps=[\n",
    "    ('encoder', OrdinalEncoder(handle_unknown='use_encoded_value', unknown_value=-1))\n",
    "])\n",
    "\n",
    "# Combine into a ColumnTransformer\n",
    "preprocessor = ColumnTransformer(transformers=[\n",
    "    ('num', numeric_transformer, numeric_features),\n",
    "    ('cat', categorical_transformer, categorical_features)\n",
    "])\n",
    "\n",
    "# Preprocess the datasets\n",
    "X_train_preprocessed = preprocessor.fit_transform(X_train)\n",
    "X_test_preprocessed = preprocessor.transform(X_test)\n",
    "\n",
    "# Get updated feature order for interpreting feature importances later\n",
    "all_features = numeric_features + categorical_features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---  \n",
    "## 🌳 4. Baseline: Decision Tree\n",
    "\n",
    "We train a simple decision tree with a `max_depth` limit of 3 to visualize how the splitting rules are constructed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dt_model = DecisionTreeClassifier(max_depth=3, random_state=42)\n",
    "dt_model.fit(X_train_preprocessed, y_train)\n",
    "\n",
    "# Plot the tree structure\n",
    "plt.figure(figsize=(18, 10))\n",
    "plot_tree(\n",
    "    dt_model, \n",
    "    feature_names=all_features, \n",
    "    class_names=[\"Stay\", \"Churn\"], \n",
    "    filled=True, \n",
    "    rounded=True, \n",
    "    fontsize=12\n",
    ")\n",
    "plt.title(\"Visualizing Decision Tree Splits (Depth=3)\", fontsize=16)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---  \n",
    "## 🌲 5. Parallel Ensembles: Random Forest & Hyperparameter Tuning\n",
    "\n",
    "We will tune a **Random Forest Classifier** using **RandomizedSearchCV** with a 5-Fold cross-validation strategy. We will also access the built-in **Out-of-Bag (OOB) Score**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the parameter grid to sample from\n",
    "rf_param_grid = {\n",
    "    'n_estimators': [50, 100, 200, 300],\n",
    "    'max_depth': [5, 10, 15, 20, None],\n",
    "    'max_features': ['sqrt', 'log2', 0.5, 0.8],\n",
    "    'min_samples_split': [2, 5, 10],\n",
    "    'min_samples_leaf': [1, 2, 4]\n",
    "}\n",
    "\n",
    "# Initialize base model with bootstrap enabled to compute OOB score\n",
    "rf_base = RandomForestClassifier(bootstrap=True, oob_score=True, random_state=42)\n",
    "\n",
    "# Setup Randomized Search\n",
    "rf_search = RandomizedSearchCV(\n",
    "    estimator=rf_base, \n",
    "    param_distributions=rf_param_grid,\n",
    "    n_iter=20, \n",
    "    cv=5,\n",
    "    scoring='roc_auc',\n",
    "    random_state=42,\n",
    "    n_jobs=-1\n",
    ")\n",
    "\n",
    "rf_search.fit(X_train_preprocessed, y_train)\n",
    "\n",
    "print(f\"Best Random Forest Parameters: {rf_search.best_params_}\")\n",
    "print(f\"Best Cross-Validation ROC-AUC: {rf_search.best_score_:.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### OOB (Out-of-Bag) Score Verification\n",
    "Let's extract the best estimator and check its OOB score. Since OOB acts as a built-in validation set, it should be very close to the cross-validation score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_rf = rf_search.best_estimator_\n",
    "print(f\"Random Forest OOB Accuracy Score: {best_rf.oob_score_:.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---  \n",
    "## ⚡ 6. Sequential Ensembles: XGBoost & Native Missing Data\n",
    "\n",
    "XGBoost handles missing values (`NaN`) natively. To demonstrate this, we will train XGBoost using **raw data** (where categorical strings are Ordinal Encoded, but numeric missing values are **not imputed**)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Preprocess data WITHOUT numeric imputation\n",
    "xgb_preprocessor = ColumnTransformer(transformers=[\n",
    "    ('num', 'passthrough', numeric_features), # Leave NaNs intact\n",
    "    ('cat', categorical_transformer, categorical_features)\n",
    "])\n",
    "\n",
    "X_train_xgb = xgb_preprocessor.fit_transform(X_train)\n",
    "X_test_xgb = xgb_preprocessor.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tuning XGBoost\n",
    "We define the tuning space for XGBoost. We tune `learning_rate`, `max_depth`, row/column subsampling, and regularization weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgb_param_grid = {\n",
    "    'learning_rate': [0.01, 0.05, 0.1, 0.2],\n",
    "    'max_depth': [3, 5, 7, 9],\n",
    "    'subsample': [0.6, 0.8, 1.0],\n",
    "    'colsample_bytree': [0.6, 0.8, 1.0],\n",
    "    'reg_lambda': [0.1, 1.0, 10.0],\n",
    "    'reg_alpha': [0.0, 0.1, 1.0],\n",
    "    'n_estimators': [100, 200, 300]\n",
    "}\n",
    "\n",
    "xgb_base = XGBClassifier(\n",
    "    use_label_encoder=False, \n",
    "    eval_metric='logloss', \n",
    "    random_state=42\n",
    ")\n",
    "\n",
    "xgb_search = RandomizedSearchCV(\n",
    "    estimator=xgb_base, \n",
    "    param_distributions=xgb_param_grid,\n",
    "    n_iter=20, \n",
    "    cv=5,\n",
    "    scoring='roc_auc',\n",
    "    random_state=42,\n",
    "    n_jobs=-1\n",
    ")\n",
    "\n",
    "xgb_search.fit(X_train_xgb, y_train)\n",
    "\n",
    "print(f\"Best XGBoost Parameters: {xgb_search.best_params_}\")\n",
    "print(f\"Best Cross-Validation ROC-AUC: {xgb_search.best_score_:.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---  \n",
    "## 📊 7. Model Comparison on Test Data\n",
    "\n",
    "Let's compare our models on the held-out test dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get predictions\n",
    "dt_preds = dt_model.predict(X_test_preprocessed)\n",
    "rf_preds = best_rf.predict(X_test_preprocessed)\n",
    "xgb_preds = xgb_search.best_estimator_.predict(X_test_xgb)\n",
    "\n",
    "print(\"=== Decision Tree Classification Report ===\")\n",
    "print(classification_report(y_test, dt_preds))\n",
    "\n",
    "print(\"=== Random Forest Classification Report ===\")\n",
    "print(classification_report(y_test, rf_preds))\n",
    "\n",
    "print(\"=== XGBoost Classification Report ===\")\n",
    "print(classification_report(y_test, xgb_preds))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plotting ROC Curves"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get probability scores\n",
    "dt_probs = dt_model.predict_proba(X_test_preprocessed)[:, 1]\n",
    "rf_probs = best_rf.predict_proba(X_test_preprocessed)[:, 1]\n",
    "xgb_probs = xgb_search.best_estimator_.predict_proba(X_test_xgb)[:, 1]\n",
    "\n",
    "# Compute ROC statistics\n",
    "dt_fpr, dt_tpr, _ = roc_curve(y_test, dt_probs)\n",
    "rf_fpr, rf_tpr, _ = roc_curve(y_test, rf_probs)\n",
    "xgb_fpr, xgb_tpr, _ = roc_curve(y_test, xgb_probs)\n",
    "\n",
    "# Plot\n",
    "plt.figure(figsize=(10, 7))\n",
    "plt.plot(dt_fpr, dt_tpr, label=f\"Decision Tree (AUC = {roc_auc_score(y_test, dt_probs):.4f})\")\n",
    "plt.plot(rf_fpr, rf_tpr, label=f\"Random Forest (AUC = {roc_auc_score(y_test, rf_probs):.4f})\")\n",
    "plt.plot(xgb_fpr, xgb_tpr, label=f\"XGBoost (AUC = {roc_auc_score(y_test, xgb_probs):.4f})\")\n",
    "plt.plot([0, 1], [0, 1], 'k--', label=\"Random Guess\")\n",
    "plt.xlabel(\"False Positive Rate\")\n",
    "plt.ylabel(\"True Positive Rate\")\n",
    "plt.title(\"ROC Curves Comparison\", fontsize=16)\n",
    "plt.legend(loc=\"lower right\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---  \n",
    "## 🔍 8. Model Interpretation: Feature Importances\n",
    "\n",
    "Let's check and compare the feature importance calculations from both Random Forest and XGBoost."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract importances\n",
    "rf_importances = best_rf.feature_importances_\n",
    "xgb_importances = xgb_search.best_estimator_.feature_importances_\n",
    "\n",
    "# Combine into a DataFrame\n",
    "importance_df = pd.DataFrame({\n",
    "    'Feature': all_features,\n",
    "    'Random Forest': rf_importances,\n",
    "    'XGBoost': xgb_importances\n",
    "}).melt(id_vars='Feature', var_name='Model', value_name='Importance')\n",
    "\n",
    "# Plot comparison\n",
    "plt.figure(figsize=(12, 8))\n",
    "sns.barplot(data=importance_df, x='Importance', y='Feature', hue='Model')\n",
    "plt.title(\"Feature Importance Comparison: RF vs. XGBoost\", fontsize=16)\n",
    "plt.xlabel(\"Relative Importance\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---  \n",
    "## 🏁 9. Key Takeaways\n",
    "\n",
    "1. **Decision Tree (Baseline)**: Simple and highly interpretable, but lacks performance on complex datasets.\n",
    "2. **Random Forest (Bagging)**: Combats variance by training independent deep trees. Very robust and easy to tune, but struggles slightly with native missing values.\n",
    "3. **XGBoost (Boosting)**: Sequential model optimizing residuals. Achieved the highest performance (AUC), handles missing values natively, and runs extremely fast due to parallel split optimizations. However, it requires careful hyperparameter tuning to prevent overfitting."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "name": "python",
   "nbformat_minor": 2,
   "packaging_version": "23.0",
   "pygments_lexer": "ipython3",
   "version": "3.10.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}