{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DataSphere ML \u2014 polyglot predictors (Colab demo)\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Harshini-Suresha/datasphere-ml/blob/main/notebooks/datasphere_ml_demo.ipynb)\n",
    "![Website](https://img.shields.io/badge/website-live-brightgreen)](https://harshini-suresha.github.io/datasphere-ml/) ![GitHub](https://img.shields.io/badge/github-repo-blue)](https://github.com/Harshini-Suresha/datasphere-ml)\n",
    "\n",
    "**Live links:** [Website (github.io)](https://harshini-suresha.github.io/datasphere-ml/) \u00b7 [GitHub repo](https://github.com/Harshini-Suresha/datasphere-ml) \u00b7 [Streamlit app](https://datasphere-ml-i9nv6eofsxkupnbfsdnqrx.streamlit.app/) \u00b7 [Download this notebook (.ipynb)](https://raw.githubusercontent.com/Harshini-Suresha/datasphere-ml/main/notebooks/datasphere_ml_demo.ipynb)\n",
    "\n",
    "One synthetic e-commerce dataset (seed `42`) spanning MySQL-style orders, Mongo-style catalogue/reviews and Neo4j-style graph centrality. Trains 3 sklearn RandomForest models:\n",
    "1. **High-value customer** classifier (top-20% spenders)\n",
    "2. **Order amount** regressor\n",
    "3. **Review rating** classifier (1\u20135 stars)\n",
    "\n",
    "> Runs in Colab in ~1\u20132 min with **no extra installs** (sklearn/pandas/matplotlib are pre-installed). For the full RF+XGBoost+LightGBM+SHAP run, clone the repo and run `python train_models.py --fast`.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0 \u00b7 Setup \u2014 run this first in Colab\n",
    "\n",
    "No installs needed. Just check versions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sklearn, pandas, numpy, matplotlib\n",
    "print('sklearn', sklearn.__version__)\n",
    "print('pandas', pandas.__version__)\n",
    "print('numpy', numpy.__version__)\n",
    "# Uncomment if you want the full repo (XGBoost/LightGBM/SHAP + Streamlit app):\n",
    "# !pip install -q xgboost lightgbm shap streamlit plotly\n",
    "# !git clone https://github.com/Harshini-Suresha/datasphere-ml.git\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1 \u00b7 Generate one synthetic dataset (seed 42)\n",
    "\n",
    "Same generator logic as `train_models.py`, smaller sizes for Colab speed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "SEED = 42\n",
    "CITIES = ['London', 'Bengaluru', 'Pune', 'Berlin', 'Austin', 'Lagos']\n",
    "CITY_P = [0.28, 0.22, 0.15, 0.14, 0.12, 0.09]\n",
    "CITY_PREM = {'London': 14, 'Austin': 10, 'Berlin': 8, 'Bengaluru': -2, 'Pune': -4, 'Lagos': -6}\n",
    "CATS = ['Shoes', 'Electronics', 'Home', 'Books', 'Beauty', 'Sports']\n",
    "\n",
    "def gen_customers(n=3000, seed=SEED):\n",
    "    rng = np.random.default_rng(seed)\n",
    "    degree = np.ones(n)\n",
    "    for i in range(1, n):\n",
    "        w = degree[:i] + 1.0\n",
    "        p = w / w.sum()\n",
    "        for _ in range(2 if i > 3 else 1):\n",
    "            j = rng.choice(i, p=p)\n",
    "            degree[i] += 1; degree[j] += 1\n",
    "    pagerank = degree / degree.sum() * n\n",
    "    city = rng.choice(CITIES, size=n, p=CITY_P)\n",
    "    prem = np.array([CITY_PREM[c] for c in city])\n",
    "    tenure = np.clip(rng.gamma(5.0, 90.0, n).astype(int), 30, 900)\n",
    "    propensity = rng.normal(0, 1, n)\n",
    "    activity = rng.gamma(2.0, 1.0, n) * (0.5 + pagerank) + 0.4 * propensity\n",
    "    n_orders = np.clip((rng.power(1.6, n) * 34 * (0.4 + activity / np.mean(activity))\n",
    "                        + (propensity > 1).astype(int) * 4).astype(int), 1, 120)\n",
    "    avg_amount = np.clip(np.round(22 + prem + rng.gamma(2.2, 16, n)\n",
    "                                    + (pagerank - 1) * 5 + propensity * 7, 2), 5, 400)\n",
    "    total = np.round(n_orders * avg_amount * rng.lognormal(0, 0.12, n), 2)\n",
    "    recency = np.clip((rng.exponential(45, n) / (0.5 + activity / 3)).astype(int), 0, 400)\n",
    "    n_cat = np.clip(rng.integers(1, 6, n) + (activity > np.mean(activity)).astype(int), 1, 6)\n",
    "    n_rev = (rng.poisson(2.5, n) + (activity > 2).astype(int) * 2).clip(0, 30)\n",
    "    fav_idx = (rng.integers(0, 6, n) + (city == 'London') * 1 + (city == 'Bengaluru') * 3) % 6\n",
    "    leniency = np.round(rng.normal(0, 0.6, n), 3)\n",
    "    df = pd.DataFrame({'customer_id': np.arange(1, n+1), 'city': city, 'tenure_days': tenure,\n",
    "                       'degree': degree.astype(int), 'pagerank': np.round(pagerank, 4),\n",
    "                       'n_orders': n_orders, 'avg_amount': avg_amount, 'recency_days': recency,\n",
    "                       'n_categories': n_cat, 'n_reviews': n_rev,\n",
    "                       'fav_category': [CATS[i] for i in fav_idx], 'leniency': leniency,\n",
    "                       'total_spent': total})\n",
    "    score = np.log(total) + rng.normal(0, 0.30, n)\n",
    "    df['high_value'] = (score >= np.quantile(score, 0.8)).astype(int)\n",
    "    return df\n",
    "\n",
    "def gen_orders(cust, n_orders=12000, seed=SEED+1):\n",
    "    rng = np.random.default_rng(seed)\n",
    "    cid = rng.choice(cust['customer_id'].values, size=n_orders,\n",
    "                     p=(cust['n_orders'] / cust['n_orders'].sum()).values)\n",
    "    c = cust.set_index('customer_id').loc[cid].reset_index(drop=True)\n",
    "    month = rng.integers(1, 13, n_orders); dow = rng.integers(0, 7, n_orders)\n",
    "    cat = [CATS[i] for i in rng.integers(0, 6, n_orders)]\n",
    "    cat_eff = {'Shoes': 8, 'Electronics': 55, 'Home': 12, 'Books': -8, 'Beauty': 2, 'Sports': 15}\n",
    "    weekend = (dow >= 5).astype(int)\n",
    "    wknd_eff = np.array([8 if x in ('Shoes','Beauty','Sports') else (-5 if x == 'Electronics' else 0) for x in cat]) * weekend\n",
    "    seas_eff = (month == 12) * 25 + (month == 11) * 10 - (month == 1) * 8\n",
    "    base = (c['avg_amount'].values * 0.50 + np.minimum(c['tenure_days'].values, 600) * 0.015\n",
    "            + np.array([cat_eff[x] for x in cat]) + wknd_eff + seas_eff)\n",
    "    amount = base * rng.lognormal(0, 0.28, n_orders)\n",
    "    return pd.DataFrame({'tenure_days': c['tenure_days'].values, 'past_avg': c['avg_amount'].values,\n",
    "                         'degree': c['degree'].values, 'pagerank': c['pagerank'].values,\n",
    "                         'recency_days': c['recency_days'].values, 'month': month,\n",
    "                         'day_of_week': dow, 'category': cat,\n",
    "                         'amount': np.round(np.clip(amount, 5, 900), 2)})\n",
    "\n",
    "def gen_reviews(cust, n_rev=8000, seed=SEED+2):\n",
    "    rng = np.random.default_rng(seed)\n",
    "    w = (cust['n_reviews'] + 0.5).values\n",
    "    cid = rng.choice(cust['customer_id'].values, size=n_rev, p=w / w.sum())\n",
    "    c = cust.set_index('customer_id').loc[cid].reset_index(drop=True)\n",
    "    price = np.round(10 + rng.power(2, n_rev) * 290, 2)\n",
    "    cat = [CATS[i] for i in rng.integers(0, 6, n_rev)]\n",
    "    reviewer_avg = np.clip(3.8 + c['leniency'].values * 0.7 + rng.normal(0, 0.15, n_rev), 1, 5)\n",
    "    quality = (np.array([0.3 if x == 'Books' else (-0.2 if x == 'Electronics' else 0.0) for x in cat])\n",
    "               - (price > 200) * 0.5 + rng.normal(0, 0.5, n_rev))\n",
    "    latent = 4.0 + quality * 0.7 + (reviewer_avg - 3.8) * 0.9\n",
    "    stars = np.clip(np.round(latent + rng.normal(0, 0.5, n_rev)), 1, 5).astype(int)\n",
    "    return pd.DataFrame({'price': price, 'category': cat,\n",
    "                         'reviewer_n_reviews': c['n_reviews'].values,\n",
    "                         'reviewer_avg_stars': np.round(reviewer_avg, 2),\n",
    "                         'reviewer_pagerank': c['pagerank'].values,\n",
    "                         'reviewer_recency': c['recency_days'].values, 'stars': stars})\n",
    "\n",
    "cust = gen_customers(); ords = gen_orders(cust); revs = gen_reviews(cust)\n",
    "print(cust.shape, ords.shape, revs.shape)\n",
    "print('high-value rate:', round(cust.high_value.mean(), 3))\n",
    "cust.head(3)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2 \u00b7 Train: high-value customer classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor\n",
    "from sklearn.metrics import accuracy_score, f1_score, mean_absolute_error, r2_score, roc_auc_score\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "\n",
    "def enc(df, cat_cols):\n",
    "    return ColumnTransformer([('cat', OneHotEncoder(handle_unknown='ignore'), cat_cols)], remainder='passthrough')\n",
    "\n",
    "feats_c = ['city','tenure_days','degree','pagerank','n_orders','avg_amount','recency_days','n_categories','n_reviews','fav_category']\n",
    "Xc_tr, Xc_te, yc_tr, yc_te = train_test_split(cust[feats_c], cust['high_value'], test_size=0.2,\n",
    "                                        stratify=cust['high_value'], random_state=42)\n",
    "pipe_c = Pipeline([('enc', enc(Xc_tr, ['city','fav_category'])),\n",
    "                   ('m', RandomForestClassifier(n_estimators=100, n_jobs=-1, random_state=42))])\n",
    "pipe_c.fit(Xc_tr, yc_tr)\n",
    "proba = pipe_c.predict_proba(Xc_te)[:, 1]; pred = (proba > 0.5).astype(int)\n",
    "print(f\"ROC-AUC {roc_auc_score(yc_te, proba):.4f} | F1 {f1_score(yc_te, pred):.4f} | acc {accuracy_score(yc_te, pred):.4f}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3 \u00b7 Train: order-amount regressor + rating classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "feats_o = ['tenure_days','past_avg','degree','pagerank','recency_days','month','day_of_week','category']\n",
    "Xo_tr, Xo_te, yo_tr, yo_te = train_test_split(ords[feats_o], ords['amount'], test_size=0.2, random_state=42)\n",
    "pipe_o = Pipeline([('enc', enc(Xo_tr, ['category'])),\n",
    "                   ('m', RandomForestRegressor(n_estimators=100, n_jobs=-1, random_state=42))])\n",
    "pipe_o.fit(Xo_tr, yo_tr)\n",
    "pr = pipe_o.predict(Xo_te)\n",
    "print(f\"R2 {r2_score(yo_te, pr):.4f} | MAE {mean_absolute_error(yo_te, pr):.2f} | RMSE {float(((yo_te-pr)**2).mean()**0.5):.2f}\")\n",
    "\n",
    "feats_r = ['price','category','reviewer_n_reviews','reviewer_avg_stars','reviewer_pagerank','reviewer_recency']\n",
    "Xr_tr, Xr_te, yr_tr, yr_te = train_test_split(revs[feats_r], revs['stars']-1, test_size=0.2,\n",
    "                                        stratify=revs['stars'], random_state=42)\n",
    "pipe_r = Pipeline([('enc', enc(Xr_tr, ['category'])),\n",
    "                   ('m', RandomForestClassifier(n_estimators=100, n_jobs=-1, random_state=42))])\n",
    "pipe_r.fit(Xr_tr, yr_tr)\n",
    "pr = pipe_r.predict(Xr_te)\n",
    "print(f\"rating acc {accuracy_score(yr_te, pr):.4f} | F1-macro {f1_score(yr_te, pr, average='macro'):.4f}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4 \u00b7 Plots: calibration, feature importance, distributions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "fig, ax = plt.subplots(1, 3, figsize=(15, 4))\n",
    "# (a) calibration: mean predicted vs observed per decile\n",
    "proba_c = pipe_c.predict_proba(Xc_te)[:, 1]\n",
    "edges = np.linspace(0, 1, 11); xs, ys = [], []\n",
    "for lo, hi in zip(edges[:-1], edges[1:]):\n",
    "    m = (proba_c > lo) & (proba_c <= hi) if lo > 0 else (proba_c >= lo) & (proba_c <= hi)\n",
    "    if m.sum() > 0:\n",
    "        xs.append(proba_c[m].mean()); ys.append(np.asarray(yc_te)[m].mean())\n",
    "ax[0].plot([0,1],[0,1],'--',color='grey'); ax[0].scatter(xs, ys); ax[0].set_title('Calibration (customer)')\n",
    "ax[0].set_xlabel('mean predicted'); ax[0].set_ylabel('observed rate')\n",
    "# (b) top importances, amount model\n",
    "names = list(pipe_o.named_steps['enc'].get_feature_names_out())\n",
    "imp = pipe_o.named_steps['m'].feature_importances_\n",
    "o = np.argsort(imp)[::-1][:8]\n",
    "ax[1].barh([names[i] for i in o][::-1], imp[o][::-1]); ax[1].set_title('Top features (amount)')\n",
    "# (c) order amount distribution\n",
    "ax[2].hist(ords['amount'], bins=40); ax[2].set_title('Order amount (\u00a3)'); ax[2].set_xlabel('\u00a3')\n",
    "plt.tight_layout(); plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5 \u00b7 Try one prediction (edit me!)\n",
    "\n",
    "Same features the Streamlit app and the `index.html` calculators use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "demo_customer = pd.DataFrame([{'city':'London','tenure_days':400,'degree':12,'pagerank':2.1,'n_orders':25,\n",
    "                                     'avg_amount':85.0,'recency_days':9,'n_categories':4,\n",
    "                                     'n_reviews':6,'fav_category':'Shoes'}])\n",
    "print('P(high-value) =', round(float(pipe_c.predict_proba(demo_customer)[0,1]), 3))\n",
    "\n",
    "demo_order = pd.DataFrame([{'tenure_days':400,'past_avg':85.0,'degree':12,'pagerank':2.1,\n",
    "                               'recency_days':9,'month':12,'day_of_week':5,'category':'Electronics'}])\n",
    "print('Predicted order \u00a3 =', round(float(pipe_o.predict(demo_order)[0]), 2))\n",
    "\n",
    "demo_review = pd.DataFrame([{'price':89.0,'category':'Shoes','reviewer_n_reviews':4,\n",
    "                                'reviewer_avg_stars':4.2,'reviewer_pagerank':1.5,'reviewer_recency':30}])\n",
    "print('Predicted stars =', int(pipe_r.predict(demo_review)[0]) + 1)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next steps\n",
    "\n",
    "- **Website:** https://harshini-suresha.github.io/datasphere-ml/ (interactive SQL lab, graph views, in-page ML calculators)\n",
    "- **Full training** (RF + XGBoost + LightGBM + SHAP + calibration JSONs):\n",
    "  ```bash\n",
    "  git clone https://github.com/Harshini-Suresha/datasphere-ml.git\n",
    "  cd datasphere-ml\n",
    "  pip install -r requirements.txt\n",
    "  python train_models.py --fast   # or full: python train_models.py\n",
    "  python export_js.py             # refresh in-page calculators (ml_models.js)\n",
    "  streamlit run streamlit_app/ml_app.py\n",
    "  ```\n",
    "- **Streamlit app:** https://datasphere-ml-i9nv6eofsxkupnbfsdnqrx.streamlit.app/\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.10.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}