{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now let's go on to our modeling step. As a reminder, our plan of action was as follows:\n",
"\n",
"1. Perform EDA on the dataset to extract valuable insight about the process generating the time series **(COMPLETED)**.\n",
"2. Build a baseline model (univariable model without exogenous variables) for benchmarking purposes. **(Covered in this notebook)**\n",
"3. Build a univariate model with all exogenous variables to check best possible performance. **(Covered in this notebook)**\n",
"4. Evaluate the model with exogenous variables and discuss any potential issues. **(Covered in this notebook)**\n",
"5. Overcome issues identified above. **(Covered in this notebook)**\n",
"6. Make future predictions with the best model.\n",
"7. Replicate flow with Automated Time Series Modeling (AutoML)"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"# Only enable critical logging (Optional)\n",
"import os\n",
"os.environ[\"PYCARET_CUSTOM_LOGGING_LEVEL\"] = \"CRITICAL\""
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"System:\n",
" python: 3.8.13 (default, Mar 28 2022, 06:59:08) [MSC v.1916 64 bit (AMD64)]\n",
"executable: C:\\Users\\Nikhil\\.conda\\envs\\pycaret_dev_sktime_0p11_2\\python.exe\n",
" machine: Windows-10-10.0.19044-SP0\n",
"\n",
"PyCaret required dependencies:\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"C:\\Users\\Nikhil\\.conda\\envs\\pycaret_dev_sktime_0p11_2\\lib\\site-packages\\_distutils_hack\\__init__.py:30: UserWarning: Setuptools is replacing distutils.\n",
" warnings.warn(\"Setuptools is replacing distutils.\")\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
" pip: 21.2.2\n",
" setuptools: 61.2.0\n",
" pycaret: 3.0.0\n",
" ipython: Not installed\n",
" ipywidgets: 7.7.0\n",
" numpy: 1.21.6\n",
" pandas: 1.4.2\n",
" jinja2: 3.1.2\n",
" scipy: 1.8.0\n",
" joblib: 1.1.0\n",
" sklearn: 1.0.2\n",
" pyod: Installed but version unavailable\n",
" imblearn: 0.9.0\n",
" category_encoders: 2.4.1\n",
" lightgbm: 3.3.2\n",
" numba: 0.55.1\n",
" requests: 2.27.1\n",
" matplotlib: 3.5.2\n",
" scikitplot: 0.3.7\n",
" yellowbrick: 1.4\n",
" plotly: 5.8.0\n",
" kaleido: 0.2.1\n",
" statsmodels: 0.13.2\n",
" sktime: 0.11.4\n",
" tbats: Installed but version unavailable\n",
" pmdarima: 1.8.5\n",
"\n",
"PyCaret optional dependencies:\n",
" shap: Not installed\n",
" interpret: Not installed\n",
" umap: Not installed\n",
" pandas_profiling: Not installed\n",
" explainerdashboard: Not installed\n",
" autoviz: Not installed\n",
" fairlearn: Not installed\n",
" xgboost: Not installed\n",
" catboost: Not installed\n",
" kmodes: Not installed\n",
" mlxtend: Not installed\n",
" statsforecast: 0.5.5\n",
" tune_sklearn: Not installed\n",
" ray: Not installed\n",
" hyperopt: Not installed\n",
" optuna: Not installed\n",
" skopt: Not installed\n",
" mlflow: 1.25.1\n",
" gradio: Not installed\n",
" fastapi: Not installed\n",
" uvicorn: Not installed\n",
" m2cgen: Not installed\n",
" evidently: Not installed\n",
" nltk: Not installed\n",
" pyLDAvis: Not installed\n",
" gensim: Not installed\n",
" spacy: Not installed\n",
" wordcloud: Not installed\n",
" textblob: Not installed\n",
" psutil: 5.9.0\n",
" fugue: Not installed\n",
" streamlit: Not installed\n",
" prophet: Not installed\n"
]
}
],
"source": [
"def what_is_installed():\n",
" from pycaret import show_versions\n",
" show_versions()\n",
"\n",
"try:\n",
" what_is_installed()\n",
"except ModuleNotFoundError:\n",
" !pip install pycaret\n",
" what_is_installed()"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import pandas as pd\n",
"from pycaret.datasets import get_data\n",
"from pycaret.time_series import TSForecastingExperiment"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"# Global Figure Settings for notebook ----\n",
"global_fig_settings = {\"renderer\": \"notebook\", \"width\": 1000, \"height\": 600}"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" Date | \n",
" Time | \n",
" CO(GT) | \n",
" PT08.S1(CO) | \n",
" NMHC(GT) | \n",
" C6H6(GT) | \n",
" PT08.S2(NMHC) | \n",
" NOx(GT) | \n",
" PT08.S3(NOx) | \n",
" NO2(GT) | \n",
" PT08.S4(NO2) | \n",
" PT08.S5(O3) | \n",
" T | \n",
" RH | \n",
" AH | \n",
"
\n",
" \n",
" \n",
" \n",
" 0 | \n",
" 2004-03-10 | \n",
" 18:00:00 | \n",
" 2.6 | \n",
" 1360 | \n",
" 150 | \n",
" 11.9 | \n",
" 1046 | \n",
" 166 | \n",
" 1056 | \n",
" 113 | \n",
" 1692 | \n",
" 1268 | \n",
" 13.6 | \n",
" 48.9 | \n",
" 0.7578 | \n",
"
\n",
" \n",
" 1 | \n",
" 2004-03-10 | \n",
" 19:00:00 | \n",
" 2.0 | \n",
" 1292 | \n",
" 112 | \n",
" 9.4 | \n",
" 955 | \n",
" 103 | \n",
" 1174 | \n",
" 92 | \n",
" 1559 | \n",
" 972 | \n",
" 13.3 | \n",
" 47.7 | \n",
" 0.7255 | \n",
"
\n",
" \n",
" 2 | \n",
" 2004-03-10 | \n",
" 20:00:00 | \n",
" 2.2 | \n",
" 1402 | \n",
" 88 | \n",
" 9.0 | \n",
" 939 | \n",
" 131 | \n",
" 1140 | \n",
" 114 | \n",
" 1555 | \n",
" 1074 | \n",
" 11.9 | \n",
" 54.0 | \n",
" 0.7502 | \n",
"
\n",
" \n",
" 3 | \n",
" 2004-03-10 | \n",
" 21:00:00 | \n",
" 2.2 | \n",
" 1376 | \n",
" 80 | \n",
" 9.2 | \n",
" 948 | \n",
" 172 | \n",
" 1092 | \n",
" 122 | \n",
" 1584 | \n",
" 1203 | \n",
" 11.0 | \n",
" 60.0 | \n",
" 0.7867 | \n",
"
\n",
" \n",
" 4 | \n",
" 2004-03-10 | \n",
" 22:00:00 | \n",
" 1.6 | \n",
" 1272 | \n",
" 51 | \n",
" 6.5 | \n",
" 836 | \n",
" 131 | \n",
" 1205 | \n",
" 116 | \n",
" 1490 | \n",
" 1110 | \n",
" 11.2 | \n",
" 59.6 | \n",
" 0.7888 | \n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" Date Time CO(GT) PT08.S1(CO) NMHC(GT) C6H6(GT) \\\n",
"0 2004-03-10 18:00:00 2.6 1360 150 11.9 \n",
"1 2004-03-10 19:00:00 2.0 1292 112 9.4 \n",
"2 2004-03-10 20:00:00 2.2 1402 88 9.0 \n",
"3 2004-03-10 21:00:00 2.2 1376 80 9.2 \n",
"4 2004-03-10 22:00:00 1.6 1272 51 6.5 \n",
"\n",
" PT08.S2(NMHC) NOx(GT) PT08.S3(NOx) NO2(GT) PT08.S4(NO2) PT08.S5(O3) \\\n",
"0 1046 166 1056 113 1692 1268 \n",
"1 955 103 1174 92 1559 972 \n",
"2 939 131 1140 114 1555 1074 \n",
"3 948 172 1092 122 1584 1203 \n",
"4 836 131 1205 116 1490 1110 \n",
"\n",
" T RH AH \n",
"0 13.6 48.9 0.7578 \n",
"1 13.3 47.7 0.7255 \n",
"2 11.9 54.0 0.7502 \n",
"3 11.0 60.0 0.7867 \n",
"4 11.2 59.6 0.7888 "
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" CO(GT) | \n",
" PT08.S1(CO) | \n",
" C6H6(GT) | \n",
" PT08.S2(NMHC) | \n",
" NOx(GT) | \n",
" PT08.S3(NOx) | \n",
" NO2(GT) | \n",
" PT08.S4(NO2) | \n",
" PT08.S5(O3) | \n",
" T | \n",
" RH | \n",
" index | \n",
"
\n",
" \n",
" \n",
" \n",
" 0 | \n",
" 2.6 | \n",
" 1360.0 | \n",
" 11.9 | \n",
" 1046.0 | \n",
" 166.0 | \n",
" 1056.0 | \n",
" 113.0 | \n",
" 1692.0 | \n",
" 1268.0 | \n",
" 13.6 | \n",
" 48.9 | \n",
" 2004-03-10 18:00:00 | \n",
"
\n",
" \n",
" 1 | \n",
" 2.0 | \n",
" 1292.0 | \n",
" 9.4 | \n",
" 955.0 | \n",
" 103.0 | \n",
" 1174.0 | \n",
" 92.0 | \n",
" 1559.0 | \n",
" 972.0 | \n",
" 13.3 | \n",
" 47.7 | \n",
" 2004-03-10 19:00:00 | \n",
"
\n",
" \n",
" 2 | \n",
" 2.2 | \n",
" 1402.0 | \n",
" 9.0 | \n",
" 939.0 | \n",
" 131.0 | \n",
" 1140.0 | \n",
" 114.0 | \n",
" 1555.0 | \n",
" 1074.0 | \n",
" 11.9 | \n",
" 54.0 | \n",
" 2004-03-10 20:00:00 | \n",
"
\n",
" \n",
" 3 | \n",
" 2.2 | \n",
" 1376.0 | \n",
" 9.2 | \n",
" 948.0 | \n",
" 172.0 | \n",
" 1092.0 | \n",
" 122.0 | \n",
" 1584.0 | \n",
" 1203.0 | \n",
" 11.0 | \n",
" 60.0 | \n",
" 2004-03-10 21:00:00 | \n",
"
\n",
" \n",
" 4 | \n",
" 1.6 | \n",
" 1272.0 | \n",
" 6.5 | \n",
" 836.0 | \n",
" 131.0 | \n",
" 1205.0 | \n",
" 116.0 | \n",
" 1490.0 | \n",
" 1110.0 | \n",
" 11.2 | \n",
" 59.6 | \n",
" 2004-03-10 22:00:00 | \n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" CO(GT) PT08.S1(CO) C6H6(GT) PT08.S2(NMHC) NOx(GT) PT08.S3(NOx) \\\n",
"0 2.6 1360.0 11.9 1046.0 166.0 1056.0 \n",
"1 2.0 1292.0 9.4 955.0 103.0 1174.0 \n",
"2 2.2 1402.0 9.0 939.0 131.0 1140.0 \n",
"3 2.2 1376.0 9.2 948.0 172.0 1092.0 \n",
"4 1.6 1272.0 6.5 836.0 131.0 1205.0 \n",
"\n",
" NO2(GT) PT08.S4(NO2) PT08.S5(O3) T RH index \n",
"0 113.0 1692.0 1268.0 13.6 48.9 2004-03-10 18:00:00 \n",
"1 92.0 1559.0 972.0 13.3 47.7 2004-03-10 19:00:00 \n",
"2 114.0 1555.0 1074.0 11.9 54.0 2004-03-10 20:00:00 \n",
"3 122.0 1584.0 1203.0 11.0 60.0 2004-03-10 21:00:00 \n",
"4 116.0 1490.0 1110.0 11.2 59.6 2004-03-10 22:00:00 "
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data = get_data(\"airquality\")\n",
"data[\"index\"] = pd.to_datetime(data[\"Date\"] + \" \" + data[\"Time\"])\n",
"data.drop(columns=[\"Date\", \"Time\"], inplace=True)\n",
"data.replace(-200, np.nan, inplace=True)\n",
"target = \"CO(GT)\"\n",
"\n",
"exclude = ['NMHC(GT)', 'AH']\n",
"data.drop(columns=exclude, inplace=True)\n",
"data.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Step 2: Baseline Model - Univariate forecasting without exogenous variables"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
" \n",
" \n",
" | \n",
" Description | \n",
" Value | \n",
"
\n",
" \n",
" \n",
" \n",
" 0 | \n",
" session_id | \n",
" 42 | \n",
"
\n",
" \n",
" 1 | \n",
" Target | \n",
" CO(GT) | \n",
"
\n",
" \n",
" 2 | \n",
" Approach | \n",
" Univariate | \n",
"
\n",
" \n",
" 3 | \n",
" Exogenous Variables | \n",
" Not Present | \n",
"
\n",
" \n",
" 4 | \n",
" Original data shape | \n",
" (9357, 1) | \n",
"
\n",
" \n",
" 5 | \n",
" Transformed data shape | \n",
" (9357, 1) | \n",
"
\n",
" \n",
" 6 | \n",
" Transformed train set shape | \n",
" (9309, 1) | \n",
"
\n",
" \n",
" 7 | \n",
" Transformed test set shape | \n",
" (48, 1) | \n",
"
\n",
" \n",
" 8 | \n",
" Rows with missing values | \n",
" 18.0% | \n",
"
\n",
" \n",
" 9 | \n",
" Fold Generator | \n",
" ExpandingWindowSplitter | \n",
"
\n",
" \n",
" 10 | \n",
" Fold Number | \n",
" 3 | \n",
"
\n",
" \n",
" 11 | \n",
" Enforce Prediction Interval | \n",
" False | \n",
"
\n",
" \n",
" 12 | \n",
" Seasonal Period(s) Tested | \n",
" 24 | \n",
"
\n",
" \n",
" 13 | \n",
" Seasonality Present | \n",
" True | \n",
"
\n",
" \n",
" 14 | \n",
" Seasonalities Detected | \n",
" [24] | \n",
"
\n",
" \n",
" 15 | \n",
" Primary Seasonality | \n",
" 24 | \n",
"
\n",
" \n",
" 16 | \n",
" Target Strictly Positive | \n",
" True | \n",
"
\n",
" \n",
" 17 | \n",
" Target White Noise | \n",
" No | \n",
"
\n",
" \n",
" 18 | \n",
" Recommended d | \n",
" 1 | \n",
"
\n",
" \n",
" 19 | \n",
" Recommended Seasonal D | \n",
" 0 | \n",
"
\n",
" \n",
" 20 | \n",
" Preprocess | \n",
" True | \n",
"
\n",
" \n",
" 21 | \n",
" Numerical Imputation (Target) | \n",
" ffill | \n",
"
\n",
" \n",
" 22 | \n",
" Transformation (Target) | \n",
" None | \n",
"
\n",
" \n",
" 23 | \n",
" Scaling (Target) | \n",
" None | \n",
"
\n",
" \n",
" 24 | \n",
" CPU Jobs | \n",
" -1 | \n",
"
\n",
" \n",
" 25 | \n",
" Use GPU | \n",
" False | \n",
"
\n",
" \n",
" 26 | \n",
" Log Experiment | \n",
" False | \n",
"
\n",
" \n",
" 27 | \n",
" Experiment Name | \n",
" ts-default-name | \n",
"
\n",
" \n",
" 28 | \n",
" USI | \n",
" 968c | \n",
"
\n",
" \n",
"
\n"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
""
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data_uni = data.copy()\n",
"data_uni.set_index(\"index\", inplace=True)\n",
"data_uni = data_uni[target]\n",
"\n",
"exp_uni = TSForecastingExperiment()\n",
"exp_uni.setup(\n",
" data=data_uni, fh=48,\n",
" numeric_imputation_target=\"ffill\", numeric_imputation_exogenous=\"ffill\",\n",
" fig_kwargs=global_fig_settings, session_id=42\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
" \n",
" \n",
" | \n",
" cutoff | \n",
" MASE | \n",
" RMSSE | \n",
" MAE | \n",
" RMSE | \n",
" MAPE | \n",
" SMAPE | \n",
" R2 | \n",
"
\n",
" \n",
" \n",
" \n",
" 0 | \n",
" 2005-03-27 14:00 | \n",
" 1.2101 | \n",
" 1.1315 | \n",
" 1.0340 | \n",
" 1.4835 | \n",
" 0.5751 | \n",
" 0.9044 | \n",
" -1.3625 | \n",
"
\n",
" \n",
" 1 | \n",
" 2005-03-29 14:00 | \n",
" 2.2086 | \n",
" 1.6277 | \n",
" 1.8839 | \n",
" 2.1316 | \n",
" 1.5146 | \n",
" 0.7456 | \n",
" -2.8068 | \n",
"
\n",
" \n",
" 2 | \n",
" 2005-03-31 14:00 | \n",
" 1.1332 | \n",
" 0.8553 | \n",
" 0.9652 | \n",
" 1.1183 | \n",
" 1.1796 | \n",
" 1.2402 | \n",
" -5.1529 | \n",
"
\n",
" \n",
" Mean | \n",
" nan | \n",
" 1.5173 | \n",
" 1.2048 | \n",
" 1.2944 | \n",
" 1.5778 | \n",
" 1.0898 | \n",
" 0.9634 | \n",
" -3.1074 | \n",
"
\n",
" \n",
" SD | \n",
" nan | \n",
" 0.4899 | \n",
" 0.3196 | \n",
" 0.4178 | \n",
" 0.4190 | \n",
" 0.3888 | \n",
" 0.2062 | \n",
" 1.5620 | \n",
"
\n",
" \n",
"
\n"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"model = exp_uni.create_model(\"arima\", order=(0,1,0), seasonal_order=(0,1,0,24))"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
" \n",
" "
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"exp_uni.plot_model(model)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"On zooming in to the forecasts, we can see that the model is able to capture some of the trends (spikes) in the dataset, but not all. The performance of our baseline model indicates that mean MASE across the CV folds is 1.52 which is not that great. Any value > 1 indicates that the model is performing worse than even a naive model with one step ahead forecasts. This model needs more improvement. Let's see if adding exogenous variables can help improve the model performance."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Step 3: Improved Model - Univariate forecasting with exogenous variables"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
" \n",
" \n",
" | \n",
" Description | \n",
" Value | \n",
"
\n",
" \n",
" \n",
" \n",
" 0 | \n",
" session_id | \n",
" 42 | \n",
"
\n",
" \n",
" 1 | \n",
" Target | \n",
" CO(GT) | \n",
"
\n",
" \n",
" 2 | \n",
" Approach | \n",
" Univariate | \n",
"
\n",
" \n",
" 3 | \n",
" Exogenous Variables | \n",
" Present | \n",
"
\n",
" \n",
" 4 | \n",
" Original data shape | \n",
" (9357, 11) | \n",
"
\n",
" \n",
" 5 | \n",
" Transformed data shape | \n",
" (9357, 11) | \n",
"
\n",
" \n",
" 6 | \n",
" Transformed train set shape | \n",
" (9309, 11) | \n",
"
\n",
" \n",
" 7 | \n",
" Transformed test set shape | \n",
" (48, 11) | \n",
"
\n",
" \n",
" 8 | \n",
" Rows with missing values | \n",
" 25.8% | \n",
"
\n",
" \n",
" 9 | \n",
" Fold Generator | \n",
" ExpandingWindowSplitter | \n",
"
\n",
" \n",
" 10 | \n",
" Fold Number | \n",
" 3 | \n",
"
\n",
" \n",
" 11 | \n",
" Enforce Prediction Interval | \n",
" False | \n",
"
\n",
" \n",
" 12 | \n",
" Seasonal Period(s) Tested | \n",
" 24 | \n",
"
\n",
" \n",
" 13 | \n",
" Seasonality Present | \n",
" True | \n",
"
\n",
" \n",
" 14 | \n",
" Seasonalities Detected | \n",
" [24] | \n",
"
\n",
" \n",
" 15 | \n",
" Primary Seasonality | \n",
" 24 | \n",
"
\n",
" \n",
" 16 | \n",
" Target Strictly Positive | \n",
" True | \n",
"
\n",
" \n",
" 17 | \n",
" Target White Noise | \n",
" No | \n",
"
\n",
" \n",
" 18 | \n",
" Recommended d | \n",
" 1 | \n",
"
\n",
" \n",
" 19 | \n",
" Recommended Seasonal D | \n",
" 0 | \n",
"
\n",
" \n",
" 20 | \n",
" Preprocess | \n",
" True | \n",
"
\n",
" \n",
" 21 | \n",
" Numerical Imputation (Target) | \n",
" ffill | \n",
"
\n",
" \n",
" 22 | \n",
" Transformation (Target) | \n",
" None | \n",
"
\n",
" \n",
" 23 | \n",
" Scaling (Target) | \n",
" None | \n",
"
\n",
" \n",
" 24 | \n",
" Numerical Imputation (Exogenous) | \n",
" ffill | \n",
"
\n",
" \n",
" 25 | \n",
" Transformation (Exogenous) | \n",
" None | \n",
"
\n",
" \n",
" 26 | \n",
" Scaling (Exogenous) | \n",
" None | \n",
"
\n",
" \n",
" 27 | \n",
" CPU Jobs | \n",
" -1 | \n",
"
\n",
" \n",
" 28 | \n",
" Use GPU | \n",
" False | \n",
"
\n",
" \n",
" 29 | \n",
" Log Experiment | \n",
" False | \n",
"
\n",
" \n",
" 30 | \n",
" Experiment Name | \n",
" ts-default-name | \n",
"
\n",
" \n",
" 31 | \n",
" USI | \n",
" 5f62 | \n",
"
\n",
" \n",
"
\n"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
""
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"exp_exo = TSForecastingExperiment()\n",
"exp_exo.setup(\n",
" data=data, target=target, index=\"index\", fh=48,\n",
" numeric_imputation_target=\"ffill\", numeric_imputation_exogenous=\"ffill\",\n",
" fig_kwargs=global_fig_settings, session_id=42\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
" \n",
" \n",
" | \n",
" cutoff | \n",
" MASE | \n",
" RMSSE | \n",
" MAE | \n",
" RMSE | \n",
" MAPE | \n",
" SMAPE | \n",
" R2 | \n",
"
\n",
" \n",
" \n",
" \n",
" 0 | \n",
" 2005-03-27 14:00 | \n",
" 0.1473 | \n",
" 0.1281 | \n",
" 0.1259 | \n",
" 0.1680 | \n",
" 0.0825 | \n",
" 0.0824 | \n",
" 0.9697 | \n",
"
\n",
" \n",
" 1 | \n",
" 2005-03-29 14:00 | \n",
" 0.1931 | \n",
" 0.1628 | \n",
" 0.1647 | \n",
" 0.2132 | \n",
" 0.1043 | \n",
" 0.1112 | \n",
" 0.9619 | \n",
"
\n",
" \n",
" 2 | \n",
" 2005-03-31 14:00 | \n",
" 0.3009 | \n",
" 0.2474 | \n",
" 0.2563 | \n",
" 0.3235 | \n",
" 0.2767 | \n",
" 0.3375 | \n",
" 0.4852 | \n",
"
\n",
" \n",
" Mean | \n",
" nan | \n",
" 0.2138 | \n",
" 0.1794 | \n",
" 0.1823 | \n",
" 0.2349 | \n",
" 0.1545 | \n",
" 0.1770 | \n",
" 0.8056 | \n",
"
\n",
" \n",
" SD | \n",
" nan | \n",
" 0.0644 | \n",
" 0.0501 | \n",
" 0.0547 | \n",
" 0.0653 | \n",
" 0.0869 | \n",
" 0.1141 | \n",
" 0.2266 | \n",
"
\n",
" \n",
"
\n"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"model_exo = exp_exo.create_model(\"arima\", order=(0,1,0), seasonal_order=(0,1,0,24))"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
" \n",
" "
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"exp_exo.plot_model(model_exo)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Step 4: Evaluate Model"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Not bad, We have managed to improve MASE to ~ 0.21 which is much better than the univariate model and also a large improvement over a naive model. We should be happy with this improvement. Let's finalize the model by training it on the entire dataset so we can make true future forecasts."
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [],
"source": [
"final_model_exo = exp_exo.finalize_model(model_exo)"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Model was trained with exogenous variables but you have not passed any for predictions. Please pass exogenous variables to make predictions.\n",
"10 exogenous variables (X) needed in order to make future predictions:\n",
"['PT08.S1(CO)', 'C6H6(GT)', 'PT08.S2(NMHC)', 'NOx(GT)', 'PT08.S3(NOx)', 'NO2(GT)', 'PT08.S4(NO2)', 'PT08.S5(O3)', 'T', 'RH']\n"
]
}
],
"source": [
"def safe_predict(exp, model):\n",
" \"\"\"Prediction wrapper for demo purposes.\"\"\"\n",
" try: \n",
" exp.predict_model(model)\n",
" except ValueError as exception:\n",
" print(exception)\n",
" exo_vars = exp.exogenous_variables\n",
" print(f\"{len(exo_vars)} exogenous variables (X) needed in order to make future predictions:\\n{exo_vars}\")\n",
"\n",
"safe_predict(exp_exo, final_model_exo)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As we can see, this approach does not come without side effects. The problem is that we have 10 exogenous variables. Hence in order to get any unknown future values for CO concentration, we will need the future values for all these exogenous variables. This is generally obtained through some forecasting process itself. But each forecast will have errors and these errors can be compounded when there are a lot of exogenous variables. \n",
"\n",
"**Let's see if we can trim down these exogenous variables to a handful of useful variables without compromising on forecasting performance.**"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Step 5: Parsimonious Model - Univariate forecasting with limited exogenous variables\n",
"\n",
"From the CCF Analysis, we found that many of the exogenous variables show a very similar correlation structure to the CO concentration. E.g. `PT08.S1(CO)`, `NOx(GT)`, `C6H6(GT)`, `PT08.S2(NMHC)` values from 24 hours before (lag = 24) show a high positive correlation to CO concentration. Instead of keeping all of them, lets pick the one with the highest positive correlation at lag 24 which is `NOx(GT)`.\n",
"\n",
"Similarly, `PT08.S3(NOx)` values from 24 hours ago shows the highest negative correlation to CO concentration. Let's keep this variable as well.\n",
"\n",
"Finally, in daily cycles, what happens 12 hours back can also impact the current value (e.g. values last night can impact the next day and vice versa). The variable with the highest correlation to CO concentration at lag = 12 is `RH`. We will keep this as well."
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
" \n",
" \n",
" | \n",
" Description | \n",
" Value | \n",
"
\n",
" \n",
" \n",
" \n",
" 0 | \n",
" session_id | \n",
" 42 | \n",
"
\n",
" \n",
" 1 | \n",
" Target | \n",
" CO(GT) | \n",
"
\n",
" \n",
" 2 | \n",
" Approach | \n",
" Univariate | \n",
"
\n",
" \n",
" 3 | \n",
" Exogenous Variables | \n",
" Present | \n",
"
\n",
" \n",
" 4 | \n",
" Original data shape | \n",
" (9357, 4) | \n",
"
\n",
" \n",
" 5 | \n",
" Transformed data shape | \n",
" (9357, 4) | \n",
"
\n",
" \n",
" 6 | \n",
" Transformed train set shape | \n",
" (9309, 4) | \n",
"
\n",
" \n",
" 7 | \n",
" Transformed test set shape | \n",
" (48, 4) | \n",
"
\n",
" \n",
" 8 | \n",
" Rows with missing values | \n",
" 25.8% | \n",
"
\n",
" \n",
" 9 | \n",
" Fold Generator | \n",
" ExpandingWindowSplitter | \n",
"
\n",
" \n",
" 10 | \n",
" Fold Number | \n",
" 3 | \n",
"
\n",
" \n",
" 11 | \n",
" Enforce Prediction Interval | \n",
" False | \n",
"
\n",
" \n",
" 12 | \n",
" Seasonal Period(s) Tested | \n",
" 24 | \n",
"
\n",
" \n",
" 13 | \n",
" Seasonality Present | \n",
" True | \n",
"
\n",
" \n",
" 14 | \n",
" Seasonalities Detected | \n",
" [24] | \n",
"
\n",
" \n",
" 15 | \n",
" Primary Seasonality | \n",
" 24 | \n",
"
\n",
" \n",
" 16 | \n",
" Target Strictly Positive | \n",
" True | \n",
"
\n",
" \n",
" 17 | \n",
" Target White Noise | \n",
" No | \n",
"
\n",
" \n",
" 18 | \n",
" Recommended d | \n",
" 1 | \n",
"
\n",
" \n",
" 19 | \n",
" Recommended Seasonal D | \n",
" 0 | \n",
"
\n",
" \n",
" 20 | \n",
" Preprocess | \n",
" True | \n",
"
\n",
" \n",
" 21 | \n",
" Numerical Imputation (Target) | \n",
" ffill | \n",
"
\n",
" \n",
" 22 | \n",
" Transformation (Target) | \n",
" None | \n",
"
\n",
" \n",
" 23 | \n",
" Scaling (Target) | \n",
" None | \n",
"
\n",
" \n",
" 24 | \n",
" Numerical Imputation (Exogenous) | \n",
" ffill | \n",
"
\n",
" \n",
" 25 | \n",
" Transformation (Exogenous) | \n",
" None | \n",
"
\n",
" \n",
" 26 | \n",
" Scaling (Exogenous) | \n",
" None | \n",
"
\n",
" \n",
" 27 | \n",
" CPU Jobs | \n",
" -1 | \n",
"
\n",
" \n",
" 28 | \n",
" Use GPU | \n",
" False | \n",
"
\n",
" \n",
" 29 | \n",
" Log Experiment | \n",
" False | \n",
"
\n",
" \n",
" 30 | \n",
" Experiment Name | \n",
" ts-default-name | \n",
"
\n",
" \n",
" 31 | \n",
" USI | \n",
" e941 | \n",
"
\n",
" \n",
"
\n"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
""
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"exp_slim = TSForecastingExperiment()\n",
"keep = [target, \"index\", 'NOx(GT)', \"PT08.S3(NOx)\", \"RH\"]\n",
"data_slim = data[keep]\n",
"exp_slim.setup(\n",
" data=data_slim, target=target, index=\"index\", fh=48,\n",
" numeric_imputation_target=\"ffill\", numeric_imputation_exogenous=\"ffill\",\n",
" fig_kwargs=global_fig_settings, session_id=42 \n",
")"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
" \n",
" \n",
" | \n",
" cutoff | \n",
" MASE | \n",
" RMSSE | \n",
" MAE | \n",
" RMSE | \n",
" MAPE | \n",
" SMAPE | \n",
" R2 | \n",
"
\n",
" \n",
" \n",
" \n",
" 0 | \n",
" 2005-03-27 14:00 | \n",
" 0.2174 | \n",
" 0.1891 | \n",
" 0.1857 | \n",
" 0.2479 | \n",
" 0.1339 | \n",
" 0.1230 | \n",
" 0.9340 | \n",
"
\n",
" \n",
" 1 | \n",
" 2005-03-29 14:00 | \n",
" 0.2644 | \n",
" 0.2209 | \n",
" 0.2255 | \n",
" 0.2893 | \n",
" 0.1358 | \n",
" 0.1524 | \n",
" 0.9299 | \n",
"
\n",
" \n",
" 2 | \n",
" 2005-03-31 14:00 | \n",
" 0.2366 | \n",
" 0.1972 | \n",
" 0.2015 | \n",
" 0.2579 | \n",
" 0.2503 | \n",
" 0.3316 | \n",
" 0.6729 | \n",
"
\n",
" \n",
" Mean | \n",
" nan | \n",
" 0.2395 | \n",
" 0.2024 | \n",
" 0.2043 | \n",
" 0.2650 | \n",
" 0.1733 | \n",
" 0.2023 | \n",
" 0.8456 | \n",
"
\n",
" \n",
" SD | \n",
" nan | \n",
" 0.0193 | \n",
" 0.0135 | \n",
" 0.0164 | \n",
" 0.0177 | \n",
" 0.0544 | \n",
" 0.0922 | \n",
" 0.1221 | \n",
"
\n",
" \n",
"
\n"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"model_slim = exp_slim.create_model(\"arima\", order=(0,1,0), seasonal_order=(0,1,0,24))"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
" \n",
" "
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"exp_slim.plot_model(model_slim)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Not bad. MASE has only increased from ~0.21 to ~0.24, but we have managed to cut our exogenous variables down from 13 to 3. This will help us when we make \"true\" unknonw future predictions since we will need the \"unknown\" future values of these exogenous variables to make the forecast for the CO concentration."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Finalize the model\n",
"\n",
"- Train the slim model on the entire dataset so we can make true future forecasts\n",
"- Save the model as a pickle file for deployment "
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [],
"source": [
"final_slim_model = exp_slim.finalize_model(model_slim)"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Transformation Pipeline and Model Successfully Saved\n"
]
},
{
"data": {
"text/plain": [
"(ForecastingPipeline(steps=[('transformer_exogenous',\n",
" TransformerPipeline(steps=[('numerical_imputer',\n",
" Imputer(method='ffill',\n",
" random_state=42))])),\n",
" ('forecaster',\n",
" TransformedTargetForecaster(steps=[('transformer_target',\n",
" TransformerPipeline(steps=[('numerical_imputer',\n",
" Imputer(method='ffill',\n",
" random_state=42))])),\n",
" ('model',\n",
" ARIMA(order=(0,\n",
" 1,\n",
" 0),\n",
" seasonal_order=(0,\n",
" 1,\n",
" 0,\n",
" 24)))]))]),\n",
" 'final_slim_model.pkl')"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"exp_slim.save_model(final_slim_model, \"final_slim_model\")"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Model was trained with exogenous variables but you have not passed any for predictions. Please pass exogenous variables to make predictions.\n",
"3 exogenous variables (X) needed in order to make future predictions:\n",
"['NOx(GT)', 'PT08.S3(NOx)', 'RH']\n"
]
}
],
"source": [
"safe_predict(exp_slim, final_slim_model)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"So we still need future values for 3 exogenous variables. We will get this in the next part using forecasting techniques."
]
}
],
"metadata": {
"interpreter": {
"hash": "c161a91f6f4623a54f30c5492a42e7cf0592610fb90c8abd312086f09f8fbe0f"
},
"kernelspec": {
"display_name": "pycaret_sktime_0p11_2",
"language": "python",
"name": "pycaret_sktime_0p11_2"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.13"
}
},
"nbformat": 4,
"nbformat_minor": 2
}