{ "cells": [ { "cell_type": "markdown", "id": "49d72278-b1a8-4f28-ab19-66357a411c7e", "metadata": {}, "source": [ "# Quick Start: Running TTM models on gift-eval benchmark\n", "\n", "**Tiny Time Mixers (TTMs)** (accepted in NeurIPS 2024) are **compact and lightweight pre-trained models** for time series forecasting, with sizes ranging from **1 to 5 million parameters**. They are designed for **fast fine-tuning** on target domain datasets. \n", "\n", "In this script, we demonstrate how to run the **TTM model** on the **GIFT-Eval benchmark** using a **20% few-shot fine-tuning setting**. For more details, see [here](https://github.com/ibm-granite/granite-tsfm/tree/gift/notebooks/hfdemo/tinytimemixer/full_benchmarking/gift_leaderboard). \n", "\n", "TTM-r2 models have been used in this evaluation. Model card can be found [here](https://huggingface.co/ibm-granite/granite-timeseries-ttm-r2).\n", "\n", "Make sure you download the gift-eval benchmark and set the `GIFT-EVAL`\n", "environment variable correctly before running this script.\n", "We will use the `Dataset` class to load the data and run the model.\n", "If you have not already please check out the [dataset.ipynb](./dataset.ipynb)\n", "notebook to learn more about the `Dataset` class. We are going to just run\n", "the model on two datasets for brevity. But feel free to run on any dataset\n", "by changing the `short_datasets` and `med_long_datasets` variables below." ] }, { "cell_type": "markdown", "id": "c4c6fef1-714e-4e3b-808a-571b69a4fe73", "metadata": {}, "source": [ "## TSFM and TTM Installation\n", "\n", "1. Clone the [GIFT-Eval repository](https://github.com/SalesforceAIResearch/gift-eval).\n", "1. Follow the instruction to set up the GIFT-Eval environment as described [here](https://github.com/SalesforceAIResearch/gift-eval?tab=readme-ov-file#installation).\n", "1. This notebook should be placed in the `notebooks` folder of the cloned repository.\n", "1. Follow the instructions below to install TSFM. " ] }, { "cell_type": "markdown", "id": "25ad68ca-a9b6-4e68-844d-539037852a99", "metadata": {}, "source": [ "### Installing `tsfm`\n", "\n", "The TTM source codes will be installed from the [granite-tsfm repository](https://github.com/ibm-granite/granite-tsfm).\n", "Note that `granite-tsfm` installs `pandas==2.2.3` but GIFT-EVAL requires `pandas==2.0.0`.\n", "Hence, after installing TTM from `granite-tsfm`, we forece reinstall `pandas==2.0.0`.\n", "\n", "\n", "Run the following code once to install granite-tsfm in your working python environment.\n" ] }, { "cell_type": "code", "execution_count": 1, "id": "8ecc0baa-2a75-42fc-ae8f-e84afd241130", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Folder 'granite-tsfm' already exists. Skipping git clone.\n" ] } ], "source": [ "import os\n", "\n", "\n", "if not os.path.exists(\"granite-tsfm\"):\n", " !git clone git@github.com:ibm-granite/granite-tsfm.git\n", " %cd granite-tsfm\n", " !pwd\n", " # Switch to the desired branch\n", " !git switch gift\n", " ! pip install \".[notebooks]\"\n", " ! pip install pandas==2.0.0\n", " %cd ..\n", "else:\n", " print(\"Folder 'granite-tsfm' already exists. Skipping git clone.\")" ] }, { "cell_type": "markdown", "id": "06614be0-eca8-45bc-91ea-ab9c47bd5c3c", "metadata": {}, "source": [ "## Imports" ] }, { "cell_type": "code", "execution_count": 2, "id": "45fd2810", "metadata": {}, "outputs": [], "source": [ "# All Required Imports\n", "import csv\n", "import json\n", "import sys\n", "\n", "import pandas as pd\n", "from dotenv import load_dotenv\n", "from gift_eval.data import Dataset\n", "from gluonts.ev.metrics import (\n", " MAE,\n", " MAPE,\n", " MASE,\n", " MSE,\n", " MSIS,\n", " ND,\n", " NRMSE,\n", " RMSE,\n", " SMAPE,\n", " MeanWeightedSumQuantileLoss,\n", ")\n", "from gluonts.model import evaluate_model\n", "from gluonts.time_feature import get_seasonality" ] }, { "cell_type": "markdown", "id": "6688a5fd-4a0e-48e6-9411-74b34495481c", "metadata": {}, "source": [ "### Add `TTMGluonTSPredictor` to `PYTHONPATH`" ] }, { "cell_type": "code", "execution_count": 3, "id": "645eeb14", "metadata": {}, "outputs": [], "source": [ "sys.path.append(os.path.realpath(\"granite-tsfm/notebooks/hfdemo/tinytimemixer/full_benchmarking/\"))\n", "from gift_leaderboard.src.ttm_gluonts_predictor import (\n", " TTM_MAX_FORECAST_HORIZON,\n", " TTMGluonTSPredictor,\n", ")\n", "from gift_leaderboard.src.utils import get_args, set_seed" ] }, { "cell_type": "markdown", "id": "9268c6f5-2220-426c-b915-782649d2f03a", "metadata": {}, "source": [ "## Set output directory and seed" ] }, { "cell_type": "code", "execution_count": 4, "id": "aa66cd16", "metadata": {}, "outputs": [], "source": [ "args = get_args()\n", "\n", "# Set out dir path\n", "OUT_DIR = f\"../results/{args.out_dir}\"\n", "\n", "# Add arguments\n", "SEED = 42\n", "\n", "# set seed\n", "set_seed(SEED)\n", "# Load environment variables\n", "load_dotenv()\n", "# Ensure the output directory exists\n", "os.makedirs(OUT_DIR, exist_ok=True)" ] }, { "cell_type": "markdown", "id": "1df14514-5253-4e9a-b60c-b41dd3c7e11d", "metadata": { "lines_to_next_cell": 2, "title": "In[4]:" }, "source": [ "## Dataset" ] }, { "cell_type": "code", "execution_count": 5, "id": "358a12fd", "metadata": { "lines_to_next_cell": 2, "title": "## Datasets" }, "outputs": [], "source": [ "# short_datasets = \"m4_yearly m4_quarterly m4_monthly m4_weekly m4_daily m4_hourly electricity/15T electricity/H electricity/D electricity/W solar/10T solar/H solar/D solar/W hospital covid_deaths us_births/D us_births/M us_births/W saugeenday/D saugeenday/M saugeenday/W temperature_rain_with_missing kdd_cup_2018_with_missing/H kdd_cup_2018_with_missing/D car_parts_with_missing restaurant hierarchical_sales/D hierarchical_sales/W LOOP_SEATTLE/5T LOOP_SEATTLE/H LOOP_SEATTLE/D SZ_TAXI/15T SZ_TAXI/H M_DENSE/H M_DENSE/D ett1/15T ett1/H ett1/D ett1/W ett2/15T ett2/H ett2/D ett2/W jena_weather/10T jena_weather/H jena_weather/D bitbrains_fast_storage/5T bitbrains_fast_storage/H bitbrains_rnd/5T bitbrains_rnd/H bizitobs_application bizitobs_service bizitobs_l2c/5T bizitobs_l2c/H\"\n", "short_datasets = \"us_births/D saugeenday/M\"\n", "\n", "# med_long_datasets = \"electricity/15T electricity/H solar/10T solar/H kdd_cup_2018_with_missing/H LOOP_SEATTLE/5T LOOP_SEATTLE/H SZ_TAXI/15T M_DENSE/H ett1/15T ett1/H ett2/15T ett2/H jena_weather/10T jena_weather/H bitbrains_fast_storage/5T bitbrains_rnd/5T bizitobs_application bizitobs_service bizitobs_l2c/5T bizitobs_l2c/H\"\n", "med_long_datasets = \"\"\n", "\n", "# Get union of short and med_long datasets\n", "all_datasets = sorted(set(short_datasets.split() + med_long_datasets.split()))\n", "\n", "dataset_properties_map = json.load(open(\"dataset_properties.json\"))" ] }, { "cell_type": "markdown", "id": "0b17ec82-c713-4b23-97ea-d5f8280d69bd", "metadata": {}, "source": [ "## Metrics" ] }, { "cell_type": "code", "execution_count": 6, "id": "8a5fb2ec", "metadata": { "title": "## Metrics" }, "outputs": [], "source": [ "# Instantiate the metrics\n", "metrics = [\n", " MSE(forecast_type=\"mean\"),\n", " MSE(forecast_type=0.5),\n", " MAE(forecast_type=\"mean\"),\n", " MAE(forecast_type=0.5),\n", " MASE(),\n", " MAPE(),\n", " SMAPE(),\n", " MSIS(),\n", " RMSE(),\n", " NRMSE(),\n", " ND(),\n", " MeanWeightedSumQuantileLoss(quantile_levels=[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]),\n", "]" ] }, { "cell_type": "markdown", "id": "a593381f-8a6e-4bb4-a267-1969f37420bf", "metadata": {}, "source": [ "## Evaluation\n", "\n", "\n", "Now that we have our predictor class `TTMGluonTSPredictor` imported,\n", "we can use it to fine-tune and predict on the gift-eval benchmark datasets.\n", "We will use the `train` function to finetune the TTM model, and\n", "`evaluate_model` function to evaluate the model.\n", "The `evaluate_model` function is a helper function to evaluate the \n", "model on the test data and return the results in a dictionary.\n", "\n", "We are going to follow the naming conventions explained in the\n", "[README](../README.md) file to store the results in a csv file\n", "called `all_results.csv` under the `results/ttm` folder.\n", "\n", "The first column in the csv file is the dataset config name which\n", "is a combination of the dataset name, frequency and the term:\n", "\n", "```python\n", "f\"{dataset_name}/{freq}/{term}\"\n", "```" ] }, { "cell_type": "markdown", "id": "c02aeb63-9665-406c-a4ef-d9c03bfe7e23", "metadata": { "title": "## Evaluation" }, "source": [ "### Define output file paths" ] }, { "cell_type": "code", "execution_count": 7, "id": "c75fc098", "metadata": { "title": "Output file" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Done datasets\n", "[]\n" ] } ], "source": [ "# Define the path for the CSV file\n", "csv_file_path = os.path.join(OUT_DIR, \"all_results.csv\")\n", "\n", "pretty_names = {\n", " \"saugeenday\": \"saugeen\",\n", " \"temperature_rain_with_missing\": \"temperature_rain\",\n", " \"kdd_cup_2018_with_missing\": \"kdd_cup_2018\",\n", " \"car_parts_with_missing\": \"car_parts\",\n", "}\n", "\n", "if not os.path.exists(csv_file_path):\n", " with open(csv_file_path, \"a\", newline=\"\") as csvfile:\n", " writer = csv.writer(csvfile)\n", "\n", " # Write the header\n", " writer.writerow(\n", " [\n", " \"dataset\",\n", " \"model\",\n", " \"eval_metrics/MSE[mean]\",\n", " \"eval_metrics/MSE[0.5]\",\n", " \"eval_metrics/MAE[mean]\",\n", " \"eval_metrics/MAE[0.5]\",\n", " \"eval_metrics/MASE[0.5]\",\n", " \"eval_metrics/MAPE[0.5]\",\n", " \"eval_metrics/sMAPE[0.5]\",\n", " \"eval_metrics/MSIS\",\n", " \"eval_metrics/RMSE[mean]\",\n", " \"eval_metrics/NRMSE[mean]\",\n", " \"eval_metrics/ND[0.5]\",\n", " \"eval_metrics/mean_weighted_sum_quantile_loss\",\n", " \"domain\",\n", " \"num_variates\",\n", " \"horizon\",\n", " \"ttm_context_len\",\n", " \"available_context_len\",\n", " \"finetune_success\",\n", " \"finetune_train_num_samples\",\n", " \"finetune_valid_num_samples\",\n", " ]\n", " )\n", "\n", "df_res = pd.read_csv(csv_file_path)\n", "done_datasets = df_res[\"dataset\"].values\n", "print(\"Done datasets\")\n", "print(done_datasets)" ] }, { "cell_type": "markdown", "id": "73ad3e4d-afb4-4785-bf8a-281a22335087", "metadata": {}, "source": [ "### Run over all defined datasets" ] }, { "cell_type": "code", "execution_count": 8, "id": "13fc22d7", "metadata": { "title": "Run over all datasets" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Processing dataset: saugeenday/M, term: short\n", "Dataset: saugeenday/M, Freq = M, H = 12\n", "Minimum context length among all time series in this dataset = 696\n", "prediction_channel_indices = [0]\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:p-1751135:t-22410562855680:get_model.py:get_model:Loading model from: ibm-granite/granite-timeseries-ttm-r2\n", "WARNING:p-1751135:t-22410562855680:get_model.py:get_model:Requested `prediction_length` (12) is not exactly equal to any of the available TTM prediction lengths. Hence, TTM will forecast using the `prediction_filter_length` argument to provide the requested prediction length. Check the model card to know more about the supported context lengths and forecast/prediction lengths.\n", "INFO:p-1751135:t-22410562855680:get_model.py:get_model:Model loaded successfully from ibm-granite/granite-timeseries-ttm-r2, revision = 52-16-ft-l1-r2.1.\n", "INFO:p-1751135:t-22410562855680:get_model.py:get_model:[TTM] context_length = 52, prediction_length = 16\n", "INFO:p-1751135:t-22410562855680:ttm_gluonts_predictor.py:_get_gift_model:The TTM has Prefix Tuning = True\n", "INFO:p-1751135:t-22410562855680:ttm_gluonts_predictor.py:train:Number of series: Train = 1, Valid = 1\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Number of channels in the dataset saugeenday/M = 1\n", "Batch size is set to None\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "1it [00:00, 1752.01it/s]\n", "1it [00:00, 10305.42it/s]\n", "1it [00:00, 4563.99it/s]\n", "INFO:p-1751135:t-22410562855680:ttm_gluonts_predictor.py:train:Global scaling done successfully.\n", "INFO:p-1751135:t-22410562855680:ttm_gluonts_predictor.py:train:Increasing fewshot fraction to 0.9 due to small dataset size.\n", "INFO:p-1751135:t-22410562855680:ttm_gluonts_predictor.py:train:Length of orginal train set = 621\n", "INFO:p-1751135:t-22410562855680:ttm_gluonts_predictor.py:train:Length of 90.0 % train set = 558\n", "INFO:p-1751135:t-22410562855680:ttm_gluonts_predictor.py:train:Number of train samples = 559, valid samples = 63\n", "INFO:p-1751135:t-22410562855680:ttm_gluonts_predictor.py:train:Using a batch size of 8, based on number of training samples = 559 and number of channels = 1.\n", "INFO:p-1751135:t-22410562855680:lr_finder.py:optimal_lr_finder:LR Finder: Running learning rate (LR) finder algorithm. If the suggested LR is very low, we suggest setting the LR manually.\n", "INFO:p-1751135:t-22410562855680:lr_finder.py:optimal_lr_finder:LR Finder: Using GPU:0.\n", "/dccstor/dnn_forecasting/conda_envs/envs/gift/lib/python3.10/site-packages/tsfm_public/toolkit/lr_finder.py:62: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n", " state = torch.load(path, map_location=device)\n", "INFO:p-1751135:t-22410562855680:lr_finder.py:optimal_lr_finder:LR Finder: Suggested learning rate = 0.004037017258596558\n", "INFO:p-1751135:t-22410562855680:ttm_gluonts_predictor.py:train:OPTIMAL SUGGESTED LEARNING RATE = 0.004037017258596558\n", "INFO:p-1751135:t-22410562855680:ttm_gluonts_predictor.py:train:Using learning rate = 0.004037017258596558\n", "WARNING:p-1751135:t-22410562855680:other.py:check_os_kernel:Detected kernel version 4.18.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.\n" ] }, { "data": { "text/html": [ "\n", "
\n", " \n", " \n", " [ 840/3500 00:37 < 02:00, 22.08 it/s, Epoch 12/50]\n", "
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
EpochTraining LossValidation Loss
10.6199000.540722
20.5457000.507937
30.5225000.500946
40.5115000.500680
50.5027000.488240
60.4937000.494512
70.4947000.476915
80.4891000.483955
90.4885000.484647
100.4860000.490387
110.4854000.493347
120.4832000.491475

" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "[TrackingCallback] Mean Epoch Time = 2.819937785466512 seconds, Total Train Time = 38.79584288597107\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 622/622 [00:00<00:00, 136646.61it/s]\n" ] }, { "data": { "text/html": [], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stderr", "output_type": "stream", "text": [ "100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 622/622 [00:00<00:00, 122418.33it/s]\n", "INFO:p-1751135:t-22410562855680:ttm_gluonts_predictor.py:get_insample_stats:Successfully, calculated the in-sample statistics.\n", "7it [00:00, 23469.33it/s]\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "b946f4697e4246a281dfd2aa2dcfdaf7", "version_major": 2, "version_minor": 0 }, "text/plain": [ "0it [00:00, ?it/s]" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stderr", "output_type": "stream", "text": [ "100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:00<00:00, 57120.87it/s]\n", "INFO:p-1751135:t-22410562855680:ttm_gluonts_predictor.py:predict:Making quantile forecasts for quantiles [0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9]\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "5402d668eb0344fe9882c1c0b20c3f30", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Processing Batches: 0%| | 0/1 [00:00\n", " \n", " \n", " [400/400 00:29, Epoch 20/20]\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
EpochTraining LossValidation Loss
10.2944000.263794
20.2737000.237946
30.2626000.229524
40.2522000.227185
50.2488000.222400
60.2460000.218649
70.2397000.216203
80.2385000.214920
90.2313000.211621
100.2253000.208062
110.2242000.206730
120.2215000.205856
130.2212000.203939
140.2204000.202302
150.2181000.204197
160.2142000.202673
170.2150000.201576
180.2146000.201187
190.2163000.201118
200.2148000.201121

" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "[TrackingCallback] Mean Epoch Time = 0.7060062766075135 seconds, Total Train Time = 29.868444204330444\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6135/6135 [00:00<00:00, 133562.00it/s]\n" ] }, { "data": { "text/html": [], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stderr", "output_type": "stream", "text": [ "100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6135/6135 [00:00<00:00, 127326.53it/s]\n", "INFO:p-1751135:t-22410562855680:ttm_gluonts_predictor.py:get_insample_stats:Successfully, calculated the in-sample statistics.\n", "20it [00:00, 42733.61it/s]\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "a84bf35f5dd94b32bcf045dca8a8e127", "version_major": 2, "version_minor": 0 }, "text/plain": [ "0it [00:00, ?it/s]" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stderr", "output_type": "stream", "text": [ "100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:00<00:00, 84733.41it/s]\n", "INFO:p-1751135:t-22410562855680:ttm_gluonts_predictor.py:predict:Making quantile forecasts for quantiles [0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9]\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "eaadc9ee84ac45b9a1db1995a6350c1b", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Processing Batches: 0%| | 0/1 [00:00 TTM_MAX_FORECAST_HORIZON:\n", " # predict all channels, needed for recursive forecast\n", " prediction_channel_indices = list(range(num_channels))\n", "\n", " print(\"prediction_channel_indices =\", prediction_channel_indices)\n", "\n", " # For very short series, force short context window creatiio for finetuning\n", " if term == \"short\":\n", " force_short_context = args.force_short_context\n", " else:\n", " force_short_context = False\n", "\n", " # Instantiate the TTM GluonTS Predictor with the minimum context length in the dataset\n", " # The predictor will automatically choose the suitable context and forecast length\n", " # of the TTM model.\n", " predictor = TTMGluonTSPredictor(\n", " context_length=min_context_length,\n", " prediction_length=dataset.prediction_length,\n", " model_path=args.model_path,\n", " test_data_label=dataset.test_data.label,\n", " random_seed=SEED,\n", " term=term,\n", " ds_name=ds_name,\n", " out_dir=OUT_DIR,\n", " scale=True,\n", " upper_bound_fewshot_samples=args.upper_bound_fewshot_samples,\n", " force_short_context=force_short_context,\n", " min_context_mult=args.min_context_mult,\n", " past_feat_dynamic_real_exist=past_feat_dynamic_real_exist,\n", " num_prediction_channels=num_prediction_channels,\n", " freq=dataset.freq,\n", " use_valid_from_train=args.use_valid_from_train,\n", " insample_forecast=args.insample_forecast,\n", " insample_use_train=args.insample_use_train,\n", " # TTM kwargs\n", " head_dropout=args.head_dropout,\n", " decoder_mode=args.decoder_mode,\n", " num_input_channels=num_channels,\n", " huber_delta=args.huber_delta,\n", " quantile=args.quantile,\n", " loss=args.loss,\n", " prediction_channel_indices=prediction_channel_indices,\n", " )\n", "\n", " print(f\"Number of channels in the dataset {ds_name} =\", num_channels)\n", " if args.batch_size is None:\n", " batch_size = None\n", " optimize_batch_size = True\n", " else:\n", " batch_size = args.batch_size\n", " optimize_batch_size = False\n", " print(\"Batch size is set to\", batch_size)\n", "\n", " finetune_train_num_samples = 0\n", " finetune_valid_num_samples = 0\n", " try:\n", " # finetune the model on the train split\n", " predictor.train(\n", " train_dataset=dataset.training_dataset,\n", " valid_dataset=dataset.validation_dataset,\n", " batch_size=batch_size,\n", " optimize_batch_size=optimize_batch_size,\n", " freeze_backbone=args.freeze_backbone,\n", " learning_rate=args.learning_rate,\n", " num_epochs=args.num_epochs,\n", " fewshot_fraction=args.fewshot_fraction,\n", " fewshot_location=args.fewshot_location,\n", " automate_fewshot_fraction=args.automate_fewshot_fraction,\n", " automate_fewshot_fraction_threshold=args.automate_fewshot_fraction_threshold,\n", " )\n", " finetune_success = True\n", " finetune_train_num_samples = predictor.train_num_samples\n", " finetune_valid_num_samples = predictor.valid_num_samples\n", " except Exception as e:\n", " print(\"Error in finetune workflow. Error =\", e)\n", " print(\"Fallback to zero-shot performance.\")\n", " finetune_success = False\n", "\n", " # Evaluate\n", " res = evaluate_model(\n", " predictor,\n", " test_data=dataset.test_data,\n", " metrics=metrics,\n", " batch_size=batch_size,\n", " axis=None,\n", " mask_invalid_label=True,\n", " allow_nan_forecast=False,\n", " seasonality=season_length,\n", " )\n", "\n", " # Append the results to the CSV file\n", " with open(csv_file_path, \"a\", newline=\"\") as csvfile:\n", " writer = csv.writer(csvfile)\n", " writer.writerow(\n", " [\n", " ds_config,\n", " \"TTM\",\n", " res[\"MSE[mean]\"][0],\n", " res[\"MSE[0.5]\"][0],\n", " res[\"MAE[mean]\"][0],\n", " res[\"MAE[0.5]\"][0],\n", " res[\"MASE[0.5]\"][0],\n", " res[\"MAPE[0.5]\"][0],\n", " res[\"sMAPE[0.5]\"][0],\n", " res[\"MSIS\"][0],\n", " res[\"RMSE[mean]\"][0],\n", " res[\"NRMSE[mean]\"][0],\n", " res[\"ND[0.5]\"][0],\n", " res[\"mean_weighted_sum_quantile_loss\"][0],\n", " dataset_properties_map[ds_key][\"domain\"],\n", " dataset_properties_map[ds_key][\"num_variates\"],\n", " dataset.prediction_length,\n", " predictor.ttm.config.context_length,\n", " min_context_length,\n", " finetune_success,\n", " finetune_train_num_samples,\n", " finetune_valid_num_samples,\n", " ]\n", " )\n", "\n", " print(f\"Results for {ds_name} have been written to {csv_file_path}\")" ] }, { "cell_type": "code", "execution_count": 9, "id": "ffd629a1", "metadata": { "title": "Save results" }, "outputs": [ { "data": { "text/html": [ "

\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
dataseteval_metrics/MASE[0.5]eval_metrics/NRMSE[mean]eval_metrics/mean_weighted_sum_quantile_loss
0saugeen/M/short0.7543080.6224160.340212
1us_births/D/short0.3884760.0410060.020413
\n", "
" ], "text/plain": [ " dataset eval_metrics/MASE[0.5] eval_metrics/NRMSE[mean] \n", "0 saugeen/M/short 0.754308 0.622416 \\\n", "1 us_births/D/short 0.388476 0.041006 \n", "\n", " eval_metrics/mean_weighted_sum_quantile_loss \n", "0 0.340212 \n", "1 0.020413 " ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Results\n", "df = pd.read_csv(f\"{OUT_DIR}/all_results.csv\")\n", "df = df.sort_values(by=\"dataset\")\n", "display(\n", " df[\n", " [\n", " \"dataset\",\n", " \"eval_metrics/MASE[0.5]\",\n", " \"eval_metrics/NRMSE[mean]\",\n", " \"eval_metrics/mean_weighted_sum_quantile_loss\",\n", " ]\n", " ]\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "08ee73d8-9843-4d25-9aae-58fab8534277", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "jupytext": { "cell_metadata_filter": "title,-all", "encoding": "# coding: utf-8", "executable": "/usr/bin/env python", "main_language": "python", "notebook_metadata_filter": "-all" }, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.4" } }, "nbformat": 4, "nbformat_minor": 5 }