{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Hierarchical time series\n",
"\n",
"[](https://mybinder.org/v2/gh/etna-team/etna/master?filepath=examples/303-hierarchical_pipeline.ipynb)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This notebook contains examples of modelling hierarchical time series.\n",
"\n",
"**Table of contents**\n",
"\n",
"* [Hierarchical time series](#chapter1)\n",
"* [Preparing dataset](#chapter2)\n",
" * [Manually setting hierarchical structure](#chapter2_1)\n",
" * [Hierarchical structure detection](#chapter2_2)\n",
"* [Reconciliation methods](#chapter3)\n",
" * [Bottom-up approach](#chapter3_1)\n",
" * [Top-down approach](#chapter3_2)\n",
"* [Exogenous variables for hierarchical forecasts](#chapter4)"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import warnings\n",
"\n",
"warnings.filterwarnings(\"ignore\")"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"\n",
"from etna.metrics import SMAPE\n",
"from etna.models import LinearPerSegmentModel\n",
"from etna.pipeline import HierarchicalPipeline\n",
"from etna.pipeline import Pipeline\n",
"from etna.transforms import LagTransform\n",
"from etna.transforms import OneHotEncoderTransform\n",
"\n",
"pd.options.display.max_columns = 100"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. Hierarchical time series <a class=\"anchor\" id=\"chapter1\"></a>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In many applications time series have a natural level structure. Time series with such properties can be disaggregated by attributes\n",
"from lower levels. On the other hand, this time series can be aggregated to higher levels to represent more general relations.\n",
"The set of possible levels forms the hierarchy of time series."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n",
"*Two level hierarchical structure*"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Image above represents relations between members of the hierarchy. Middle and top levels can be disaggregated using members from\n",
"lower levels. For example\n",
"\n",
"$$\n",
"y_{A,t} = y_{AA,t} + y_{AB,t}\n",
"$$\n",
"\n",
"$$\n",
"y_{t} = y_{A,t} + y_{B,t}\n",
"$$\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In matrix notation level aggregation could be written as\n",
"\n",
"$$\n",
"\\begin{equation}\n",
" \\begin{bmatrix}\n",
" y_{A,t} \\\\\n",
" y_t\n",
" \\end{bmatrix}\n",
" =\n",
" \\begin{bmatrix}\n",
" 1 & 1 & 0 \\\\\n",
" 1 & 1 & 1\n",
" \\end{bmatrix}\n",
" \\begin{bmatrix}\n",
" y_{AA,t} \\\\ y_{AB,t} \\\\ y_{B,t}\n",
" \\end{bmatrix}\n",
" =\n",
" S\n",
" \\begin{bmatrix}\n",
" y_{AA,t} \\\\ y_{AB,t} \\\\ y_{B,t}\n",
" \\end{bmatrix}\n",
"\\end{equation}\n",
"$$\n",
"\n",
"where $S$ - summing matrix."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. Preparing dataset <a class=\"anchor\" id=\"chapter2\"></a>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Consider the Australian tourism dataset.\n",
"\n",
"This dataset consists of the following components:\n",
"\n",
"* `Total` - total domestic tourism demand,\n",
"* Tourism reasons components (`Hol` for holiday, `Bus` for business, etc)\n",
"* Components representing the \"region-reason\" division (`NSW - hol`, `NSW - bus`, etc)\n",
"* Components representing \"region - reason - city\" division (`NSW - hol - city`, `NSW - hol - noncity`, etc)\n",
"\n",
"We can see that these components form a hierarchy with the following levels:\n",
"\n",
"1. Total\n",
"2. Tourism reason\n",
"3. Region\n",
"4. City"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" % Total % Received % Xferd Average Speed Time Time Time Current\n",
" Dload Upload Total Spent Left Speed\n",
"100 15664 100 15664 0 0 16533 0 --:--:-- --:--:-- --:--:-- 16646\n"
]
}
],
"source": [
"!curl \"https://robjhyndman.com/data/hier1_with_names.csv\" --ssl-no-revoke -o \"hier1_with_names.csv\""
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Total</th>\n",
" <th>Hol</th>\n",
" <th>VFR</th>\n",
" <th>Bus</th>\n",
" <th>Oth</th>\n",
" <th>NSW - hol</th>\n",
" <th>VIC - hol</th>\n",
" <th>QLD - hol</th>\n",
" <th>SA - hol</th>\n",
" <th>WA - hol</th>\n",
" <th>TAS - hol</th>\n",
" <th>NT - hol</th>\n",
" <th>NSW - vfr</th>\n",
" <th>VIC - vfr</th>\n",
" <th>QLD - vfr</th>\n",
" <th>SA - vfr</th>\n",
" <th>WA - vfr</th>\n",
" <th>TAS - vfr</th>\n",
" <th>NT - vfr</th>\n",
" <th>NSW - bus</th>\n",
" <th>VIC - bus</th>\n",
" <th>QLD - bus</th>\n",
" <th>SA - bus</th>\n",
" <th>WA - bus</th>\n",
" <th>TAS - bus</th>\n",
" <th>NT - bus</th>\n",
" <th>NSW - oth</th>\n",
" <th>VIC - oth</th>\n",
" <th>QLD - oth</th>\n",
" <th>SA - oth</th>\n",
" <th>WA - oth</th>\n",
" <th>TAS - oth</th>\n",
" <th>NT - oth</th>\n",
" <th>NSW - hol - city</th>\n",
" <th>NSW - hol - noncity</th>\n",
" <th>VIC - hol - city</th>\n",
" <th>VIC - hol - noncity</th>\n",
" <th>QLD - hol - city</th>\n",
" <th>QLD - hol - noncity</th>\n",
" <th>SA - hol - city</th>\n",
" <th>SA - hol - noncity</th>\n",
" <th>WA - hol - city</th>\n",
" <th>WA - hol - noncity</th>\n",
" <th>TAS - hol - city</th>\n",
" <th>TAS - hol - noncity</th>\n",
" <th>NT - hol - city</th>\n",
" <th>NT - hol - noncity</th>\n",
" <th>NSW - vfr - city</th>\n",
" <th>NSW - vfr - noncity</th>\n",
" <th>VIC - vfr - city</th>\n",
" <th>VIC - vfr - noncity</th>\n",
" <th>QLD - vfr - city</th>\n",
" <th>QLD - vfr - noncity</th>\n",
" <th>SA - vfr - city</th>\n",
" <th>SA - vfr - noncity</th>\n",
" <th>WA - vfr - city</th>\n",
" <th>WA - vfr - noncity</th>\n",
" <th>TAS - vfr - city</th>\n",
" <th>TAS - vfr - noncity</th>\n",
" <th>NT - vfr - city</th>\n",
" <th>NT - vfr - noncity</th>\n",
" <th>NSW - bus - city</th>\n",
" <th>NSW - bus - noncity</th>\n",
" <th>VIC - bus - city</th>\n",
" <th>VIC - bus - noncity</th>\n",
" <th>QLD - bus - city</th>\n",
" <th>QLD - bus - noncity</th>\n",
" <th>SA - bus - city</th>\n",
" <th>SA - bus - noncity</th>\n",
" <th>WA - bus - city</th>\n",
" <th>WA - bus - noncity</th>\n",
" <th>TAS - bus - city</th>\n",
" <th>TAS - bus - noncity</th>\n",
" <th>NT - bus - city</th>\n",
" <th>NT - bus - noncity</th>\n",
" <th>NSW - oth - city</th>\n",
" <th>NSW - oth - noncity</th>\n",
" <th>VIC - oth - city</th>\n",
" <th>VIC - oth - noncity</th>\n",
" <th>QLD - oth - city</th>\n",
" <th>QLD - oth - noncity</th>\n",
" <th>SA - oth - city</th>\n",
" <th>SA - oth - noncity</th>\n",
" <th>WA - oth - city</th>\n",
" <th>WA - oth - noncity</th>\n",
" <th>TAS - oth - city</th>\n",
" <th>TAS - oth - noncity</th>\n",
" <th>NT - oth - city</th>\n",
" <th>NT - oth - noncity</th>\n",
" </tr>\n",
" <tr>\n",
" <th>timestamp</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>2006-01-01</th>\n",
" <td>84503</td>\n",
" <td>45906</td>\n",
" <td>26042</td>\n",
" <td>9815</td>\n",
" <td>2740</td>\n",
" <td>17589</td>\n",
" <td>10412</td>\n",
" <td>9078</td>\n",
" <td>3089</td>\n",
" <td>3449</td>\n",
" <td>2102</td>\n",
" <td>187</td>\n",
" <td>9398</td>\n",
" <td>5993</td>\n",
" <td>5290</td>\n",
" <td>2193</td>\n",
" <td>1781</td>\n",
" <td>1350</td>\n",
" <td>37</td>\n",
" <td>2885</td>\n",
" <td>2148</td>\n",
" <td>2093</td>\n",
" <td>844</td>\n",
" <td>1406</td>\n",
" <td>223</td>\n",
" <td>216</td>\n",
" <td>906</td>\n",
" <td>467</td>\n",
" <td>702</td>\n",
" <td>317</td>\n",
" <td>205</td>\n",
" <td>100</td>\n",
" <td>43</td>\n",
" <td>3096</td>\n",
" <td>14493</td>\n",
" <td>2531</td>\n",
" <td>7881</td>\n",
" <td>4688</td>\n",
" <td>4390</td>\n",
" <td>888</td>\n",
" <td>2201</td>\n",
" <td>1383</td>\n",
" <td>2066</td>\n",
" <td>619</td>\n",
" <td>1483</td>\n",
" <td>101</td>\n",
" <td>86</td>\n",
" <td>2709</td>\n",
" <td>6689</td>\n",
" <td>2565</td>\n",
" <td>3428</td>\n",
" <td>3003</td>\n",
" <td>2287</td>\n",
" <td>1324</td>\n",
" <td>869</td>\n",
" <td>1019</td>\n",
" <td>762</td>\n",
" <td>602</td>\n",
" <td>748</td>\n",
" <td>28</td>\n",
" <td>9</td>\n",
" <td>1201</td>\n",
" <td>1684</td>\n",
" <td>1164</td>\n",
" <td>984</td>\n",
" <td>1111</td>\n",
" <td>982</td>\n",
" <td>388</td>\n",
" <td>456</td>\n",
" <td>532</td>\n",
" <td>874</td>\n",
" <td>116</td>\n",
" <td>107</td>\n",
" <td>136</td>\n",
" <td>80</td>\n",
" <td>396</td>\n",
" <td>510</td>\n",
" <td>181</td>\n",
" <td>286</td>\n",
" <td>431</td>\n",
" <td>271</td>\n",
" <td>244</td>\n",
" <td>73</td>\n",
" <td>168</td>\n",
" <td>37</td>\n",
" <td>76</td>\n",
" <td>24</td>\n",
" <td>35</td>\n",
" <td>8</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-02-01</th>\n",
" <td>65312</td>\n",
" <td>29347</td>\n",
" <td>20676</td>\n",
" <td>11823</td>\n",
" <td>3466</td>\n",
" <td>11027</td>\n",
" <td>6025</td>\n",
" <td>6310</td>\n",
" <td>1935</td>\n",
" <td>2454</td>\n",
" <td>1098</td>\n",
" <td>498</td>\n",
" <td>7829</td>\n",
" <td>4107</td>\n",
" <td>4902</td>\n",
" <td>1445</td>\n",
" <td>1353</td>\n",
" <td>523</td>\n",
" <td>517</td>\n",
" <td>4301</td>\n",
" <td>1825</td>\n",
" <td>2224</td>\n",
" <td>749</td>\n",
" <td>2043</td>\n",
" <td>373</td>\n",
" <td>308</td>\n",
" <td>1238</td>\n",
" <td>552</td>\n",
" <td>839</td>\n",
" <td>363</td>\n",
" <td>269</td>\n",
" <td>97</td>\n",
" <td>108</td>\n",
" <td>1479</td>\n",
" <td>9548</td>\n",
" <td>1439</td>\n",
" <td>4586</td>\n",
" <td>2320</td>\n",
" <td>3990</td>\n",
" <td>521</td>\n",
" <td>1414</td>\n",
" <td>1059</td>\n",
" <td>1395</td>\n",
" <td>409</td>\n",
" <td>689</td>\n",
" <td>201</td>\n",
" <td>297</td>\n",
" <td>2184</td>\n",
" <td>5645</td>\n",
" <td>1852</td>\n",
" <td>2255</td>\n",
" <td>1957</td>\n",
" <td>2945</td>\n",
" <td>806</td>\n",
" <td>639</td>\n",
" <td>750</td>\n",
" <td>603</td>\n",
" <td>257</td>\n",
" <td>266</td>\n",
" <td>168</td>\n",
" <td>349</td>\n",
" <td>2020</td>\n",
" <td>2281</td>\n",
" <td>1014</td>\n",
" <td>811</td>\n",
" <td>776</td>\n",
" <td>1448</td>\n",
" <td>346</td>\n",
" <td>403</td>\n",
" <td>356</td>\n",
" <td>1687</td>\n",
" <td>83</td>\n",
" <td>290</td>\n",
" <td>138</td>\n",
" <td>170</td>\n",
" <td>657</td>\n",
" <td>581</td>\n",
" <td>229</td>\n",
" <td>323</td>\n",
" <td>669</td>\n",
" <td>170</td>\n",
" <td>142</td>\n",
" <td>221</td>\n",
" <td>170</td>\n",
" <td>99</td>\n",
" <td>36</td>\n",
" <td>61</td>\n",
" <td>69</td>\n",
" <td>39</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-03-01</th>\n",
" <td>72753</td>\n",
" <td>32492</td>\n",
" <td>20582</td>\n",
" <td>13565</td>\n",
" <td>6114</td>\n",
" <td>8910</td>\n",
" <td>5060</td>\n",
" <td>11733</td>\n",
" <td>1569</td>\n",
" <td>3398</td>\n",
" <td>458</td>\n",
" <td>1364</td>\n",
" <td>7277</td>\n",
" <td>3811</td>\n",
" <td>5489</td>\n",
" <td>1453</td>\n",
" <td>1687</td>\n",
" <td>391</td>\n",
" <td>474</td>\n",
" <td>4093</td>\n",
" <td>1944</td>\n",
" <td>3379</td>\n",
" <td>750</td>\n",
" <td>1560</td>\n",
" <td>303</td>\n",
" <td>1536</td>\n",
" <td>1433</td>\n",
" <td>446</td>\n",
" <td>1434</td>\n",
" <td>712</td>\n",
" <td>1546</td>\n",
" <td>55</td>\n",
" <td>488</td>\n",
" <td>1609</td>\n",
" <td>7301</td>\n",
" <td>1488</td>\n",
" <td>3572</td>\n",
" <td>4758</td>\n",
" <td>6975</td>\n",
" <td>476</td>\n",
" <td>1093</td>\n",
" <td>1101</td>\n",
" <td>2297</td>\n",
" <td>127</td>\n",
" <td>331</td>\n",
" <td>619</td>\n",
" <td>745</td>\n",
" <td>2225</td>\n",
" <td>5052</td>\n",
" <td>1882</td>\n",
" <td>1929</td>\n",
" <td>2619</td>\n",
" <td>2870</td>\n",
" <td>1078</td>\n",
" <td>375</td>\n",
" <td>953</td>\n",
" <td>734</td>\n",
" <td>130</td>\n",
" <td>261</td>\n",
" <td>390</td>\n",
" <td>84</td>\n",
" <td>1975</td>\n",
" <td>2118</td>\n",
" <td>1153</td>\n",
" <td>791</td>\n",
" <td>1079</td>\n",
" <td>2300</td>\n",
" <td>390</td>\n",
" <td>360</td>\n",
" <td>440</td>\n",
" <td>1120</td>\n",
" <td>196</td>\n",
" <td>107</td>\n",
" <td>452</td>\n",
" <td>1084</td>\n",
" <td>540</td>\n",
" <td>893</td>\n",
" <td>128</td>\n",
" <td>318</td>\n",
" <td>270</td>\n",
" <td>1164</td>\n",
" <td>397</td>\n",
" <td>315</td>\n",
" <td>380</td>\n",
" <td>1166</td>\n",
" <td>32</td>\n",
" <td>23</td>\n",
" <td>150</td>\n",
" <td>338</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-04-01</th>\n",
" <td>70880</td>\n",
" <td>31813</td>\n",
" <td>21613</td>\n",
" <td>11478</td>\n",
" <td>5976</td>\n",
" <td>10658</td>\n",
" <td>5481</td>\n",
" <td>8109</td>\n",
" <td>2270</td>\n",
" <td>3561</td>\n",
" <td>1320</td>\n",
" <td>414</td>\n",
" <td>8303</td>\n",
" <td>5090</td>\n",
" <td>4441</td>\n",
" <td>1209</td>\n",
" <td>1714</td>\n",
" <td>394</td>\n",
" <td>462</td>\n",
" <td>3463</td>\n",
" <td>1753</td>\n",
" <td>2880</td>\n",
" <td>890</td>\n",
" <td>1791</td>\n",
" <td>298</td>\n",
" <td>403</td>\n",
" <td>1902</td>\n",
" <td>606</td>\n",
" <td>749</td>\n",
" <td>454</td>\n",
" <td>1549</td>\n",
" <td>91</td>\n",
" <td>625</td>\n",
" <td>1520</td>\n",
" <td>9138</td>\n",
" <td>1906</td>\n",
" <td>3575</td>\n",
" <td>3328</td>\n",
" <td>4781</td>\n",
" <td>571</td>\n",
" <td>1699</td>\n",
" <td>1128</td>\n",
" <td>2433</td>\n",
" <td>371</td>\n",
" <td>949</td>\n",
" <td>164</td>\n",
" <td>250</td>\n",
" <td>2918</td>\n",
" <td>5385</td>\n",
" <td>2208</td>\n",
" <td>2882</td>\n",
" <td>2097</td>\n",
" <td>2344</td>\n",
" <td>568</td>\n",
" <td>641</td>\n",
" <td>999</td>\n",
" <td>715</td>\n",
" <td>137</td>\n",
" <td>257</td>\n",
" <td>244</td>\n",
" <td>218</td>\n",
" <td>1500</td>\n",
" <td>1963</td>\n",
" <td>1245</td>\n",
" <td>508</td>\n",
" <td>1128</td>\n",
" <td>1752</td>\n",
" <td>255</td>\n",
" <td>635</td>\n",
" <td>539</td>\n",
" <td>1252</td>\n",
" <td>70</td>\n",
" <td>228</td>\n",
" <td>243</td>\n",
" <td>160</td>\n",
" <td>745</td>\n",
" <td>1157</td>\n",
" <td>270</td>\n",
" <td>336</td>\n",
" <td>214</td>\n",
" <td>535</td>\n",
" <td>194</td>\n",
" <td>260</td>\n",
" <td>410</td>\n",
" <td>1139</td>\n",
" <td>48</td>\n",
" <td>43</td>\n",
" <td>172</td>\n",
" <td>453</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-05-01</th>\n",
" <td>86893</td>\n",
" <td>46793</td>\n",
" <td>26947</td>\n",
" <td>10027</td>\n",
" <td>3126</td>\n",
" <td>16152</td>\n",
" <td>10958</td>\n",
" <td>10047</td>\n",
" <td>3023</td>\n",
" <td>4287</td>\n",
" <td>2113</td>\n",
" <td>213</td>\n",
" <td>10386</td>\n",
" <td>6152</td>\n",
" <td>5636</td>\n",
" <td>1685</td>\n",
" <td>2026</td>\n",
" <td>784</td>\n",
" <td>278</td>\n",
" <td>3347</td>\n",
" <td>1522</td>\n",
" <td>2751</td>\n",
" <td>666</td>\n",
" <td>1023</td>\n",
" <td>335</td>\n",
" <td>383</td>\n",
" <td>984</td>\n",
" <td>558</td>\n",
" <td>1015</td>\n",
" <td>180</td>\n",
" <td>190</td>\n",
" <td>137</td>\n",
" <td>62</td>\n",
" <td>1958</td>\n",
" <td>14194</td>\n",
" <td>2517</td>\n",
" <td>8441</td>\n",
" <td>4930</td>\n",
" <td>5117</td>\n",
" <td>873</td>\n",
" <td>2150</td>\n",
" <td>1560</td>\n",
" <td>2727</td>\n",
" <td>523</td>\n",
" <td>1590</td>\n",
" <td>62</td>\n",
" <td>151</td>\n",
" <td>3154</td>\n",
" <td>7232</td>\n",
" <td>2988</td>\n",
" <td>3164</td>\n",
" <td>2703</td>\n",
" <td>2933</td>\n",
" <td>887</td>\n",
" <td>798</td>\n",
" <td>1396</td>\n",
" <td>630</td>\n",
" <td>347</td>\n",
" <td>437</td>\n",
" <td>153</td>\n",
" <td>125</td>\n",
" <td>1196</td>\n",
" <td>2151</td>\n",
" <td>950</td>\n",
" <td>572</td>\n",
" <td>1192</td>\n",
" <td>1559</td>\n",
" <td>386</td>\n",
" <td>280</td>\n",
" <td>582</td>\n",
" <td>441</td>\n",
" <td>130</td>\n",
" <td>205</td>\n",
" <td>194</td>\n",
" <td>189</td>\n",
" <td>426</td>\n",
" <td>558</td>\n",
" <td>265</td>\n",
" <td>293</td>\n",
" <td>458</td>\n",
" <td>557</td>\n",
" <td>147</td>\n",
" <td>33</td>\n",
" <td>162</td>\n",
" <td>28</td>\n",
" <td>77</td>\n",
" <td>60</td>\n",
" <td>15</td>\n",
" <td>47</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Total Hol VFR Bus Oth NSW - hol VIC - hol QLD - hol \\\n",
"timestamp \n",
"2006-01-01 84503 45906 26042 9815 2740 17589 10412 9078 \n",
"2006-02-01 65312 29347 20676 11823 3466 11027 6025 6310 \n",
"2006-03-01 72753 32492 20582 13565 6114 8910 5060 11733 \n",
"2006-04-01 70880 31813 21613 11478 5976 10658 5481 8109 \n",
"2006-05-01 86893 46793 26947 10027 3126 16152 10958 10047 \n",
"\n",
" SA - hol WA - hol TAS - hol NT - hol NSW - vfr VIC - vfr \\\n",
"timestamp \n",
"2006-01-01 3089 3449 2102 187 9398 5993 \n",
"2006-02-01 1935 2454 1098 498 7829 4107 \n",
"2006-03-01 1569 3398 458 1364 7277 3811 \n",
"2006-04-01 2270 3561 1320 414 8303 5090 \n",
"2006-05-01 3023 4287 2113 213 10386 6152 \n",
"\n",
" QLD - vfr SA - vfr WA - vfr TAS - vfr NT - vfr NSW - bus \\\n",
"timestamp \n",
"2006-01-01 5290 2193 1781 1350 37 2885 \n",
"2006-02-01 4902 1445 1353 523 517 4301 \n",
"2006-03-01 5489 1453 1687 391 474 4093 \n",
"2006-04-01 4441 1209 1714 394 462 3463 \n",
"2006-05-01 5636 1685 2026 784 278 3347 \n",
"\n",
" VIC - bus QLD - bus SA - bus WA - bus TAS - bus NT - bus \\\n",
"timestamp \n",
"2006-01-01 2148 2093 844 1406 223 216 \n",
"2006-02-01 1825 2224 749 2043 373 308 \n",
"2006-03-01 1944 3379 750 1560 303 1536 \n",
"2006-04-01 1753 2880 890 1791 298 403 \n",
"2006-05-01 1522 2751 666 1023 335 383 \n",
"\n",
" NSW - oth VIC - oth QLD - oth SA - oth WA - oth TAS - oth \\\n",
"timestamp \n",
"2006-01-01 906 467 702 317 205 100 \n",
"2006-02-01 1238 552 839 363 269 97 \n",
"2006-03-01 1433 446 1434 712 1546 55 \n",
"2006-04-01 1902 606 749 454 1549 91 \n",
"2006-05-01 984 558 1015 180 190 137 \n",
"\n",
" NT - oth NSW - hol - city NSW - hol - noncity VIC - hol - city \\\n",
"timestamp \n",
"2006-01-01 43 3096 14493 2531 \n",
"2006-02-01 108 1479 9548 1439 \n",
"2006-03-01 488 1609 7301 1488 \n",
"2006-04-01 625 1520 9138 1906 \n",
"2006-05-01 62 1958 14194 2517 \n",
"\n",
" VIC - hol - noncity QLD - hol - city QLD - hol - noncity \\\n",
"timestamp \n",
"2006-01-01 7881 4688 4390 \n",
"2006-02-01 4586 2320 3990 \n",
"2006-03-01 3572 4758 6975 \n",
"2006-04-01 3575 3328 4781 \n",
"2006-05-01 8441 4930 5117 \n",
"\n",
" SA - hol - city SA - hol - noncity WA - hol - city \\\n",
"timestamp \n",
"2006-01-01 888 2201 1383 \n",
"2006-02-01 521 1414 1059 \n",
"2006-03-01 476 1093 1101 \n",
"2006-04-01 571 1699 1128 \n",
"2006-05-01 873 2150 1560 \n",
"\n",
" WA - hol - noncity TAS - hol - city TAS - hol - noncity \\\n",
"timestamp \n",
"2006-01-01 2066 619 1483 \n",
"2006-02-01 1395 409 689 \n",
"2006-03-01 2297 127 331 \n",
"2006-04-01 2433 371 949 \n",
"2006-05-01 2727 523 1590 \n",
"\n",
" NT - hol - city NT - hol - noncity NSW - vfr - city \\\n",
"timestamp \n",
"2006-01-01 101 86 2709 \n",
"2006-02-01 201 297 2184 \n",
"2006-03-01 619 745 2225 \n",
"2006-04-01 164 250 2918 \n",
"2006-05-01 62 151 3154 \n",
"\n",
" NSW - vfr - noncity VIC - vfr - city VIC - vfr - noncity \\\n",
"timestamp \n",
"2006-01-01 6689 2565 3428 \n",
"2006-02-01 5645 1852 2255 \n",
"2006-03-01 5052 1882 1929 \n",
"2006-04-01 5385 2208 2882 \n",
"2006-05-01 7232 2988 3164 \n",
"\n",
" QLD - vfr - city QLD - vfr - noncity SA - vfr - city \\\n",
"timestamp \n",
"2006-01-01 3003 2287 1324 \n",
"2006-02-01 1957 2945 806 \n",
"2006-03-01 2619 2870 1078 \n",
"2006-04-01 2097 2344 568 \n",
"2006-05-01 2703 2933 887 \n",
"\n",
" SA - vfr - noncity WA - vfr - city WA - vfr - noncity \\\n",
"timestamp \n",
"2006-01-01 869 1019 762 \n",
"2006-02-01 639 750 603 \n",
"2006-03-01 375 953 734 \n",
"2006-04-01 641 999 715 \n",
"2006-05-01 798 1396 630 \n",
"\n",
" TAS - vfr - city TAS - vfr - noncity NT - vfr - city \\\n",
"timestamp \n",
"2006-01-01 602 748 28 \n",
"2006-02-01 257 266 168 \n",
"2006-03-01 130 261 390 \n",
"2006-04-01 137 257 244 \n",
"2006-05-01 347 437 153 \n",
"\n",
" NT - vfr - noncity NSW - bus - city NSW - bus - noncity \\\n",
"timestamp \n",
"2006-01-01 9 1201 1684 \n",
"2006-02-01 349 2020 2281 \n",
"2006-03-01 84 1975 2118 \n",
"2006-04-01 218 1500 1963 \n",
"2006-05-01 125 1196 2151 \n",
"\n",
" VIC - bus - city VIC - bus - noncity QLD - bus - city \\\n",
"timestamp \n",
"2006-01-01 1164 984 1111 \n",
"2006-02-01 1014 811 776 \n",
"2006-03-01 1153 791 1079 \n",
"2006-04-01 1245 508 1128 \n",
"2006-05-01 950 572 1192 \n",
"\n",
" QLD - bus - noncity SA - bus - city SA - bus - noncity \\\n",
"timestamp \n",
"2006-01-01 982 388 456 \n",
"2006-02-01 1448 346 403 \n",
"2006-03-01 2300 390 360 \n",
"2006-04-01 1752 255 635 \n",
"2006-05-01 1559 386 280 \n",
"\n",
" WA - bus - city WA - bus - noncity TAS - bus - city \\\n",
"timestamp \n",
"2006-01-01 532 874 116 \n",
"2006-02-01 356 1687 83 \n",
"2006-03-01 440 1120 196 \n",
"2006-04-01 539 1252 70 \n",
"2006-05-01 582 441 130 \n",
"\n",
" TAS - bus - noncity NT - bus - city NT - bus - noncity \\\n",
"timestamp \n",
"2006-01-01 107 136 80 \n",
"2006-02-01 290 138 170 \n",
"2006-03-01 107 452 1084 \n",
"2006-04-01 228 243 160 \n",
"2006-05-01 205 194 189 \n",
"\n",
" NSW - oth - city NSW - oth - noncity VIC - oth - city \\\n",
"timestamp \n",
"2006-01-01 396 510 181 \n",
"2006-02-01 657 581 229 \n",
"2006-03-01 540 893 128 \n",
"2006-04-01 745 1157 270 \n",
"2006-05-01 426 558 265 \n",
"\n",
" VIC - oth - noncity QLD - oth - city QLD - oth - noncity \\\n",
"timestamp \n",
"2006-01-01 286 431 271 \n",
"2006-02-01 323 669 170 \n",
"2006-03-01 318 270 1164 \n",
"2006-04-01 336 214 535 \n",
"2006-05-01 293 458 557 \n",
"\n",
" SA - oth - city SA - oth - noncity WA - oth - city \\\n",
"timestamp \n",
"2006-01-01 244 73 168 \n",
"2006-02-01 142 221 170 \n",
"2006-03-01 397 315 380 \n",
"2006-04-01 194 260 410 \n",
"2006-05-01 147 33 162 \n",
"\n",
" WA - oth - noncity TAS - oth - city TAS - oth - noncity \\\n",
"timestamp \n",
"2006-01-01 37 76 24 \n",
"2006-02-01 99 36 61 \n",
"2006-03-01 1166 32 23 \n",
"2006-04-01 1139 48 43 \n",
"2006-05-01 28 77 60 \n",
"\n",
" NT - oth - city NT - oth - noncity \n",
"timestamp \n",
"2006-01-01 35 8 \n",
"2006-02-01 69 39 \n",
"2006-03-01 150 338 \n",
"2006-04-01 172 453 \n",
"2006-05-01 15 47 "
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df = pd.read_csv(\"hier1_with_names.csv\")\n",
"\n",
"periods = len(df)\n",
"df[\"timestamp\"] = pd.date_range(\"2006-01-01\", periods=periods, freq=\"MS\")\n",
"df.set_index(\"timestamp\", inplace=True)\n",
"\n",
"df.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 2.1 Manually setting hierarchical structure <a class=\"anchor\" id=\"chapter2_1\"></a>\n",
"This section presents how to set hierarchical structure and prepare data. We are going to create a hierarchical\n",
"dataset with two levels: total demand and demand per tourism reason."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"from etna.datasets import TSDataset"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Consider the **Reason** level of the hierarchy."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Hol</th>\n",
" <th>VFR</th>\n",
" <th>Bus</th>\n",
" <th>Oth</th>\n",
" </tr>\n",
" <tr>\n",
" <th>timestamp</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>2006-01-01</th>\n",
" <td>45906</td>\n",
" <td>26042</td>\n",
" <td>9815</td>\n",
" <td>2740</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-02-01</th>\n",
" <td>29347</td>\n",
" <td>20676</td>\n",
" <td>11823</td>\n",
" <td>3466</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-03-01</th>\n",
" <td>32492</td>\n",
" <td>20582</td>\n",
" <td>13565</td>\n",
" <td>6114</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-04-01</th>\n",
" <td>31813</td>\n",
" <td>21613</td>\n",
" <td>11478</td>\n",
" <td>5976</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-05-01</th>\n",
" <td>46793</td>\n",
" <td>26947</td>\n",
" <td>10027</td>\n",
" <td>3126</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Hol VFR Bus Oth\n",
"timestamp \n",
"2006-01-01 45906 26042 9815 2740\n",
"2006-02-01 29347 20676 11823 3466\n",
"2006-03-01 32492 20582 13565 6114\n",
"2006-04-01 31813 21613 11478 5976\n",
"2006-05-01 46793 26947 10027 3126"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"reason_segments = [\"Hol\", \"VFR\", \"Bus\", \"Oth\"]\n",
"\n",
"df[reason_segments].head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 2.1.1 Convert dataset to ETNA wide format <a class=\"anchor\" id=\"chapter2_1_1\"></a>\n",
"First, convert dataframe to ETNA long format."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>timestamp</th>\n",
" <th>target</th>\n",
" <th>segment</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>2006-01-01</td>\n",
" <td>45906</td>\n",
" <td>Hol</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2006-02-01</td>\n",
" <td>29347</td>\n",
" <td>Hol</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>2006-03-01</td>\n",
" <td>32492</td>\n",
" <td>Hol</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>2006-04-01</td>\n",
" <td>31813</td>\n",
" <td>Hol</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>2006-05-01</td>\n",
" <td>46793</td>\n",
" <td>Hol</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" timestamp target segment\n",
"0 2006-01-01 45906 Hol\n",
"1 2006-02-01 29347 Hol\n",
"2 2006-03-01 32492 Hol\n",
"3 2006-04-01 31813 Hol\n",
"4 2006-05-01 46793 Hol"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"hierarchical_df = []\n",
"for segment_name in reason_segments:\n",
" segment = df[segment_name]\n",
"\n",
" segment_slice = pd.DataFrame(\n",
" {\"timestamp\": segment.index, \"target\": segment.values, \"segment\": [segment_name] * periods}\n",
" )\n",
" hierarchical_df.append(segment_slice)\n",
"\n",
"hierarchical_df = pd.concat(hierarchical_df, axis=0)\n",
"\n",
"hierarchical_df.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, the dataframe could be converted to ETNA wide format."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"hierarchical_df = TSDataset.to_dataset(df=hierarchical_df)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 2.1.2 Creat HierarchicalStructure <a class=\"anchor\" id=\"chapter2_1_2\"></a>\n",
"For handling information about hierarchical structure, there is a dedicated object in the ETNA library: `HierarchicalStructure`.\n",
"\n",
"To create `HierarchicalStructure` define relationships between segments at different levels. This relation should be\n",
"described as mapping between levels members, where keys are parent segments and values are lists of child segments\n",
"from the lower level. Also provide a list of level names, where ordering corresponds to hierarchical relationships\n",
"between levels."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
"from etna.datasets import HierarchicalStructure"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"HierarchicalStructure(level_structure = {'total': ['Hol', 'VFR', 'Bus', 'Oth']}, level_names = ['total', 'reason'], )"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"hierarchical_structure = HierarchicalStructure(\n",
" level_structure={\"total\": [\"Hol\", \"VFR\", \"Bus\", \"Oth\"]}, level_names=[\"total\", \"reason\"]\n",
")\n",
"\n",
"hierarchical_structure"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 2.1.3 Create hierarchical dataset <a class=\"anchor\" id=\"chapter2_1_3\"></a>\n",
"When all the data is prepared, call the `TSDataset` constructor to create a hierarchical dataset."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead tr th {\n",
" text-align: left;\n",
" }\n",
"\n",
" .dataframe thead tr:last-of-type th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr>\n",
" <th>segment</th>\n",
" <th>Bus</th>\n",
" <th>Hol</th>\n",
" <th>Oth</th>\n",
" <th>VFR</th>\n",
" </tr>\n",
" <tr>\n",
" <th>feature</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" </tr>\n",
" <tr>\n",
" <th>timestamp</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>2006-01-01</th>\n",
" <td>9815</td>\n",
" <td>45906</td>\n",
" <td>2740</td>\n",
" <td>26042</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-02-01</th>\n",
" <td>11823</td>\n",
" <td>29347</td>\n",
" <td>3466</td>\n",
" <td>20676</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-03-01</th>\n",
" <td>13565</td>\n",
" <td>32492</td>\n",
" <td>6114</td>\n",
" <td>20582</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-04-01</th>\n",
" <td>11478</td>\n",
" <td>31813</td>\n",
" <td>5976</td>\n",
" <td>21613</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-05-01</th>\n",
" <td>10027</td>\n",
" <td>46793</td>\n",
" <td>3126</td>\n",
" <td>26947</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
"segment Bus Hol Oth VFR\n",
"feature target target target target\n",
"timestamp \n",
"2006-01-01 9815 45906 2740 26042\n",
"2006-02-01 11823 29347 3466 20676\n",
"2006-03-01 13565 32492 6114 20582\n",
"2006-04-01 11478 31813 5976 21613\n",
"2006-05-01 10027 46793 3126 26947"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"hierarchical_ts = TSDataset(df=hierarchical_df, freq=\"MS\", hierarchical_structure=hierarchical_structure)\n",
"\n",
"hierarchical_ts.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Ensure that the dataset is at the desired level."
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'reason'"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"hierarchical_ts.current_df_level"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 2.2 Hierarchical structure detection <a class=\"anchor\" id=\"chapter2_2\"></a>\n",
"\n",
"This section presents how to prepare data and detect hierarchical structure.\n",
"The main advantage of this approach for creating hierarchical structures is that you don't need to define an adjacency list.\n",
"All hierarchical relationships would be detected from the dataframe columns.\n",
"\n",
"The main applications for this approach are when defining the adjacency list is not desirable or when some columns of\n",
"the dataframe already have information about hierarchy (e.g. related categorical columns).\n",
"\n",
"A data frame must be prepared in a specific format for detection to work. The following sections show how to do so.\n",
"\n",
"Consider the City level of the hierarchy."
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>NSW - hol - city</th>\n",
" <th>NSW - hol - noncity</th>\n",
" <th>VIC - hol - city</th>\n",
" <th>VIC - hol - noncity</th>\n",
" <th>QLD - hol - city</th>\n",
" <th>QLD - hol - noncity</th>\n",
" <th>SA - hol - city</th>\n",
" <th>SA - hol - noncity</th>\n",
" <th>WA - hol - city</th>\n",
" <th>WA - hol - noncity</th>\n",
" <th>TAS - hol - city</th>\n",
" <th>TAS - hol - noncity</th>\n",
" <th>NT - hol - city</th>\n",
" <th>NT - hol - noncity</th>\n",
" <th>NSW - vfr - city</th>\n",
" <th>NSW - vfr - noncity</th>\n",
" <th>VIC - vfr - city</th>\n",
" <th>VIC - vfr - noncity</th>\n",
" <th>QLD - vfr - city</th>\n",
" <th>QLD - vfr - noncity</th>\n",
" <th>SA - vfr - city</th>\n",
" <th>SA - vfr - noncity</th>\n",
" <th>WA - vfr - city</th>\n",
" <th>WA - vfr - noncity</th>\n",
" <th>TAS - vfr - city</th>\n",
" <th>TAS - vfr - noncity</th>\n",
" <th>NT - vfr - city</th>\n",
" <th>NT - vfr - noncity</th>\n",
" <th>NSW - bus - city</th>\n",
" <th>NSW - bus - noncity</th>\n",
" <th>VIC - bus - city</th>\n",
" <th>VIC - bus - noncity</th>\n",
" <th>QLD - bus - city</th>\n",
" <th>QLD - bus - noncity</th>\n",
" <th>SA - bus - city</th>\n",
" <th>SA - bus - noncity</th>\n",
" <th>WA - bus - city</th>\n",
" <th>WA - bus - noncity</th>\n",
" <th>TAS - bus - city</th>\n",
" <th>TAS - bus - noncity</th>\n",
" <th>NT - bus - city</th>\n",
" <th>NT - bus - noncity</th>\n",
" <th>NSW - oth - city</th>\n",
" <th>NSW - oth - noncity</th>\n",
" <th>VIC - oth - city</th>\n",
" <th>VIC - oth - noncity</th>\n",
" <th>QLD - oth - city</th>\n",
" <th>QLD - oth - noncity</th>\n",
" <th>SA - oth - city</th>\n",
" <th>SA - oth - noncity</th>\n",
" <th>WA - oth - city</th>\n",
" <th>WA - oth - noncity</th>\n",
" <th>TAS - oth - city</th>\n",
" <th>TAS - oth - noncity</th>\n",
" <th>NT - oth - city</th>\n",
" <th>NT - oth - noncity</th>\n",
" </tr>\n",
" <tr>\n",
" <th>timestamp</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>2006-01-01</th>\n",
" <td>3096</td>\n",
" <td>14493</td>\n",
" <td>2531</td>\n",
" <td>7881</td>\n",
" <td>4688</td>\n",
" <td>4390</td>\n",
" <td>888</td>\n",
" <td>2201</td>\n",
" <td>1383</td>\n",
" <td>2066</td>\n",
" <td>619</td>\n",
" <td>1483</td>\n",
" <td>101</td>\n",
" <td>86</td>\n",
" <td>2709</td>\n",
" <td>6689</td>\n",
" <td>2565</td>\n",
" <td>3428</td>\n",
" <td>3003</td>\n",
" <td>2287</td>\n",
" <td>1324</td>\n",
" <td>869</td>\n",
" <td>1019</td>\n",
" <td>762</td>\n",
" <td>602</td>\n",
" <td>748</td>\n",
" <td>28</td>\n",
" <td>9</td>\n",
" <td>1201</td>\n",
" <td>1684</td>\n",
" <td>1164</td>\n",
" <td>984</td>\n",
" <td>1111</td>\n",
" <td>982</td>\n",
" <td>388</td>\n",
" <td>456</td>\n",
" <td>532</td>\n",
" <td>874</td>\n",
" <td>116</td>\n",
" <td>107</td>\n",
" <td>136</td>\n",
" <td>80</td>\n",
" <td>396</td>\n",
" <td>510</td>\n",
" <td>181</td>\n",
" <td>286</td>\n",
" <td>431</td>\n",
" <td>271</td>\n",
" <td>244</td>\n",
" <td>73</td>\n",
" <td>168</td>\n",
" <td>37</td>\n",
" <td>76</td>\n",
" <td>24</td>\n",
" <td>35</td>\n",
" <td>8</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-02-01</th>\n",
" <td>1479</td>\n",
" <td>9548</td>\n",
" <td>1439</td>\n",
" <td>4586</td>\n",
" <td>2320</td>\n",
" <td>3990</td>\n",
" <td>521</td>\n",
" <td>1414</td>\n",
" <td>1059</td>\n",
" <td>1395</td>\n",
" <td>409</td>\n",
" <td>689</td>\n",
" <td>201</td>\n",
" <td>297</td>\n",
" <td>2184</td>\n",
" <td>5645</td>\n",
" <td>1852</td>\n",
" <td>2255</td>\n",
" <td>1957</td>\n",
" <td>2945</td>\n",
" <td>806</td>\n",
" <td>639</td>\n",
" <td>750</td>\n",
" <td>603</td>\n",
" <td>257</td>\n",
" <td>266</td>\n",
" <td>168</td>\n",
" <td>349</td>\n",
" <td>2020</td>\n",
" <td>2281</td>\n",
" <td>1014</td>\n",
" <td>811</td>\n",
" <td>776</td>\n",
" <td>1448</td>\n",
" <td>346</td>\n",
" <td>403</td>\n",
" <td>356</td>\n",
" <td>1687</td>\n",
" <td>83</td>\n",
" <td>290</td>\n",
" <td>138</td>\n",
" <td>170</td>\n",
" <td>657</td>\n",
" <td>581</td>\n",
" <td>229</td>\n",
" <td>323</td>\n",
" <td>669</td>\n",
" <td>170</td>\n",
" <td>142</td>\n",
" <td>221</td>\n",
" <td>170</td>\n",
" <td>99</td>\n",
" <td>36</td>\n",
" <td>61</td>\n",
" <td>69</td>\n",
" <td>39</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-03-01</th>\n",
" <td>1609</td>\n",
" <td>7301</td>\n",
" <td>1488</td>\n",
" <td>3572</td>\n",
" <td>4758</td>\n",
" <td>6975</td>\n",
" <td>476</td>\n",
" <td>1093</td>\n",
" <td>1101</td>\n",
" <td>2297</td>\n",
" <td>127</td>\n",
" <td>331</td>\n",
" <td>619</td>\n",
" <td>745</td>\n",
" <td>2225</td>\n",
" <td>5052</td>\n",
" <td>1882</td>\n",
" <td>1929</td>\n",
" <td>2619</td>\n",
" <td>2870</td>\n",
" <td>1078</td>\n",
" <td>375</td>\n",
" <td>953</td>\n",
" <td>734</td>\n",
" <td>130</td>\n",
" <td>261</td>\n",
" <td>390</td>\n",
" <td>84</td>\n",
" <td>1975</td>\n",
" <td>2118</td>\n",
" <td>1153</td>\n",
" <td>791</td>\n",
" <td>1079</td>\n",
" <td>2300</td>\n",
" <td>390</td>\n",
" <td>360</td>\n",
" <td>440</td>\n",
" <td>1120</td>\n",
" <td>196</td>\n",
" <td>107</td>\n",
" <td>452</td>\n",
" <td>1084</td>\n",
" <td>540</td>\n",
" <td>893</td>\n",
" <td>128</td>\n",
" <td>318</td>\n",
" <td>270</td>\n",
" <td>1164</td>\n",
" <td>397</td>\n",
" <td>315</td>\n",
" <td>380</td>\n",
" <td>1166</td>\n",
" <td>32</td>\n",
" <td>23</td>\n",
" <td>150</td>\n",
" <td>338</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-04-01</th>\n",
" <td>1520</td>\n",
" <td>9138</td>\n",
" <td>1906</td>\n",
" <td>3575</td>\n",
" <td>3328</td>\n",
" <td>4781</td>\n",
" <td>571</td>\n",
" <td>1699</td>\n",
" <td>1128</td>\n",
" <td>2433</td>\n",
" <td>371</td>\n",
" <td>949</td>\n",
" <td>164</td>\n",
" <td>250</td>\n",
" <td>2918</td>\n",
" <td>5385</td>\n",
" <td>2208</td>\n",
" <td>2882</td>\n",
" <td>2097</td>\n",
" <td>2344</td>\n",
" <td>568</td>\n",
" <td>641</td>\n",
" <td>999</td>\n",
" <td>715</td>\n",
" <td>137</td>\n",
" <td>257</td>\n",
" <td>244</td>\n",
" <td>218</td>\n",
" <td>1500</td>\n",
" <td>1963</td>\n",
" <td>1245</td>\n",
" <td>508</td>\n",
" <td>1128</td>\n",
" <td>1752</td>\n",
" <td>255</td>\n",
" <td>635</td>\n",
" <td>539</td>\n",
" <td>1252</td>\n",
" <td>70</td>\n",
" <td>228</td>\n",
" <td>243</td>\n",
" <td>160</td>\n",
" <td>745</td>\n",
" <td>1157</td>\n",
" <td>270</td>\n",
" <td>336</td>\n",
" <td>214</td>\n",
" <td>535</td>\n",
" <td>194</td>\n",
" <td>260</td>\n",
" <td>410</td>\n",
" <td>1139</td>\n",
" <td>48</td>\n",
" <td>43</td>\n",
" <td>172</td>\n",
" <td>453</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-05-01</th>\n",
" <td>1958</td>\n",
" <td>14194</td>\n",
" <td>2517</td>\n",
" <td>8441</td>\n",
" <td>4930</td>\n",
" <td>5117</td>\n",
" <td>873</td>\n",
" <td>2150</td>\n",
" <td>1560</td>\n",
" <td>2727</td>\n",
" <td>523</td>\n",
" <td>1590</td>\n",
" <td>62</td>\n",
" <td>151</td>\n",
" <td>3154</td>\n",
" <td>7232</td>\n",
" <td>2988</td>\n",
" <td>3164</td>\n",
" <td>2703</td>\n",
" <td>2933</td>\n",
" <td>887</td>\n",
" <td>798</td>\n",
" <td>1396</td>\n",
" <td>630</td>\n",
" <td>347</td>\n",
" <td>437</td>\n",
" <td>153</td>\n",
" <td>125</td>\n",
" <td>1196</td>\n",
" <td>2151</td>\n",
" <td>950</td>\n",
" <td>572</td>\n",
" <td>1192</td>\n",
" <td>1559</td>\n",
" <td>386</td>\n",
" <td>280</td>\n",
" <td>582</td>\n",
" <td>441</td>\n",
" <td>130</td>\n",
" <td>205</td>\n",
" <td>194</td>\n",
" <td>189</td>\n",
" <td>426</td>\n",
" <td>558</td>\n",
" <td>265</td>\n",
" <td>293</td>\n",
" <td>458</td>\n",
" <td>557</td>\n",
" <td>147</td>\n",
" <td>33</td>\n",
" <td>162</td>\n",
" <td>28</td>\n",
" <td>77</td>\n",
" <td>60</td>\n",
" <td>15</td>\n",
" <td>47</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" NSW - hol - city NSW - hol - noncity VIC - hol - city \\\n",
"timestamp \n",
"2006-01-01 3096 14493 2531 \n",
"2006-02-01 1479 9548 1439 \n",
"2006-03-01 1609 7301 1488 \n",
"2006-04-01 1520 9138 1906 \n",
"2006-05-01 1958 14194 2517 \n",
"\n",
" VIC - hol - noncity QLD - hol - city QLD - hol - noncity \\\n",
"timestamp \n",
"2006-01-01 7881 4688 4390 \n",
"2006-02-01 4586 2320 3990 \n",
"2006-03-01 3572 4758 6975 \n",
"2006-04-01 3575 3328 4781 \n",
"2006-05-01 8441 4930 5117 \n",
"\n",
" SA - hol - city SA - hol - noncity WA - hol - city \\\n",
"timestamp \n",
"2006-01-01 888 2201 1383 \n",
"2006-02-01 521 1414 1059 \n",
"2006-03-01 476 1093 1101 \n",
"2006-04-01 571 1699 1128 \n",
"2006-05-01 873 2150 1560 \n",
"\n",
" WA - hol - noncity TAS - hol - city TAS - hol - noncity \\\n",
"timestamp \n",
"2006-01-01 2066 619 1483 \n",
"2006-02-01 1395 409 689 \n",
"2006-03-01 2297 127 331 \n",
"2006-04-01 2433 371 949 \n",
"2006-05-01 2727 523 1590 \n",
"\n",
" NT - hol - city NT - hol - noncity NSW - vfr - city \\\n",
"timestamp \n",
"2006-01-01 101 86 2709 \n",
"2006-02-01 201 297 2184 \n",
"2006-03-01 619 745 2225 \n",
"2006-04-01 164 250 2918 \n",
"2006-05-01 62 151 3154 \n",
"\n",
" NSW - vfr - noncity VIC - vfr - city VIC - vfr - noncity \\\n",
"timestamp \n",
"2006-01-01 6689 2565 3428 \n",
"2006-02-01 5645 1852 2255 \n",
"2006-03-01 5052 1882 1929 \n",
"2006-04-01 5385 2208 2882 \n",
"2006-05-01 7232 2988 3164 \n",
"\n",
" QLD - vfr - city QLD - vfr - noncity SA - vfr - city \\\n",
"timestamp \n",
"2006-01-01 3003 2287 1324 \n",
"2006-02-01 1957 2945 806 \n",
"2006-03-01 2619 2870 1078 \n",
"2006-04-01 2097 2344 568 \n",
"2006-05-01 2703 2933 887 \n",
"\n",
" SA - vfr - noncity WA - vfr - city WA - vfr - noncity \\\n",
"timestamp \n",
"2006-01-01 869 1019 762 \n",
"2006-02-01 639 750 603 \n",
"2006-03-01 375 953 734 \n",
"2006-04-01 641 999 715 \n",
"2006-05-01 798 1396 630 \n",
"\n",
" TAS - vfr - city TAS - vfr - noncity NT - vfr - city \\\n",
"timestamp \n",
"2006-01-01 602 748 28 \n",
"2006-02-01 257 266 168 \n",
"2006-03-01 130 261 390 \n",
"2006-04-01 137 257 244 \n",
"2006-05-01 347 437 153 \n",
"\n",
" NT - vfr - noncity NSW - bus - city NSW - bus - noncity \\\n",
"timestamp \n",
"2006-01-01 9 1201 1684 \n",
"2006-02-01 349 2020 2281 \n",
"2006-03-01 84 1975 2118 \n",
"2006-04-01 218 1500 1963 \n",
"2006-05-01 125 1196 2151 \n",
"\n",
" VIC - bus - city VIC - bus - noncity QLD - bus - city \\\n",
"timestamp \n",
"2006-01-01 1164 984 1111 \n",
"2006-02-01 1014 811 776 \n",
"2006-03-01 1153 791 1079 \n",
"2006-04-01 1245 508 1128 \n",
"2006-05-01 950 572 1192 \n",
"\n",
" QLD - bus - noncity SA - bus - city SA - bus - noncity \\\n",
"timestamp \n",
"2006-01-01 982 388 456 \n",
"2006-02-01 1448 346 403 \n",
"2006-03-01 2300 390 360 \n",
"2006-04-01 1752 255 635 \n",
"2006-05-01 1559 386 280 \n",
"\n",
" WA - bus - city WA - bus - noncity TAS - bus - city \\\n",
"timestamp \n",
"2006-01-01 532 874 116 \n",
"2006-02-01 356 1687 83 \n",
"2006-03-01 440 1120 196 \n",
"2006-04-01 539 1252 70 \n",
"2006-05-01 582 441 130 \n",
"\n",
" TAS - bus - noncity NT - bus - city NT - bus - noncity \\\n",
"timestamp \n",
"2006-01-01 107 136 80 \n",
"2006-02-01 290 138 170 \n",
"2006-03-01 107 452 1084 \n",
"2006-04-01 228 243 160 \n",
"2006-05-01 205 194 189 \n",
"\n",
" NSW - oth - city NSW - oth - noncity VIC - oth - city \\\n",
"timestamp \n",
"2006-01-01 396 510 181 \n",
"2006-02-01 657 581 229 \n",
"2006-03-01 540 893 128 \n",
"2006-04-01 745 1157 270 \n",
"2006-05-01 426 558 265 \n",
"\n",
" VIC - oth - noncity QLD - oth - city QLD - oth - noncity \\\n",
"timestamp \n",
"2006-01-01 286 431 271 \n",
"2006-02-01 323 669 170 \n",
"2006-03-01 318 270 1164 \n",
"2006-04-01 336 214 535 \n",
"2006-05-01 293 458 557 \n",
"\n",
" SA - oth - city SA - oth - noncity WA - oth - city \\\n",
"timestamp \n",
"2006-01-01 244 73 168 \n",
"2006-02-01 142 221 170 \n",
"2006-03-01 397 315 380 \n",
"2006-04-01 194 260 410 \n",
"2006-05-01 147 33 162 \n",
"\n",
" WA - oth - noncity TAS - oth - city TAS - oth - noncity \\\n",
"timestamp \n",
"2006-01-01 37 76 24 \n",
"2006-02-01 99 36 61 \n",
"2006-03-01 1166 32 23 \n",
"2006-04-01 1139 48 43 \n",
"2006-05-01 28 77 60 \n",
"\n",
" NT - oth - city NT - oth - noncity \n",
"timestamp \n",
"2006-01-01 35 8 \n",
"2006-02-01 69 39 \n",
"2006-03-01 150 338 \n",
"2006-04-01 172 453 \n",
"2006-05-01 15 47 "
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"city_segments = list(filter(lambda name: name.count(\"-\") == 2, df.columns))\n",
"\n",
"df[city_segments].head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 2.2.1 Prepare data in ETNA hierarchical long format <a class=\"anchor\" id=\"chapter2_2_1\"></a>\n",
"Before trying to detect a hierarchical structure, data must be transformed to hierarchical long format. In this format,\n",
"your `DataFrame` must contain `timestamp`, `target` and level columns. Each level column represents membership of the\n",
"observation at higher levels of the hierarchy."
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>timestamp</th>\n",
" <th>target</th>\n",
" <th>city_level</th>\n",
" <th>region_level</th>\n",
" <th>reason_level</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>2006-01-01</td>\n",
" <td>3096</td>\n",
" <td>city</td>\n",
" <td>NSW</td>\n",
" <td>Hol</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2006-02-01</td>\n",
" <td>1479</td>\n",
" <td>city</td>\n",
" <td>NSW</td>\n",
" <td>Hol</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>2006-03-01</td>\n",
" <td>1609</td>\n",
" <td>city</td>\n",
" <td>NSW</td>\n",
" <td>Hol</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>2006-04-01</td>\n",
" <td>1520</td>\n",
" <td>city</td>\n",
" <td>NSW</td>\n",
" <td>Hol</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>2006-05-01</td>\n",
" <td>1958</td>\n",
" <td>city</td>\n",
" <td>NSW</td>\n",
" <td>Hol</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" timestamp target city_level region_level reason_level\n",
"0 2006-01-01 3096 city NSW Hol\n",
"1 2006-02-01 1479 city NSW Hol\n",
"2 2006-03-01 1609 city NSW Hol\n",
"3 2006-04-01 1520 city NSW Hol\n",
"4 2006-05-01 1958 city NSW Hol"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"hierarchical_df = []\n",
"for segment_name in city_segments:\n",
" segment = df[segment_name]\n",
" region, reason, city = segment_name.split(\" - \")\n",
"\n",
" seg_df = pd.DataFrame(\n",
" data={\n",
" \"timestamp\": segment.index,\n",
" \"target\": segment.values,\n",
" \"city_level\": [city] * periods,\n",
" \"region_level\": [region] * periods,\n",
" \"reason_level\": [reason] * periods,\n",
" },\n",
" )\n",
" hierarchical_df.append(seg_df)\n",
"\n",
"hierarchical_df = pd.concat(hierarchical_df, axis=0)\n",
"\n",
"hierarchical_df[\"reason_level\"].replace({\"hol\": \"Hol\", \"vfr\": \"VFR\", \"bus\": \"Bus\", \"oth\": \"Oth\"}, inplace=True)\n",
"hierarchical_df.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here we can omit total level as it will be added automatically in hierarchy detection."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 2.2.2 Convert data to etna wide format with `to_hierarchical_dataset`<a class=\"anchor\" id=\"chapter2_2_2\"></a>\n",
"To detect hierarchical structure and convert `DataFrame` to ETNA wide format, call `TSDataset.to_hierarchical_dataset`,\n",
"provided with prepared data and level columns names in order from top to bottom."
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead tr th {\n",
" text-align: left;\n",
" }\n",
"\n",
" .dataframe thead tr:last-of-type th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr>\n",
" <th>segment</th>\n",
" <th>Bus_NSW_city</th>\n",
" <th>Bus_NSW_noncity</th>\n",
" <th>Bus_NT_city</th>\n",
" <th>Bus_NT_noncity</th>\n",
" <th>Bus_QLD_city</th>\n",
" <th>Bus_QLD_noncity</th>\n",
" <th>Bus_SA_city</th>\n",
" <th>Bus_SA_noncity</th>\n",
" <th>Bus_TAS_city</th>\n",
" <th>Bus_TAS_noncity</th>\n",
" <th>Bus_VIC_city</th>\n",
" <th>Bus_VIC_noncity</th>\n",
" <th>Bus_WA_city</th>\n",
" <th>Bus_WA_noncity</th>\n",
" <th>Hol_NSW_city</th>\n",
" <th>Hol_NSW_noncity</th>\n",
" <th>Hol_NT_city</th>\n",
" <th>Hol_NT_noncity</th>\n",
" <th>Hol_QLD_city</th>\n",
" <th>Hol_QLD_noncity</th>\n",
" <th>Hol_SA_city</th>\n",
" <th>Hol_SA_noncity</th>\n",
" <th>Hol_TAS_city</th>\n",
" <th>Hol_TAS_noncity</th>\n",
" <th>Hol_VIC_city</th>\n",
" <th>Hol_VIC_noncity</th>\n",
" <th>Hol_WA_city</th>\n",
" <th>Hol_WA_noncity</th>\n",
" <th>Oth_NSW_city</th>\n",
" <th>Oth_NSW_noncity</th>\n",
" <th>Oth_NT_city</th>\n",
" <th>Oth_NT_noncity</th>\n",
" <th>Oth_QLD_city</th>\n",
" <th>Oth_QLD_noncity</th>\n",
" <th>Oth_SA_city</th>\n",
" <th>Oth_SA_noncity</th>\n",
" <th>Oth_TAS_city</th>\n",
" <th>Oth_TAS_noncity</th>\n",
" <th>Oth_VIC_city</th>\n",
" <th>Oth_VIC_noncity</th>\n",
" <th>Oth_WA_city</th>\n",
" <th>Oth_WA_noncity</th>\n",
" <th>VFR_NSW_city</th>\n",
" <th>VFR_NSW_noncity</th>\n",
" <th>VFR_NT_city</th>\n",
" <th>VFR_NT_noncity</th>\n",
" <th>VFR_QLD_city</th>\n",
" <th>VFR_QLD_noncity</th>\n",
" <th>VFR_SA_city</th>\n",
" <th>VFR_SA_noncity</th>\n",
" <th>VFR_TAS_city</th>\n",
" <th>VFR_TAS_noncity</th>\n",
" <th>VFR_VIC_city</th>\n",
" <th>VFR_VIC_noncity</th>\n",
" <th>VFR_WA_city</th>\n",
" <th>VFR_WA_noncity</th>\n",
" </tr>\n",
" <tr>\n",
" <th>feature</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" </tr>\n",
" <tr>\n",
" <th>timestamp</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>2006-01-01</th>\n",
" <td>1201</td>\n",
" <td>1684</td>\n",
" <td>136</td>\n",
" <td>80</td>\n",
" <td>1111</td>\n",
" <td>982</td>\n",
" <td>388</td>\n",
" <td>456</td>\n",
" <td>116</td>\n",
" <td>107</td>\n",
" <td>1164</td>\n",
" <td>984</td>\n",
" <td>532</td>\n",
" <td>874</td>\n",
" <td>3096</td>\n",
" <td>14493</td>\n",
" <td>101</td>\n",
" <td>86</td>\n",
" <td>4688</td>\n",
" <td>4390</td>\n",
" <td>888</td>\n",
" <td>2201</td>\n",
" <td>619</td>\n",
" <td>1483</td>\n",
" <td>2531</td>\n",
" <td>7881</td>\n",
" <td>1383</td>\n",
" <td>2066</td>\n",
" <td>396</td>\n",
" <td>510</td>\n",
" <td>35</td>\n",
" <td>8</td>\n",
" <td>431</td>\n",
" <td>271</td>\n",
" <td>244</td>\n",
" <td>73</td>\n",
" <td>76</td>\n",
" <td>24</td>\n",
" <td>181</td>\n",
" <td>286</td>\n",
" <td>168</td>\n",
" <td>37</td>\n",
" <td>2709</td>\n",
" <td>6689</td>\n",
" <td>28</td>\n",
" <td>9</td>\n",
" <td>3003</td>\n",
" <td>2287</td>\n",
" <td>1324</td>\n",
" <td>869</td>\n",
" <td>602</td>\n",
" <td>748</td>\n",
" <td>2565</td>\n",
" <td>3428</td>\n",
" <td>1019</td>\n",
" <td>762</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-02-01</th>\n",
" <td>2020</td>\n",
" <td>2281</td>\n",
" <td>138</td>\n",
" <td>170</td>\n",
" <td>776</td>\n",
" <td>1448</td>\n",
" <td>346</td>\n",
" <td>403</td>\n",
" <td>83</td>\n",
" <td>290</td>\n",
" <td>1014</td>\n",
" <td>811</td>\n",
" <td>356</td>\n",
" <td>1687</td>\n",
" <td>1479</td>\n",
" <td>9548</td>\n",
" <td>201</td>\n",
" <td>297</td>\n",
" <td>2320</td>\n",
" <td>3990</td>\n",
" <td>521</td>\n",
" <td>1414</td>\n",
" <td>409</td>\n",
" <td>689</td>\n",
" <td>1439</td>\n",
" <td>4586</td>\n",
" <td>1059</td>\n",
" <td>1395</td>\n",
" <td>657</td>\n",
" <td>581</td>\n",
" <td>69</td>\n",
" <td>39</td>\n",
" <td>669</td>\n",
" <td>170</td>\n",
" <td>142</td>\n",
" <td>221</td>\n",
" <td>36</td>\n",
" <td>61</td>\n",
" <td>229</td>\n",
" <td>323</td>\n",
" <td>170</td>\n",
" <td>99</td>\n",
" <td>2184</td>\n",
" <td>5645</td>\n",
" <td>168</td>\n",
" <td>349</td>\n",
" <td>1957</td>\n",
" <td>2945</td>\n",
" <td>806</td>\n",
" <td>639</td>\n",
" <td>257</td>\n",
" <td>266</td>\n",
" <td>1852</td>\n",
" <td>2255</td>\n",
" <td>750</td>\n",
" <td>603</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-03-01</th>\n",
" <td>1975</td>\n",
" <td>2118</td>\n",
" <td>452</td>\n",
" <td>1084</td>\n",
" <td>1079</td>\n",
" <td>2300</td>\n",
" <td>390</td>\n",
" <td>360</td>\n",
" <td>196</td>\n",
" <td>107</td>\n",
" <td>1153</td>\n",
" <td>791</td>\n",
" <td>440</td>\n",
" <td>1120</td>\n",
" <td>1609</td>\n",
" <td>7301</td>\n",
" <td>619</td>\n",
" <td>745</td>\n",
" <td>4758</td>\n",
" <td>6975</td>\n",
" <td>476</td>\n",
" <td>1093</td>\n",
" <td>127</td>\n",
" <td>331</td>\n",
" <td>1488</td>\n",
" <td>3572</td>\n",
" <td>1101</td>\n",
" <td>2297</td>\n",
" <td>540</td>\n",
" <td>893</td>\n",
" <td>150</td>\n",
" <td>338</td>\n",
" <td>270</td>\n",
" <td>1164</td>\n",
" <td>397</td>\n",
" <td>315</td>\n",
" <td>32</td>\n",
" <td>23</td>\n",
" <td>128</td>\n",
" <td>318</td>\n",
" <td>380</td>\n",
" <td>1166</td>\n",
" <td>2225</td>\n",
" <td>5052</td>\n",
" <td>390</td>\n",
" <td>84</td>\n",
" <td>2619</td>\n",
" <td>2870</td>\n",
" <td>1078</td>\n",
" <td>375</td>\n",
" <td>130</td>\n",
" <td>261</td>\n",
" <td>1882</td>\n",
" <td>1929</td>\n",
" <td>953</td>\n",
" <td>734</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-04-01</th>\n",
" <td>1500</td>\n",
" <td>1963</td>\n",
" <td>243</td>\n",
" <td>160</td>\n",
" <td>1128</td>\n",
" <td>1752</td>\n",
" <td>255</td>\n",
" <td>635</td>\n",
" <td>70</td>\n",
" <td>228</td>\n",
" <td>1245</td>\n",
" <td>508</td>\n",
" <td>539</td>\n",
" <td>1252</td>\n",
" <td>1520</td>\n",
" <td>9138</td>\n",
" <td>164</td>\n",
" <td>250</td>\n",
" <td>3328</td>\n",
" <td>4781</td>\n",
" <td>571</td>\n",
" <td>1699</td>\n",
" <td>371</td>\n",
" <td>949</td>\n",
" <td>1906</td>\n",
" <td>3575</td>\n",
" <td>1128</td>\n",
" <td>2433</td>\n",
" <td>745</td>\n",
" <td>1157</td>\n",
" <td>172</td>\n",
" <td>453</td>\n",
" <td>214</td>\n",
" <td>535</td>\n",
" <td>194</td>\n",
" <td>260</td>\n",
" <td>48</td>\n",
" <td>43</td>\n",
" <td>270</td>\n",
" <td>336</td>\n",
" <td>410</td>\n",
" <td>1139</td>\n",
" <td>2918</td>\n",
" <td>5385</td>\n",
" <td>244</td>\n",
" <td>218</td>\n",
" <td>2097</td>\n",
" <td>2344</td>\n",
" <td>568</td>\n",
" <td>641</td>\n",
" <td>137</td>\n",
" <td>257</td>\n",
" <td>2208</td>\n",
" <td>2882</td>\n",
" <td>999</td>\n",
" <td>715</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-05-01</th>\n",
" <td>1196</td>\n",
" <td>2151</td>\n",
" <td>194</td>\n",
" <td>189</td>\n",
" <td>1192</td>\n",
" <td>1559</td>\n",
" <td>386</td>\n",
" <td>280</td>\n",
" <td>130</td>\n",
" <td>205</td>\n",
" <td>950</td>\n",
" <td>572</td>\n",
" <td>582</td>\n",
" <td>441</td>\n",
" <td>1958</td>\n",
" <td>14194</td>\n",
" <td>62</td>\n",
" <td>151</td>\n",
" <td>4930</td>\n",
" <td>5117</td>\n",
" <td>873</td>\n",
" <td>2150</td>\n",
" <td>523</td>\n",
" <td>1590</td>\n",
" <td>2517</td>\n",
" <td>8441</td>\n",
" <td>1560</td>\n",
" <td>2727</td>\n",
" <td>426</td>\n",
" <td>558</td>\n",
" <td>15</td>\n",
" <td>47</td>\n",
" <td>458</td>\n",
" <td>557</td>\n",
" <td>147</td>\n",
" <td>33</td>\n",
" <td>77</td>\n",
" <td>60</td>\n",
" <td>265</td>\n",
" <td>293</td>\n",
" <td>162</td>\n",
" <td>28</td>\n",
" <td>3154</td>\n",
" <td>7232</td>\n",
" <td>153</td>\n",
" <td>125</td>\n",
" <td>2703</td>\n",
" <td>2933</td>\n",
" <td>887</td>\n",
" <td>798</td>\n",
" <td>347</td>\n",
" <td>437</td>\n",
" <td>2988</td>\n",
" <td>3164</td>\n",
" <td>1396</td>\n",
" <td>630</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
"segment Bus_NSW_city Bus_NSW_noncity Bus_NT_city Bus_NT_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 1201 1684 136 80 \n",
"2006-02-01 2020 2281 138 170 \n",
"2006-03-01 1975 2118 452 1084 \n",
"2006-04-01 1500 1963 243 160 \n",
"2006-05-01 1196 2151 194 189 \n",
"\n",
"segment Bus_QLD_city Bus_QLD_noncity Bus_SA_city Bus_SA_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 1111 982 388 456 \n",
"2006-02-01 776 1448 346 403 \n",
"2006-03-01 1079 2300 390 360 \n",
"2006-04-01 1128 1752 255 635 \n",
"2006-05-01 1192 1559 386 280 \n",
"\n",
"segment Bus_TAS_city Bus_TAS_noncity Bus_VIC_city Bus_VIC_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 116 107 1164 984 \n",
"2006-02-01 83 290 1014 811 \n",
"2006-03-01 196 107 1153 791 \n",
"2006-04-01 70 228 1245 508 \n",
"2006-05-01 130 205 950 572 \n",
"\n",
"segment Bus_WA_city Bus_WA_noncity Hol_NSW_city Hol_NSW_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 532 874 3096 14493 \n",
"2006-02-01 356 1687 1479 9548 \n",
"2006-03-01 440 1120 1609 7301 \n",
"2006-04-01 539 1252 1520 9138 \n",
"2006-05-01 582 441 1958 14194 \n",
"\n",
"segment Hol_NT_city Hol_NT_noncity Hol_QLD_city Hol_QLD_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 101 86 4688 4390 \n",
"2006-02-01 201 297 2320 3990 \n",
"2006-03-01 619 745 4758 6975 \n",
"2006-04-01 164 250 3328 4781 \n",
"2006-05-01 62 151 4930 5117 \n",
"\n",
"segment Hol_SA_city Hol_SA_noncity Hol_TAS_city Hol_TAS_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 888 2201 619 1483 \n",
"2006-02-01 521 1414 409 689 \n",
"2006-03-01 476 1093 127 331 \n",
"2006-04-01 571 1699 371 949 \n",
"2006-05-01 873 2150 523 1590 \n",
"\n",
"segment Hol_VIC_city Hol_VIC_noncity Hol_WA_city Hol_WA_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 2531 7881 1383 2066 \n",
"2006-02-01 1439 4586 1059 1395 \n",
"2006-03-01 1488 3572 1101 2297 \n",
"2006-04-01 1906 3575 1128 2433 \n",
"2006-05-01 2517 8441 1560 2727 \n",
"\n",
"segment Oth_NSW_city Oth_NSW_noncity Oth_NT_city Oth_NT_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 396 510 35 8 \n",
"2006-02-01 657 581 69 39 \n",
"2006-03-01 540 893 150 338 \n",
"2006-04-01 745 1157 172 453 \n",
"2006-05-01 426 558 15 47 \n",
"\n",
"segment Oth_QLD_city Oth_QLD_noncity Oth_SA_city Oth_SA_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 431 271 244 73 \n",
"2006-02-01 669 170 142 221 \n",
"2006-03-01 270 1164 397 315 \n",
"2006-04-01 214 535 194 260 \n",
"2006-05-01 458 557 147 33 \n",
"\n",
"segment Oth_TAS_city Oth_TAS_noncity Oth_VIC_city Oth_VIC_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 76 24 181 286 \n",
"2006-02-01 36 61 229 323 \n",
"2006-03-01 32 23 128 318 \n",
"2006-04-01 48 43 270 336 \n",
"2006-05-01 77 60 265 293 \n",
"\n",
"segment Oth_WA_city Oth_WA_noncity VFR_NSW_city VFR_NSW_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 168 37 2709 6689 \n",
"2006-02-01 170 99 2184 5645 \n",
"2006-03-01 380 1166 2225 5052 \n",
"2006-04-01 410 1139 2918 5385 \n",
"2006-05-01 162 28 3154 7232 \n",
"\n",
"segment VFR_NT_city VFR_NT_noncity VFR_QLD_city VFR_QLD_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 28 9 3003 2287 \n",
"2006-02-01 168 349 1957 2945 \n",
"2006-03-01 390 84 2619 2870 \n",
"2006-04-01 244 218 2097 2344 \n",
"2006-05-01 153 125 2703 2933 \n",
"\n",
"segment VFR_SA_city VFR_SA_noncity VFR_TAS_city VFR_TAS_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 1324 869 602 748 \n",
"2006-02-01 806 639 257 266 \n",
"2006-03-01 1078 375 130 261 \n",
"2006-04-01 568 641 137 257 \n",
"2006-05-01 887 798 347 437 \n",
"\n",
"segment VFR_VIC_city VFR_VIC_noncity VFR_WA_city VFR_WA_noncity \n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 2565 3428 1019 762 \n",
"2006-02-01 1852 2255 750 603 \n",
"2006-03-01 1882 1929 953 734 \n",
"2006-04-01 2208 2882 999 715 \n",
"2006-05-01 2988 3164 1396 630 "
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"hierarchical_df, hierarchical_structure = TSDataset.to_hierarchical_dataset(\n",
" df=hierarchical_df, level_columns=[\"reason_level\", \"region_level\", \"city_level\"]\n",
")\n",
"\n",
"hierarchical_df.head()"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"HierarchicalStructure(level_structure = {'total': ['Hol', 'VFR', 'Bus', 'Oth'], 'Bus': ['Bus_NSW', 'Bus_VIC', 'Bus_QLD', 'Bus_SA', 'Bus_WA', 'Bus_TAS', 'Bus_NT'], 'Hol': ['Hol_NSW', 'Hol_VIC', 'Hol_QLD', 'Hol_SA', 'Hol_WA', 'Hol_TAS', 'Hol_NT'], 'Oth': ['Oth_NSW', 'Oth_VIC', 'Oth_QLD', 'Oth_SA', 'Oth_WA', 'Oth_TAS', 'Oth_NT'], 'VFR': ['VFR_NSW', 'VFR_VIC', 'VFR_QLD', 'VFR_SA', 'VFR_WA', 'VFR_TAS', 'VFR_NT'], 'Bus_NSW': ['Bus_NSW_city', 'Bus_NSW_noncity'], 'Bus_NT': ['Bus_NT_city', 'Bus_NT_noncity'], 'Bus_QLD': ['Bus_QLD_city', 'Bus_QLD_noncity'], 'Bus_SA': ['Bus_SA_city', 'Bus_SA_noncity'], 'Bus_TAS': ['Bus_TAS_city', 'Bus_TAS_noncity'], 'Bus_VIC': ['Bus_VIC_city', 'Bus_VIC_noncity'], 'Bus_WA': ['Bus_WA_city', 'Bus_WA_noncity'], 'Hol_NSW': ['Hol_NSW_city', 'Hol_NSW_noncity'], 'Hol_NT': ['Hol_NT_city', 'Hol_NT_noncity'], 'Hol_QLD': ['Hol_QLD_city', 'Hol_QLD_noncity'], 'Hol_SA': ['Hol_SA_city', 'Hol_SA_noncity'], 'Hol_TAS': ['Hol_TAS_city', 'Hol_TAS_noncity'], 'Hol_VIC': ['Hol_VIC_city', 'Hol_VIC_noncity'], 'Hol_WA': ['Hol_WA_city', 'Hol_WA_noncity'], 'Oth_NSW': ['Oth_NSW_city', 'Oth_NSW_noncity'], 'Oth_NT': ['Oth_NT_city', 'Oth_NT_noncity'], 'Oth_QLD': ['Oth_QLD_city', 'Oth_QLD_noncity'], 'Oth_SA': ['Oth_SA_city', 'Oth_SA_noncity'], 'Oth_TAS': ['Oth_TAS_city', 'Oth_TAS_noncity'], 'Oth_VIC': ['Oth_VIC_city', 'Oth_VIC_noncity'], 'Oth_WA': ['Oth_WA_city', 'Oth_WA_noncity'], 'VFR_NSW': ['VFR_NSW_city', 'VFR_NSW_noncity'], 'VFR_NT': ['VFR_NT_city', 'VFR_NT_noncity'], 'VFR_QLD': ['VFR_QLD_city', 'VFR_QLD_noncity'], 'VFR_SA': ['VFR_SA_city', 'VFR_SA_noncity'], 'VFR_TAS': ['VFR_TAS_city', 'VFR_TAS_noncity'], 'VFR_VIC': ['VFR_VIC_city', 'VFR_VIC_noncity'], 'VFR_WA': ['VFR_WA_city', 'VFR_WA_noncity']}, level_names = ['total', 'reason_level', 'region_level', 'city_level'], )"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"hierarchical_structure"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here we see that `hierarchical_structure` has a mapping between higher level segments and adjacent lower level segments."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 2.2.3 Create the hierarchical dataset<a class=\"anchor\" id=\"chapter2_2_3\"></a>\n",
"To convert data to `TSDataset` call the constructor and pass detected `hierarchical_structure`."
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead tr th {\n",
" text-align: left;\n",
" }\n",
"\n",
" .dataframe thead tr:last-of-type th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr>\n",
" <th>segment</th>\n",
" <th>Bus_NSW_city</th>\n",
" <th>Bus_NSW_noncity</th>\n",
" <th>Bus_NT_city</th>\n",
" <th>Bus_NT_noncity</th>\n",
" <th>Bus_QLD_city</th>\n",
" <th>Bus_QLD_noncity</th>\n",
" <th>Bus_SA_city</th>\n",
" <th>Bus_SA_noncity</th>\n",
" <th>Bus_TAS_city</th>\n",
" <th>Bus_TAS_noncity</th>\n",
" <th>Bus_VIC_city</th>\n",
" <th>Bus_VIC_noncity</th>\n",
" <th>Bus_WA_city</th>\n",
" <th>Bus_WA_noncity</th>\n",
" <th>Hol_NSW_city</th>\n",
" <th>Hol_NSW_noncity</th>\n",
" <th>Hol_NT_city</th>\n",
" <th>Hol_NT_noncity</th>\n",
" <th>Hol_QLD_city</th>\n",
" <th>Hol_QLD_noncity</th>\n",
" <th>Hol_SA_city</th>\n",
" <th>Hol_SA_noncity</th>\n",
" <th>Hol_TAS_city</th>\n",
" <th>Hol_TAS_noncity</th>\n",
" <th>Hol_VIC_city</th>\n",
" <th>Hol_VIC_noncity</th>\n",
" <th>Hol_WA_city</th>\n",
" <th>Hol_WA_noncity</th>\n",
" <th>Oth_NSW_city</th>\n",
" <th>Oth_NSW_noncity</th>\n",
" <th>Oth_NT_city</th>\n",
" <th>Oth_NT_noncity</th>\n",
" <th>Oth_QLD_city</th>\n",
" <th>Oth_QLD_noncity</th>\n",
" <th>Oth_SA_city</th>\n",
" <th>Oth_SA_noncity</th>\n",
" <th>Oth_TAS_city</th>\n",
" <th>Oth_TAS_noncity</th>\n",
" <th>Oth_VIC_city</th>\n",
" <th>Oth_VIC_noncity</th>\n",
" <th>Oth_WA_city</th>\n",
" <th>Oth_WA_noncity</th>\n",
" <th>VFR_NSW_city</th>\n",
" <th>VFR_NSW_noncity</th>\n",
" <th>VFR_NT_city</th>\n",
" <th>VFR_NT_noncity</th>\n",
" <th>VFR_QLD_city</th>\n",
" <th>VFR_QLD_noncity</th>\n",
" <th>VFR_SA_city</th>\n",
" <th>VFR_SA_noncity</th>\n",
" <th>VFR_TAS_city</th>\n",
" <th>VFR_TAS_noncity</th>\n",
" <th>VFR_VIC_city</th>\n",
" <th>VFR_VIC_noncity</th>\n",
" <th>VFR_WA_city</th>\n",
" <th>VFR_WA_noncity</th>\n",
" </tr>\n",
" <tr>\n",
" <th>feature</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" </tr>\n",
" <tr>\n",
" <th>timestamp</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>2006-01-01</th>\n",
" <td>1201</td>\n",
" <td>1684</td>\n",
" <td>136</td>\n",
" <td>80</td>\n",
" <td>1111</td>\n",
" <td>982</td>\n",
" <td>388</td>\n",
" <td>456</td>\n",
" <td>116</td>\n",
" <td>107</td>\n",
" <td>1164</td>\n",
" <td>984</td>\n",
" <td>532</td>\n",
" <td>874</td>\n",
" <td>3096</td>\n",
" <td>14493</td>\n",
" <td>101</td>\n",
" <td>86</td>\n",
" <td>4688</td>\n",
" <td>4390</td>\n",
" <td>888</td>\n",
" <td>2201</td>\n",
" <td>619</td>\n",
" <td>1483</td>\n",
" <td>2531</td>\n",
" <td>7881</td>\n",
" <td>1383</td>\n",
" <td>2066</td>\n",
" <td>396</td>\n",
" <td>510</td>\n",
" <td>35</td>\n",
" <td>8</td>\n",
" <td>431</td>\n",
" <td>271</td>\n",
" <td>244</td>\n",
" <td>73</td>\n",
" <td>76</td>\n",
" <td>24</td>\n",
" <td>181</td>\n",
" <td>286</td>\n",
" <td>168</td>\n",
" <td>37</td>\n",
" <td>2709</td>\n",
" <td>6689</td>\n",
" <td>28</td>\n",
" <td>9</td>\n",
" <td>3003</td>\n",
" <td>2287</td>\n",
" <td>1324</td>\n",
" <td>869</td>\n",
" <td>602</td>\n",
" <td>748</td>\n",
" <td>2565</td>\n",
" <td>3428</td>\n",
" <td>1019</td>\n",
" <td>762</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-02-01</th>\n",
" <td>2020</td>\n",
" <td>2281</td>\n",
" <td>138</td>\n",
" <td>170</td>\n",
" <td>776</td>\n",
" <td>1448</td>\n",
" <td>346</td>\n",
" <td>403</td>\n",
" <td>83</td>\n",
" <td>290</td>\n",
" <td>1014</td>\n",
" <td>811</td>\n",
" <td>356</td>\n",
" <td>1687</td>\n",
" <td>1479</td>\n",
" <td>9548</td>\n",
" <td>201</td>\n",
" <td>297</td>\n",
" <td>2320</td>\n",
" <td>3990</td>\n",
" <td>521</td>\n",
" <td>1414</td>\n",
" <td>409</td>\n",
" <td>689</td>\n",
" <td>1439</td>\n",
" <td>4586</td>\n",
" <td>1059</td>\n",
" <td>1395</td>\n",
" <td>657</td>\n",
" <td>581</td>\n",
" <td>69</td>\n",
" <td>39</td>\n",
" <td>669</td>\n",
" <td>170</td>\n",
" <td>142</td>\n",
" <td>221</td>\n",
" <td>36</td>\n",
" <td>61</td>\n",
" <td>229</td>\n",
" <td>323</td>\n",
" <td>170</td>\n",
" <td>99</td>\n",
" <td>2184</td>\n",
" <td>5645</td>\n",
" <td>168</td>\n",
" <td>349</td>\n",
" <td>1957</td>\n",
" <td>2945</td>\n",
" <td>806</td>\n",
" <td>639</td>\n",
" <td>257</td>\n",
" <td>266</td>\n",
" <td>1852</td>\n",
" <td>2255</td>\n",
" <td>750</td>\n",
" <td>603</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-03-01</th>\n",
" <td>1975</td>\n",
" <td>2118</td>\n",
" <td>452</td>\n",
" <td>1084</td>\n",
" <td>1079</td>\n",
" <td>2300</td>\n",
" <td>390</td>\n",
" <td>360</td>\n",
" <td>196</td>\n",
" <td>107</td>\n",
" <td>1153</td>\n",
" <td>791</td>\n",
" <td>440</td>\n",
" <td>1120</td>\n",
" <td>1609</td>\n",
" <td>7301</td>\n",
" <td>619</td>\n",
" <td>745</td>\n",
" <td>4758</td>\n",
" <td>6975</td>\n",
" <td>476</td>\n",
" <td>1093</td>\n",
" <td>127</td>\n",
" <td>331</td>\n",
" <td>1488</td>\n",
" <td>3572</td>\n",
" <td>1101</td>\n",
" <td>2297</td>\n",
" <td>540</td>\n",
" <td>893</td>\n",
" <td>150</td>\n",
" <td>338</td>\n",
" <td>270</td>\n",
" <td>1164</td>\n",
" <td>397</td>\n",
" <td>315</td>\n",
" <td>32</td>\n",
" <td>23</td>\n",
" <td>128</td>\n",
" <td>318</td>\n",
" <td>380</td>\n",
" <td>1166</td>\n",
" <td>2225</td>\n",
" <td>5052</td>\n",
" <td>390</td>\n",
" <td>84</td>\n",
" <td>2619</td>\n",
" <td>2870</td>\n",
" <td>1078</td>\n",
" <td>375</td>\n",
" <td>130</td>\n",
" <td>261</td>\n",
" <td>1882</td>\n",
" <td>1929</td>\n",
" <td>953</td>\n",
" <td>734</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-04-01</th>\n",
" <td>1500</td>\n",
" <td>1963</td>\n",
" <td>243</td>\n",
" <td>160</td>\n",
" <td>1128</td>\n",
" <td>1752</td>\n",
" <td>255</td>\n",
" <td>635</td>\n",
" <td>70</td>\n",
" <td>228</td>\n",
" <td>1245</td>\n",
" <td>508</td>\n",
" <td>539</td>\n",
" <td>1252</td>\n",
" <td>1520</td>\n",
" <td>9138</td>\n",
" <td>164</td>\n",
" <td>250</td>\n",
" <td>3328</td>\n",
" <td>4781</td>\n",
" <td>571</td>\n",
" <td>1699</td>\n",
" <td>371</td>\n",
" <td>949</td>\n",
" <td>1906</td>\n",
" <td>3575</td>\n",
" <td>1128</td>\n",
" <td>2433</td>\n",
" <td>745</td>\n",
" <td>1157</td>\n",
" <td>172</td>\n",
" <td>453</td>\n",
" <td>214</td>\n",
" <td>535</td>\n",
" <td>194</td>\n",
" <td>260</td>\n",
" <td>48</td>\n",
" <td>43</td>\n",
" <td>270</td>\n",
" <td>336</td>\n",
" <td>410</td>\n",
" <td>1139</td>\n",
" <td>2918</td>\n",
" <td>5385</td>\n",
" <td>244</td>\n",
" <td>218</td>\n",
" <td>2097</td>\n",
" <td>2344</td>\n",
" <td>568</td>\n",
" <td>641</td>\n",
" <td>137</td>\n",
" <td>257</td>\n",
" <td>2208</td>\n",
" <td>2882</td>\n",
" <td>999</td>\n",
" <td>715</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-05-01</th>\n",
" <td>1196</td>\n",
" <td>2151</td>\n",
" <td>194</td>\n",
" <td>189</td>\n",
" <td>1192</td>\n",
" <td>1559</td>\n",
" <td>386</td>\n",
" <td>280</td>\n",
" <td>130</td>\n",
" <td>205</td>\n",
" <td>950</td>\n",
" <td>572</td>\n",
" <td>582</td>\n",
" <td>441</td>\n",
" <td>1958</td>\n",
" <td>14194</td>\n",
" <td>62</td>\n",
" <td>151</td>\n",
" <td>4930</td>\n",
" <td>5117</td>\n",
" <td>873</td>\n",
" <td>2150</td>\n",
" <td>523</td>\n",
" <td>1590</td>\n",
" <td>2517</td>\n",
" <td>8441</td>\n",
" <td>1560</td>\n",
" <td>2727</td>\n",
" <td>426</td>\n",
" <td>558</td>\n",
" <td>15</td>\n",
" <td>47</td>\n",
" <td>458</td>\n",
" <td>557</td>\n",
" <td>147</td>\n",
" <td>33</td>\n",
" <td>77</td>\n",
" <td>60</td>\n",
" <td>265</td>\n",
" <td>293</td>\n",
" <td>162</td>\n",
" <td>28</td>\n",
" <td>3154</td>\n",
" <td>7232</td>\n",
" <td>153</td>\n",
" <td>125</td>\n",
" <td>2703</td>\n",
" <td>2933</td>\n",
" <td>887</td>\n",
" <td>798</td>\n",
" <td>347</td>\n",
" <td>437</td>\n",
" <td>2988</td>\n",
" <td>3164</td>\n",
" <td>1396</td>\n",
" <td>630</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
"segment Bus_NSW_city Bus_NSW_noncity Bus_NT_city Bus_NT_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 1201 1684 136 80 \n",
"2006-02-01 2020 2281 138 170 \n",
"2006-03-01 1975 2118 452 1084 \n",
"2006-04-01 1500 1963 243 160 \n",
"2006-05-01 1196 2151 194 189 \n",
"\n",
"segment Bus_QLD_city Bus_QLD_noncity Bus_SA_city Bus_SA_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 1111 982 388 456 \n",
"2006-02-01 776 1448 346 403 \n",
"2006-03-01 1079 2300 390 360 \n",
"2006-04-01 1128 1752 255 635 \n",
"2006-05-01 1192 1559 386 280 \n",
"\n",
"segment Bus_TAS_city Bus_TAS_noncity Bus_VIC_city Bus_VIC_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 116 107 1164 984 \n",
"2006-02-01 83 290 1014 811 \n",
"2006-03-01 196 107 1153 791 \n",
"2006-04-01 70 228 1245 508 \n",
"2006-05-01 130 205 950 572 \n",
"\n",
"segment Bus_WA_city Bus_WA_noncity Hol_NSW_city Hol_NSW_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 532 874 3096 14493 \n",
"2006-02-01 356 1687 1479 9548 \n",
"2006-03-01 440 1120 1609 7301 \n",
"2006-04-01 539 1252 1520 9138 \n",
"2006-05-01 582 441 1958 14194 \n",
"\n",
"segment Hol_NT_city Hol_NT_noncity Hol_QLD_city Hol_QLD_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 101 86 4688 4390 \n",
"2006-02-01 201 297 2320 3990 \n",
"2006-03-01 619 745 4758 6975 \n",
"2006-04-01 164 250 3328 4781 \n",
"2006-05-01 62 151 4930 5117 \n",
"\n",
"segment Hol_SA_city Hol_SA_noncity Hol_TAS_city Hol_TAS_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 888 2201 619 1483 \n",
"2006-02-01 521 1414 409 689 \n",
"2006-03-01 476 1093 127 331 \n",
"2006-04-01 571 1699 371 949 \n",
"2006-05-01 873 2150 523 1590 \n",
"\n",
"segment Hol_VIC_city Hol_VIC_noncity Hol_WA_city Hol_WA_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 2531 7881 1383 2066 \n",
"2006-02-01 1439 4586 1059 1395 \n",
"2006-03-01 1488 3572 1101 2297 \n",
"2006-04-01 1906 3575 1128 2433 \n",
"2006-05-01 2517 8441 1560 2727 \n",
"\n",
"segment Oth_NSW_city Oth_NSW_noncity Oth_NT_city Oth_NT_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 396 510 35 8 \n",
"2006-02-01 657 581 69 39 \n",
"2006-03-01 540 893 150 338 \n",
"2006-04-01 745 1157 172 453 \n",
"2006-05-01 426 558 15 47 \n",
"\n",
"segment Oth_QLD_city Oth_QLD_noncity Oth_SA_city Oth_SA_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 431 271 244 73 \n",
"2006-02-01 669 170 142 221 \n",
"2006-03-01 270 1164 397 315 \n",
"2006-04-01 214 535 194 260 \n",
"2006-05-01 458 557 147 33 \n",
"\n",
"segment Oth_TAS_city Oth_TAS_noncity Oth_VIC_city Oth_VIC_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 76 24 181 286 \n",
"2006-02-01 36 61 229 323 \n",
"2006-03-01 32 23 128 318 \n",
"2006-04-01 48 43 270 336 \n",
"2006-05-01 77 60 265 293 \n",
"\n",
"segment Oth_WA_city Oth_WA_noncity VFR_NSW_city VFR_NSW_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 168 37 2709 6689 \n",
"2006-02-01 170 99 2184 5645 \n",
"2006-03-01 380 1166 2225 5052 \n",
"2006-04-01 410 1139 2918 5385 \n",
"2006-05-01 162 28 3154 7232 \n",
"\n",
"segment VFR_NT_city VFR_NT_noncity VFR_QLD_city VFR_QLD_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 28 9 3003 2287 \n",
"2006-02-01 168 349 1957 2945 \n",
"2006-03-01 390 84 2619 2870 \n",
"2006-04-01 244 218 2097 2344 \n",
"2006-05-01 153 125 2703 2933 \n",
"\n",
"segment VFR_SA_city VFR_SA_noncity VFR_TAS_city VFR_TAS_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 1324 869 602 748 \n",
"2006-02-01 806 639 257 266 \n",
"2006-03-01 1078 375 130 261 \n",
"2006-04-01 568 641 137 257 \n",
"2006-05-01 887 798 347 437 \n",
"\n",
"segment VFR_VIC_city VFR_VIC_noncity VFR_WA_city VFR_WA_noncity \n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 2565 3428 1019 762 \n",
"2006-02-01 1852 2255 750 603 \n",
"2006-03-01 1882 1929 953 734 \n",
"2006-04-01 2208 2882 999 715 \n",
"2006-05-01 2988 3164 1396 630 "
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"hierarchical_ts = TSDataset(df=hierarchical_df, freq=\"MS\", hierarchical_structure=hierarchical_structure)\n",
"hierarchical_ts.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now the dataset converted to hierarchical. We can examine what hierarchical levels were detected with the following code."
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['total', 'reason_level', 'region_level', 'city_level']"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"hierarchical_ts.hierarchical_structure.level_names"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Ensure that the dataset is at the desired level."
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'city_level'"
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"hierarchical_ts.current_df_level"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The hierarchical dataset could be aggregated to higher levels with the `get_level_dataset` method."
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead tr th {\n",
" text-align: left;\n",
" }\n",
"\n",
" .dataframe thead tr:last-of-type th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr>\n",
" <th>segment</th>\n",
" <th>Bus</th>\n",
" <th>Hol</th>\n",
" <th>Oth</th>\n",
" <th>VFR</th>\n",
" </tr>\n",
" <tr>\n",
" <th>feature</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" </tr>\n",
" <tr>\n",
" <th>timestamp</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>2006-01-01</th>\n",
" <td>9815</td>\n",
" <td>45906</td>\n",
" <td>2740</td>\n",
" <td>26042</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-02-01</th>\n",
" <td>11823</td>\n",
" <td>29347</td>\n",
" <td>3466</td>\n",
" <td>20676</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-03-01</th>\n",
" <td>13565</td>\n",
" <td>32492</td>\n",
" <td>6114</td>\n",
" <td>20582</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-04-01</th>\n",
" <td>11478</td>\n",
" <td>31813</td>\n",
" <td>5976</td>\n",
" <td>21613</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-05-01</th>\n",
" <td>10027</td>\n",
" <td>46793</td>\n",
" <td>3126</td>\n",
" <td>26947</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
"segment Bus Hol Oth VFR\n",
"feature target target target target\n",
"timestamp \n",
"2006-01-01 9815 45906 2740 26042\n",
"2006-02-01 11823 29347 3466 20676\n",
"2006-03-01 13565 32492 6114 20582\n",
"2006-04-01 11478 31813 5976 21613\n",
"2006-05-01 10027 46793 3126 26947"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"hierarchical_ts.get_level_dataset(target_level=\"reason_level\").head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. Reconciliation methods <a class=\"anchor\" id=\"chapter3\"></a>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this section we will examine the reconciliation methods available in ETNA.\n",
"Hierarchical time series reconciliation allows for the readjustment of predictions produced by separate models on\n",
"a set of hierarchically linked time series in order to make the forecasts more accurate, and ensure that they are coherent.\n",
"\n",
"There are several reconciliation methods in the ETNA library:\n",
"\n",
"* Bottom-up approach\n",
"* Top-down approach\n",
"\n",
"Middle-out reconciliation approach could be viewed as a composition of bottom-up and top-down approaches. This method could\n",
"be implemented using functionality from the library. For aggregation to higher levels, one could use provided bottom-up methods,\n",
"and for disaggregation to lower levels -- top-down methods.\n",
"\n",
"Reconciliation methods estimate mapping matrices to perform transitions between levels. These matrices are sparse.\n",
"ETNA uses `scipy.sparse.csr_matrix` to efficiently store them and perform computation.\n",
"\n",
"More detailed information about this and other reconciliation methods can be found [here](https://otexts.com/fpp2/hierarchical.html)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Some important definitions:\n",
"\n",
"* **source level** - level of the hierarchy for model estimation, reconciliation applied to this level\n",
"* **target level** - desired level of the hierarchy, after reconciliation we want to have series at this level."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 3.1. Bottom-up approach <a class=\"anchor\" id=\"chapter3_1\"></a>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The main idea of this approach is to produce forecasts for time series at lower hierarchical levels and then perform\n",
"aggregation to higher levels.\n",
"\n",
"$$\n",
"\\hat y_{A,h} = \\hat y_{AA,h} + \\hat y_{AB,h}\n",
"$$\n",
"\n",
"$$\n",
"\\hat y_{B,h} = \\hat y_{BA,h} + \\hat y_{BB,h} + \\hat y_{BC,h}\n",
"$$\n",
"\n",
"where $h$ - forecast horizon."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In matrix notation:\n",
"\n",
"$$\n",
"\\begin{equation}\n",
" \\begin{bmatrix}\n",
" \\hat y_{A,h} \\\\ \\hat y_{B,h}\n",
" \\end{bmatrix}\n",
" =\n",
" \\begin{bmatrix}\n",
" 1 & 1 & 0 & 0 & 0 \\\\\n",
" 0 & 0 & 1 & 1 & 1\n",
" \\end{bmatrix}\n",
" \\begin{bmatrix}\n",
" \\hat y_{AA,h} \\\\ \\hat y_{AB,h} \\\\ \\hat y_{BA,h} \\\\ \\hat y_{BB,h} \\\\ \\hat y_{BC,h}\n",
" \\end{bmatrix}\n",
"\\end{equation}\n",
"$$\n",
"\n",
"An advantage of this approach is that we are forecasting at the bottom-level of a structure and are able to capture\n",
"all the patterns of the individual series. On the other hand, bottom-level data can be quite noisy and more challenging\n",
"to model and forecast."
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [],
"source": [
"from etna.reconciliation import BottomUpReconciliator"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To create `BottomUpReconciliator` specify \"source\" and \"target\" levels for aggregation. Make sure that the source\n",
"level is lower in the hierarchy than the target level."
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [],
"source": [
"reconciliator = BottomUpReconciliator(target_level=\"region_level\", source_level=\"city_level\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The next step is mapping matrix estimation. To do so pass hierarchical dataset to `fit` method. Current dataset level\n",
"should be lower or equal to source level."
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[1, 1, 0, ..., 0, 0, 0],\n",
" [0, 0, 1, ..., 0, 0, 0],\n",
" [0, 0, 0, ..., 0, 0, 0],\n",
" ...,\n",
" [0, 0, 0, ..., 0, 0, 0],\n",
" [0, 0, 0, ..., 1, 0, 0],\n",
" [0, 0, 0, ..., 0, 1, 1]], dtype=int32)"
]
},
"execution_count": 23,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"reconciliator.fit(ts=hierarchical_ts)\n",
"reconciliator.mapping_matrix.toarray()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now `reconciliator` is ready to perform aggregation to target level."
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead tr th {\n",
" text-align: left;\n",
" }\n",
"\n",
" .dataframe thead tr:last-of-type th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr>\n",
" <th>segment</th>\n",
" <th>Bus_NSW_city</th>\n",
" <th>Bus_NSW_noncity</th>\n",
" <th>Bus_NT_city</th>\n",
" <th>Bus_NT_noncity</th>\n",
" <th>Bus_QLD_city</th>\n",
" <th>Bus_QLD_noncity</th>\n",
" <th>Bus_SA_city</th>\n",
" <th>Bus_SA_noncity</th>\n",
" <th>Bus_TAS_city</th>\n",
" <th>Bus_TAS_noncity</th>\n",
" <th>Bus_VIC_city</th>\n",
" <th>Bus_VIC_noncity</th>\n",
" <th>Bus_WA_city</th>\n",
" <th>Bus_WA_noncity</th>\n",
" <th>Hol_NSW_city</th>\n",
" <th>Hol_NSW_noncity</th>\n",
" <th>Hol_NT_city</th>\n",
" <th>Hol_NT_noncity</th>\n",
" <th>Hol_QLD_city</th>\n",
" <th>Hol_QLD_noncity</th>\n",
" <th>Hol_SA_city</th>\n",
" <th>Hol_SA_noncity</th>\n",
" <th>Hol_TAS_city</th>\n",
" <th>Hol_TAS_noncity</th>\n",
" <th>Hol_VIC_city</th>\n",
" <th>Hol_VIC_noncity</th>\n",
" <th>Hol_WA_city</th>\n",
" <th>Hol_WA_noncity</th>\n",
" <th>Oth_NSW_city</th>\n",
" <th>Oth_NSW_noncity</th>\n",
" <th>Oth_NT_city</th>\n",
" <th>Oth_NT_noncity</th>\n",
" <th>Oth_QLD_city</th>\n",
" <th>Oth_QLD_noncity</th>\n",
" <th>Oth_SA_city</th>\n",
" <th>Oth_SA_noncity</th>\n",
" <th>Oth_TAS_city</th>\n",
" <th>Oth_TAS_noncity</th>\n",
" <th>Oth_VIC_city</th>\n",
" <th>Oth_VIC_noncity</th>\n",
" <th>Oth_WA_city</th>\n",
" <th>Oth_WA_noncity</th>\n",
" <th>VFR_NSW_city</th>\n",
" <th>VFR_NSW_noncity</th>\n",
" <th>VFR_NT_city</th>\n",
" <th>VFR_NT_noncity</th>\n",
" <th>VFR_QLD_city</th>\n",
" <th>VFR_QLD_noncity</th>\n",
" <th>VFR_SA_city</th>\n",
" <th>VFR_SA_noncity</th>\n",
" <th>VFR_TAS_city</th>\n",
" <th>VFR_TAS_noncity</th>\n",
" <th>VFR_VIC_city</th>\n",
" <th>VFR_VIC_noncity</th>\n",
" <th>VFR_WA_city</th>\n",
" <th>VFR_WA_noncity</th>\n",
" </tr>\n",
" <tr>\n",
" <th>feature</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" </tr>\n",
" <tr>\n",
" <th>timestamp</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>2006-01-01</th>\n",
" <td>1201</td>\n",
" <td>1684</td>\n",
" <td>136</td>\n",
" <td>80</td>\n",
" <td>1111</td>\n",
" <td>982</td>\n",
" <td>388</td>\n",
" <td>456</td>\n",
" <td>116</td>\n",
" <td>107</td>\n",
" <td>1164</td>\n",
" <td>984</td>\n",
" <td>532</td>\n",
" <td>874</td>\n",
" <td>3096</td>\n",
" <td>14493</td>\n",
" <td>101</td>\n",
" <td>86</td>\n",
" <td>4688</td>\n",
" <td>4390</td>\n",
" <td>888</td>\n",
" <td>2201</td>\n",
" <td>619</td>\n",
" <td>1483</td>\n",
" <td>2531</td>\n",
" <td>7881</td>\n",
" <td>1383</td>\n",
" <td>2066</td>\n",
" <td>396</td>\n",
" <td>510</td>\n",
" <td>35</td>\n",
" <td>8</td>\n",
" <td>431</td>\n",
" <td>271</td>\n",
" <td>244</td>\n",
" <td>73</td>\n",
" <td>76</td>\n",
" <td>24</td>\n",
" <td>181</td>\n",
" <td>286</td>\n",
" <td>168</td>\n",
" <td>37</td>\n",
" <td>2709</td>\n",
" <td>6689</td>\n",
" <td>28</td>\n",
" <td>9</td>\n",
" <td>3003</td>\n",
" <td>2287</td>\n",
" <td>1324</td>\n",
" <td>869</td>\n",
" <td>602</td>\n",
" <td>748</td>\n",
" <td>2565</td>\n",
" <td>3428</td>\n",
" <td>1019</td>\n",
" <td>762</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-02-01</th>\n",
" <td>2020</td>\n",
" <td>2281</td>\n",
" <td>138</td>\n",
" <td>170</td>\n",
" <td>776</td>\n",
" <td>1448</td>\n",
" <td>346</td>\n",
" <td>403</td>\n",
" <td>83</td>\n",
" <td>290</td>\n",
" <td>1014</td>\n",
" <td>811</td>\n",
" <td>356</td>\n",
" <td>1687</td>\n",
" <td>1479</td>\n",
" <td>9548</td>\n",
" <td>201</td>\n",
" <td>297</td>\n",
" <td>2320</td>\n",
" <td>3990</td>\n",
" <td>521</td>\n",
" <td>1414</td>\n",
" <td>409</td>\n",
" <td>689</td>\n",
" <td>1439</td>\n",
" <td>4586</td>\n",
" <td>1059</td>\n",
" <td>1395</td>\n",
" <td>657</td>\n",
" <td>581</td>\n",
" <td>69</td>\n",
" <td>39</td>\n",
" <td>669</td>\n",
" <td>170</td>\n",
" <td>142</td>\n",
" <td>221</td>\n",
" <td>36</td>\n",
" <td>61</td>\n",
" <td>229</td>\n",
" <td>323</td>\n",
" <td>170</td>\n",
" <td>99</td>\n",
" <td>2184</td>\n",
" <td>5645</td>\n",
" <td>168</td>\n",
" <td>349</td>\n",
" <td>1957</td>\n",
" <td>2945</td>\n",
" <td>806</td>\n",
" <td>639</td>\n",
" <td>257</td>\n",
" <td>266</td>\n",
" <td>1852</td>\n",
" <td>2255</td>\n",
" <td>750</td>\n",
" <td>603</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-03-01</th>\n",
" <td>1975</td>\n",
" <td>2118</td>\n",
" <td>452</td>\n",
" <td>1084</td>\n",
" <td>1079</td>\n",
" <td>2300</td>\n",
" <td>390</td>\n",
" <td>360</td>\n",
" <td>196</td>\n",
" <td>107</td>\n",
" <td>1153</td>\n",
" <td>791</td>\n",
" <td>440</td>\n",
" <td>1120</td>\n",
" <td>1609</td>\n",
" <td>7301</td>\n",
" <td>619</td>\n",
" <td>745</td>\n",
" <td>4758</td>\n",
" <td>6975</td>\n",
" <td>476</td>\n",
" <td>1093</td>\n",
" <td>127</td>\n",
" <td>331</td>\n",
" <td>1488</td>\n",
" <td>3572</td>\n",
" <td>1101</td>\n",
" <td>2297</td>\n",
" <td>540</td>\n",
" <td>893</td>\n",
" <td>150</td>\n",
" <td>338</td>\n",
" <td>270</td>\n",
" <td>1164</td>\n",
" <td>397</td>\n",
" <td>315</td>\n",
" <td>32</td>\n",
" <td>23</td>\n",
" <td>128</td>\n",
" <td>318</td>\n",
" <td>380</td>\n",
" <td>1166</td>\n",
" <td>2225</td>\n",
" <td>5052</td>\n",
" <td>390</td>\n",
" <td>84</td>\n",
" <td>2619</td>\n",
" <td>2870</td>\n",
" <td>1078</td>\n",
" <td>375</td>\n",
" <td>130</td>\n",
" <td>261</td>\n",
" <td>1882</td>\n",
" <td>1929</td>\n",
" <td>953</td>\n",
" <td>734</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-04-01</th>\n",
" <td>1500</td>\n",
" <td>1963</td>\n",
" <td>243</td>\n",
" <td>160</td>\n",
" <td>1128</td>\n",
" <td>1752</td>\n",
" <td>255</td>\n",
" <td>635</td>\n",
" <td>70</td>\n",
" <td>228</td>\n",
" <td>1245</td>\n",
" <td>508</td>\n",
" <td>539</td>\n",
" <td>1252</td>\n",
" <td>1520</td>\n",
" <td>9138</td>\n",
" <td>164</td>\n",
" <td>250</td>\n",
" <td>3328</td>\n",
" <td>4781</td>\n",
" <td>571</td>\n",
" <td>1699</td>\n",
" <td>371</td>\n",
" <td>949</td>\n",
" <td>1906</td>\n",
" <td>3575</td>\n",
" <td>1128</td>\n",
" <td>2433</td>\n",
" <td>745</td>\n",
" <td>1157</td>\n",
" <td>172</td>\n",
" <td>453</td>\n",
" <td>214</td>\n",
" <td>535</td>\n",
" <td>194</td>\n",
" <td>260</td>\n",
" <td>48</td>\n",
" <td>43</td>\n",
" <td>270</td>\n",
" <td>336</td>\n",
" <td>410</td>\n",
" <td>1139</td>\n",
" <td>2918</td>\n",
" <td>5385</td>\n",
" <td>244</td>\n",
" <td>218</td>\n",
" <td>2097</td>\n",
" <td>2344</td>\n",
" <td>568</td>\n",
" <td>641</td>\n",
" <td>137</td>\n",
" <td>257</td>\n",
" <td>2208</td>\n",
" <td>2882</td>\n",
" <td>999</td>\n",
" <td>715</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-05-01</th>\n",
" <td>1196</td>\n",
" <td>2151</td>\n",
" <td>194</td>\n",
" <td>189</td>\n",
" <td>1192</td>\n",
" <td>1559</td>\n",
" <td>386</td>\n",
" <td>280</td>\n",
" <td>130</td>\n",
" <td>205</td>\n",
" <td>950</td>\n",
" <td>572</td>\n",
" <td>582</td>\n",
" <td>441</td>\n",
" <td>1958</td>\n",
" <td>14194</td>\n",
" <td>62</td>\n",
" <td>151</td>\n",
" <td>4930</td>\n",
" <td>5117</td>\n",
" <td>873</td>\n",
" <td>2150</td>\n",
" <td>523</td>\n",
" <td>1590</td>\n",
" <td>2517</td>\n",
" <td>8441</td>\n",
" <td>1560</td>\n",
" <td>2727</td>\n",
" <td>426</td>\n",
" <td>558</td>\n",
" <td>15</td>\n",
" <td>47</td>\n",
" <td>458</td>\n",
" <td>557</td>\n",
" <td>147</td>\n",
" <td>33</td>\n",
" <td>77</td>\n",
" <td>60</td>\n",
" <td>265</td>\n",
" <td>293</td>\n",
" <td>162</td>\n",
" <td>28</td>\n",
" <td>3154</td>\n",
" <td>7232</td>\n",
" <td>153</td>\n",
" <td>125</td>\n",
" <td>2703</td>\n",
" <td>2933</td>\n",
" <td>887</td>\n",
" <td>798</td>\n",
" <td>347</td>\n",
" <td>437</td>\n",
" <td>2988</td>\n",
" <td>3164</td>\n",
" <td>1396</td>\n",
" <td>630</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
"segment Bus_NSW_city Bus_NSW_noncity Bus_NT_city Bus_NT_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 1201 1684 136 80 \n",
"2006-02-01 2020 2281 138 170 \n",
"2006-03-01 1975 2118 452 1084 \n",
"2006-04-01 1500 1963 243 160 \n",
"2006-05-01 1196 2151 194 189 \n",
"\n",
"segment Bus_QLD_city Bus_QLD_noncity Bus_SA_city Bus_SA_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 1111 982 388 456 \n",
"2006-02-01 776 1448 346 403 \n",
"2006-03-01 1079 2300 390 360 \n",
"2006-04-01 1128 1752 255 635 \n",
"2006-05-01 1192 1559 386 280 \n",
"\n",
"segment Bus_TAS_city Bus_TAS_noncity Bus_VIC_city Bus_VIC_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 116 107 1164 984 \n",
"2006-02-01 83 290 1014 811 \n",
"2006-03-01 196 107 1153 791 \n",
"2006-04-01 70 228 1245 508 \n",
"2006-05-01 130 205 950 572 \n",
"\n",
"segment Bus_WA_city Bus_WA_noncity Hol_NSW_city Hol_NSW_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 532 874 3096 14493 \n",
"2006-02-01 356 1687 1479 9548 \n",
"2006-03-01 440 1120 1609 7301 \n",
"2006-04-01 539 1252 1520 9138 \n",
"2006-05-01 582 441 1958 14194 \n",
"\n",
"segment Hol_NT_city Hol_NT_noncity Hol_QLD_city Hol_QLD_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 101 86 4688 4390 \n",
"2006-02-01 201 297 2320 3990 \n",
"2006-03-01 619 745 4758 6975 \n",
"2006-04-01 164 250 3328 4781 \n",
"2006-05-01 62 151 4930 5117 \n",
"\n",
"segment Hol_SA_city Hol_SA_noncity Hol_TAS_city Hol_TAS_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 888 2201 619 1483 \n",
"2006-02-01 521 1414 409 689 \n",
"2006-03-01 476 1093 127 331 \n",
"2006-04-01 571 1699 371 949 \n",
"2006-05-01 873 2150 523 1590 \n",
"\n",
"segment Hol_VIC_city Hol_VIC_noncity Hol_WA_city Hol_WA_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 2531 7881 1383 2066 \n",
"2006-02-01 1439 4586 1059 1395 \n",
"2006-03-01 1488 3572 1101 2297 \n",
"2006-04-01 1906 3575 1128 2433 \n",
"2006-05-01 2517 8441 1560 2727 \n",
"\n",
"segment Oth_NSW_city Oth_NSW_noncity Oth_NT_city Oth_NT_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 396 510 35 8 \n",
"2006-02-01 657 581 69 39 \n",
"2006-03-01 540 893 150 338 \n",
"2006-04-01 745 1157 172 453 \n",
"2006-05-01 426 558 15 47 \n",
"\n",
"segment Oth_QLD_city Oth_QLD_noncity Oth_SA_city Oth_SA_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 431 271 244 73 \n",
"2006-02-01 669 170 142 221 \n",
"2006-03-01 270 1164 397 315 \n",
"2006-04-01 214 535 194 260 \n",
"2006-05-01 458 557 147 33 \n",
"\n",
"segment Oth_TAS_city Oth_TAS_noncity Oth_VIC_city Oth_VIC_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 76 24 181 286 \n",
"2006-02-01 36 61 229 323 \n",
"2006-03-01 32 23 128 318 \n",
"2006-04-01 48 43 270 336 \n",
"2006-05-01 77 60 265 293 \n",
"\n",
"segment Oth_WA_city Oth_WA_noncity VFR_NSW_city VFR_NSW_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 168 37 2709 6689 \n",
"2006-02-01 170 99 2184 5645 \n",
"2006-03-01 380 1166 2225 5052 \n",
"2006-04-01 410 1139 2918 5385 \n",
"2006-05-01 162 28 3154 7232 \n",
"\n",
"segment VFR_NT_city VFR_NT_noncity VFR_QLD_city VFR_QLD_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 28 9 3003 2287 \n",
"2006-02-01 168 349 1957 2945 \n",
"2006-03-01 390 84 2619 2870 \n",
"2006-04-01 244 218 2097 2344 \n",
"2006-05-01 153 125 2703 2933 \n",
"\n",
"segment VFR_SA_city VFR_SA_noncity VFR_TAS_city VFR_TAS_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 1324 869 602 748 \n",
"2006-02-01 806 639 257 266 \n",
"2006-03-01 1078 375 130 261 \n",
"2006-04-01 568 641 137 257 \n",
"2006-05-01 887 798 347 437 \n",
"\n",
"segment VFR_VIC_city VFR_VIC_noncity VFR_WA_city VFR_WA_noncity \n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 2565 3428 1019 762 \n",
"2006-02-01 1852 2255 750 603 \n",
"2006-03-01 1882 1929 953 734 \n",
"2006-04-01 2208 2882 999 715 \n",
"2006-05-01 2988 3164 1396 630 "
]
},
"execution_count": 24,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"reconciliator.aggregate(ts=hierarchical_ts).head(5)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`HierarchicalPipeline` provides the ability to perform forecasts reconciliation in pipeline.\n",
"A couple of points to keep in mind while working with this type of pipeline:\n",
"\n",
"1. Transforms and model work with the dataset on the **source** level.\n",
"2. Forecasts are automatically reconciliated to the **target** level, metrics reported for **target** level as well."
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
"[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.3s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 2 out of 2 | elapsed: 0.7s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 3 out of 3 | elapsed: 1.0s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 3 out of 3 | elapsed: 1.0s finished\n",
"[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
"[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.4s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 2 out of 2 | elapsed: 0.7s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 3 out of 3 | elapsed: 1.0s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 3 out of 3 | elapsed: 1.0s finished\n",
"[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
"[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.1s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 2 out of 2 | elapsed: 0.2s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 3 out of 3 | elapsed: 0.3s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 3 out of 3 | elapsed: 0.3s finished\n"
]
}
],
"source": [
"pipeline = HierarchicalPipeline(\n",
" transforms=[\n",
" LagTransform(in_column=\"target\", lags=[1, 2, 3, 4, 6, 12]),\n",
" ],\n",
" model=LinearPerSegmentModel(),\n",
" reconciliator=BottomUpReconciliator(target_level=\"region_level\", source_level=\"city_level\"),\n",
")\n",
"\n",
"bottom_up_metrics, _, _ = pipeline.backtest(ts=hierarchical_ts, metrics=[SMAPE()], n_folds=3, aggregate_metrics=True)\n",
"bottom_up_metrics = bottom_up_metrics.set_index(\"segment\").add_suffix(\"_bottom_up\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 3.2. Top-down approach <a class=\"anchor\" id=\"chapter3_2\"></a>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Top-down approach is based on the idea of generating forecasts for time series at higher hierarchy levels and then\n",
"performing disaggregation to lower levels. This approach can be expressed with the following formula:\n",
"\n",
"$$\n",
"\\begin{align*}\n",
"\\hat y_{AA,h} = p_{AA} \\hat y_A, &&\n",
"\\hat y_{AB,h} = p_{AB} \\hat y_A, &&\n",
"\\hat y_{BA,h} = p_{BA} \\hat y_B, &&\n",
"\\hat y_{BB,h} = p_{BB} \\hat y_B, &&\n",
"\\hat y_{BC,h} = p_{BC} \\hat y_B\n",
"\\end{align*}\n",
"$$\n",
"\n",
"In matrix notations this equation could be rewritten as:\n",
"\n",
"$$\n",
"\\begin{equation}\n",
" \\begin{bmatrix}\n",
" \\hat y_{AA,h} \\\\ \\hat y_{AB,h} \\\\ \\hat y_{BA,h} \\\\ \\hat y_{BB,h} \\\\ \\hat y_{BC,h}\n",
" \\end{bmatrix}\n",
" =\n",
" \\begin{bmatrix}\n",
" p_{AA} & 0 & 0 & 0 & 0 \\\\\n",
" 0 & p_{AB} & 0 & 0 & 0 \\\\\n",
" 0 & 0 & p_{BA} & 0 & 0 \\\\\n",
" 0 & 0 & 0 & p_{BB} & 0 \\\\\n",
" 0 & 0 & 0 & 0 & p_{BC} \\\\\n",
" \\end{bmatrix}\n",
" \\begin{bmatrix}\n",
" 1 & 0 \\\\\n",
" 1 & 0 \\\\\n",
" 0 & 1 \\\\\n",
" 0 & 1 \\\\\n",
" 0 & 1 \\\\\n",
" \\end{bmatrix}\n",
" \\begin{bmatrix}\n",
" \\hat y_{A,h} \\\\ \\hat y_{B,h}\n",
" \\end{bmatrix}\n",
" =\n",
" P S^T\n",
" \\begin{bmatrix}\n",
" \\hat y_{A,h} \\\\ \\hat y_{B,h}\n",
" \\end{bmatrix}\n",
"\\end{equation}\n",
"$$\n",
"\n",
"The main challenge of this approach is proportions estimation.\n",
"In ETNA library, there are two methods available:\n",
"\n",
"* Average historical proportions (AHP)\n",
"* Proportions of the historical averages (PHA)\n",
"\n",
"**Average historical proportions**\n",
"\n",
"This method is based on averaging historical proportions:\n",
"\n",
"$$\n",
"\\begin{equation}\n",
"\\large p_i = \\frac{1}{n} \\sum_{t = T - n}^{T} \\frac{y_{i, t}}{y_t}\n",
"\\end{equation}\n",
"$$\n",
"\n",
"where $n$ - window size, $T$ - latest timestamp, $y_{i, t}$ - time series at the lower level, $y_t$ - time series at\n",
"the higher level. Both $y_{i, t}$ and $y_t$ have hierarchical relationship.\n",
"\n",
"**Proportions of the historical averages**\n",
"This approach uses a proportion of the averages for estimation:\n",
"\n",
"$$\n",
"\\begin{equation}\n",
"\\large p_i = \\sum_{t = T - n}^{T} \\frac{y_{i, t}}{n} \\Bigg / \\sum_{t = T - n}^{T} \\frac{y_t}{n}\n",
"\\end{equation}\n",
"$$\n",
"\n",
"where $n$ - window size, $T$ - latest timestamp, $y_{i, t}$ - time series at the lower level, $y_t$ - time series at\n",
"the higher level. Both $y_{i, t}$ and $y_t$ have hierarchical relationship.\n",
"\n",
"Described methods require only series at the higher level for forecasting. Advantages of this approach are: simplicity and\n",
"reliability. Loss of information is main disadvantage of the approach.\n",
"\n",
"This method can be useful when it is needed to forecast lower level series, but some of them have more noise.\n",
"Aggregation to a higher level and reconciliation back helps to use more meaningful information while modeling."
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [],
"source": [
"from etna.reconciliation import TopDownReconciliator"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`TopDownReconciliator` accepts four arguments in its constructor. You need to specify reconciliation levels,\n",
"a method and a window size. First, let's look at the AHP top-down reconciliation method."
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [],
"source": [
"ahp_reconciliator = TopDownReconciliator(\n",
" target_level=\"region_level\", source_level=\"reason_level\", method=\"AHP\", period=6\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The top-down approach has slightly different dataset levels requirements in comparison to the bottom-up method.\n",
"Here source level should be higher than the target level, and the current dataset level should not be higher\n",
"than the target level.\n",
"\n",
"After all level requirements met and the reconciliator is fitted, we can obtain the mapping matrix. Note, that now\n",
"mapping matrix contains reconciliation proportions, and not only zeros and ones.\n",
"\n",
"Columns of the top-down mapping matrix correspond to segments at the source level of the hierarchy, and rows to\n",
"the segments at the target level."
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[0.29517217, 0. , 0. , 0. ],\n",
" [0.17522331, 0. , 0. , 0. ],\n",
" [0.29906179, 0. , 0. , 0. ],\n",
" [0.06509802, 0. , 0. , 0. ],\n",
" [0.10138424, 0. , 0. , 0. ],\n",
" [0.0348691 , 0. , 0. , 0. ],\n",
" [0.02919136, 0. , 0. , 0. ],\n",
" [0. , 0.35663824, 0. , 0. ],\n",
" [0. , 0.19596791, 0. , 0. ],\n",
" [0. , 0.25065754, 0. , 0. ],\n",
" [0. , 0.06313639, 0. , 0. ],\n",
" [0. , 0.09261382, 0. , 0. ],\n",
" [0. , 0.02383924, 0. , 0. ],\n",
" [0. , 0.01714686, 0. , 0. ],\n",
" [0. , 0. , 0.29766462, 0. ],\n",
" [0. , 0. , 0.16667059, 0. ],\n",
" [0. , 0. , 0.27550314, 0. ],\n",
" [0. , 0. , 0.0654707 , 0. ],\n",
" [0. , 0. , 0.13979554, 0. ],\n",
" [0. , 0. , 0.0245672 , 0. ],\n",
" [0. , 0. , 0.03032821, 0. ],\n",
" [0. , 0. , 0. , 0.29191277],\n",
" [0. , 0. , 0. , 0.15036933],\n",
" [0. , 0. , 0. , 0.25667986],\n",
" [0. , 0. , 0. , 0.09445469],\n",
" [0. , 0. , 0. , 0.1319362 ],\n",
" [0. , 0. , 0. , 0.03209989],\n",
" [0. , 0. , 0. , 0.04254726]])"
]
},
"execution_count": 28,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ahp_reconciliator.fit(ts=hierarchical_ts)\n",
"ahp_reconciliator.mapping_matrix.toarray()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let’s fit `HierarchicalPipeline` with **AHP** method."
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
"[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.2s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 2 out of 2 | elapsed: 0.3s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 3 out of 3 | elapsed: 0.4s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 3 out of 3 | elapsed: 0.4s finished\n",
"[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
"[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.1s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 2 out of 2 | elapsed: 0.2s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 3 out of 3 | elapsed: 0.3s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 3 out of 3 | elapsed: 0.3s finished\n",
"[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
"[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.1s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 2 out of 2 | elapsed: 0.2s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 3 out of 3 | elapsed: 0.3s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 3 out of 3 | elapsed: 0.3s finished\n"
]
}
],
"source": [
"reconciliator = TopDownReconciliator(target_level=\"region_level\", source_level=\"reason_level\", method=\"AHP\", period=9)\n",
"\n",
"pipeline = HierarchicalPipeline(\n",
" transforms=[\n",
" LagTransform(in_column=\"target\", lags=[1, 2, 3, 4, 6, 12]),\n",
" ],\n",
" model=LinearPerSegmentModel(),\n",
" reconciliator=reconciliator,\n",
")\n",
"\n",
"ahp_metrics, _, _ = pipeline.backtest(ts=hierarchical_ts, metrics=[SMAPE()], n_folds=3, aggregate_metrics=True)\n",
"ahp_metrics = ahp_metrics.set_index(\"segment\").add_suffix(\"_ahp\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To use another top-down proportion estimation method pass different method name to the `TopDownReconciliator` constructor.\n",
"Let's take a look at the PHA method."
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {},
"outputs": [],
"source": [
"pha_reconciliator = TopDownReconciliator(\n",
" target_level=\"region_level\", source_level=\"reason_level\", method=\"PHA\", period=6\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It should be noted that the fitted mapping matrix has the same structure as in the previous method, but with slightly\n",
"different non-zero values."
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[0.29761574, 0. , 0. , 0. ],\n",
" [0.17910202, 0. , 0. , 0. ],\n",
" [0.29400697, 0. , 0. , 0. ],\n",
" [0.0651224 , 0. , 0. , 0. ],\n",
" [0.10000206, 0. , 0. , 0. ],\n",
" [0.03596948, 0. , 0. , 0. ],\n",
" [0.02818132, 0. , 0. , 0. ],\n",
" [0. , 0.35710317, 0. , 0. ],\n",
" [0. , 0.19744442, 0. , 0. ],\n",
" [0. , 0.24879185, 0. , 0. ],\n",
" [0. , 0.06362301, 0. , 0. ],\n",
" [0. , 0.09206311, 0. , 0. ],\n",
" [0. , 0.02404128, 0. , 0. ],\n",
" [0. , 0.01693316, 0. , 0. ],\n",
" [0. , 0. , 0.29730368, 0. ],\n",
" [0. , 0. , 0.16779538, 0. ],\n",
" [0. , 0. , 0.27544335, 0. ],\n",
" [0. , 0. , 0.06506127, 0. ],\n",
" [0. , 0. , 0.139399 , 0. ],\n",
" [0. , 0. , 0.02441176, 0. ],\n",
" [0. , 0. , 0.03058557, 0. ],\n",
" [0. , 0. , 0. , 0.28940705],\n",
" [0. , 0. , 0. , 0.14772684],\n",
" [0. , 0. , 0. , 0.26106345],\n",
" [0. , 0. , 0. , 0.09481879],\n",
" [0. , 0. , 0. , 0.13193001],\n",
" [0. , 0. , 0. , 0.03034655],\n",
" [0. , 0. , 0. , 0.04470731]])"
]
},
"execution_count": 31,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pha_reconciliator.fit(ts=hierarchical_ts)\n",
"pha_reconciliator.mapping_matrix.toarray()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let’s fit `HierarchicalPipeline` with **PHA** method."
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
"[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.1s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 2 out of 2 | elapsed: 0.2s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 3 out of 3 | elapsed: 0.3s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 3 out of 3 | elapsed: 0.3s finished\n",
"[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
"[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.1s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 2 out of 2 | elapsed: 0.2s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 3 out of 3 | elapsed: 0.3s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 3 out of 3 | elapsed: 0.3s finished\n",
"[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
"[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.1s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 2 out of 2 | elapsed: 0.2s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 3 out of 3 | elapsed: 0.3s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 3 out of 3 | elapsed: 0.3s finished\n"
]
}
],
"source": [
"reconciliator = TopDownReconciliator(target_level=\"region_level\", source_level=\"reason_level\", method=\"PHA\", period=9)\n",
"\n",
"pipeline = HierarchicalPipeline(\n",
" transforms=[\n",
" LagTransform(in_column=\"target\", lags=[1, 2, 3, 4, 6, 12]),\n",
" ],\n",
" model=LinearPerSegmentModel(),\n",
" reconciliator=reconciliator,\n",
")\n",
"\n",
"pha_metrics, _, _ = pipeline.backtest(ts=hierarchical_ts, metrics=[SMAPE()], n_folds=3, aggregate_metrics=True)\n",
"pha_metrics = pha_metrics.set_index(\"segment\").add_suffix(\"_pha\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Finally, let's forecast the middle level series directly."
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
"[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.2s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 2 out of 2 | elapsed: 0.3s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 3 out of 3 | elapsed: 0.5s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 3 out of 3 | elapsed: 0.5s finished\n",
"[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
"[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.2s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 2 out of 2 | elapsed: 0.4s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 3 out of 3 | elapsed: 0.6s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 3 out of 3 | elapsed: 0.6s finished\n",
"[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
"[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.1s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 2 out of 2 | elapsed: 0.2s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 3 out of 3 | elapsed: 0.2s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 3 out of 3 | elapsed: 0.2s finished\n"
]
}
],
"source": [
"region_level_ts = hierarchical_ts.get_level_dataset(target_level=\"region_level\")\n",
"\n",
"pipeline = Pipeline(\n",
" transforms=[\n",
" LagTransform(in_column=\"target\", lags=[1, 2, 3, 4, 6, 12]),\n",
" ],\n",
" model=LinearPerSegmentModel(),\n",
")\n",
"\n",
"region_level_metric, _, _ = pipeline.backtest(ts=region_level_ts, metrics=[SMAPE()], n_folds=3, aggregate_metrics=True)\n",
"\n",
"region_level_metric = region_level_metric.set_index(\"segment\").add_suffix(\"_region_level\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we can take a look at metrics and compare methods."
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>SMAPE_bottom_up</th>\n",
" <th>SMAPE_ahp</th>\n",
" <th>SMAPE_pha</th>\n",
" <th>SMAPE_region_level</th>\n",
" </tr>\n",
" <tr>\n",
" <th>segment</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>Bus_NSW</th>\n",
" <td>5.270422</td>\n",
" <td>5.015657</td>\n",
" <td>4.814298</td>\n",
" <td>7.059593</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Bus_NT</th>\n",
" <td>25.765018</td>\n",
" <td>16.629957</td>\n",
" <td>16.210839</td>\n",
" <td>12.718326</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Bus_QLD</th>\n",
" <td>18.254162</td>\n",
" <td>3.653159</td>\n",
" <td>3.769619</td>\n",
" <td>16.946310</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Bus_SA</th>\n",
" <td>15.282322</td>\n",
" <td>16.694209</td>\n",
" <td>16.940077</td>\n",
" <td>16.967729</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Bus_TAS</th>\n",
" <td>30.695013</td>\n",
" <td>27.307973</td>\n",
" <td>26.523334</td>\n",
" <td>24.133560</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Bus_VIC</th>\n",
" <td>15.116212</td>\n",
" <td>4.571119</td>\n",
" <td>4.359536</td>\n",
" <td>14.334224</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Bus_WA</th>\n",
" <td>10.009304</td>\n",
" <td>17.026675</td>\n",
" <td>16.926661</td>\n",
" <td>13.688767</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Hol_NSW</th>\n",
" <td>14.454165</td>\n",
" <td>12.399630</td>\n",
" <td>11.994909</td>\n",
" <td>17.043741</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Hol_NT</th>\n",
" <td>53.250687</td>\n",
" <td>44.744583</td>\n",
" <td>49.765493</td>\n",
" <td>55.005842</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Hol_QLD</th>\n",
" <td>9.624166</td>\n",
" <td>11.851425</td>\n",
" <td>13.983468</td>\n",
" <td>9.840566</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Hol_SA</th>\n",
" <td>8.202269</td>\n",
" <td>29.610151</td>\n",
" <td>29.559022</td>\n",
" <td>9.343553</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Hol_TAS</th>\n",
" <td>51.592386</td>\n",
" <td>45.207863</td>\n",
" <td>45.832426</td>\n",
" <td>48.459946</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Hol_VIC</th>\n",
" <td>7.125269</td>\n",
" <td>17.840854</td>\n",
" <td>16.883132</td>\n",
" <td>7.421170</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Hol_WA</th>\n",
" <td>16.415138</td>\n",
" <td>23.793483</td>\n",
" <td>25.208175</td>\n",
" <td>18.636453</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Oth_NSW</th>\n",
" <td>29.987238</td>\n",
" <td>34.912603</td>\n",
" <td>34.685216</td>\n",
" <td>23.790318</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Oth_NT</th>\n",
" <td>98.032493</td>\n",
" <td>49.537936</td>\n",
" <td>54.553245</td>\n",
" <td>94.936352</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Oth_QLD</th>\n",
" <td>31.464303</td>\n",
" <td>25.776690</td>\n",
" <td>26.905804</td>\n",
" <td>33.150413</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Oth_SA</th>\n",
" <td>24.098806</td>\n",
" <td>31.959505</td>\n",
" <td>32.030506</td>\n",
" <td>28.638613</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Oth_TAS</th>\n",
" <td>55.187208</td>\n",
" <td>36.601269</td>\n",
" <td>34.369745</td>\n",
" <td>37.551688</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Oth_VIC</th>\n",
" <td>31.365795</td>\n",
" <td>29.007615</td>\n",
" <td>26.359892</td>\n",
" <td>34.186411</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Oth_WA</th>\n",
" <td>26.894592</td>\n",
" <td>22.888604</td>\n",
" <td>25.102690</td>\n",
" <td>24.231688</td>\n",
" </tr>\n",
" <tr>\n",
" <th>VFR_NSW</th>\n",
" <td>4.977585</td>\n",
" <td>10.066277</td>\n",
" <td>10.065904</td>\n",
" <td>4.568697</td>\n",
" </tr>\n",
" <tr>\n",
" <th>VFR_NT</th>\n",
" <td>46.565888</td>\n",
" <td>23.537350</td>\n",
" <td>23.650015</td>\n",
" <td>40.369501</td>\n",
" </tr>\n",
" <tr>\n",
" <th>VFR_QLD</th>\n",
" <td>12.675037</td>\n",
" <td>13.975730</td>\n",
" <td>14.587780</td>\n",
" <td>4.299708</td>\n",
" </tr>\n",
" <tr>\n",
" <th>VFR_SA</th>\n",
" <td>15.613376</td>\n",
" <td>16.390740</td>\n",
" <td>15.695589</td>\n",
" <td>5.758916</td>\n",
" </tr>\n",
" <tr>\n",
" <th>VFR_TAS</th>\n",
" <td>33.182773</td>\n",
" <td>25.492236</td>\n",
" <td>25.177505</td>\n",
" <td>33.747570</td>\n",
" </tr>\n",
" <tr>\n",
" <th>VFR_VIC</th>\n",
" <td>9.237164</td>\n",
" <td>17.037192</td>\n",
" <td>16.725030</td>\n",
" <td>12.894987</td>\n",
" </tr>\n",
" <tr>\n",
" <th>VFR_WA</th>\n",
" <td>17.416115</td>\n",
" <td>15.014164</td>\n",
" <td>15.509316</td>\n",
" <td>14.953232</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" SMAPE_bottom_up SMAPE_ahp SMAPE_pha SMAPE_region_level\n",
"segment \n",
"Bus_NSW 5.270422 5.015657 4.814298 7.059593\n",
"Bus_NT 25.765018 16.629957 16.210839 12.718326\n",
"Bus_QLD 18.254162 3.653159 3.769619 16.946310\n",
"Bus_SA 15.282322 16.694209 16.940077 16.967729\n",
"Bus_TAS 30.695013 27.307973 26.523334 24.133560\n",
"Bus_VIC 15.116212 4.571119 4.359536 14.334224\n",
"Bus_WA 10.009304 17.026675 16.926661 13.688767\n",
"Hol_NSW 14.454165 12.399630 11.994909 17.043741\n",
"Hol_NT 53.250687 44.744583 49.765493 55.005842\n",
"Hol_QLD 9.624166 11.851425 13.983468 9.840566\n",
"Hol_SA 8.202269 29.610151 29.559022 9.343553\n",
"Hol_TAS 51.592386 45.207863 45.832426 48.459946\n",
"Hol_VIC 7.125269 17.840854 16.883132 7.421170\n",
"Hol_WA 16.415138 23.793483 25.208175 18.636453\n",
"Oth_NSW 29.987238 34.912603 34.685216 23.790318\n",
"Oth_NT 98.032493 49.537936 54.553245 94.936352\n",
"Oth_QLD 31.464303 25.776690 26.905804 33.150413\n",
"Oth_SA 24.098806 31.959505 32.030506 28.638613\n",
"Oth_TAS 55.187208 36.601269 34.369745 37.551688\n",
"Oth_VIC 31.365795 29.007615 26.359892 34.186411\n",
"Oth_WA 26.894592 22.888604 25.102690 24.231688\n",
"VFR_NSW 4.977585 10.066277 10.065904 4.568697\n",
"VFR_NT 46.565888 23.537350 23.650015 40.369501\n",
"VFR_QLD 12.675037 13.975730 14.587780 4.299708\n",
"VFR_SA 15.613376 16.390740 15.695589 5.758916\n",
"VFR_TAS 33.182773 25.492236 25.177505 33.747570\n",
"VFR_VIC 9.237164 17.037192 16.725030 12.894987\n",
"VFR_WA 17.416115 15.014164 15.509316 14.953232"
]
},
"execution_count": 34,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"all_metrics = pd.concat([bottom_up_metrics, ahp_metrics, pha_metrics, region_level_metric], axis=1)\n",
"\n",
"all_metrics"
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"SMAPE_bottom_up 25.634104\n",
"SMAPE_ahp 22.448023\n",
"SMAPE_pha 22.792472\n",
"SMAPE_region_level 23.738495\n",
"dtype: float64"
]
},
"execution_count": 35,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"all_metrics.mean()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The results presented above show that using reconciliation methods can improve forecasting quality\n",
"for some segments."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4. Exogenous variables for hierarchical forecasts <a class=\"anchor\" id=\"chapter4\"></a>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This section shows how exogenous variables can be added to a hierarchical `TSDataset`.\n",
"\n",
"Before adding exogenous variables to the dataset, we should decide at which level we should place them. Model fitting and\n",
"initial forecasting in the `HierarchicalPipeline` are made at the **source level**. So exogenous variables should be at the\n",
"**source level** as well.\n",
"\n",
"Let's try to add monthly indicators to our model."
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {},
"outputs": [],
"source": [
"from etna.datasets.utils import duplicate_data\n",
"\n",
"horizon = 3\n",
"exog_index = pd.date_range(\"2006-01-01\", periods=periods + horizon, freq=\"MS\")\n",
"\n",
"months_df = pd.DataFrame({\"timestamp\": exog_index.values, \"month\": exog_index.month.astype(\"category\")})\n",
"\n",
"reason_level_segments = hierarchical_ts.hierarchical_structure.get_level_segments(level_name=\"reason_level\")"
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead tr th {\n",
" text-align: left;\n",
" }\n",
"\n",
" .dataframe thead tr:last-of-type th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr>\n",
" <th>segment</th>\n",
" <th>Bus</th>\n",
" <th>Hol</th>\n",
" <th>Oth</th>\n",
" <th>VFR</th>\n",
" </tr>\n",
" <tr>\n",
" <th>feature</th>\n",
" <th>month</th>\n",
" <th>month</th>\n",
" <th>month</th>\n",
" <th>month</th>\n",
" </tr>\n",
" <tr>\n",
" <th>timestamp</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>2006-01-01</th>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-02-01</th>\n",
" <td>2</td>\n",
" <td>2</td>\n",
" <td>2</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-03-01</th>\n",
" <td>3</td>\n",
" <td>3</td>\n",
" <td>3</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-04-01</th>\n",
" <td>4</td>\n",
" <td>4</td>\n",
" <td>4</td>\n",
" <td>4</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-05-01</th>\n",
" <td>5</td>\n",
" <td>5</td>\n",
" <td>5</td>\n",
" <td>5</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
"segment Bus Hol Oth VFR\n",
"feature month month month month\n",
"timestamp \n",
"2006-01-01 1 1 1 1\n",
"2006-02-01 2 2 2 2\n",
"2006-03-01 3 3 3 3\n",
"2006-04-01 4 4 4 4\n",
"2006-05-01 5 5 5 5"
]
},
"execution_count": 37,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"months_ts = duplicate_data(df=months_df, segments=reason_level_segments)\n",
"months_ts.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Previous block showed how to create exogenous variables and convert to a hierarchical format manually.\n",
"Another way to convert exogenous variables to a hierarchical dataset is to use `TSDataset.to_hierarchical_dataset`."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"First, let's convert the dataframe to hierarchical long format."
]
},
{
"cell_type": "code",
"execution_count": 38,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>timestamp</th>\n",
" <th>month</th>\n",
" <th>reason</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>2006-01-01</td>\n",
" <td>1</td>\n",
" <td>Hol</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2006-02-01</td>\n",
" <td>2</td>\n",
" <td>Hol</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>2006-03-01</td>\n",
" <td>3</td>\n",
" <td>Hol</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>2006-04-01</td>\n",
" <td>4</td>\n",
" <td>Hol</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>2006-05-01</td>\n",
" <td>5</td>\n",
" <td>Hol</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" timestamp month reason\n",
"0 2006-01-01 1 Hol\n",
"1 2006-02-01 2 Hol\n",
"2 2006-03-01 3 Hol\n",
"3 2006-04-01 4 Hol\n",
"4 2006-05-01 5 Hol"
]
},
"execution_count": 38,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"months_ts = duplicate_data(df=months_df, segments=reason_level_segments, format=\"long\")\n",
"months_ts.rename(columns={\"segment\": \"reason\"}, inplace=True)\n",
"months_ts.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we are ready to use `to_hierarchical_dataset` method. When using this method with exogenous data\n",
"pass `return_hierarchy=False`, because we want to use hierarchical structure from target variables.\n",
"Setting `keep_level_columns=True` will add level columns to the result dataframe."
]
},
{
"cell_type": "code",
"execution_count": 39,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead tr th {\n",
" text-align: left;\n",
" }\n",
"\n",
" .dataframe thead tr:last-of-type th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr>\n",
" <th>segment</th>\n",
" <th>Bus</th>\n",
" <th>Hol</th>\n",
" <th>Oth</th>\n",
" <th>VFR</th>\n",
" </tr>\n",
" <tr>\n",
" <th>feature</th>\n",
" <th>month</th>\n",
" <th>month</th>\n",
" <th>month</th>\n",
" <th>month</th>\n",
" </tr>\n",
" <tr>\n",
" <th>timestamp</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>2006-01-01</th>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-02-01</th>\n",
" <td>2</td>\n",
" <td>2</td>\n",
" <td>2</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-03-01</th>\n",
" <td>3</td>\n",
" <td>3</td>\n",
" <td>3</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-04-01</th>\n",
" <td>4</td>\n",
" <td>4</td>\n",
" <td>4</td>\n",
" <td>4</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-05-01</th>\n",
" <td>5</td>\n",
" <td>5</td>\n",
" <td>5</td>\n",
" <td>5</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
"segment Bus Hol Oth VFR\n",
"feature month month month month\n",
"timestamp \n",
"2006-01-01 1 1 1 1\n",
"2006-02-01 2 2 2 2\n",
"2006-03-01 3 3 3 3\n",
"2006-04-01 4 4 4 4\n",
"2006-05-01 5 5 5 5"
]
},
"execution_count": 39,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"months_ts, _ = TSDataset.to_hierarchical_dataset(df=months_ts, level_columns=[\"reason\"], return_hierarchy=False)\n",
"months_ts.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"When dataframe with exogenous variables is prepared, create new hierarchical dataset with exogenous variables."
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {},
"outputs": [],
"source": [
"hierarchical_ts_w_exog = TSDataset(\n",
" df=hierarchical_df,\n",
" df_exog=months_ts,\n",
" hierarchical_structure=hierarchical_structure,\n",
" freq=\"MS\",\n",
" known_future=\"all\",\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 41,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'df_level=city_level, df_exog_level=reason_level'"
]
},
"execution_count": 41,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"f\"df_level={hierarchical_ts_w_exog.current_df_level}, df_exog_level={hierarchical_ts_w_exog.current_df_exog_level}\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here we can see different levels for the dataframes inside the dataset. In such case exogenous variables wouldn't be merged to target\n",
"variables."
]
},
{
"cell_type": "code",
"execution_count": 42,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead tr th {\n",
" text-align: left;\n",
" }\n",
"\n",
" .dataframe thead tr:last-of-type th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr>\n",
" <th>segment</th>\n",
" <th>Bus_NSW_city</th>\n",
" <th>Bus_NSW_noncity</th>\n",
" <th>Bus_NT_city</th>\n",
" <th>Bus_NT_noncity</th>\n",
" <th>Bus_QLD_city</th>\n",
" <th>Bus_QLD_noncity</th>\n",
" <th>Bus_SA_city</th>\n",
" <th>Bus_SA_noncity</th>\n",
" <th>Bus_TAS_city</th>\n",
" <th>Bus_TAS_noncity</th>\n",
" <th>Bus_VIC_city</th>\n",
" <th>Bus_VIC_noncity</th>\n",
" <th>Bus_WA_city</th>\n",
" <th>Bus_WA_noncity</th>\n",
" <th>Hol_NSW_city</th>\n",
" <th>Hol_NSW_noncity</th>\n",
" <th>Hol_NT_city</th>\n",
" <th>Hol_NT_noncity</th>\n",
" <th>Hol_QLD_city</th>\n",
" <th>Hol_QLD_noncity</th>\n",
" <th>Hol_SA_city</th>\n",
" <th>Hol_SA_noncity</th>\n",
" <th>Hol_TAS_city</th>\n",
" <th>Hol_TAS_noncity</th>\n",
" <th>Hol_VIC_city</th>\n",
" <th>Hol_VIC_noncity</th>\n",
" <th>Hol_WA_city</th>\n",
" <th>Hol_WA_noncity</th>\n",
" <th>Oth_NSW_city</th>\n",
" <th>Oth_NSW_noncity</th>\n",
" <th>Oth_NT_city</th>\n",
" <th>Oth_NT_noncity</th>\n",
" <th>Oth_QLD_city</th>\n",
" <th>Oth_QLD_noncity</th>\n",
" <th>Oth_SA_city</th>\n",
" <th>Oth_SA_noncity</th>\n",
" <th>Oth_TAS_city</th>\n",
" <th>Oth_TAS_noncity</th>\n",
" <th>Oth_VIC_city</th>\n",
" <th>Oth_VIC_noncity</th>\n",
" <th>Oth_WA_city</th>\n",
" <th>Oth_WA_noncity</th>\n",
" <th>VFR_NSW_city</th>\n",
" <th>VFR_NSW_noncity</th>\n",
" <th>VFR_NT_city</th>\n",
" <th>VFR_NT_noncity</th>\n",
" <th>VFR_QLD_city</th>\n",
" <th>VFR_QLD_noncity</th>\n",
" <th>VFR_SA_city</th>\n",
" <th>VFR_SA_noncity</th>\n",
" <th>VFR_TAS_city</th>\n",
" <th>VFR_TAS_noncity</th>\n",
" <th>VFR_VIC_city</th>\n",
" <th>VFR_VIC_noncity</th>\n",
" <th>VFR_WA_city</th>\n",
" <th>VFR_WA_noncity</th>\n",
" </tr>\n",
" <tr>\n",
" <th>feature</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" <th>target</th>\n",
" </tr>\n",
" <tr>\n",
" <th>timestamp</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>2006-01-01</th>\n",
" <td>1201</td>\n",
" <td>1684</td>\n",
" <td>136</td>\n",
" <td>80</td>\n",
" <td>1111</td>\n",
" <td>982</td>\n",
" <td>388</td>\n",
" <td>456</td>\n",
" <td>116</td>\n",
" <td>107</td>\n",
" <td>1164</td>\n",
" <td>984</td>\n",
" <td>532</td>\n",
" <td>874</td>\n",
" <td>3096</td>\n",
" <td>14493</td>\n",
" <td>101</td>\n",
" <td>86</td>\n",
" <td>4688</td>\n",
" <td>4390</td>\n",
" <td>888</td>\n",
" <td>2201</td>\n",
" <td>619</td>\n",
" <td>1483</td>\n",
" <td>2531</td>\n",
" <td>7881</td>\n",
" <td>1383</td>\n",
" <td>2066</td>\n",
" <td>396</td>\n",
" <td>510</td>\n",
" <td>35</td>\n",
" <td>8</td>\n",
" <td>431</td>\n",
" <td>271</td>\n",
" <td>244</td>\n",
" <td>73</td>\n",
" <td>76</td>\n",
" <td>24</td>\n",
" <td>181</td>\n",
" <td>286</td>\n",
" <td>168</td>\n",
" <td>37</td>\n",
" <td>2709</td>\n",
" <td>6689</td>\n",
" <td>28</td>\n",
" <td>9</td>\n",
" <td>3003</td>\n",
" <td>2287</td>\n",
" <td>1324</td>\n",
" <td>869</td>\n",
" <td>602</td>\n",
" <td>748</td>\n",
" <td>2565</td>\n",
" <td>3428</td>\n",
" <td>1019</td>\n",
" <td>762</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-02-01</th>\n",
" <td>2020</td>\n",
" <td>2281</td>\n",
" <td>138</td>\n",
" <td>170</td>\n",
" <td>776</td>\n",
" <td>1448</td>\n",
" <td>346</td>\n",
" <td>403</td>\n",
" <td>83</td>\n",
" <td>290</td>\n",
" <td>1014</td>\n",
" <td>811</td>\n",
" <td>356</td>\n",
" <td>1687</td>\n",
" <td>1479</td>\n",
" <td>9548</td>\n",
" <td>201</td>\n",
" <td>297</td>\n",
" <td>2320</td>\n",
" <td>3990</td>\n",
" <td>521</td>\n",
" <td>1414</td>\n",
" <td>409</td>\n",
" <td>689</td>\n",
" <td>1439</td>\n",
" <td>4586</td>\n",
" <td>1059</td>\n",
" <td>1395</td>\n",
" <td>657</td>\n",
" <td>581</td>\n",
" <td>69</td>\n",
" <td>39</td>\n",
" <td>669</td>\n",
" <td>170</td>\n",
" <td>142</td>\n",
" <td>221</td>\n",
" <td>36</td>\n",
" <td>61</td>\n",
" <td>229</td>\n",
" <td>323</td>\n",
" <td>170</td>\n",
" <td>99</td>\n",
" <td>2184</td>\n",
" <td>5645</td>\n",
" <td>168</td>\n",
" <td>349</td>\n",
" <td>1957</td>\n",
" <td>2945</td>\n",
" <td>806</td>\n",
" <td>639</td>\n",
" <td>257</td>\n",
" <td>266</td>\n",
" <td>1852</td>\n",
" <td>2255</td>\n",
" <td>750</td>\n",
" <td>603</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-03-01</th>\n",
" <td>1975</td>\n",
" <td>2118</td>\n",
" <td>452</td>\n",
" <td>1084</td>\n",
" <td>1079</td>\n",
" <td>2300</td>\n",
" <td>390</td>\n",
" <td>360</td>\n",
" <td>196</td>\n",
" <td>107</td>\n",
" <td>1153</td>\n",
" <td>791</td>\n",
" <td>440</td>\n",
" <td>1120</td>\n",
" <td>1609</td>\n",
" <td>7301</td>\n",
" <td>619</td>\n",
" <td>745</td>\n",
" <td>4758</td>\n",
" <td>6975</td>\n",
" <td>476</td>\n",
" <td>1093</td>\n",
" <td>127</td>\n",
" <td>331</td>\n",
" <td>1488</td>\n",
" <td>3572</td>\n",
" <td>1101</td>\n",
" <td>2297</td>\n",
" <td>540</td>\n",
" <td>893</td>\n",
" <td>150</td>\n",
" <td>338</td>\n",
" <td>270</td>\n",
" <td>1164</td>\n",
" <td>397</td>\n",
" <td>315</td>\n",
" <td>32</td>\n",
" <td>23</td>\n",
" <td>128</td>\n",
" <td>318</td>\n",
" <td>380</td>\n",
" <td>1166</td>\n",
" <td>2225</td>\n",
" <td>5052</td>\n",
" <td>390</td>\n",
" <td>84</td>\n",
" <td>2619</td>\n",
" <td>2870</td>\n",
" <td>1078</td>\n",
" <td>375</td>\n",
" <td>130</td>\n",
" <td>261</td>\n",
" <td>1882</td>\n",
" <td>1929</td>\n",
" <td>953</td>\n",
" <td>734</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-04-01</th>\n",
" <td>1500</td>\n",
" <td>1963</td>\n",
" <td>243</td>\n",
" <td>160</td>\n",
" <td>1128</td>\n",
" <td>1752</td>\n",
" <td>255</td>\n",
" <td>635</td>\n",
" <td>70</td>\n",
" <td>228</td>\n",
" <td>1245</td>\n",
" <td>508</td>\n",
" <td>539</td>\n",
" <td>1252</td>\n",
" <td>1520</td>\n",
" <td>9138</td>\n",
" <td>164</td>\n",
" <td>250</td>\n",
" <td>3328</td>\n",
" <td>4781</td>\n",
" <td>571</td>\n",
" <td>1699</td>\n",
" <td>371</td>\n",
" <td>949</td>\n",
" <td>1906</td>\n",
" <td>3575</td>\n",
" <td>1128</td>\n",
" <td>2433</td>\n",
" <td>745</td>\n",
" <td>1157</td>\n",
" <td>172</td>\n",
" <td>453</td>\n",
" <td>214</td>\n",
" <td>535</td>\n",
" <td>194</td>\n",
" <td>260</td>\n",
" <td>48</td>\n",
" <td>43</td>\n",
" <td>270</td>\n",
" <td>336</td>\n",
" <td>410</td>\n",
" <td>1139</td>\n",
" <td>2918</td>\n",
" <td>5385</td>\n",
" <td>244</td>\n",
" <td>218</td>\n",
" <td>2097</td>\n",
" <td>2344</td>\n",
" <td>568</td>\n",
" <td>641</td>\n",
" <td>137</td>\n",
" <td>257</td>\n",
" <td>2208</td>\n",
" <td>2882</td>\n",
" <td>999</td>\n",
" <td>715</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-05-01</th>\n",
" <td>1196</td>\n",
" <td>2151</td>\n",
" <td>194</td>\n",
" <td>189</td>\n",
" <td>1192</td>\n",
" <td>1559</td>\n",
" <td>386</td>\n",
" <td>280</td>\n",
" <td>130</td>\n",
" <td>205</td>\n",
" <td>950</td>\n",
" <td>572</td>\n",
" <td>582</td>\n",
" <td>441</td>\n",
" <td>1958</td>\n",
" <td>14194</td>\n",
" <td>62</td>\n",
" <td>151</td>\n",
" <td>4930</td>\n",
" <td>5117</td>\n",
" <td>873</td>\n",
" <td>2150</td>\n",
" <td>523</td>\n",
" <td>1590</td>\n",
" <td>2517</td>\n",
" <td>8441</td>\n",
" <td>1560</td>\n",
" <td>2727</td>\n",
" <td>426</td>\n",
" <td>558</td>\n",
" <td>15</td>\n",
" <td>47</td>\n",
" <td>458</td>\n",
" <td>557</td>\n",
" <td>147</td>\n",
" <td>33</td>\n",
" <td>77</td>\n",
" <td>60</td>\n",
" <td>265</td>\n",
" <td>293</td>\n",
" <td>162</td>\n",
" <td>28</td>\n",
" <td>3154</td>\n",
" <td>7232</td>\n",
" <td>153</td>\n",
" <td>125</td>\n",
" <td>2703</td>\n",
" <td>2933</td>\n",
" <td>887</td>\n",
" <td>798</td>\n",
" <td>347</td>\n",
" <td>437</td>\n",
" <td>2988</td>\n",
" <td>3164</td>\n",
" <td>1396</td>\n",
" <td>630</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
"segment Bus_NSW_city Bus_NSW_noncity Bus_NT_city Bus_NT_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 1201 1684 136 80 \n",
"2006-02-01 2020 2281 138 170 \n",
"2006-03-01 1975 2118 452 1084 \n",
"2006-04-01 1500 1963 243 160 \n",
"2006-05-01 1196 2151 194 189 \n",
"\n",
"segment Bus_QLD_city Bus_QLD_noncity Bus_SA_city Bus_SA_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 1111 982 388 456 \n",
"2006-02-01 776 1448 346 403 \n",
"2006-03-01 1079 2300 390 360 \n",
"2006-04-01 1128 1752 255 635 \n",
"2006-05-01 1192 1559 386 280 \n",
"\n",
"segment Bus_TAS_city Bus_TAS_noncity Bus_VIC_city Bus_VIC_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 116 107 1164 984 \n",
"2006-02-01 83 290 1014 811 \n",
"2006-03-01 196 107 1153 791 \n",
"2006-04-01 70 228 1245 508 \n",
"2006-05-01 130 205 950 572 \n",
"\n",
"segment Bus_WA_city Bus_WA_noncity Hol_NSW_city Hol_NSW_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 532 874 3096 14493 \n",
"2006-02-01 356 1687 1479 9548 \n",
"2006-03-01 440 1120 1609 7301 \n",
"2006-04-01 539 1252 1520 9138 \n",
"2006-05-01 582 441 1958 14194 \n",
"\n",
"segment Hol_NT_city Hol_NT_noncity Hol_QLD_city Hol_QLD_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 101 86 4688 4390 \n",
"2006-02-01 201 297 2320 3990 \n",
"2006-03-01 619 745 4758 6975 \n",
"2006-04-01 164 250 3328 4781 \n",
"2006-05-01 62 151 4930 5117 \n",
"\n",
"segment Hol_SA_city Hol_SA_noncity Hol_TAS_city Hol_TAS_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 888 2201 619 1483 \n",
"2006-02-01 521 1414 409 689 \n",
"2006-03-01 476 1093 127 331 \n",
"2006-04-01 571 1699 371 949 \n",
"2006-05-01 873 2150 523 1590 \n",
"\n",
"segment Hol_VIC_city Hol_VIC_noncity Hol_WA_city Hol_WA_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 2531 7881 1383 2066 \n",
"2006-02-01 1439 4586 1059 1395 \n",
"2006-03-01 1488 3572 1101 2297 \n",
"2006-04-01 1906 3575 1128 2433 \n",
"2006-05-01 2517 8441 1560 2727 \n",
"\n",
"segment Oth_NSW_city Oth_NSW_noncity Oth_NT_city Oth_NT_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 396 510 35 8 \n",
"2006-02-01 657 581 69 39 \n",
"2006-03-01 540 893 150 338 \n",
"2006-04-01 745 1157 172 453 \n",
"2006-05-01 426 558 15 47 \n",
"\n",
"segment Oth_QLD_city Oth_QLD_noncity Oth_SA_city Oth_SA_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 431 271 244 73 \n",
"2006-02-01 669 170 142 221 \n",
"2006-03-01 270 1164 397 315 \n",
"2006-04-01 214 535 194 260 \n",
"2006-05-01 458 557 147 33 \n",
"\n",
"segment Oth_TAS_city Oth_TAS_noncity Oth_VIC_city Oth_VIC_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 76 24 181 286 \n",
"2006-02-01 36 61 229 323 \n",
"2006-03-01 32 23 128 318 \n",
"2006-04-01 48 43 270 336 \n",
"2006-05-01 77 60 265 293 \n",
"\n",
"segment Oth_WA_city Oth_WA_noncity VFR_NSW_city VFR_NSW_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 168 37 2709 6689 \n",
"2006-02-01 170 99 2184 5645 \n",
"2006-03-01 380 1166 2225 5052 \n",
"2006-04-01 410 1139 2918 5385 \n",
"2006-05-01 162 28 3154 7232 \n",
"\n",
"segment VFR_NT_city VFR_NT_noncity VFR_QLD_city VFR_QLD_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 28 9 3003 2287 \n",
"2006-02-01 168 349 1957 2945 \n",
"2006-03-01 390 84 2619 2870 \n",
"2006-04-01 244 218 2097 2344 \n",
"2006-05-01 153 125 2703 2933 \n",
"\n",
"segment VFR_SA_city VFR_SA_noncity VFR_TAS_city VFR_TAS_noncity \\\n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 1324 869 602 748 \n",
"2006-02-01 806 639 257 266 \n",
"2006-03-01 1078 375 130 261 \n",
"2006-04-01 568 641 137 257 \n",
"2006-05-01 887 798 347 437 \n",
"\n",
"segment VFR_VIC_city VFR_VIC_noncity VFR_WA_city VFR_WA_noncity \n",
"feature target target target target \n",
"timestamp \n",
"2006-01-01 2565 3428 1019 762 \n",
"2006-02-01 1852 2255 750 603 \n",
"2006-03-01 1882 1929 953 734 \n",
"2006-04-01 2208 2882 999 715 \n",
"2006-05-01 2988 3164 1396 630 "
]
},
"execution_count": 42,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"hierarchical_ts_w_exog.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Exogenous data will be merged only when both dataframes are at the same level, so we can perform reconciliation to do this.\n",
"Right now, our dataset is lower, than the exogenous variables, so they aren't merged.\n",
"To perform aggregation to higher levels, we can use `get_level_dataset` method."
]
},
{
"cell_type": "code",
"execution_count": 43,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead tr th {\n",
" text-align: left;\n",
" }\n",
"\n",
" .dataframe thead tr:last-of-type th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr>\n",
" <th>segment</th>\n",
" <th colspan=\"2\" halign=\"left\">Bus</th>\n",
" <th colspan=\"2\" halign=\"left\">Hol</th>\n",
" <th colspan=\"2\" halign=\"left\">Oth</th>\n",
" <th colspan=\"2\" halign=\"left\">VFR</th>\n",
" </tr>\n",
" <tr>\n",
" <th>feature</th>\n",
" <th>month</th>\n",
" <th>target</th>\n",
" <th>month</th>\n",
" <th>target</th>\n",
" <th>month</th>\n",
" <th>target</th>\n",
" <th>month</th>\n",
" <th>target</th>\n",
" </tr>\n",
" <tr>\n",
" <th>timestamp</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>2006-01-01</th>\n",
" <td>1</td>\n",
" <td>9815.0</td>\n",
" <td>1</td>\n",
" <td>45906.0</td>\n",
" <td>1</td>\n",
" <td>2740.0</td>\n",
" <td>1</td>\n",
" <td>26042.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-02-01</th>\n",
" <td>2</td>\n",
" <td>11823.0</td>\n",
" <td>2</td>\n",
" <td>29347.0</td>\n",
" <td>2</td>\n",
" <td>3466.0</td>\n",
" <td>2</td>\n",
" <td>20676.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-03-01</th>\n",
" <td>3</td>\n",
" <td>13565.0</td>\n",
" <td>3</td>\n",
" <td>32492.0</td>\n",
" <td>3</td>\n",
" <td>6114.0</td>\n",
" <td>3</td>\n",
" <td>20582.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-04-01</th>\n",
" <td>4</td>\n",
" <td>11478.0</td>\n",
" <td>4</td>\n",
" <td>31813.0</td>\n",
" <td>4</td>\n",
" <td>5976.0</td>\n",
" <td>4</td>\n",
" <td>21613.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2006-05-01</th>\n",
" <td>5</td>\n",
" <td>10027.0</td>\n",
" <td>5</td>\n",
" <td>46793.0</td>\n",
" <td>5</td>\n",
" <td>3126.0</td>\n",
" <td>5</td>\n",
" <td>26947.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
"segment Bus Hol Oth VFR \n",
"feature month target month target month target month target\n",
"timestamp \n",
"2006-01-01 1 9815.0 1 45906.0 1 2740.0 1 26042.0\n",
"2006-02-01 2 11823.0 2 29347.0 2 3466.0 2 20676.0\n",
"2006-03-01 3 13565.0 3 32492.0 3 6114.0 3 20582.0\n",
"2006-04-01 4 11478.0 4 31813.0 4 5976.0 4 21613.0\n",
"2006-05-01 5 10027.0 5 46793.0 5 3126.0 5 26947.0"
]
},
"execution_count": 43,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"hierarchical_ts_w_exog.get_level_dataset(target_level=\"reason_level\").head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The modeling process stays the same as in the previous cases."
]
},
{
"cell_type": "code",
"execution_count": 44,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
"[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.2s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 2 out of 2 | elapsed: 0.5s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 3 out of 3 | elapsed: 0.7s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 3 out of 3 | elapsed: 0.7s finished\n",
"[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
"[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.2s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 2 out of 2 | elapsed: 0.4s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 3 out of 3 | elapsed: 0.6s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 3 out of 3 | elapsed: 0.6s finished\n",
"[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
"[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.1s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 2 out of 2 | elapsed: 0.2s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 3 out of 3 | elapsed: 0.2s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 3 out of 3 | elapsed: 0.2s finished\n"
]
}
],
"source": [
"region_level_ts_w_exog = hierarchical_ts_w_exog.get_level_dataset(target_level=\"region_level\")\n",
"\n",
"pipeline = HierarchicalPipeline(\n",
" transforms=[\n",
" OneHotEncoderTransform(in_column=\"month\"),\n",
" LagTransform(in_column=\"target\", lags=[1, 2, 3, 4, 6, 12]),\n",
" ],\n",
" model=LinearPerSegmentModel(),\n",
" reconciliator=TopDownReconciliator(\n",
" target_level=\"region_level\", source_level=\"reason_level\", period=9, method=\"AHP\"\n",
" ),\n",
")\n",
"\n",
"metric, _, _ = pipeline.backtest(ts=region_level_ts_w_exog, metrics=[SMAPE()], n_folds=3, aggregate_metrics=True)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.6"
}
},
"nbformat": 4,
"nbformat_minor": 4
}