{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "## [02_Probabilistic.ipynb](https://github.com/raybellwaves/xskillscore-tutorial/blob/master/02_Probabilistic.ipynb)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This notebook shows how to use probabilistic metrics in a typical data science task where the data is a pandas.DataFrame.\n", "\n", "The metric Continuous Ranked Probability Score (CRPS) is used to verify multiple forecasts for the same target." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import xarray as xr\n", "import pandas as pd\n", "import numpy as np\n", "import xskillscore as xs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Use the same data as in [01_Deterministic.ipynb](https://github.com/raybellwaves/xskillscore-tutorial/blob/master/01_Determinisitic.ipynb):" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
y
DATESTORESKU
2020-01-01009
12
24
101
15
\n", "
" ], "text/plain": [ " y\n", "DATE STORE SKU \n", "2020-01-01 0 0 9\n", " 1 2\n", " 2 4\n", " 1 0 1\n", " 1 5" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "stores = np.arange(4)\n", "skus = np.arange(3)\n", "dates = pd.date_range(\"1/1/2020\", \"1/5/2020\", freq=\"D\")\n", "\n", "rows = []\n", "for _, date in enumerate(dates):\n", " for _, store in enumerate(stores):\n", " for _, sku in enumerate(skus):\n", " rows.append(\n", " dict(\n", " {\n", " \"DATE\": date,\n", " \"STORE\": store,\n", " \"SKU\": sku,\n", " \"QUANTITY_SOLD\": np.random.randint(9) + 1,\n", " }\n", " )\n", " )\n", "df = pd.DataFrame(rows)\n", "df.rename(columns={\"QUANTITY_SOLD\": \"y\"}, inplace=True)\n", "df.set_index(['DATE', 'STORE', 'SKU'], inplace=True)\n", "df.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Instead of making a single prediction as in [01_Deterministic.ipynb](https://github.com/raybellwaves/xskillscore-tutorial/blob/master/01_Determinisitic.ipynb) we will make multiple forecasts (ensemble forecast). This is akin to more complex methods such as **bagging**, **boosting** and **stacking**. \n", "\n", "Do 6 forecasts and append them to the `pandas.DataFrame` using an extra field called `member`. This will be saved in a new `pandas.DataFrame` called `df_yhat`:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
ymemberyhat
DATESTORESKU
2020-01-01009110
1210
2412
10110
1515
..................
2020-01-0521867
2260
30868
1361
2569
\n", "

360 rows × 3 columns

\n", "
" ], "text/plain": [ " y member yhat\n", "DATE STORE SKU \n", "2020-01-01 0 0 9 1 10\n", " 1 2 1 0\n", " 2 4 1 2\n", " 1 0 1 1 0\n", " 1 5 1 5\n", "... .. ... ...\n", "2020-01-05 2 1 8 6 7\n", " 2 2 6 0\n", " 3 0 8 6 8\n", " 1 3 6 1\n", " 2 5 6 9\n", "\n", "[360 rows x 3 columns]" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tmp = df.copy()\n", "for i in range(1, 7):\n", " tmp['member'] = i\n", " noise = np.random.uniform(-1, 1, size=len(df['y']))\n", " tmp['yhat'] = (df['y'] + (df['y'] * noise)).astype(int)\n", " if i == 1:\n", " df_yhat = tmp.copy()\n", " else:\n", " df_yhat = df_yhat.append(tmp)\n", "df_yhat" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Drop the `y` column from `df_yhat` and add `member` to the MultiIndex:" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
yhat
DATESTORESKUmember
2020-01-0100110
110
212
1010
115
...............
2020-01-052167
260
3068
161
269
\n", "

360 rows × 1 columns

\n", "
" ], "text/plain": [ " yhat\n", "DATE STORE SKU member \n", "2020-01-01 0 0 1 10\n", " 1 1 0\n", " 2 1 2\n", " 1 0 1 0\n", " 1 1 5\n", "... ...\n", "2020-01-05 2 1 6 7\n", " 2 6 0\n", " 3 0 6 8\n", " 1 6 1\n", " 2 6 9\n", "\n", "[360 rows x 1 columns]" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df_yhat.drop('y', axis=1, inplace=True)\n", "df_yhat.set_index(['member'], append=True, inplace=True)\n", "df_yhat" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Convert the target `pandas.DataFrame` (`df`) to an `xarray.Dataset`:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "Show/Hide data repr\n", "\n", "\n", "\n", "\n", "\n", "Show/Hide attributes\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
xarray.Dataset
" ], "text/plain": [ "\n", "Dimensions: (DATE: 5, SKU: 3, STORE: 4)\n", "Coordinates:\n", " * DATE (DATE) datetime64[ns] 2020-01-01 2020-01-02 ... 2020-01-05\n", " * STORE (STORE) int64 0 1 2 3\n", " * SKU (SKU) int64 0 1 2\n", "Data variables:\n", " y (DATE, STORE, SKU) int64 9 2 4 1 5 8 6 3 1 6 ... 7 3 4 8 8 2 8 3 5" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ds = df.to_xarray()\n", "ds" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now add the predicted `pandas.DataFrame` (`df`) as an `xarray.DataArray` called `yhat` to the `xarray.Dataset`:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "Show/Hide data repr\n", "\n", "\n", "\n", "\n", "\n", "Show/Hide attributes\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
xarray.Dataset
    • DATE: 5
    • SKU: 3
    • STORE: 4
    • member: 6
    • DATE
      (DATE)
      datetime64[ns]
      2020-01-01 ... 2020-01-05
      array(['2020-01-01T00:00:00.000000000', '2020-01-02T00:00:00.000000000',\n",
       "       '2020-01-03T00:00:00.000000000', '2020-01-04T00:00:00.000000000',\n",
       "       '2020-01-05T00:00:00.000000000'], dtype='datetime64[ns]')
    • STORE
      (STORE)
      int64
      0 1 2 3
      array([0, 1, 2, 3])
    • SKU
      (SKU)
      int64
      0 1 2
      array([0, 1, 2])
    • member
      (member)
      int64
      1 2 3 4 5 6
      array([1, 2, 3, 4, 5, 6])
    • y
      (DATE, STORE, SKU)
      int64
      9 2 4 1 5 8 6 3 ... 3 4 8 8 2 8 3 5
      array([[[9, 2, 4],\n",
       "        [1, 5, 8],\n",
       "        [6, 3, 1],\n",
       "        [6, 4, 5]],\n",
       "\n",
       "       [[3, 4, 5],\n",
       "        [5, 6, 8],\n",
       "        [5, 4, 1],\n",
       "        [8, 2, 7]],\n",
       "\n",
       "       [[8, 3, 2],\n",
       "        [4, 4, 2],\n",
       "        [2, 3, 5],\n",
       "        [1, 4, 6]],\n",
       "\n",
       "       [[2, 3, 9],\n",
       "        [2, 7, 1],\n",
       "        [4, 1, 9],\n",
       "        [4, 3, 9]],\n",
       "\n",
       "       [[2, 8, 7],\n",
       "        [7, 3, 4],\n",
       "        [8, 8, 2],\n",
       "        [8, 3, 5]]])
    • yhat
      (DATE, STORE, SKU, member)
      int64
      10 10 1 10 16 11 0 ... 5 4 5 8 1 9
      array([[[[10, 10,  1, 10, 16, 11],\n",
       "         [ 0,  3,  1,  0,  2,  0],\n",
       "         [ 2,  5,  7,  2,  0,  1]],\n",
       "\n",
       "        [[ 0,  1,  0,  0,  0,  0],\n",
       "         [ 5,  0,  8,  8,  9,  1],\n",
       "         [ 1,  4, 15,  2,  1, 10]],\n",
       "\n",
       "        [[ 0,  3,  5,  4,  5,  5],\n",
       "         [ 3,  2,  0,  1,  4,  1],\n",
       "         [ 1,  1,  0,  0,  0,  1]],\n",
       "\n",
       "        [[10,  3, 11, 10, 10,  0],\n",
       "         [ 3,  5,  3,  3,  1,  2],\n",
       "         [ 4,  3,  6,  4,  0,  2]]],\n",
       "\n",
       "\n",
       "       [[[ 4,  3,  0,  1,  0,  0],\n",
       "         [ 5,  3,  7,  6,  1,  1],\n",
       "         [ 0,  7,  8,  6,  5,  1]],\n",
       "\n",
       "        [[ 5,  1,  8,  6,  3,  6],\n",
       "         [ 6,  9, 11,  4, 10,  6],\n",
       "         [ 0,  5,  5,  6,  3, 15]],\n",
       "\n",
       "        [[ 2,  5,  9,  2,  4,  5],\n",
       "         [ 2,  3,  1,  3,  6,  6],\n",
       "         [ 0,  0,  1,  1,  0,  0]],\n",
       "\n",
       "        [[15,  2, 13, 10,  4,  4],\n",
       "         [ 0,  2,  0,  0,  3,  2],\n",
       "         [ 6,  7,  8,  1, 10,  7]]],\n",
       "\n",
       "\n",
       "       [[[ 6,  1, 12,  7,  5, 10],\n",
       "         [ 3,  0,  0,  1,  1,  3],\n",
       "         [ 1,  2,  0,  2,  2,  3]],\n",
       "\n",
       "        [[ 6,  5,  5,  7,  1,  1],\n",
       "         [ 6,  3,  2,  3,  1,  2],\n",
       "         [ 0,  0,  3,  0,  1,  3]],\n",
       "\n",
       "        [[ 1,  3,  0,  1,  0,  3],\n",
       "         [ 0,  5,  0,  2,  2,  0],\n",
       "         [ 7,  9,  6,  5,  2,  1]],\n",
       "\n",
       "        [[ 0,  1,  0,  1,  0,  0],\n",
       "         [ 4,  5,  6,  1,  7,  4],\n",
       "         [ 0,  7,  4,  6,  5,  0]]],\n",
       "\n",
       "\n",
       "       [[[ 0,  3,  0,  1,  1,  2],\n",
       "         [ 3,  4,  3,  5,  5,  0],\n",
       "         [ 2, 15, 11, 16, 11, 14]],\n",
       "\n",
       "        [[ 3,  2,  2,  3,  1,  1],\n",
       "         [ 8,  9,  2,  6,  0, 12],\n",
       "         [ 0,  0,  0,  1,  0,  1]],\n",
       "\n",
       "        [[ 5,  0,  2,  1,  3,  2],\n",
       "         [ 1,  0,  1,  0,  0,  0],\n",
       "         [ 7,  8,  4,  5,  6,  4]],\n",
       "\n",
       "        [[ 7,  6,  5,  5,  3,  7],\n",
       "         [ 0,  0,  4,  2,  0,  3],\n",
       "         [ 5, 10,  1,  8,  0,  0]]],\n",
       "\n",
       "\n",
       "       [[[ 3,  1,  2,  3,  1,  3],\n",
       "         [ 8, 11, 15,  6, 11, 13],\n",
       "         [12,  7, 13,  5, 12,  8]],\n",
       "\n",
       "        [[ 6,  6,  4,  1,  9, 13],\n",
       "         [ 3,  1,  4,  0,  4,  3],\n",
       "         [ 6,  7,  0,  7,  4,  3]],\n",
       "\n",
       "        [[ 1,  7, 11,  0, 11,  2],\n",
       "         [ 7,  0,  3, 15,  6,  7],\n",
       "         [ 3,  1,  2,  2,  1,  0]],\n",
       "\n",
       "        [[ 6,  2,  1,  2,  9,  8],\n",
       "         [ 5,  0,  4,  1,  2,  1],\n",
       "         [ 5,  4,  5,  8,  1,  9]]]])
" ], "text/plain": [ "\n", "Dimensions: (DATE: 5, SKU: 3, STORE: 4, member: 6)\n", "Coordinates:\n", " * DATE (DATE) datetime64[ns] 2020-01-01 2020-01-02 ... 2020-01-05\n", " * STORE (STORE) int64 0 1 2 3\n", " * SKU (SKU) int64 0 1 2\n", " * member (member) int64 1 2 3 4 5 6\n", "Data variables:\n", " y (DATE, STORE, SKU) int64 9 2 4 1 5 8 6 3 1 6 ... 7 3 4 8 8 2 8 3 5\n", " yhat (DATE, STORE, SKU, member) int64 10 10 1 10 16 11 0 ... 5 4 5 8 1 9" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ds['yhat'] = df_yhat.to_xarray()['yhat']\n", "ds" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Notice how an `xarray.Dataset` can handle Data variables which have different shape but share some dimensions" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Using xskillscore - CRPS" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Continuous Ranked Probability Score (CRPS) can also be considered as the probabilistic Mean Absolute Error. It compares the empirical distribution of an ensemble forecast to a scalar observation it is given as:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\\begin{align}\n", "CRPS = \\int_{-\\infty}^{\\infty} (F(f) - H(f - o))^{2} df\n", "\\end{align}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "where where `F(f)` is the cumulative distribution function (CDF) of the forecast and `H()` is the Heaviside step function where the value is 1 if the argument is positive (the prediction overestimates the target or 0 (the prediction is equal to or lower than the target data).\n", "\n", "See https://climpred.readthedocs.io/en/stable/metrics.html#continuous-ranked-probability-score-crps for further documentation." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It is not a common verification metric and in most cases the predictions are averaged then verified using deterministic metrics.\n", "\n", "For example, you can see averaging on the `member` dimension gives a better prediction than any individual prediction:\n", "\n", "Note: for this we will use the function itself instead of the Accessor method:" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "avg_member_rmse: 1.5302505241517372\n", "member 1:\n", "ind_member_rmse: 2.958039891549808\n", "member 2:\n", "ind_member_rmse: 2.717228980659034\n", "member 3:\n", "ind_member_rmse: 3.361547262794322\n", "member 4:\n", "ind_member_rmse: 2.869378562220979\n", "member 5:\n", "ind_member_rmse: 3.082207001484488\n", "member 6:\n", "ind_member_rmse: 3.119829055146024\n" ] } ], "source": [ "avg_member_rmse = xs.rmse(\n", " ds[\"y\"], ds[\"yhat\"].mean(dim=\"member\"), [\"DATE\", \"STORE\", \"SKU\"]\n", ")\n", "print(\"avg_member_rmse: \", avg_member_rmse.values)\n", "for i in range(len(ds.coords[\"member\"])):\n", " print(f\"member {i + 1}:\")\n", " ind_member_rmse = xs.rmse(\n", " ds[\"y\"], ds[\"yhat\"].sel(member=i + 1), [\"DATE\", \"STORE\", \"SKU\"]\n", " )\n", " print(\"ind_member_rmse: \", ind_member_rmse.values)\n", " \n", " assert avg_member_rmse < ind_member_rmse" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "However, you will see it appear in some Kaggle competitions such as the [NFL Big Data Bowl](https://www.kaggle.com/c/nfl-big-data-bowl-2020/overview/evaluation) and the [Second Annual Data Science Bowl](https://www.kaggle.com/c/second-annual-data-science-bowl/overview/evaluation) so it's good to have in your arsenal." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The CRPS is only valid over the `member` dimension and therefore only takes 2 arguments:" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "Show/Hide data repr\n", "\n", "\n", "\n", "\n", "\n", "Show/Hide attributes\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
xarray.DataArray
  • DATE: 5
  • STORE: 4
  • SKU: 3
  • 1.167 0.7222 1.194 0.6944 1.25 ... 1.556 0.3056 1.944 0.8611 0.5556
    array([[[1.16666667, 0.72222222, 1.19444444],\n",
       "        [0.69444444, 1.25      , 2.75      ],\n",
       "        [1.44444444, 0.75      , 0.25      ],\n",
       "        [2.22222222, 0.86111111, 1.13888889]],\n",
       "\n",
       "       [[1.16666667, 0.86111111, 0.86111111],\n",
       "        [0.58333333, 0.94444444, 2.33333333],\n",
       "        [0.58333333, 0.80555556, 0.44444444],\n",
       "        [1.94444444, 0.52777778, 0.41666667]],\n",
       "\n",
       "       [[1.19444444, 1.        , 0.16666667],\n",
       "        [0.91666667, 1.02777778, 0.80555556],\n",
       "        [0.66666667, 1.25      , 0.77777778],\n",
       "        [0.44444444, 0.47222222, 1.16666667]],\n",
       "\n",
       "       [[0.58333333, 0.44444444, 2.47222222],\n",
       "        [0.22222222, 1.19444444, 0.44444444],\n",
       "        [1.30555556, 0.44444444, 2.5       ],\n",
       "        [1.08333333, 0.97222222, 3.16666667]],\n",
       "\n",
       "       [[0.36111111, 1.66666667, 1.52777778],\n",
       "        [1.08333333, 0.36111111, 0.80555556],\n",
       "        [2.16666667, 1.55555556, 0.30555556],\n",
       "        [1.94444444, 0.86111111, 0.55555556]]])
    • DATE
      (DATE)
      datetime64[ns]
      2020-01-01 ... 2020-01-05
      array(['2020-01-01T00:00:00.000000000', '2020-01-02T00:00:00.000000000',\n",
       "       '2020-01-03T00:00:00.000000000', '2020-01-04T00:00:00.000000000',\n",
       "       '2020-01-05T00:00:00.000000000'], dtype='datetime64[ns]')
    • STORE
      (STORE)
      int64
      0 1 2 3
      array([0, 1, 2, 3])
    • SKU
      (SKU)
      int64
      0 1 2
      array([0, 1, 2])
" ], "text/plain": [ "\n", "array([[[1.16666667, 0.72222222, 1.19444444],\n", " [0.69444444, 1.25 , 2.75 ],\n", " [1.44444444, 0.75 , 0.25 ],\n", " [2.22222222, 0.86111111, 1.13888889]],\n", "\n", " [[1.16666667, 0.86111111, 0.86111111],\n", " [0.58333333, 0.94444444, 2.33333333],\n", " [0.58333333, 0.80555556, 0.44444444],\n", " [1.94444444, 0.52777778, 0.41666667]],\n", "\n", " [[1.19444444, 1. , 0.16666667],\n", " [0.91666667, 1.02777778, 0.80555556],\n", " [0.66666667, 1.25 , 0.77777778],\n", " [0.44444444, 0.47222222, 1.16666667]],\n", "\n", " [[0.58333333, 0.44444444, 2.47222222],\n", " [0.22222222, 1.19444444, 0.44444444],\n", " [1.30555556, 0.44444444, 2.5 ],\n", " [1.08333333, 0.97222222, 3.16666667]],\n", "\n", " [[0.36111111, 1.66666667, 1.52777778],\n", " [1.08333333, 0.36111111, 0.80555556],\n", " [2.16666667, 1.55555556, 0.30555556],\n", " [1.94444444, 0.86111111, 0.55555556]]])\n", "Coordinates:\n", " * DATE (DATE) datetime64[ns] 2020-01-01 2020-01-02 ... 2020-01-05\n", " * STORE (STORE) int64 0 1 2 3\n", " * SKU (SKU) int64 0 1 2" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ds.xs.crps_ensemble('y', 'yhat')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To return an overall CRPS it is recommended to average over all dimensions before using `crps`:" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "Show/Hide data repr\n", "\n", "\n", "\n", "\n", "\n", "Show/Hide attributes\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
xarray.DataArray
  • 0.5648
    array(0.56481481)
    " ], "text/plain": [ "\n", "array(0.56481481)" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "y = ds['y'].mean(dim=['DATE', 'STORE', 'SKU'])\n", "yhat = ds['yhat'].mean(dim=['DATE', 'STORE', 'SKU'])\n", "xs.crps_ensemble(y, yhat)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.2" } }, "nbformat": 4, "nbformat_minor": 4 }