{
"cells": [
{
"cell_type": "markdown",
"id": "57b22edd-ff30-4e05-8b54-9cd6ba52755c",
"metadata": {},
"source": [
"# Flip-Flop Index"
]
},
{
"cell_type": "markdown",
"id": "3023fa66-1f96-4961-afbc-c4cf91fe232a",
"metadata": {},
"source": [
"The Flip-Flop Index quantifies the stability of a forecast without penalising a trend toward more accurate values at shorter lead times. The Flip-Flop Index has the same units as the forecast, with smaller values indicating greater stability (fewer \"flip-flops\"). The Flip-Flop Index cannot be used to verify a forecast. Indeed, it does not use the verifying observation in its calculation. However, it describes an (arguably important) characteristic of a sequence of forecasts for an event. \n",
"\n",
"Read about the Flip-Flop Index at: \n",
"\n",
"- Griffiths D. et al. (2019). Flip-Flop Index: Quantifying Revision Stability for Fixed-Event Forecasts, \n",
"*Meteorological Applications*, 26, 30-35. [https://doi.org/10.1002/met.1732](https://doi.org/10.1002/met.1732) \n",
"- Griffiths D. et al. (2021). Circular Flip-Flop Index: quantifying revision stability of forecasts of direction, *Journal of Southern Hemisphere Earth Systems Science*, 71, 266–271. [https://doi.org/10.1071/ES21010](https://doi.org/10.1071/ES21010)\n",
"\n",
"The two functions demonstrated here are `flip_flop_index` which calculates the index for individual forecast revision series, and `flip_flop_index_proportion_exceeding` which summarises the Flip-Flop Index values over many forecast revision series by reporting the frequency with which it exceeded values of interest."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "91abca47",
"metadata": {
"ExecuteTime": {
"end_time": "2023-11-09T04:58:29.280799Z",
"start_time": "2023-11-09T04:58:29.275796Z"
}
},
"outputs": [],
"source": [
"from scipy.stats import skewnorm\n",
"import numpy as np\n",
"import xarray as xr\n",
"import pandas as pd\n",
"import plotly.express as px\n",
"\n",
"import matplotlib.pyplot as plt\n",
"\n",
"from scores.continuous import mse\n",
"from scores.continuous import flip_flop_index_proportion_exceeding, flip_flop_index"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "41766974-8695-4361-9eb5-c52d82109713",
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"# help(flip_flop_index) # Uncomment this to see the help message"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "d8bea123-8e4e-4006-b9f6-7df5e27f4b74",
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"# help(flip_flop_index_proportion_exceeding) # Uncomment this to see the help message"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "dcf1849b-c5f7-4631-9685-a0277c945c14",
"metadata": {},
"outputs": [],
"source": [
"# Create two synthetic forecast revision series.\n",
"# These might represent a forecast of 28 degrees a week in advance of the event, updated daily with a forecast of 21 degrees the day before the event.\n",
"# Inspection (and the graph below) shows that the unstable_fcst jumps around while the stable_fcst has a cooler trend each day.\n",
"unstable_fcst = xr.DataArray(data=[21., 23, 18, 20, 26, 25, 28], dims=\"lead_day\", coords={\"lead_day\": np.arange(1, 8)})\n",
"stable_fcst = xr.DataArray(data=[21., 23, 23, 24, 25, 26, 28], dims=\"lead_day\", coords={\"lead_day\": np.arange(1, 8)})"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3917ba0e-6db2-4ef5-b268-20f1fa8884da",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
" \n",
" "
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.plotly.v1+json": {
"config": {
"plotlyServerURL": "https://plot.ly"
},
"data": [
{
"hovertemplate": "
<xarray.DataArray ()>\n",
"array(1.8)\n",
"Attributes:\n",
" sampling_dim: lead_day<xarray.DataArray ()>\n",
"array(0.)\n",
"Attributes:\n",
" sampling_dim: lead_day<xarray.DataArray (lead_day: 7)>\n",
"array([0.99, 1.32, 1.66, 2.04, 2.48, 2.88, 3.42])\n",
"Coordinates:\n",
" * lead_day (lead_day) int64 1 2 3 4 5 6 7<xarray.DataArray (lead_day: 7)>\n",
"array([0.99, 1.32, 1.66, 2.04, 2.48, 2.88, 3.42])\n",
"Coordinates:\n",
" * lead_day (lead_day) int64 1 2 3 4 5 6 7<xarray.DataArray (time: 100, location number: 100)>\n",
"array([[1.09, 0.29, 0.15, ..., 0.25, 1.27, 0.59],\n",
" [0.63, 0.44, 0.96, ..., 1.35, 1.39, 0.23],\n",
" [0.97, 0.47, 0.19, ..., 1.39, 0.91, 0.97],\n",
" ...,\n",
" [0.62, 0.57, 1.09, ..., 0.97, 1.52, 0.33],\n",
" [0.84, 0.58, 0.53, ..., 0.25, 0.33, 0.63],\n",
" [0.71, 1.37, 1.13, ..., 0.64, 0.58, 0.62]])\n",
"Coordinates:\n",
" * time (time) datetime64[ns] 2023-01-01 2023-01-02 ... 2023-04-10\n",
" * location number (location number) int64 0 1 2 3 4 5 6 ... 94 95 96 97 98 99\n",
"Attributes:\n",
" sampling_dim: lead_day