{ "cells": [ { "cell_type": "markdown", "id": "sf-download-01", "metadata": {}, "source": [ "\n", "\n", "> **Download this notebook:** [`01_backtest_robustness.ipynb`](https://raw.githubusercontent.com/sablier-ai/sablier-flow/main/examples/01_backtest_robustness.ipynb) (right-click \u2192 Save Link As)\n", "> \u00b7 [View source on GitHub](https://github.com/sablier-ai/sablier-flow/blob/main/examples/01_backtest_robustness.ipynb)\n", "> \u00b7 Or `pip install sablier-flow && sablier-flow notebook` to copy a fresh local copy from the wheel.\n" ] }, { "cell_type": "markdown", "id": "4464fefc", "metadata": {}, "source": [ "# Backtest Robustness \u2014 Catching Lucky Strategies from a Parameter Search\n", "\n", "**The question this notebook answers, falsifiably:** when you run a large\n", "parameter search and pick the strategies that look best on training data,\n", "some of them are real signal and some are *lucky* \u2014 selected upward by\n", "chance, not by skill. How do you tell them apart *before* committing capital?\n", "\n", "**How we measure it.** Train a FLOW model on your data; FLOW learns the joint\n", "dynamics of returns, vol, and cross-asset correlations. Generate 200 synthetic\n", "alternative versions of the training period. For every strategy,\n", "`sf.evaluate_family(...)` computes a `per_strategy_overfit_score` \u2014 the\n", "fraction of synth paths where the strategy's synth Sharpe is below its real\n", "Sharpe. If the strategy genuinely has no edge but was selected for high real\n", "Sharpe by chance, FLOW's synth distribution (which doesn't replicate the\n", "specific luck of the real period) will sit below the real Sharpe \u2192 score\n", "elevates toward 1.0.\n", "\n", "**The demo, falsifiably:** we build **500 pure-noise strategies** (each one\n", "a fixed random sign sequence applied to an equal-weight portfolio \u2014 zero\n", "underlying skill by construction). We select the **top 30 by training\n", "Sharpe**. By selection alone they must have inflated real Sharpes; whether\n", "FLOW catches them is what we test. We compare against a **12-variant honest\n", "designed family** that has not been selected from any pool.\n", "\n", "If FLOW's per-strategy `overfit_score` is a real selection-bias signal,\n", "the selected-30 distribution should sit **clearly above** the honest-12\n", "distribution. If they overlap, the diagnostic adds no value beyond\n", "classical CSCV-PBO and this notebook fails." ] }, { "cell_type": "markdown", "id": "b624b48a", "metadata": {}, "source": [ "## Operating envelope\n", "\n", "> **What this notebook demonstrates:** sablier-flow's per-strategy\n", "> `overfit_score` distinguishes selection-biased lucky strategies from\n", "> genuinely-designed strategies \u2014 exactly the diagnostic a quant wants\n", "> before deploying the winners of a parameter search.\n", ">\n", "> **Where this works best:** multi-asset (3\u20138 features) daily panels,\n", "> \u22655 years of training data, families where you can articulate a\n", "> \"what would the null look like\" question.\n", ">\n", "> **Where to be careful:** single-asset, intraday, sparse data, or\n", "> regimes where the synth distribution might miss key dynamics.\n", ">\n", "> **How to validate on your data:** run cells below against the demo\n", "> data to confirm the diagnostic differentiates, then swap in your own\n", "> panel via the *Try this on your own data* cell at the end." ] }, { "cell_type": "markdown", "id": "d278c23f", "metadata": {}, "source": [ "## Setup" ] }, { "cell_type": "code", "execution_count": 1, "id": "17060663", "metadata": { "execution": { "iopub.execute_input": "2026-06-01T19:22:01.908096Z", "iopub.status.busy": "2026-06-01T19:22:01.907936Z", "iopub.status.idle": "2026-06-01T19:22:01.911613Z", "shell.execute_reply": "2026-06-01T19:22:01.910921Z" } }, "outputs": [], "source": [ "# One-time install. Pinned to a known-good wheel floor for the helpers used below.\n", "# %pip install -q \"sablier-flow>=1.1\"\n" ] }, { "cell_type": "code", "execution_count": 2, "id": "ca5321ef", "metadata": { "execution": { "iopub.execute_input": "2026-06-01T19:22:01.914001Z", "iopub.status.busy": "2026-06-01T19:22:01.913789Z", "iopub.status.idle": "2026-06-01T19:22:02.273520Z", "shell.execute_reply": "2026-06-01T19:22:02.273156Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "sablier-flow 1.1.0\n" ] } ], "source": [ "import os\n", "import json\n", "import pathlib\n", "import time\n", "import warnings\n", "warnings.filterwarnings('ignore', message='.*estimated wall-clock.*')\n", "\n", "import numpy as np\n", "import pandas as pd\n", "\n", "import sablier_flow as sf\n", "from sablier_flow.demo import DEMO_DATA_TYPES\n", "\n", "print(f'sablier-flow {sf.__version__}')\n" ] }, { "cell_type": "markdown", "id": "472a29eb", "metadata": {}, "source": [ "### Authenticate" ] }, { "cell_type": "code", "execution_count": 3, "id": "129a1f41", "metadata": { "execution": { "iopub.execute_input": "2026-06-01T19:22:02.274675Z", "iopub.status.busy": "2026-06-01T19:22:02.274575Z", "iopub.status.idle": "2026-06-01T19:22:02.553068Z", "shell.execute_reply": "2026-06-01T19:22:02.552438Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "signed in as: you@example.com\n" ] } ], "source": [ "if not os.environ.get('SABLIER_FLOW_API_KEY'):\n", " cred = pathlib.Path.home() / '.sablier/credentials'\n", " if cred.exists():\n", " os.environ['SABLIER_FLOW_API_KEY'] = json.load(open(cred))['default']['api_key']\n", " else:\n", " sf.login()\n", "\n", "print('signed in as:', sf.whoami()['email'])\n" ] }, { "cell_type": "markdown", "id": "2d1f10f0", "metadata": {}, "source": [ "## Section 1 \u2014 Load the demo panel\n", "\n", "The canonical demo: 4 tradeable ETFs across equity + bonds, plus 3 macro\n", "features (VIX volatility, TNX 10-year yield, DXY dollar). FLOW fits on all\n", "7 features so it learns the regime-conditional structure; strategies trade\n", "only the 4 tradeable columns." ] }, { "cell_type": "code", "execution_count": 4, "id": "1dafd2b1", "metadata": { "execution": { "iopub.execute_input": "2026-06-01T19:22:02.555228Z", "iopub.status.busy": "2026-06-01T19:22:02.554997Z", "iopub.status.idle": "2026-06-01T19:22:02.610944Z", "shell.execute_reply": "2026-06-01T19:22:02.610590Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "features: ['IWM', 'QQQ', 'SPY', 'TLT', 'VIX', 'TNX', 'DXY']\n", "tradeable subset: ['SPY', 'QQQ', 'IWM', 'TLT']\n", "train: (2013, 7) range 2012-01-03 \u2192 2019-12-31\n" ] } ], "source": [ "df = sf.demo_data(name='us_equities_macro_2010_2023')\n", "data_types = DEMO_DATA_TYPES['us_equities_macro_2010_2023']\n", "features = list(df.columns)\n", "data = df.dropna()\n", "\n", "TRADEABLE = ['SPY', 'QQQ', 'IWM', 'TLT']\n", "\n", "# 8-year training window \u2014 long enough that random-position strategies have\n", "# stable Sharpe statistics under the null.\n", "train = data.loc['2012':'2019']\n", "\n", "print(f'features: {features}')\n", "print(f'tradeable subset: {TRADEABLE}')\n", "print(f'train: {train.shape} range {train.index[0].date()} \u2192 {train.index[-1].date()}')\n" ] }, { "cell_type": "markdown", "id": "eb8af28e", "metadata": {}, "source": [ "## Section 2 \u2014 A 12-variant honest baseline family\n", "\n", "First we establish what `per_strategy_overfit_score` looks like on a\n", "*designed* family \u2014 every variant is a sensible mean-reversion or\n", "momentum strategy that a quant might write by hand. No selection step.\n", "We expect per-strategy `overfit_score` values around 0.5 (no per-strategy overfit signal beyond what FLOW's null\n", "already accounts for)." ] }, { "cell_type": "code", "execution_count": 5, "id": "41fd55ad", "metadata": { "execution": { "iopub.execute_input": "2026-06-01T19:22:02.612167Z", "iopub.status.busy": "2026-06-01T19:22:02.612099Z", "iopub.status.idle": "2026-06-01T19:22:02.615092Z", "shell.execute_reply": "2026-06-01T19:22:02.614785Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "honest_family: 12 variants\n", "sample: ['momentum_lb10_thr0.5', 'momentum_lb10_thr1.0', 'momentum_lb10_thr1.5', 'momentum_lb21_thr0.5', 'momentum_lb21_thr1.0', 'momentum_lb21_thr1.5']\n" ] } ], "source": [ "def backtest_variant(data, kind, lb, thr):\n", " \"\"\"Equal-weighted long/short on z-score, restricted to TRADEABLE columns.\"\"\"\n", " avail = [c for c in TRADEABLE if c in data.columns]\n", " prices = data[avail]\n", " rets = prices.pct_change()\n", " z = (rets - rets.rolling(lb).mean()) / rets.rolling(lb).std()\n", " pos = (z > thr).astype(float) - (z < -thr).astype(float)\n", " if kind == 'mean_rev':\n", " pos = -pos\n", " pos = pos.shift(1).fillna(0)\n", " n_active = pos.abs().sum(axis=1).clip(lower=1)\n", " pos = pos.div(n_active, axis=0)\n", " pnl = (pos * rets).sum(axis=1)\n", " if pnl.std() > 1e-10:\n", " return float(pnl.mean() / pnl.std() * np.sqrt(252))\n", " return 0.0\n", "\n", "def make_designed_strategy(kind, lb, thr):\n", " return lambda data: {'sharpe': backtest_variant(data, kind, lb, thr)}\n", "\n", "# 12 honest designed variants \u2014 lookback \u00d7 threshold \u00d7 direction\n", "honest_family = {\n", " f'{kind}_lb{lb}_thr{thr:.1f}': make_designed_strategy(kind, lb, thr)\n", " for kind in ['momentum', 'mean_rev']\n", " for lb in [10, 21]\n", " for thr in [0.5, 1.0, 1.5]\n", "}\n", "print(f'honest_family: {len(honest_family)} variants')\n", "print(f'sample: {list(honest_family.keys())[:6]}')\n" ] }, { "cell_type": "markdown", "id": "a77f1ac5", "metadata": {}, "source": [ "## Section 3 \u2014 Fit a FLOW model (horizon = 252 days)\n", "\n", "`horizon=252` (~1 year synth paths) keeps the Sharpe-variance comparison\n", "between real-data backtest and synth-path backtest in a reasonable range.\n", "With shorter horizons (e.g., 63 days), synth-path Sharpe has very high\n", "variance per path and the comparison becomes insensitive.\n", "\n", "**Cost:** ~200 credits. **Time:** ~5 min on the hosted L4 / H100." ] }, { "cell_type": "code", "execution_count": 6, "id": "9f696d06", "metadata": { "execution": { "iopub.execute_input": "2026-06-01T19:22:02.616092Z", "iopub.status.busy": "2026-06-01T19:22:02.616024Z", "iopub.status.idle": "2026-06-01T19:22:03.462715Z", "shell.execute_reply": "2026-06-01T19:22:03.462274Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "sablier-flow: fitting 7 feature(s) over 2013 bars [row cadence: daily (median \u0394t=1 days 00:00:00)]\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "dispatched: ac3f14ad-aea8-44c8-8b3c-10ce1bf88cdc\n" ] } ], "source": [ "handle = sf.fit_async(\n", " train,\n", " features=features,\n", " data_types=data_types,\n", " horizon=252,\n", " seed=42,\n", ")\n", "print(f'dispatched: {handle.job_id}')\n" ] }, { "cell_type": "code", "execution_count": 7, "id": "34b85ed7", "metadata": { "execution": { "iopub.execute_input": "2026-06-01T19:22:03.464028Z", "iopub.status.busy": "2026-06-01T19:22:03.463955Z", "iopub.status.idle": "2026-06-01T19:24:07.303829Z", "shell.execute_reply": "2026-06-01T19:24:07.303077Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "fit complete in 2.1 min\n", "model_id: 90a655b4-b2d3-47a3-8a1c-ccf82a0a4712\n", "training_loss: 1.0646\n" ] } ], "source": [ "t0 = time.time()\n", "result = sf.fetch_result(handle, poll_timeout_s=2400)\n", "print(f'fit complete in {(time.time()-t0)/60:.1f} min')\n", "print(f'model_id: {result.model_id}')\n", "print(f'training_loss: {result.training_loss:.4f}')\n" ] }, { "cell_type": "markdown", "id": "f08b856d", "metadata": {}, "source": [ "## Section 4 \u2014 Evaluate the honest baseline family\n", "\n", "`sf.evaluate_family(...)` generates 200 synthetic alternative versions of\n", "the training window (using `model_id`), backtests every strategy on every\n", "path, and returns a `FamilyReport` with PBO + per-strategy diagnostics.\n", "\n", "> **A note on the aggregate `verdict` label vs the headline metric.**\n", "> `FamilyReport.verdict` collapses two independent signals (realistic-null DSR\n", "> and classical CSCV PBO) into one bucket. On a small honest family (S=12), the\n", "> CSCV partition count is C(12, 6)=924, which is below the SDK's recommended\n", "> floor of 16 splits and produces unstable PBO estimates. The aggregate verdict\n", "> may print `'overfit_selection'` on the honest family AND `'looks_like_noise'`\n", "> on the lucky-30 \u2014 the OPPOSITE of the demo's claim \u2014 because PBO is a\n", "> family-level statistic that doesn't see the per-strategy selection bias the\n", "> demo is testing for. **The claim of this notebook is on `per_strategy_overfit_score`**:\n", "> a per-strategy distribution lookup against FLOW's realistic null. Read the\n", "> aggregate verdict only as background and the per-strategy distribution as\n", "> the headline." ] }, { "cell_type": "code", "execution_count": 8, "id": "99c109bf", "metadata": { "execution": { "iopub.execute_input": "2026-06-01T19:24:07.305832Z", "iopub.status.busy": "2026-06-01T19:24:07.305684Z", "iopub.status.idle": "2026-06-01T19:27:38.118936Z", "shell.execute_reply": "2026-06-01T19:27:38.118563Z" } }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "evaluate_family: ~12870 partitions, ~311280 strategy evaluations (use progress=True for live updates).\n", "/tmp/claude-501/ipykernel_9560/3692565224.py:2: UserWarning: evaluate_family workload is large (311280 strategy evaluations); consider progress=True and/or executor='thread'.\n", " honest_report = sf.evaluate_family(\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "sablier-flow: estimated cost 2 credits (use sf.estimate_cost(...) to preview before charging).\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "sablier-flow: actual cost 0 credits (remaining balance: 10000).\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/tmp/claude-501/ipykernel_9560/3692565224.py:2: UserWarning: evaluate_family: real_data has 2013 bars but the synthetic horizon is 252. The real Sharpe is computed over 2013 bars while each synthetic Sharpe is computed over 252 bars \u2014 different sample sizes mean the synthetic distribution is a biased null for the DSR-vs-real comparison. PBO is unaffected. To match horizons, either pass `like=real_data.iloc[-252:]` (truncates synth to your window) or window `real_data` yourself to 252 bars before calling.\n", " honest_report = sf.evaluate_family(\n", " [evaluate_family] 10/200 synthetic paths complete\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " [evaluate_family] 20/200 synthetic paths complete\n", " [evaluate_family] 30/200 synthetic paths complete\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " [evaluate_family] 40/200 synthetic paths complete\n", " [evaluate_family] 50/200 synthetic paths complete\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " [evaluate_family] 60/200 synthetic paths complete\n", " [evaluate_family] 70/200 synthetic paths complete\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " [evaluate_family] 80/200 synthetic paths complete\n", " [evaluate_family] 90/200 synthetic paths complete\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " [evaluate_family] 100/200 synthetic paths complete\n", " [evaluate_family] 110/200 synthetic paths complete\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " [evaluate_family] 120/200 synthetic paths complete\n", " [evaluate_family] 130/200 synthetic paths complete\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " [evaluate_family] 140/200 synthetic paths complete\n", " [evaluate_family] 150/200 synthetic paths complete\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " [evaluate_family] 160/200 synthetic paths complete\n", " [evaluate_family] 170/200 synthetic paths complete\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " [evaluate_family] 180/200 synthetic paths complete\n", " [evaluate_family] 190/200 synthetic paths complete\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " [evaluate_family] 200/200 synthetic paths complete\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/private/tmp/sf_e2e/lib/python3.12/site-packages/sablier_flow/analytics/family.py:731: UserWarning: deflated_sharpe: synthetic_sharpes appears inconsistent with n_trials=12; realistic and analytical nulls will disagree (realistic E[max]=1.469 vs analytical E[max]=1.665). Pass n_trials matching the actual selection process or pass synthetic_sharpes matching N=12.\n", " dsr = _dsr(\n", "probability_of_backtest_overfitting: ~12870 partitions, ~308880 strategy evaluations (use progress=True for live updates).\n", "/private/tmp/sf_e2e/lib/python3.12/site-packages/sablier_flow/analytics/family.py:769: UserWarning: probability_of_backtest_overfitting workload is large (308880 strategy evaluations); consider progress=True and/or executor='thread'.\n", " pbo_value, n_partitions = probability_of_backtest_overfitting(\n", "/private/tmp/sf_e2e/lib/python3.12/site-packages/sablier_flow/analytics/family.py:769: UserWarning: probability_of_backtest_overfitting: truncated 13 trailing rows so each of the 16 chunks has 125 rows. Pass real_data with a length divisible by cscv_splits to avoid this.\n", " pbo_value, n_partitions = probability_of_backtest_overfitting(\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "evaluate_family done in 3.5 min\n", "\n", "aggregate PBO (classical CSCV, no synth): 0.658\n", "verdict label: overfit_selection\n", "\n", "per_strategy_overfit_score distribution:\n", " min : 0.320\n", " median: 0.500\n", " max : 0.680\n", " # strategies above 0.7: 0\n", " # strategies above 0.9: 0\n" ] } ], "source": [ "t0 = time.time()\n", "honest_report = sf.evaluate_family(\n", " honest_family, train,\n", " model_id=result.model_id,\n", " n_paths=200,\n", " primary_metric='sharpe',\n", " higher_is_better=True,\n", ")\n", "print(f'evaluate_family done in {(time.time()-t0)/60:.1f} min')\n", "print()\n", "print(f'aggregate PBO (classical CSCV, no synth): {honest_report.pbo:.3f}')\n", "print(f'verdict label: {honest_report.verdict}')\n", "\n", "honest_scores = dict(honest_report.per_strategy_overfit_score)\n", "print()\n", "print(f'per_strategy_overfit_score distribution:')\n", "print(f' min : {min(honest_scores.values()):.3f}')\n", "print(f' median: {np.median(list(honest_scores.values())):.3f}')\n", "print(f' max : {max(honest_scores.values()):.3f}')\n", "print(f' # strategies above 0.7: {sum(1 for v in honest_scores.values() if v >= 0.7)}')\n", "print(f' # strategies above 0.9: {sum(1 for v in honest_scores.values() if v >= 0.9)}')\n" ] }, { "cell_type": "markdown", "id": "b2c86ea8", "metadata": {}, "source": [ "## Section 5 \u2014 Simulate a parameter search with PURE NOISE strategies\n", "\n", "Now we set up the selection-bias experiment. We build **500 pure-noise\n", "strategies**: each one is a fixed random \u00b11 position sequence applied to\n", "the equal-weight portfolio of the 4 tradeable assets. The sequence is\n", "deterministic per strategy (seeded), but uncorrelated with any market\n", "signal \u2014 by construction, *zero underlying skill*.\n", "\n", "Some of these 500 will have positive realized Sharpe on the real training\n", "data purely by chance \u2014 their random sequence happened to align with\n", "days where the portfolio went up. By selecting the **top 30 by training\n", "Sharpe**, we get a family that is *guaranteed lucky*: positive realized\n", "Sharpe with no underlying edge.\n", "\n", "This is the canonical \"data snooping\" or \"selection bias\" setup. The\n", "question: does FLOW's per-strategy `overfit_score` flag it?" ] }, { "cell_type": "code", "execution_count": 9, "id": "4ea1ac9d", "metadata": { "execution": { "iopub.execute_input": "2026-06-01T19:27:38.121399Z", "iopub.status.busy": "2026-06-01T19:27:38.121273Z", "iopub.status.idle": "2026-06-01T19:27:38.446301Z", "shell.execute_reply": "2026-06-01T19:27:38.445895Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "noise pool: 500 strategies (pure random sign sequences, zero skill)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "training Sharpe distribution across 500 pure-noise candidates:\n", " expected std under null = sqrt(252/2013) \u2248 0.354\n", " observed: min=-1.450, median=-0.006, max=+1.045\n", " std=0.374 (close to theoretical \u2248 0.354 \u2192 null is well-behaved)\n", "\n", "selected top 30 by in-sample Sharpe (these are LUCKY by construction):\n", " #1 noise_0375 train_sharpe = +1.045\n", " #2 noise_0237 train_sharpe = +0.975\n", " #3 noise_0387 train_sharpe = +0.939\n", " #4 noise_0297 train_sharpe = +0.932\n", " #5 noise_0452 train_sharpe = +0.919\n", " ... (25 more)\n", "\n", "selected median train Sharpe : +0.755\n", "pool median train Sharpe : -0.006\n" ] } ], "source": [ "def make_noise_strategy(seed, max_len=10000):\n", " \"\"\"Pure-noise strategy: fixed random \u00b11 sequence applied to the\n", " equal-weight portfolio of TRADEABLE assets. No underlying skill.\"\"\"\n", " rng = np.random.default_rng(seed)\n", " fixed_positions = rng.choice([-1.0, 1.0], size=max_len)\n", "\n", " def fn(data):\n", " avail = [c for c in TRADEABLE if c in data.columns]\n", " rets = data[avail].pct_change().mean(axis=1)\n", " n = len(rets)\n", " pos = pd.Series(fixed_positions[:n], index=rets.index)\n", " pnl = pos.shift(1).fillna(0) * rets\n", " if pnl.std() > 1e-10:\n", " return {'sharpe': float(pnl.mean() / pnl.std() * np.sqrt(252))}\n", " return {'sharpe': 0.0}\n", " return fn\n", "\n", "# 500 pure-noise strategies, each with a unique seed\n", "N_POOL = 500\n", "noise_pool = {f'noise_{i:04d}': make_noise_strategy(seed=1000 + i) for i in range(N_POOL)}\n", "print(f'noise pool: {len(noise_pool)} strategies (pure random sign sequences, zero skill)')\n", "\n", "# Backtest all 500 on training, sort by realized Sharpe\n", "noise_train_sharpes = {name: fn(train)['sharpe'] for name, fn in noise_pool.items()}\n", "sorted_by_train = sorted(noise_train_sharpes.items(), key=lambda x: -x[1])\n", "\n", "vals = np.array(list(noise_train_sharpes.values()))\n", "print()\n", "print(f'training Sharpe distribution across {N_POOL} pure-noise candidates:')\n", "print(f' expected std under null = sqrt(252/{len(train)}) \u2248 {np.sqrt(252/len(train)):.3f}')\n", "print(f' observed: min={vals.min():+.3f}, median={np.median(vals):+.3f}, max={vals.max():+.3f}')\n", "print(f' std={vals.std():.3f} (close to theoretical \u2248 {np.sqrt(252/len(train)):.3f} \u2192 null is well-behaved)')\n", "\n", "# Selection step \u2014 top 30 by in-sample Sharpe\n", "N_TOP = 30\n", "top_names = [name for name, _ in sorted_by_train[:N_TOP]]\n", "selected_family = {name: noise_pool[name] for name in top_names}\n", "\n", "print()\n", "print(f'selected top {N_TOP} by in-sample Sharpe (these are LUCKY by construction):')\n", "for i, (name, sh) in enumerate(sorted_by_train[:5]):\n", " print(f' #{i+1} {name} train_sharpe = {sh:+.3f}')\n", "print(f' ... ({N_TOP - 5} more)')\n", "print()\n", "print(f'selected median train Sharpe : {np.median([noise_train_sharpes[n] for n in top_names]):+.3f}')\n", "print(f'pool median train Sharpe : {np.median(vals):+.3f}')\n" ] }, { "cell_type": "markdown", "id": "937eea33", "metadata": {}, "source": [ "## Section 6 \u2014 Run FLOW's diagnostic on the lucky 30\n", "\n", "These 30 strategies have zero underlying skill (they're random sign\n", "sequences). Their positive training Sharpe is *entirely* selection bias.\n", "\n", "**The honest test:** does `per_strategy_overfit_score` cluster *high* for\n", "them, separating them clearly from the honest designed family of\n", "Section 4? If yes, FLOW catches selection bias. If the two distributions\n", "overlap, this diagnostic isn't differentiating and we drop the notebook." ] }, { "cell_type": "code", "execution_count": 10, "id": "366e4e48", "metadata": { "execution": { "iopub.execute_input": "2026-06-01T19:27:38.447425Z", "iopub.status.busy": "2026-06-01T19:27:38.447347Z", "iopub.status.idle": "2026-06-01T19:31:20.419565Z", "shell.execute_reply": "2026-06-01T19:31:20.417693Z" } }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "evaluate_family: ~12870 partitions, ~778200 strategy evaluations (use progress=True for live updates).\n", "/tmp/claude-501/ipykernel_9560/738322035.py:2: UserWarning: evaluate_family workload is large (778200 strategy evaluations); consider progress=True and/or executor='thread'.\n", " selected_report = sf.evaluate_family(\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "sablier-flow: estimated cost 2 credits (use sf.estimate_cost(...) to preview before charging).\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "sablier-flow: actual cost 0 credits (remaining balance: 10000).\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/tmp/claude-501/ipykernel_9560/738322035.py:2: UserWarning: evaluate_family: real_data has 2013 bars but the synthetic horizon is 252. The real Sharpe is computed over 2013 bars while each synthetic Sharpe is computed over 252 bars \u2014 different sample sizes mean the synthetic distribution is a biased null for the DSR-vs-real comparison. PBO is unaffected. To match horizons, either pass `like=real_data.iloc[-252:]` (truncates synth to your window) or window `real_data` yourself to 252 bars before calling.\n", " selected_report = sf.evaluate_family(\n", " [evaluate_family] 10/200 synthetic paths complete\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " [evaluate_family] 20/200 synthetic paths complete\n", " [evaluate_family] 30/200 synthetic paths complete\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " [evaluate_family] 40/200 synthetic paths complete\n", " [evaluate_family] 50/200 synthetic paths complete\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " [evaluate_family] 60/200 synthetic paths complete\n", " [evaluate_family] 70/200 synthetic paths complete\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " [evaluate_family] 80/200 synthetic paths complete\n", " [evaluate_family] 90/200 synthetic paths complete\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " [evaluate_family] 100/200 synthetic paths complete\n", " [evaluate_family] 110/200 synthetic paths complete\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " [evaluate_family] 120/200 synthetic paths complete\n", " [evaluate_family] 130/200 synthetic paths complete\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " [evaluate_family] 140/200 synthetic paths complete\n", " [evaluate_family] 150/200 synthetic paths complete\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " [evaluate_family] 160/200 synthetic paths complete\n", " [evaluate_family] 170/200 synthetic paths complete\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " [evaluate_family] 180/200 synthetic paths complete\n", " [evaluate_family] 190/200 synthetic paths complete\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " [evaluate_family] 200/200 synthetic paths complete\n", "probability_of_backtest_overfitting: ~12870 partitions, ~772200 strategy evaluations (use progress=True for live updates).\n", "/private/tmp/sf_e2e/lib/python3.12/site-packages/sablier_flow/analytics/family.py:769: UserWarning: probability_of_backtest_overfitting workload is large (772200 strategy evaluations); consider progress=True and/or executor='thread'.\n", " pbo_value, n_partitions = probability_of_backtest_overfitting(\n", "/private/tmp/sf_e2e/lib/python3.12/site-packages/sablier_flow/analytics/family.py:769: UserWarning: probability_of_backtest_overfitting: truncated 13 trailing rows so each of the 16 chunks has 125 rows. Pass real_data with a length divisible by cscv_splits to avoid this.\n", " pbo_value, n_partitions = probability_of_backtest_overfitting(\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "evaluate_family done in 3.7 min\n", "\n", "aggregate PBO (classical CSCV): 0.360\n", "verdict label: looks_like_noise\n", "\n", "per_strategy_overfit_score distribution (selection-biased lucky-30):\n", " min : 0.690\n", " median: 0.772\n", " max : 0.885\n", " # strategies above 0.7: 29\n", " # strategies above 0.9: 0\n" ] } ], "source": [ "t0 = time.time()\n", "selected_report = sf.evaluate_family(\n", " selected_family, train,\n", " model_id=result.model_id,\n", " n_paths=200,\n", " primary_metric='sharpe',\n", " higher_is_better=True,\n", ")\n", "print(f'evaluate_family done in {(time.time()-t0)/60:.1f} min')\n", "print()\n", "print(f'aggregate PBO (classical CSCV): {selected_report.pbo:.3f}')\n", "print(f'verdict label: {selected_report.verdict}')\n", "\n", "selected_scores = dict(selected_report.per_strategy_overfit_score)\n", "print()\n", "print(f'per_strategy_overfit_score distribution (selection-biased lucky-30):')\n", "print(f' min : {min(selected_scores.values()):.3f}')\n", "print(f' median: {np.median(list(selected_scores.values())):.3f}')\n", "print(f' max : {max(selected_scores.values()):.3f}')\n", "print(f' # strategies above 0.7: {sum(1 for v in selected_scores.values() if v >= 0.7)}')\n", "print(f' # strategies above 0.9: {sum(1 for v in selected_scores.values() if v >= 0.9)}')\n" ] }, { "cell_type": "markdown", "id": "686cc1f1", "metadata": {}, "source": [ "## Section 7 \u2014 Side-by-side comparison\n", "\n", "The 12-variant honest designed family vs the 30-variant selection-biased\n", "lucky family. The predication, falsifiably: the selection-biased family's\n", "overfit_score distribution should sit clearly *to the right* of the\n", "honest family's." ] }, { "cell_type": "code", "execution_count": 11, "id": "c1f65d27", "metadata": { "execution": { "iopub.execute_input": "2026-06-01T19:31:20.424902Z", "iopub.status.busy": "2026-06-01T19:31:20.424800Z", "iopub.status.idle": "2026-06-01T19:31:20.770626Z", "shell.execute_reply": "2026-06-01T19:31:20.769223Z" } }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAABEAAAAGGCAYAAAB7dvRBAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjksIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvJkbTWQAAAAlwSFlzAAAPYQAAD2EBqD+naQAAbNxJREFUeJzt3Qm8VfP+//FP86SJZp1KqGgkpJRQVNzEdUOGMhRSpgi5kTljxhSRuKTkJq7clDQgSSUJRSmVJqU6DZpO6/94f3//te/e++x9pvY5eziv5+OxOp211177u75rnf39rs/6DkU8z/MMAAAAAAAghRWNdwIAAAAAAADyGwEQAAAAAACQ8giAAAAAAACAlEcABAAAAAAApDwCIAAAAAAAIOURAAEAAAAAACmPAAgAAAAAAEh5BEAAAAAAAEDKIwACAAAAAABSHgEQoJC66qqrrF69evm2/yJFitj9999viWzmzJkunfqZnW+++cbatGlj5cqVc+9ZtGiRFZRVq1a5zxwzZkxgnfJW62Lp4MGD1qRJE3vkkUcs2Z166ql25513xjsZAApRuZeb7/CC/uynnnoq223zo1yJJZ07ncN41z+Ujr/97W+Wn6ZMmWItWrSw0qVLu+PZtm2bxbNulB9/Ozt37rRq1arZ22+/bclsy5Ytrm748ccfxzspyCECIEh5qnDoi3z+/PkRXz/jjDPcTV+y+PHHH13BrkoNCsb+/fute/fu9ueff9ozzzxj//rXv6xu3bopl/3vvPOOrVmzxvr371+gnzt+/Hi74oor7Nhjj3V/q/qbjBaEUtoaN27sKht16tSxiy++2H7++edM29511102fPhw27BhQwEcAYD88P3339s//vEP932rG8EjjzzSzj77bHvhhRcSMsPHjh1rzz77bLyTgSSnG2qVbWXKlHHlmOocKvNSzXPPPWfly5e3Sy+9tMA+86+//rJrr73W1fsrVqxohx12mDVv3tylRXW9cAo8XXfddVa1alV3Ds4880xbuHBhyDZHHHGE9e7d2+69994COw4cmuKH+H4AcQiAPPDAA+4mMR5PsnJTyBQvnhpfMStWrLDffvvNRo0a5Qq5gqbKv/KzRIkS+fo5Tz75pKuIqFJQkEaMGGELFiywk08+2VX8onn88cftyy+/dMGoZs2aueDGiy++aCeeeKLNnTs3JJDZrVs3q1Chgr300kv24IMPFtCRAIiVOXPmuJsNBTr79OljNWrUcAFa/a3rZuWmm25KyADIkiVL7NZbb43Ld/ihGjx4sN19993xTkahp2D/jh077KGHHrKOHTsWeH6cfvrp7notWbJkvn2Ggg36O77tttusWLFiVlB0XD/88IOde+65rg5dtGhR912jdHz99dfubzi4Vex5551n3333nQ0cONCqVKni6hSqf6vOooc2vhtuuMGef/55++yzz+yss84qsONB3qTG3QmAhKOndali06ZN7melSpXi8vlqFZHf+fntt9+6Qv7pp5+2gqanW3qyq4pIVq2xBgwY4ConwZWySy65xJo2bWqPPfaYvfXWW4H12peeHL/55psuYJjIzboBZKaueArG6mYw/LvX/05OFgXxHR4LemiRKg8uklm86xwqP/P7ev3oo4/sjz/+cC1dCtLhhx/ugqjBFLzQd40eqAwbNswFW+W9995zwZEJEya4+oQovQ0aNLAhQ4aEBEuOO+44V39Rq3MCIImPLjBABAcOHHCR96OPPtpKlSrlosT33HOP7d27N2I/0C+++MJOOeUUV2DUr1/f3XRFakanp0JpaWlun8ccc4x7oq0Ic7Bx48ZZy5YtXbNAPcHWzZ2i5KIvVj39Fj0ZU6UqJ2NYTJo0yX0xK336+f7770fcTmlR8111MdC21atXt+uvv962bt0asp26E3Xq1MlFw9VE86ijjrJrrrkm2z64SudJJ53k9q28ffnllyP2Odbv6urgp1v5pTSpT2y433//3X220upvN3r06EzbrV271i644ALXhFF9ThXtDz+fkajfa/v27d3/lffBXTQWL17sXtc51zGp0FRawlsx+Meorhrq6qGCVs0p1VzS8zz3VNNvsaB9hAchctJ/XGlUM85IGjZs6M5XVpTXCizoyU+ktC9fvtwdqypkSv/VV19tu3fvtljQ34QqXNnRGCzhT6T0BEbn/Keffsq0vZrKq+VOQY7XAiB2Le/0tx3pJlDf4eEUAFXZqTJJNzlqzabv1uzktNyT//73v+671i+f1WrNvwlSuTB58mT3neOXzX4rzWjf4Xpa3K5dO1cu6ThVDoR/l8XyO1hdONUaRXmk41BrlUifFez11193N3TKc5Wxxx9/vGu1Fy4n9YKc5rXKxYcffthq165tZcuWdfUdPbXPq2jjV0Qb80TXkup0+uzKlSu7cnHq1KlZfsYbb7zhgkdqKaCbY7X20Q1+OHWn0Dncs2dPxP3oOurVq5f7v64vpc8f9+Tzzz939RC1itK5UNmpuoxaNYQfr7p2rF692tVR9X89ZFB3Gr9rmc6prjtdD8E38jkZH03nR/mp6zWcjkvXp85rdnUO7UN1wUhpV91OdTb9X/WlO+64wzIyMiy/+NdH8FgrCoDoGv373/8eWKe0KAjywQcfZKpDqs7xn//8x+UPEhthXhQa27dvt82bN2daH6nPn7o5qDBTxPf22293zeKGDh3qKibhwQNVSrSd+hSq0NLNt77AVRFTIS+qpKiyoS90FQoqvBRVHjRokK1fvz7QZ3jatGnWo0cP69ChgwuOiD5Tzf5vueUWVwjffPPNrpmdAjKKOIv/MxIV2hdddJGrtOgYdHOuipMqFuGUNlXQ9Lo+Z+XKlS4irtYBSoMKdD2ZOOecc1whoKayKshVuZs4cWKW+a99dO7c2WrWrOmeyKsgU9cE7ScSBZW0zxtvvNFVNnXMOg4V6OpvKRs3bnSDXfoBE+1LFVSdi/T09EAzZFUOlKd6r46rVq1artWBKp/ZUZ6o4vDoo4+696pCogLRP1+//vqryy8FLlRBe+WVV9xPPWEIr1iptYLOlVorqKKsCp4q6goEqTKic67BwFTQ63PCgxFZufLKK10zcVVog1tR6OmpAi9q2pwVXY96X7Qm2irwVaHVNaT+r6+++qqrEPvXqf83FunvKZwqvqrUxIIqGroO/L+1YPobFF27J5xwQkw+D0DB0I3ZV199lek7LVprEQWU9T2l8ls3nhonRN+hKnuyepKek3JPtI1u6PVdo7Jb+9Q2Csxfdtll9s9//tN9ByrYrkCDZPU99+mnn1qXLl1cAF034iqnlObTTjvNfceG37Dn5Ds4K3owo24V/fr1czeperCickc3w36ZFomCHTrm888/393g6wZP5bKCGdqX5LRekNO8vu+++1z5qG4KWnS82v++ffssv6l+ovOhgLvqKAq6qw6o+oLSEInKfbUiUL1M6Va9UO/V+FbBY2op/bqpVl0mWgsLXUd6aKF9ah86536QQC0RVJ/s27evqwfNmzfPXTO65vRaMNWxdH3pb+CJJ55wdQulRUEPfcbll1/ubuxHjhxpPXv2tNatW7vPygnVbfQwR/vV2Giqx/h0faj+pdezq3Oo+2okSruCaa1atXKD9+pvRQ+GlA86dp8CZzkJiiiQpSWYzoXSqb87Be/0OfrO0cNJn65LpTH8AY2CYzo/qlvpIWVwnUN/+6oDJtPYgoWSB6S4119/XaHYLJfGjRsHtl+0aJFb17t375D93HHHHW79Z599FlhXt25dt2727NmBdZs2bfJKlSrl3X777YF1Dz30kFeuXDnv559/Dtnn3Xff7RUrVsxbvXq1+/2WW27xKlSo4B04cCDq8UyYMMF95owZM3J0/C1atPBq1qzpbdu2LbBu6tSpbh9Kv+/zzz93695+++2Q90+ZMiVk/fvvv+9+/+abb7L8XG0zZMiQwO9du3b1ypYt6/3++++Bdb/88otXvHhxt234e0uWLOktX748sO67775z61944YXAumuvvdYd2+bNm0Pef+mll3oVK1b0du/e7X5/9tln3XvffffdwDa7du3yjjnmmBzlpV7Xdsr7YP7+g73zzjuZrgnlg9Zdd911gXU6x7Vr1/aKFCniPfbYY4H1W7du9cqUKeP16tUrsG7lypXu/bqWw/fp0/ktXbq0d9ddd4Wk5+abb3bX3s6dO7M8RqXloosuyrTe/5xrrrkmZP2FF17oHXHEESHr2rdvn+3fmpbgYwunv0XtJ6f+9a9/uX2+9tprEV/XddS3b98c7w9AYlA5pfJRS+vWrb0777zT++STT7x9+/aFbLdq1Sq3zSOPPBKy/vvvv3flS/B6fffkpdzT92v58uW9Vq1aeX/99VfItgcPHgz8/7zzzgvZf1bf4Sqbq1Wr5m3ZsiWknCtatKjXs2fPPH0HR+J/tsqVtWvXBtZ//fXXbv1tt92W6bOyK+c6derk1a9fP/B7TuoFOc1r1aH0va28DM7be+65J9vyI1r9I/y8Rzte1UmU/8rbjIyMkG2D06J9KX3y3HPPuXJc9bxgumZ1vQSbOHFijuocfr01PD8jnYuhQ4e6z//tt99Cjlfvf/TRRzPVLbTtuHHjAuuXLl2aKb/8Ok9wOsPzcNmyZW6bESNGhKTn/PPP9+rVqxeSX+H279/v0hFcTw5P+4MPPhiy/oQTTvBatmwZss6vg2e3BB9beF3NX0466SRv8eLFIduo7hT+dyeTJ09279G1G2zOnDlu/fjx46MeOxIDXWBQaKjpn57Yhy8aTDGYP42VxhsIppYgoif3wdSyQk1YfXoCoui9Wgb4FJnXNmpKqVYo/qLBrRS9nj17tttOT0127drl0hULal2i5v9qmRI8sKWa6SndwZRGbaPXgtOoiLaeYs2YMSOQRr//Zk6e9ouOURF8NWdU6wufIu16QhGJ8ia4aaTOk5oc+/mqOs6///1v69q1q/t/cJr15EBP4vyRunVO1fLE78MpehqgpqiHQs18fXqips9WixQJHyVcggdQ1aBf6g6ktKvFik/5G3795ITOnZqjaiYXv/ml8l1PoPyuP1lRyyBdn9Ho6VYwXc96j56g+PSEJtLfWPgSq+lply5d6p5A6smV32Q4nP83ByC5qCxSCxC1PND4RHrarO92tcj78MMPA9uplYFaI6iFRHA5oFZ56iLnl12R5LTc0/eWWk+odUP4k/u8jC/kl81qLRr89FzlnNISaTrNnHwHZ0XlgPIu+Cm2nrBnN3VncDnnt6RVi1aVUfo9p/WCnOa16gp6Oq9BboPzNnxg2fygbhm6ltQCJfypf6TzrGtSrXPVCie8laVaVajliLpy+dQKQ91W/G61uRV8LlRXVP6ppYrKfLVWyKrO4dctVBcIHndD6/RabuscGgdD10/wFLZqDaJWuGpdktXfhbZTmnNb5whPoz47J3UOnYtw6lal13Rd6rPU+kh5GkytQ9TVKJz/HRDe9cg/HuociY8uMCg0VNjrhjO7GyT131XBF9wMTlSZUiGh14OpO0ukfQb3af3ll1/ceBHRunv4A16pWem7777rggKqqKi5pQoqdR3JCz+twSNVBxd6wTfpSqMqM5H6VgenUQW3mm+qmaia+qm/qipWagIcqaDw36uCIjxPJdK6nOSrmjirr6aaIWrJKs3KB31OeIGsPDgUKsSVDxq3JXxQPr9imNUxqTKoglR9psPXZzUbSjQq5BXwUD9hNXtVRVLdQ9Q9Jiey6rcanna/oNf5UGAquMtJQdAMMBqdXXmlJsXRRpHXMTEAKpCc1BVQAQ7dECsIoi6oKncUzFYAQYF8lV36O49UzklWM6/ktNzzb2Jj1azdL5sjlUHqJvnJJ5+4m7HgwHV238Eqj4K7iOhmOfjBR6T80U2s6hxZUdcUjWmhYFT4mCPKO31GTuoFOc3raPUW1Z+yumGOBZ1n1f/CHxBFMmvWLPdATFOua9yPcOryqqCNbtIVUNGxK0CkMTvyWiapG6/2pQBg+Lgp4XUO1S3C65w6V+r+HP75Wh9pzJuc1DnUrUbnTN1HFExQAOxQ6xyR0h5erxZ1F8srdfvyu37p+0TdnBWc03XqD4Kqv6FIY8X547cEB6SCj4c6R+IjAAJEkdMvsKxuvHx6oqAv1mhPvlUJEVUMVKlT5UdRdC0agEyFjMYkyU9Koz4/OJofzC+MlC+64dQYF+rrqbSqX7Se/mtdrMZ2yC5f/cFj1c802tP/8NY9sabglPqxqvLTokULd+xKlwJW4YPbRjumnFw/OaWnoyrQNYCbAiD6qYI8J9PoqT9xVhWgnKQzvAIeTXjFPLdU0VOQUAEwBXuCWxWF0zbhASYAyUXjMCgYokXlpcaQ0M2Wbsz1XatySeVlpO+prMqknJZ7iSC772CN56Cbcp/KxawGzs5pQEDjZzVq1MjNjqHWCzoXajWiQIdfzuWkXhDPvI5WnzuUQTU1LorKF40nprFNwsfP0A27BiD1AyDKH91MZzc2RjRKq+qRKmcVdNE5UYBMY8upJVF4nSPa9RLLOocGGlZAR8eo8U9U59CDxuweLqnVk85JtDpHTqfF1YOwnJxDXX/Z1U0VBNHYKBrc1B/AVS2H1VornL8uvO7hHw91jsRHAAQIoyi2ChJFgYMHF9WTdBV2ej231JVj586dOboRVeVC3Tq0KB1qFaJBMjXAW6RWDFnx06pjCbds2bJMaVSLAUXUw6PakairhxYNPqcRxNXkUS0hgptc+lTpUURfA4OFi7QuJ1RZ0uCoKvyyy1flgwbSC28NEJ4HuaGCbvr06e6Jlyo3vkh5XVBUadATN1V61SRXzXk1MGpOKhOqTGlAukMRXgGP5lAq5nryor8NDT6m6zWrJ3WqGCogk9UgwQCSi9+S078JUdml73bdgPoPE3Iqp+We3x1T5Ui0VouS0/LZL5sjlUHq2qcbqOy6LYZTsCH4hjL85ixS2aTv0Uizo/gUzNBNu1ocBLdAidatKKt6QU7zOrjeogFig29289JKwQ9GBM/u4Qtv0as0qt71448/uocaWdE5UlCjbdu2LkikgdvD81wPr9Q1VYORK0igwbgjDdidExqsVudLD8OCu3TEqst0XiiQoZaYOjada7UW8gf1z4oG01VeH2qdQwHR8HMYiQKl4bMShvO7swS3pNE1oIcsuiaCu0Spa5O6UYd/3/jHQ50j8TEGCBBGI45L+Je4nn6Ivuzz0lJAzUf1VCScCmVNuyvh3R70heu3YvCb4fmVokiFeThFr/UFrgIz+EtdBaYK+PA0Kpig6X/DKX3+56kCEv6kwK8oRJtWVjfgClLohnzdunUhwQ89tcsL7VNNbjUOSPhUfhI8/ZzOqT5XlRWfmvJG6zqT08+X8LzISeGfn9T0VOdITzAUdMvp0yaNo6F8zMnUwNHk9xgguj7VrFh/S3r6qzRnZcGCBe6n+kgDSC66yY70VNofs8J/yqzAq76PFYwO316/Z9WdMKflnrqjKuCuGVjCpy8N/kyVz5G6P2ZVNgeX5foO1sxtfj0kN9QFUeWsv4QHh1X+Kijs0wwiupGLNg5XtHJOx6eWqcFyUi/IaV4r7eq2pNlNgvd5KGWrbraVbnVF9imAFj6rn7rtqN6l2VfCW1REuhbVnURBHd08q3VG+LWmvFWgRA8k9HAgr60/op0L/V+z+cS7zqH6pFrCKo1qFZITKr81+8qhyMsYIOryHulcalYlCe4qr1YhevgZPJuR3q/6hx7EhHf7Vp1DrVvzGuRCwaEFCBCmefPm7gm1bo5VIKtvqyoKqqiocNTASbmlgkFPUNQc0p8iV/17FdHXTbmmi1Mhqackat6oqelUsCqyrUqAKhJ+RFn/VyGjAlUFur6AtX20frWqsCloo6cUapKq/Wuf+oLWDbJPx6mbZm2vbjiq8KkSoqcw+rJXIavCQPnw0ksv2YUXXugqFRoYbtSoUa4PclaVNkXfVbHT0x9NY6aKkKa/U59qfV5eaDpZVZI1EJdaOqjCp+PT2CaqlOj/otf0WSoEVUCp8qlmq+HTouWGjtefXk59XjVmi47vUJ9oHCo9YVKe6pzpmok2zVw4PaVSxVSVtGhT/WUnr2OAaBBgfyBgBa70t6GpBEV57E8HrIGI9XekiofOrZrbBguvXKrio6eWTIELJB8NgqlAtcoatVBTay51OdQ4R2q1oG4wonJI3xeamlZlqcppBSv0XawbXA12ranFI8lpuafve3X5UBmtp85qaadWBRqXRGn0u6jqO1Dp0yDq2k7N7vV9FcmTTz7pbpB1I6iBsP1pcHUDld3T6rxQyxXVA1T+KiihgIK6PmYVkFZ++K1S/aC6ynvVN4K7BuSkXpDTvFbrTp0vbac6k96vAT71sCSvXQt0U65uI0qfpt/VOdP0vnqCHzwWmvJI3SBUFmrQTQXXVMdSCw617lCaIuWryn6Ne6JuqJou1x8XS8enz1b9Q/W2Hj16WF7pb0B5q7xRIEufoQdAeW0VEyuqX+o60jnU9RytLhqpzqF6mFq15Lbl1qGMAaJ6g6b+1feEWhjpWtXDSdUXdJ2rPu3T9agWTfquUZBH15+uc9VfFXAN5++DMUCSQLynoQHyW7TpxHyacjN4Glx/iq4HHnjAO+qoo7wSJUp4aWlp3qBBg7w9e/aEbBc8FVr4PsOn8tyxY4fbh6Ze1RRvVapU8dq0aeM99dRTgWn93nvvPe+cc85xU+Npmzp16njXX3+9t379+pB9jRo1yk1Bp6n/cjKl2r///W/vuOOOc9PzHn/88W4qtmjTwr3yyituqjFNl6Zp/5o2beqmH1y3bp17feHChV6PHj1c2rQ/pfVvf/ubN3/+/JD9RJp6bPr06W4qMx3b0Ucf7b366qtuGjRN3xr+3n79+mVKm9IbPgXexo0b3bY6RzpXNWrU8Dp06OCOI5imiNP0bJqKV3mvKYf96ffyOg2uphTUdHmVKlVy0+52797d5VP4sftT7f3xxx8h79exaJq17K7JnEyDG+yJJ57INAVeTjRr1sxNLRwsWtr9vyul7VD5n5Hd9HXZTbMbTFMYaorkwYMHH3L6ABS8//73v24KykaNGnmHHXaYKzdUft50003uez9SOde2bVv3napF71PZoOk6fXkt93wffvihK7e1naasP+WUU9x0mj5NN37ZZZe5MiF4qvlI3+Hy6aefeqeddlpgf5ou/scff4zpd7D/2U8++aT39NNPu7JSZXe7du3ctLuRPiv8mFU2qJzW9KaPP/64N3r06JDPzmm9IKd5re9v1cH0Ha7tzjjjDG/JkiUR6wCRRKp/aFrlJk2auOuoYcOG3ltvvRW1HNXxqa6iY6lcubIre6ZNm5Zl3U/TCut4Tj/99JDpaufNm+c+Q3W7Q6236tro2LGj+3tQPaZPnz7uHIZfWzmtW0Q7npxMgxvsxhtvdNuPHTs2x8e4d+9edwzh0wdHS3tWdZ7cUJ6qruZfq/qsE0880Rs2bJir+4f7888/Xb1IU06r/qg8jHQ/8dNPP7n06W8aia+I/ol3EAZA4aUo/A8//BDXsTNSjZ6kaWAyPQ2NNJtONHoao2llNdK8P61hslJzbz2l1SB+avEDAEBBUyshtdx98803czw7SrJRfeO1115zs7PlpmWtWtqoO5Xqfzkd+DRRacYftWRVK2NagCQ+xgABUGDC50xXoaf+3Go6ithQTFsVETU3zk3wQzSImd4zfPjwpD8d6iKm6fkIfgAA4kVdgdQVSt1pUpHGxFG3Eo3JlttuxQqcqFuVBspNZhr7RWOIqCsewY/kwBggAAqM+ltqDBT91Pgm6oOr/sV5HRQT/6NxMzQ+hsZE0dgymsottzT4W6QBZZORBkoFACAeNIOOxo3QeHIKxud2Vp9Et2nTJjfWmsaxUwDglltuyfU+FBjSfpKdxkAJHlMPiY8uMAAKjAaS0g26mklqYDEN/vboo4/meKBORKfuLpoGUl1XNHWypiEEAAAFTwP1agYRDYyq7qUamDeVzJw5000KoEFP7733XhfkAZIFARAAAAAAAJDyGAMEAAAAAACkPAIgAAAAAAAg5REAAQAAAAAAKY8ACICYOXjwoD3xxBNuMM7SpUtbs2bN7J133snVPjSq+FlnnWUVK1Z0g4a1bNnSxo8fn2nataFDh9rxxx/vpl078sgjrXv37vbDDz/E7FguvPBC69GjR2Bq2cqVK9uYMWMsv82ZM8fatm3rjqtGjRp2880352h0caVN069FW95+++2YpO+nn36yzp07u9HbDz/8cLvyyivtjz/+iMm+AQCFs/xfsGCB/e1vf3PlnsoXvf/555+3jIyMkO1UHt56661Wu3ZtN5j6cccd52aUiyXK/8znVnWM888/39LS0tyMNk2aNHHTvqo+Fk6Dv2rQew2QWqZMGTfQ/YQJE2J6joBDwTS4AGLmn//8pz322GPWp08fO/nkk91UrJdddpm7Ab/00kuzff/rr79u1157rZ199tludphixYrZsmXLbM2aNSHbXX755W7KV32OCtZ169bZ8OHD3awymgK2bt26h3ws8+bNC0zPq5v+bdu22amnnmr5adGiRdahQwdXoRs2bJitXbvWnnrqKfvll1/sv//9b5bvPf30091I8+GeeeYZ++6779x+D5XSo89RcErnRxVRpU95rvzSlMYAgMLnUMp/BT/atGljxx57rN11113uAYDKPE2tumLFCnvuuefcdgqGaFaV+fPnW79+/dz2n3zyiZv5bOvWrXbPPffE5Fgo/0Pt3r3bBTRUB7rhhhtcYENTzQ8ZMsSmT59un332mTvPkp6e7h7iKAii86eA1rvvvmsXX3yxexCjawKIOw8AYmDt2rVeiRIlvH79+gXWHTx40GvXrp1Xu3Zt78CBA1m+f+XKlV6ZMmW8m2++OdvP0VfXHXfcEbL+s88+c+uHDRt2iEfieWvWrHH7mjt3rvv91Vdf9SpWrOiOJz916dLFq1mzprd9+/bAulGjRrm0fPLJJ7ne3+7du73y5ct7Z599dkzS17dvX3eOfvvtt8C6adOmufS9/PLLMfkMAEDhKv/79OnjlSxZ0tuyZUvI+tNPP92rUKFC4Pd3333XlTevvfZayHYXXXSRV7p0aW/jxo2HfCyU/5nt3bvX+/LLLzOtf+CBB9z5UD3A98QTT7h106dPD6zLyMjwTj75ZK9GjRpuX0C80QUGiLP777/fRc5//vlnu+KKK9zT9apVq7p51dX1Qq0funXrZhUqVHCR9Keffjrk/fv27bP77rvPdRXRe9U0sV27djZjxoyQ7RSpL1q0qIvWB7vuuuvck3u1EjgUetqzf/9+9yTGp+Pq27evazmgpwVZGTlypHu68+CDD7rf1bpAxx9ux44d7mf16tVD1tesWdP9VHPLvNi7d69t3rzZLcq7EiVKuKae+n327NmuOe6WLVvc72oOmpUHHnjAbrvtNtedR+cnJ/TUZNq0ae4a0Ln29ezZ0zUH1hOU3PrPf/7j8kstZmLh3//+t2uiXKdOncC6jh07WoMGDfKUPgAozCj//1f+qdtMpUqVMpXrwWX6559/7n6GtyjR7+qKoXpIXlD+Z011RLXQidRVyG8lG3yOVIdVV2af6p5qAbJhwwabNWtWns4REFPxjsAAhd2QIUNctLxFixZejx49vJdeesk777zzAq0ZGjZs6J68a/1pp53m1s+aNSvw/j/++MO1GhgwYIA3YsQIF33Xe/Q05ttvvw1st2/fPu+EE07w6tat66Wnp7t1U6ZMcft76KGHQtKkfeZk2bNnT+A9vXv39sqVK5eplcTy5cvdZzz//PNZ5kPLli29Zs2aeWPHjvWOPPJI957KlSt7gwcPdk8Pgo9DT5T0JOHDDz90T2u+/vprr3379t5RRx3lbd26NU/n4fXXX3efmZNFrVWyonPlH8Nhhx3mXXjhha4Vybp166K+54svvnDbjx8/PtNrbdu29U488cRcH9P555/vWmz459u3a9euHJ3fP//8M1PLm8cffzzT51xxxRXe4Ycfnuv0AUBhRvn/f1R3UfmiesSPP/7orVq1yq1TPebZZ58N5Nd1113nFStWzNu/f39IPk6ePNm9//rrr8/TeaD8z7r8j2bq1Kku31Vv851zzjlenTp1Mm07fPhwt+3QoUPzdI6AWCIAAiRIBUgFu0/NRXWTX6RIEe+xxx4LrNfNvW5oe/XqFbJteJNCbVe9enXvmmuuCVn//fffu2amqmRoG92kn3TSSZkqEzkNBKjS4FPQpn79+pmOTzfb2vbuu+/OMh/UzFUBj1KlSnn33nuv995773mXXXZZxPcq4HH00UeHpEUBlPXr13t5peCEmnFqUZCoZ8+e7v/vvPNOIIDjv/7XX3/laJ8LFy50waVTTz3VK1q0qDufCmTo+NS9JjiwM2HCBPc5s2fPzrSf7t27u4BPbqgpsc71xRdfHPWay25RPvi++eYbt+7NN9/MtL+BAwe614IDYgCArFH+/68e079/fxfw8MsfBToUBAn29NNPu9c+//zzkPWqI2j93/72tzxdcpT/WZf/0XTs2NHV3YIfPN10002uvqMgVrBLL73U7VfnGYg3BkEFEkTv3r0D/9fgnyeddJLrOqJBQX1qHtqwYUP79ddfQ7bVIuqaocE69VPvX7hwYchnaNRudc8YNGiQLV682HXnmDp1qhUvHvpVoK4YOdG4cePA///66y83Ins4NWv1X8+Kurwo3RpETYOgyUUXXWR//vmnGwBNg5tpVhjRjCwtWrRwM79oUK7ly5e7WWH0u9Luf2ZuqKmtFuWfuh2p24i6d7z33ntuf+oqFOn4snLCCSe4ZfDgwW6mFA3qpia6jzzyiD300EOuiajfJcnPn2h5mF3+hVO61f0mUvcXdavRIGXZCW56nF36/G1ym0cAUNgV9vJfx3D00Ue7AU5Vjut9mkHmpptucl1/L7jgAredBtBUN9lrrrnGDXyuQVB1DC+99FKOPicayv/MsutOrIHQ1c1XeR/cdUnXsro0q8uLBmFXd2V1kX3//fcP6RwBsUQABEgQweMqiMbzUCWgSpUqmdZrLIpgb7zxhhsbZOnSpW4cDp+mows3cOBAGzdunBvlXAWYppINpxv/3FJhqX604fwp0rIrTPX6rl27AlPP+vT7lClT7Ntvv3UzkGzfvt2NcaLjuP322wPbqcJ3xhlnuJlkNO5IbijPtF/RiPLqr9qoUSNXQdTvCmJoLA0tyn+ND5IbOl+qpOk4NL6IKqiaQvaUU04JOX6Jloe5HdtEo63rM7p06ZLptfr167slN7JLX/A2AICcK+zlvx586EGHZjzTmFeiG+gzzzzTzfaisacUqFEwRDPAafr1c845x22nMbNeeOEF69WrV+C9uUH5n3vjx493D3YUoAuvb2m8tLFjx7rZYk477TS3Tuft2Wefddvm5RwBsUYABEgQ/lOc7NZJ8OCgb731ll111VXuCYkqN5qeTO9TiwhNHxdOT49UyRBNXxqJBqrKCVXG/IqNnqDo5l5p86dDk/Xr17uftWrVynJfel3pCh/cVMcjmuLOH4hT06tpPvpg7du3dxWhL7/8MtcBEL1HFa1g4VPpalAv0TEq0JIdDSo7efJkt8ydO9cFPfQE7vrrr7fzzjvPTdkbfH79QVz9/AqmddnlX7DVq1e7gcjUaiVSsEatbbRkR+nzjzu79CnYQusPAMi9wl7+qxWBWkSG3xyrnB8wYICtWrXKjjnmGLdOD0J0HEq/Hpo0b97c1q1b517TgNy5Rfmfffkf3kJIrUhVj1FLj0j+8Y9/uHOnepAGtz/xxBNt5syZeT5HQKwRAAGSnLo66Gn+xIkTQyoemvUlnG7CVVlSoODWW291T4BUUP39738P2c6/2c2OWltof6IuKa+++qobDTz4qdLXX38deD0rmsVGFbPff/89pHWCX7HxC2IFP0SFajBVvLTuwIEDlluqQPnNfhU8UbcaPU1SqxDlj55M+cekbbOjfahioMqhAisvvviiqyyEP+ULpuCInnDNnz/fPfnyqRvLokWLQtZlR02HlR/RZn956qmnXFPo7CgIpIqnHHnkke4cKH3h9DQxu/MLAIitVCn/Va6Hl+nit2gJL9d1cx68T3XFyGvrFcr/7Mv/4POpmV/U4lbdWsK7T4XPHHPyySfH5BwBsUYABEhy/lOi4CcvKqQ07Wz4DfewYcNszpw5rgmpbsgVkdfNup6oBDe1zUsfYE3Vq6lf9SRHN/x+mhQI0M1z8BRqeiqk4IL6/PotFC655BLXNPe1115zY2T4FTZVstS6QAGS4KcH2lZTCPp0THoapO4quaUxRVQoq7muWk88+eST7nd1WVH+qpmnphfOKT3JUpNd/cxptxA9TdNn6omepkD2xzv517/+5VprqF+0b/fu3S6dOmfhTaRFzU917qON85GXMUD8MVnU3FpjpGiKYNEYJprCWeceAFBwUqX8V7muz1X3niOOOMKtU0BEN9kqC7VtNBpf6/HHH3ddL/Jyc035n7PyX8EtXTf16tWzjz76KFddXvVwS9eC6kW0AEEiIAACJDkVKHr6o6i8CqeVK1e6gkZPYYK7Oajw0o21nth07drVrRszZox7inLjjTe6ioYvL5WI2rVru6dKCh7oqY0i/5MmTXJdMTQeRXBzXg3CphtppVWFqV+B6tChg2u6q7E39FRG7//iiy/s5ZdfDnSvUNpV8dJAaL/99ltgEFRVuvTkKnjQOD29UD9otebQsWZHrRvU4sKvrKmyqEpVboIfosrksmXL3DFmRd19dN58Cvzos9WdR91XNAie+narr3Pnzp1DWlyoZYme8gUHgWTJkiVugLu777475IngoY4BIhqIdsKECe6zb7nlFnd96Xw3bdrUrr766lzvDwCQd6lS/qu8uuKKK6xVq1au7NPNtVoyLliwwB5++OGQrpwqH9WFVF1i1F3nlVdecceqm3KN3+Wj/I9d+a/xzzRArboiq6uVuvYGU4BK58Sn608PbRSE03keMWKEe5AVrcsMUODiPQ0NUNj50+Bp3vVgmuq2XLlymbZv376917hx48DvBw8e9B599FE3ZZmmkD3hhBO8jz76yL3fn8ZMU8ydfPLJbmrdbdu2hezvueeec58/fvz4Qz4WTevqp0VTsCqdb731VqbtlDZ95sqVK0PW79ixw7vlllvclK96f9OmTSO+X/PT33bbbV6DBg3cMVepUsVNsfbrr79mmvY3J1Pw+jTlsKbXDZ7irV+/frnIgdDjy25p1apVpvdqer82bdp4pUuX9qpWreo+Pz09PWSbGTNmuPfr2gnnTwe4ePFiLz8sWbLEO+ecc7yyZct6lSpV8i6//HJvw4YN+fJZAJDKKP//Z8qUKa5+o/LcL/9HjhyZKc9U9tevX9+V/SojL7vsMm/FihWZtqP8jx3V1bKqy6jOE0z1sbS0NHcea9Wq5d1www3exo0bY5gi4NAU0T8FH3YBgPyn5rh33nmnGwwufHBVAACQmij/AUTzv7ZiAJBiNCr9zTffTPADAIBChPIfQDS0AAEAAAAAACmPFiAAAAAAACDlEQABAAAAAAApjwAIAAAAAABIeQRAAAAAAABAyise7wQkooMHD9q6deusfPnyVqRIkXgnBwAA/H+e59mOHTusVq1aVrRowTzHoV4AAEBq1AsIgESg4EdaWlp+nB8AABADa9assdq1axdIXlIvAAAgNeoFBEAiUMsPPxMrVKgQ+7MDIN8sWrTI2rdvb7NmzbIWLVqQ00CKSU9Pdw8p/LK6IFAvAIDkRv0wdaXnsl5AACQCv9uLgh8EQIDkcthhhwV+8vcLpK6C7KJKvQAAkhv1w9RXJIf1AgZBBQAAAAAAKY8ACAAAAAAASHkEQAAAAAAAQMpjDBAAQL7JyMiw/fv3k8PIsRIlSlixYsWSMse43pFbJUuWLLDpnAEABEAAAPk0J/uGDRts27Zt5C9yrVKlSlajRo0CHej0UHC9I68U/DjqqKNcIAQAkP9oAQIAiDk/+FGtWjUrW7Zs0tzIIv6BhN27d9umTZvc7zVr1kyKU8L1jrw4ePCgrVu3ztavX2916tThexIAUj0AMnToUJs4caItXbrUypQpY23atLHHH3/cGjZsGNhmz549dvvtt9u4ceNs79691qlTJ3vppZesevXqWVaghgwZYqNGjXIV8NNOO81GjBhhxx57bAEdGQAUXuoG4Ac/jjjiiHgnB0lG9QFREETXUKJ3h+F6x6GoWrWqC4IcOHDAdf8CAOSvuHY6nDVrlvXr18/mzp1r06ZNc/3EzznnHNu1a1dgm9tuu83+85//2IQJE9z2KiT+/ve/Z7nfJ554wp5//nkbOXKkff3111auXDkXOFEwBQCQv/wxP9TyA8gL/9pJhvFjuN5xKPyuLwqkAQBSvAXIlClTQn4fM2aMe9qzYMECO/3002379u322muv2dixY+2ss85y27z++ut23HHHuaDJqaeeGrH1x7PPPmuDBw+2bt26uXVvvvmmazEyadIku/TSSwvo6ACgcKPbCwrTtZOMaUb8cd0AQMFKqGGnFfCQww8/3P1UIERPVjp27BjYplGjRq6f5FdffRVxHytXrnR9cYPfU7FiRWvVqlXU9wAAAAAAgNRWPJEGgrr11lvdeB1NmjRx6xTIUNNAjQYfTK059Fok/vrwMUKyeo/GFtHiS09PP+TjAQAgmjPOOMNatGjhWixKvXr1XBmoBfFHvSC2uN6BwkvDF2zdujXeybAVK1YEfpYqVSrH76tcubLVqlUrH1OGQhsA0VggS5YssS+++CIug7E+8MADBf65AADIN99848ariqe8DDp+1VVX2RtvvBGyTu8L7uL6559/2k033eTG89KUnxdddJE999xzdthhh1miol6Qv5L1eo/WXUVjzw0cONBmzpxpZ555ZsRt5s2bZyeffHLM0g8kS/Cj81ln2u4EeLi89/+PKTXgxr5WKhcDDpetUMGmfDaDIEgKSYgASP/+/e2jjz6y2bNnW+3atQPra9SoYfv27XOzCQS3Atm4caN7LRJ/vbYJnj5Pv+tpWySDBg2yAQMGhLQASUtLi8mxAQCQk5kg4k2Djk+ePNkNOq6uoyqbNej4l19+meX7Onfu7Mbn8oU/Wbv88svdNJ/+YOdXX321XXfddW58r0RFvSB/Jev1rus42H//+1+79tprXVBPNJth+Db33nuvTZ8+3U466aR8OhIgcanlh4Ifd9Wpamll/2+Gr3hZvmOX3bR5i91Vr4YdUz5nAdg1u/+yx1f/4Y6DViCpI65jgGjAUhU477//vn322Wd21FFHhbzesmVLNyWYCg7fsmXLbPXq1da6deuI+9Q+FAQJfo8CGpoNJtp7VFmrUKFCyAIAKJxN9dVaQV1R1OxVT4M1pbpmJ9ONe/ny5e2YY45xNz4+tV7s0qWLa9Gg7a+88krbvHlz4HW9t2fPnu51BeaffvrpTJ+rLjB+dxgZNmyYNW3a1D0lV0D+xhtvtJ07d4YMGq4HA5988okbGFz7ViAi/OYrp/xBx/W5GnRc5a+CGnPmzHGDjmdFZajKXX9Rvvl++ukn1xrk1VdfdWNxtW3b1l544QX31F1PBhNVYakXcL3n7noPvs61fPDBB67FR/369d3r6rYd/LqmAdc2+u5gsFMUZgp+NKhQLq5LnXL/F4DRz5y+J95BG6RgAETdXt566y33FEiVSo3RoeWvv/5yrysir8i6WmfMmDHDDYqqQkSBjOAZYDQwqoIoogJGFdeHH37YPvzwQ/v+++9dxVNRuwsuuCBuxwoASA7q0lGlShXXZF3BkL59+1r37t3d092FCxe66doV5Ni9e7droagbqBNOOMHmz5/vbvbV4vDiiy8O7E9N4zWNu26Epk6d6prJaz9ZUVcRTef+ww8/uPToIcGdd94Zso0+/6mnnrJ//etfrgWlHg7ccccdgdfffvttFxjJavn888/zPOi4T8ejGdwaNmzo8mrLli2B1/ReBWqCn37rM3R8ejCB+ON6z9317tPfuVqQqJ4ajeqh+ntQ3RUAkBji2gVmxIgRgScQwRSFV79ieeaZZwJ9hoP7aAZTqxB/BhlRJVFP3NTEVpVTPXFSpbR06dIFclwAgMx0w7506dICzxrd2JQtWzbH2zdv3txNpe53hXjsscdcQKRPnz5u3X333efKr8WLF9unn37qgh+PPvpo4P2jR492rTZ+/vlnF3xXywoF+zt06BC44Qzu7hlJ8GCoah2ioP4NN9wQUv4pYDFy5Eg7+uij3e9qUfnggw8GXj///PNdq4usHHnkkXkedFzU6kTdBtT6UgPL3XPPPa41jG4iixUr5t6r4Eiw4sWLu9nestpvskuWa1243nN+vQfT37Ee3un6j0Z/+6q3Zvf3DgAoJAEQdYHJjoIWw4cPd0tO96NWIKoEBlcEAQDxpRtCNTUvaGrdcOKJJ+Z4+2bNmgX+r5t4NWNXdxSfP0jipk2b7LvvvnMtFCMN6KmAgFo0aiyr4ECEbv7VWiIrCqxoIE7lmbpxHjhwwA3aqBtr/wZXP/3gh6h7jdLk082Zlvx06aWXBv6vPFLeKU1qFeIHfAqjZLnWhes9bxTo1Pg20R6urV271nVRe/fdd/P4CQCAlB0EFQCQ+vR0Wjdo8fjc3NDYU+FB9eB1fl9+Td+ucTm6du1qjz/+eKb9KCCxfPnyXKd31apV9re//c11J3nkkUdcwEQzpKmpvYIpfgAkUjqDHwioC8z111+f5WdpLJN27drladDxSDQWglrL6LgVANF7g4MyomCOZobJzX6TTbJc68L1nvvrXV3H1Pp4/PjxUbdRa2YFT9USCwCQOAiAAAAKhG7cc/t0OtHpeP7973+7birq2hFOrSF0g6nxLjS+gGg0eXWPad++fcR96sZZwRUNlqouoJKXp8i56QITPOi4P6NFdoOOR3vqrTEP/FnY9F4FVXRMfosIjWei48subcksFa914Xr/X9cWXc/qPhSJApEKgGgMuvAAEwAgvgiAAABwCIN5a5aYHj16uPGn1FpDrR80y4lmPlHXGLXc0ECoehqs8TD++c9/BgIbkWiWGY3vodlS1LpE03JqrI/cyk0XmOBBx3UMmvVEA8BGGnRcXXMuvPBC1/rlgQcecAETPTVXlx/lgdKvcQ9EM9RonBCNn6Jj0HFprBJ1nWFKweRTmK93n7qkaercSLM5+RTkW7lypfXu3TvXxwEASOFZYAAASGa6idcNW0ZGhpsdRuNgaABTdSPxb/qefPJJ181EN3eaAUUDc2c1PoSeKms6WnWradKkievKopuw/KZBx9X1RgGN008/3QU1Jk6cGHXQcY2PooFg1dKkQYMG7oZSx6XuAZpG1qf060ZSXWLOPfdcd/yvvPJKvh8PYq8wX+8+BXvUwkNBoKxaiGjWqLx0SQIA5K8iXk5GIi1kFN3X0wEVenoqACB5aHpRVbbzMhggYkODderpp2YGYfYtxPoaikcZndVncr3jUHD9IJVpKvduZ3e0FxvVsQYVysU1LUu27bBus+bbB+1PsiaVctZa7Of0XdZ/6Wr7YNqn1rhx43xPI/Imt/UCWoAAAAAAAICURwAEAAAAAACkPAIgAAAAAAAg5REAAQAAAAAAKY8ACAAgXzDGNgrTtZOMaUb8cd0AQMEiAAIAiKkSJUq4n7t37yZnkSf+teNfS4mM6x2HYt++fYFppQEA+a94AXwGAKAQUUW+UqVKtmnTJvd72bJlrUiRIvFOFpLkabiCH7p2dA0lw00h1zvy6uDBg/bHH3+478jixamSA0BB4NsWABBzNWrUcD/9IAiQGwp++NdQMuB6R14VLVrU6tSpQ5AYAAoIARAAQMypxUfNmjWtWrVqtn//fnIYuepSkgwtP4JxvSOvSpYs6YIgAICCQQAEAJBvdCObbDezQF5xvQMAkNgIOQMAAAAAgJRHAAQAAAAAAKQ8AiAAAAAAACDlEQABAAAAAAApjwAIAAAAAABIeQRAAAAAAABAyiMAAgAAAAAAUh4BEAAAAAAAkPLiGgCZPXu2de3a1WrVqmVFihSxSZMmhbyudZGWJ598Muo+77///kzbN2rUqACOBgAAAAAAJKq4BkB27dplzZs3t+HDh0d8ff369SHL6NGjXUDjoosuynK/jRs3DnnfF198kU9HAAAAAAAAkkHxeH54ly5d3BJNjRo1Qn7/4IMP7Mwzz7T69etnud/ixYtnei8AAAAAACi84hoAyY2NGzfa5MmT7Y033sh2219++cV1qyldurS1bt3ahg4danXq1Im6/d69e93iS09Pj1m6AQBAcqFeAABAakqaQVAV+Chfvrz9/e9/z3K7Vq1a2ZgxY2zKlCk2YsQIW7lypbVr18527NgR9T0KkFSsWDGwpKWl5cMRAACAZEC9AACA1JQ0ARCN/3H55Ze7Vh1ZUZea7t27W7NmzaxTp0728ccf27Zt2+zdd9+N+p5BgwbZ9u3bA8uaNWvy4QgAAEAyoF4AAEBqSoouMJ9//rktW7bMxo8fn+v3VqpUyRo0aGDLly+Puk2pUqXcAgAAQL0AAIDUlBQtQF577TVr2bKlmzEmt3bu3GkrVqywmjVr5kvaAAAAAABA4otrAETBiUWLFrlFNF6H/r969eqQAUknTJhgvXv3jriPDh062Isvvhj4/Y477rBZs2bZqlWrbM6cOXbhhRdasWLFrEePHgVwRAAAAAAAIBHFtQvM/Pnz3bS2vgEDBrifvXr1cgOZyrhx48zzvKgBDLXu2Lx5c+D3tWvXum23bNliVatWtbZt29rcuXPd/wEAAAAAQOEU1wDIGWec4YIbWbnuuuvcEo1aegRTwAQAAAAAACDpxgABAAAAAAA4FARAAAAAAABAyiMAAgAAAAAAUh4BEAAAAAAAkPIIgAAAAAAAgJRHAAQAAAAAAKQ8AiAAAAAAACDlEQABAAAAAAApjwAIAAAAAABIeQRAAAAAAABAyiMAAgAAAAAAUh4BEAAAAAAAkPIIgAAAAAAAgJRHAAQAAAAAAKQ8AiAAAAAAACDlEQABAAAAAAApjwAIAAAAAABIeQRAAAAAAABAyiMAAgAAAAAAUh4BEAAAAAAAkPIIgAAAAAAAgJRHAAQAAAAAAKQ8AiAAAAAAACDlxTUAMnv2bOvatavVqlXLihQpYpMmTQp5/aqrrnLrg5fOnTtnu9/hw4dbvXr1rHTp0taqVSubN29ePh4FAAAAAABIdHENgOzatcuaN2/uAhbRKOCxfv36wPLOO+9kuc/x48fbgAEDbMiQIbZw4UK3/06dOtmmTZvy4QgAAAAAAEAyKB7PD+/SpYtbslKqVCmrUaNGjvc5bNgw69Onj1199dXu95EjR9rkyZNt9OjRdvfddx9ymgEAAAAAQPJJ+DFAZs6cadWqVbOGDRta3759bcuWLVG33bdvny1YsMA6duwYWFe0aFH3+1dffRX1fXv37rX09PSQBQAAFE7UCwAASE0JHQBR95c333zTpk+fbo8//rjNmjXLtRjJyMiIuP3mzZvda9WrVw9Zr983bNgQ9XOGDh1qFStWDCxpaWkxPxYAAJAcqBcAAJCaEjoAcumll9r5559vTZs2tQsuuMA++ugj++abb1yrkFgaNGiQbd++PbCsWbMmpvsHAADJg3oBAACpKa5jgORW/fr1rUqVKrZ8+XLr0KFDptf1WrFixWzjxo0h6/V7VuOIaJwRLQAAANQLAABITQndAiTc2rVr3RggNWvWjPh6yZIlrWXLlq7LjO/gwYPu99atWxdgSgEAAAAAQCKJawBk586dtmjRIrfIypUr3f9Xr17tXhs4cKDNnTvXVq1a5YIY3bp1s2OOOcZNa+tTS5AXX3wx8LumwB01apS98cYb9tNPP7mBUzXdrj8rDAAAAAAAKHzi2gVm/vz5duaZZ4YEL6RXr142YsQIW7x4sQtkbNu2zWrVqmXnnHOOPfTQQyHdVVasWOEGP/Vdcskl9scff9h9993nBj5t0aKFTZkyJdPAqAAAAAAAoPCIawDkjDPOMM/zor7+ySefZLsPtQ4J179/f7cAAAAAAAAk3RggAAAAAAAAeUEABAAAAAAApDwCIAAAAAAAIOURAAEAAAAAACmPAAgAAAAAAEh5BEAAAAAAAEDKIwACAAAAAABSHgEQAAAAAACQ8giAAAAAAACAlEcABAAAAAAApDwCIAAAAAAAIOURAAEAAAAAACmPAAgAAAAAAEh5BEAAAAAAAEDKIwACAAAAAABSHgEQAAAAAACQ8giAAAAAAACAlEcABAAAAAAApDwCIAAAAAAAIOURAAEAAAAAACmPAAgAAAAAAEh5BEAAAAAAAEDKi2sAZPbs2da1a1erVauWFSlSxCZNmhR4bf/+/XbXXXdZ06ZNrVy5cm6bnj172rp167Lc5/333+/2Fbw0atSoAI4GAAAAAAAkqrgGQHbt2mXNmze34cOHZ3pt9+7dtnDhQrv33nvdz4kTJ9qyZcvs/PPPz3a/jRs3tvXr1weWL774Ip+OAAAAAAAAJIPi8fzwLl26uCWSihUr2rRp00LWvfjii3bKKafY6tWrrU6dOlH3W7x4catRo0bM0wsAAAAAAJJTUo0Bsn37dtelpVKlSllu98svv7guM/Xr17fLL7/cBUwAAAAAAEDhFdcWILmxZ88eNyZIjx49rEKFClG3a9WqlY0ZM8YaNmzour888MAD1q5dO1uyZImVL18+4nv27t3rFl96enq+HAMAAEh81AsAAEhNSdECRAOiXnzxxeZ5no0YMSLLbdWlpnv37tasWTPr1KmTffzxx7Zt2zZ79913o75n6NChrsuNv6SlpeXDUQAAgGRAvQAAgNRUNFmCH7/99psbEySr1h+RqLtMgwYNbPny5VG3GTRokOte4y9r1qyJQcoBAEAyol4AAEBqKp4MwQ+N6TFjxgw74ogjcr2PnTt32ooVK+zKK6+Muk2pUqXcAgAAQL0AAIDUFNcWIApOLFq0yC2ycuVK938NWqrgxz/+8Q+bP3++vf3225aRkWEbNmxwy759+wL76NChg5sdxnfHHXfYrFmzbNWqVTZnzhy78MILrVixYm7sEAAAAAAAUDjFtQWIghtnnnlm4PcBAwa4n7169bL777/fPvzwQ/d7ixYtQt6n1iBnnHGG+79ad2zevDnw2tq1a12wY8uWLVa1alVr27atzZ071/0fAAAAAAAUTnENgCiIoYFNo8nqNZ9aegQbN25cTNIGAAAAAABSR8IPggoAAAAAAHCoCIAAAAAAAICURwAEAAAAAACkvDwFQM466yzbtm1bpvXp6enuNQAAAAAAgKQPgMycOTNkKlrfnj177PPPP49FugAAAAAAAOIzC8zixYsD///xxx9tw4YNgd8zMjJsypQpduSRR8YudQAAAAAAAAUdAGnRooUVKVLELZG6upQpU8ZeeOGFWKQLAAAAAAAgPgGQlStXmud5Vr9+fZs3b55VrVo18FrJkiWtWrVqVqxYsdilDgAAAAAAoKADIHXr1nU/Dx48GIvPBgAAAAAASLwASLBffvnFZsyYYZs2bcoUELnvvvtikTYAAAAAAID4BUBGjRplffv2tSpVqliNGjXcmCA+/Z8ACAAAAAAASPoAyMMPP2yPPPKI3XXXXbFPEQAAAAAAQIwVzcubtm7dat27d491WgAAAAAAABInAKLgx9SpU2OfGgAAAAAAgETpAnPMMcfYvffea3PnzrWmTZtaiRIlQl6/+eabY5U+AAAAAACA+ARAXnnlFTvssMNs1qxZbgmmQVAJgAAAAAAAgKQPgKxcuTL2KQEAAAAAAEikMUAAAAAAAABSvgXINddck+Xro0ePzmt6AAAAAAAAEiMAomlwg+3fv9+WLFli27Zts7POOitWaQMAAAAAAIhfAOT999/PtO7gwYPWt29fO/roo2ORLgAAAAAAgMQbA6Ro0aI2YMAAe+aZZ2K1SwAAAAAAgMQbBHXFihV24MCBWO4SAAAAAAAgPgEQtfQIXm677Ta79NJL7ZJLLnFLTs2ePdu6du1qtWrVsiJFitikSZNCXvc8z+677z6rWbOmlSlTxjp27Gi//PJLtvsdPny41atXz0qXLm2tWrWyefPm5eUwAQAAAABAYQ6AfPvttyHL4sWL3fqnn37ann322RzvZ9euXda8eXMXsIjkiSeesOeff95GjhxpX3/9tZUrV846depke/bsibrP8ePHu6DMkCFDbOHChW7/es+mTZvycKQAAAAAAKDQDoI6Y8aMmHx4ly5d3BKJWn8omDJ48GDr1q2bW/fmm29a9erVXUsRtTiJZNiwYdanTx+7+uqr3e8KnkyePNlNzXv33XfHJN0AAAAAAKAQjQHyxx9/2BdffOEW/T+WVq5caRs2bHDdXnwVK1Z0XVq++uqriO/Zt2+fLViwIOQ9GpxVv0d7DwAAAAAASH15agGiris33XSTa5Gh6W+lWLFi1rNnT3vhhResbNmyh5wwBT9ELT6C6Xf/tXCbN2+2jIyMiO9ZunRp1M/au3evW3zp6emHmHoAAJCsqBcAAJCa8jwI6qxZs+w///mPbdu2zS0ffPCBW3f77bdbshk6dKhrXeIvaWlp8U4SAACIE+oFAACkpjwFQP7973/ba6+95sbvqFChglvOPfdcGzVqlL333nsxSViNGjXcz40bN4as1+/+a+GqVKniWqLk5j0yaNAg2759e2BZs2ZNTI4BAAAkH+oFAACkpjwFQHbv3p2pm4lUq1bNvRYLRx11lAtaTJ8+PaRrimaDad26dcT3lCxZ0lq2bBnyHnXR0e/R3iOlSpUKBHL8BQAAFE7UCwAASE15CoAomKBpZoOno/3rr7/sgQceyDLQEG7nzp22aNEit/gDn+r/q1evtiJFititt95qDz/8sH344Yf2/fffuzFGatWqZRdccEFgHx06dLAXX3wxpHuOWqK88cYb9tNPP1nfvn3dmCX+rDAAAAAAAKDwydMgqJqetnPnzla7dm1r3ry5W/fdd9+5JyZTp07N8X7mz59vZ555ZkjwQnr16mVjxoyxO++80wUvrrvuOjfOSNu2bW3KlClWunTpwHtWrFjhBj/1XXLJJW5Gmvvuu88NltqiRQv3nkgtVgAAAAAAQOGQpwBI06ZN7ZdffrG33347MLtKjx497PLLL7cyZcrkeD9nnHGGeZ4X9XW1AnnwwQfdEs2qVasyrevfv79bAAAAAAAA8hwA0ejoalHRp0+fkPWjR492rS/uuusuchcAAAAAACT3GCAvv/yyNWrUKNP6xo0b28iRI2ORLgAAAAAAgPgGQDS2Rs2aNTOtr1q1qq1fvz4W6QIAAAAAAIhvACQtLc2+/PLLTOu1TrO0AAAAAAAAJP0YIBr7Q1PU7t+/38466yy3bvr06W7Wlttvvz3WaQQAAAAAACj4AMjAgQNty5YtduONN9q+ffvcOk1Nq8FPBw0adGgpAgAAAAAASIQAiKanffzxx+3ee++1n376yU19e+yxx1qpUqVinT4AAAAAAArcgYwMW758edLmfOXKlRmiIhYBEN9hhx1mJ5988qHsAgAAAACAhLJl7z77c+tWu7VPbytW/JBum+OmbIUKNuWzGQRBgiTnmQQAAAAAIJ/sPHDAiplnA9OqWP2KFZIun9fs/sseX/2Hbd26lQBIEAIgAAAAAABEULtMaWtQoRx5U5inwQUAAAAAAEgmBEAAAAAAAEDKIwACAAAAAABSHgEQAAAAAACQ8giAAAAAAACAlEcABAAAAAAApDwCIAAAAAAAIOURAAEAAAAAACmPAAgAAAAAAEh5BEAAAAAAAEDKIwACAAAAAABSHgEQAAAAAACQ8giAAAAAAACAlJfwAZB69epZkSJFMi39+vWLuP2YMWMybVu6dOkCTzcAAAAAAEgcxS3BffPNN5aRkRH4fcmSJXb22Wdb9+7do76nQoUKtmzZssDvCoIAAAAAAIDCK+EDIFWrVg35/bHHHrOjjz7a2rdvH/U9CnjUqFGjAFIHAAAAAACSQcJ3gQm2b98+e+utt+yaa67JslXHzp07rW7dupaWlmbdunWzH374Icv97t2719LT00MWAABQOFEvAAAgNSVVAGTSpEm2bds2u+qqq6Ju07BhQxs9erR98MEHLlhy8OBBa9Omja1duzbqe4YOHWoVK1YMLAqcAACAwol6AQAAqSmpAiCvvfaadenSxWrVqhV1m9atW1vPnj2tRYsWrpvMxIkTXTeal19+Oep7Bg0aZNu3bw8sa9asyacjAAAAiY56AQAAqSnhxwDx/fbbb/bpp5+6gEZulChRwk444QRbvnx51G1KlSrlFgAAAOoFAACkpqRpAfL6669btWrV7LzzzsvV+zSDzPfff281a9bMt7QBAAAAAIDElhQBEI3joQBIr169rHjx0EYr6u6ipqq+Bx980KZOnWq//vqrLVy40K644grXeqR3795xSDkAAAAAAEgESdEFRl1fVq9e7WZ/Caf1RYv+L46zdetW69Onj23YsMEqV65sLVu2tDlz5tjxxx9fwKkGAAAAAACJIikCIOecc455nhfxtZkzZ4b8/swzz7gFAAAAAAAgqbrAAAAAAAAAHAoCIAAAAAAAIOURAAEAAAAAACmPAAgAAAAAAEh5BEAAAAAAAEDKIwACAAAAAABSHgEQAAAAAACQ8giAAAAAAACAlEcABAAAAAAApDwCIAAAAAAAIOURAAEAAAAAACmPAAgAAAAAAEh5BEAAAAAAAEDKIwACAAAAAABSHgEQAAAAAACQ8giAAAAAAACAlEcABAAAAAAApDwCIAAAAAAAIOURAAEAAAAAACmPAAgAAAAAAEh5BEAAAAAAAEDKIwACAAAAAABSHgEQAAAAAACQ8hI6AHL//fdbkSJFQpZGjRpl+Z4JEya4bUqXLm1Nmza1jz/+uMDSCwAAAAAAElNCB0CkcePGtn79+sDyxRdfRN12zpw51qNHD7v22mvt22+/tQsuuMAtS5YsKdA0AwAAAACAxJLwAZDixYtbjRo1AkuVKlWibvvcc89Z586dbeDAgXbcccfZQw89ZCeeeKK9+OKLBZpmAAAAAACQWBI+APLLL79YrVq1rH79+nb55Zfb6tWro2771VdfWceOHUPWderUya3Pyt69ey09PT1kAQAAhRP1AgAAUlNCB0BatWplY8aMsSlTptiIESNs5cqV1q5dO9uxY0fE7Tds2GDVq1cPWafftT4rQ4cOtYoVKwaWtLS0mB4HAABIHtQLAABITQkdAOnSpYt1797dmjVr5lpyaEDTbdu22bvvvhvTzxk0aJBt3749sKxZsyam+wcAAMmDegEAAKmpuCWRSpUqWYMGDWz58uURX9cYIRs3bgxZp9+1PiulSpVyCwAAAPUCAABSU0K3AAm3c+dOW7FihdWsWTPi661bt7bp06eHrJs2bZpbDwAAAAAACq+EDoDccccdNmvWLFu1apWb4vbCCy+0YsWKualupWfPnq6Zqu+WW25x44U8/fTTtnTpUrv//vtt/vz51r9//zgeBQAAAAAAiLeE7gKzdu1aF+zYsmWLVa1a1dq2bWtz5851/xfNCFO06P9iOG3atLGxY8fa4MGD7Z577rFjjz3WJk2aZE2aNInjUQAAAAAAgHhL6ADIuHHjsnx95syZmdZp0FQtAAAAAAAASdEFBgAAAAAAIBYIgAAAAAAAgJRHAAQAAAAAAKQ8AiAAAAAAACDlEQABAAAAAAApjwAIAAAAAABIeQk9DS6A+Fi3bp1t3bo1KbN/xYoVgZ+lSpWyZFO5cmWrVatWvJMBAAAApBwCIAAyBT/OOruTbd+5KylzZs/u3e7n9f1utlJlyliyqXhYOfts2icEQQAAAIAYIwACIIRafij4kXZ2Hyt7RPK1RFj7zSe2bcY7VvWUblazcWtLJru3rLM100a5c0ArEAAAACC2CIAAiEjBj/I1j0q63ClZvrL7WapilaRMPwAAAID8wSCoAAAAAAAg5REAAQAAAAAAKY8ACAAAAAAASHkEQAAAAAAAQMojAAIAAAAAAFIeARAAAAAAAJDyCIAAAAAAAICURwAEAAAAAACkPAIgAAAAAAAg5REAAQAAAAAAKY8ACAAAAAAASHkEQAAAAAAAQMpL6ADI0KFD7eSTT7by5ctbtWrV7IILLrBly5Zl+Z4xY8ZYkSJFQpbSpUsXWJoBAAAAAEDiSegAyKxZs6xfv342d+5cmzZtmu3fv9/OOecc27VrV5bvq1Chgq1fvz6w/PbbbwWWZgAAAAAAkHiKWwKbMmVKptYdagmyYMECO/3006O+T60+atSoUQApBAAAAAAAySChW4CE2759u/t5+OGHZ7ndzp07rW7dupaWlmbdunWzH374oYBSCAAAAAAAElFCtwAJdvDgQbv11lvttNNOsyZNmkTdrmHDhjZ69Ghr1qyZC5g89dRT1qZNGxcEqV27dsT37N271y2+9PT0fDkGAACQ+KgXAACQmpKmBYjGAlmyZImNGzcuy+1at25tPXv2tBYtWlj79u1t4sSJVrVqVXv55ZezHGy1YsWKgUUtRwAAQOFEvQAAgNSUFAGQ/v3720cffWQzZsyI2oojmhIlStgJJ5xgy5cvj7rNoEGDXGsRf1mzZk0MUg0AAJIR9QIAAFJTQneB8TzPbrrpJnv//fdt5syZdtRRR+V6HxkZGfb999/bueeeG3WbUqVKuQUAAIB6AQAAqal4ond7GTt2rH3wwQdWvnx527Bhg1uvbiplypRx/1d3lyOPPNI1V5UHH3zQTj31VDvmmGNs27Zt9uSTT7ppcHv37h3XYwEAAAAAAPGT0AGQESNGuJ9nnHFGyPrXX3/drrrqKvf/1atXW9Gi/+vJs3XrVuvTp48LllSuXNlatmxpc+bMseOPP76AUw8AAAAAABJFwneByY66xgR75pln3AIAAAAAAJAUARAgWa1bt861RkpGGjD4YMbBeCej0Mo4cCDLQZsTnVre1apVK97JAAAAADIhAALkQ/DjrLM72fadu5Iyb/fv22c7duy0AxkZ8U5KobN3x1bbtvVP63PjzVaseDFLRhUPK2efTfuEIAgAAAASDgEQIMbU8kPBj7Sz+1jZI5LvSfiW5d/a9imv28GDBEAK2oG9u80rUsyO7HitVaxR15LN7i3rbM20Ue5vgFYgAAAASDQEQIB8ouBH+Zq5n7o53nZt/j3eSSj0yhxRMymvHQAAACCR/W/6FAAAAAAAgBRFAAQAAAAAAKQ8AiAAAAAAACDlEQABAAAAAAApjwAIAAAAAABIeQRAAAAAAABAyiMAAgAAAAAAUh4BEAAAAAAAkPIIgAAAAAAAgJRXPN4JAAAAAABktm7dOtu6dWtSZs3y5cvtYEZGvJMBhCAAAgAAAAAJGPzofNaZtjs93ZLRvv37bWd6umVkHIh3UoAAAiAAAAAAkGDU8kPBj7vqVLW0smUs2Xy9eas9u32bZWQcjHdSgAACIAAAAACQoBT8aFChnCWb33btjncSgEwYBBUAAAAAAKQ8AiAAAAAAACDlEQABAAAAAAApjwAIAAAAAABIeQRAAAAAAABAyiMAAgAAAAAAUl5SBECGDx9u9erVs9KlS1urVq1s3rx5WW4/YcIEa9Sokdu+adOm9vHHHxdYWgEAAAAAQOJJ+ADI+PHjbcCAATZkyBBbuHChNW/e3Dp16mSbNm2KuP2cOXOsR48edu2119q3335rF1xwgVuWLFlS4GkHAAAAAACJIeEDIMOGDbM+ffrY1Vdfbccff7yNHDnSypYta6NHj464/XPPPWedO3e2gQMH2nHHHWcPPfSQnXjiifbiiy8WeNoBAAAAAEBiSOgAyL59+2zBggXWsWPHwLqiRYu637/66quI79H64O1FLUaibQ8AAAAAAFJfcUtgmzdvtoyMDKtevXrIev2+dOnSiO/ZsGFDxO21Ppq9e/e6xbd9+3b3c/78+VauXDlLRgoUHTx40JJVMqf/119/tYwDGbZj/a92YO9flmx2/bHGPM+znetXWtGDGZZs9mz9v+5xuzevs62rfrRkkux5/9ef6+3A/v22ePFi27lzpyWjZP7uSYX0V61a1apUqZLlNunp6e6n/lbyC/WCxJPs1zbpJ+/zXqc8YD+n77RdB5KvXrBq5273Xf3Lzl22v2ixuKbl15273M9l6Ttt/0Ev6dKfF7/v/sv27z+Q1PWyqvlRL/AS2O+//66j8ObMmROyfuDAgd4pp5wS8T0lSpTwxo4dG7Ju+PDhXrVq1aJ+zpAhQ9znsJAHXANcA1wDXANcA8lxDaxZsyZGtQ3qBfE+lyzkAdcA1wDXANeAFVC9IKFbgCjaU6xYMdu4cWPIev1eo0aNiO/R+txsL4MGDXIDrfq2bdtmdevWtdWrV1vFihUP+TiQO4ripaWl2Zo1a6xChQpkXwEj/+OHvI8v8j858l9PeHbs2GG1atXKt7RQL0gs/G2S/4UV1z75X5il51O9IKEDICVLlrSWLVva9OnT3UwuouaP+r1///4R39O6dWv3+q233hpYN23aNLc+mlKlSrklnIIf3IDHj/Ke/Cf/CyOuffK/MMvJ9Z/fDyeoFyQmvhvJ/8KKa5/8L8wqxLhekNABEFHLjF69etlJJ51kp5xyij377LO2a9cuNyuM9OzZ04488kgbOnSo+/2WW26x9u3b29NPP23nnXeejRs3zo3l8corr8T5SAAAAAAAQLwkfADkkksusT/++MPuu+8+N5BpixYtbMqUKYGBTtVNRQNL+dq0aWNjx461wYMH2z333GPHHnusTZo0yZo0aRLHowAAAAAAAPGU8AEQUXeXaF1eZs6cmWld9+7d3XIoTV+HDBkSsVsM8h/5H1/kP3lfWHHtk/9cG4mJv03yv7Di2if/C7NS+XRPXkQjocZ0jwAAAAAAAAnmf31HAAAAAAAAUhQBEAAAAAAAkPIIgAAAAAAAgJRXaAMgw4cPt3r16lnp0qWtVatWNm/evCy3nzBhgjVq1Mht37RpU/v4448LLK2FPf9HjRpl7dq1s8qVK7ulY8eO2Z4vxC7/g2la6SJFitgFF1xAFhdQ3m/bts369etnNWvWdINANWjQgO+fAsx/Tb3esGFDK1OmjKWlpdltt91me/bsOZQkFFqzZ8+2rl27Wq1atdz3iGZoy44GOj/xxBPdtX/MMcfYmDFj8i191Avii3pB8uR/MOoFBZ/31Atii3pBIawXeIXQuHHjvJIlS3qjR4/2fvjhB69Pnz5epUqVvI0bN0bc/ssvv/SKFSvmPfHEE96PP/7oDR482CtRooT3/fffF3jaC2P+X3bZZd7w4cO9b7/91vvpp5+8q666yqtYsaK3du3aAk97Ycx/38qVK70jjzzSa9eundetW7cCS29hzvu9e/d6J510knfuued6X3zxhTsHM2fO9BYtWlTgaS+M+f/22297pUqVcj+V95988olXs2ZN77bbbivwtKeCjz/+2PvnP//pTZw4UYOve++//36W2//6669e2bJlvQEDBriy94UXXnBl8ZQpU2KeNuoF8UW9ILny30e9oODznnpBbFEvKJz1gkIZADnllFO8fv36BX7PyMjwatWq5Q0dOjTi9hdffLF33nnnhaxr1aqVd/311+d7WlNRbvM/3IEDB7zy5ct7b7zxRj6mMnXlJf+V523atPFeffVVr1evXgRACijvR4wY4dWvX9/bt29fXj8Sh5D/2vass84KWadC97TTTiNfD1FOKjp33nmn17hx45B1l1xyidepU6eY5z/1gviiXpB8+U+9ID55T70gtqgXFM56QaHrArNv3z5bsGCB60bhK1q0qPv9q6++ivgerQ/eXjp16hR1e8Q2/8Pt3r3b9u/fb4cffjhZXUD5/+CDD1q1atXs2muvJc8LMO8//PBDa926tesCU716dWvSpIk9+uijlpGRwXkogPxv06aNe4/fHPnXX3913Y/OPfdc8r8AFFTZS70gvqgXJGf+Uy+IT95TL4gd6gWFt15Q3AqZzZs3u5sH3UwE0+9Lly6N+J4NGzZE3F7rkf/5H+6uu+5yfcXC/wCQP/n/xRdf2GuvvWaLFi0iiws473XD/dlnn9nll1/ubryXL19uN954owsADhkyhPORz/l/2WWXufe1bdtWrSXtwIEDdsMNN9g999xD3heAaGVvenq6/fXXX25clligXhBf1AuSL/+pF8Qv76kXxA71gsJbLyh0LUCQ3B577DE34Nb777/vBotC/tqxY4ddeeWVbiDaKlWqkN0F7ODBg67lzSuvvGItW7a0Sy65xP75z3/ayJEjORcFQANtqcXNSy+9ZAsXLrSJEyfa5MmT7aGHHiL/gQRBvaBgUS+IL+oF8UW9IDUUuhYguokrVqyYbdy4MWS9fq9Ro0bE92h9brZHbPPf99RTT7mKzqeffmrNmjUjmwsg/1esWGGrVq1yIzQHF75SvHhxW7ZsmR199NGci3zIe9HMLyVKlHDv8x133HEuAq6mmyVLliTv8zH/7733XhcA7N27t/tdM4Dt2rXLrrvuOheIUlNl5J9oZW+FChVi1vpDqBfEF/WC5Mp/6gXxy3uhXhDf/KdekBr1gkJXe9MNg56kTp8+PeSGTr+rr30kWh+8vUybNi3q9oht/ssTTzzhnrpOmTLFTjrpJLK4gPJfUz9///33rvuLv5x//vl25plnuv9rWlDkT97Laaed5rq9+EEn+fnnn10FiOBH/ue/xhsKD3L4waj/G68L+amgyl7qBfFFvSC58p96QfzyXqgXxDf/qRekSL3AK6RTHmlqwzFjxrgpdK677jo35dSGDRvc61deeaV39913h0yDW7x4ce+pp55y07AOGTKEaXALMP8fe+wxN0XYe++9561fvz6w7Nix41CSUWjlNv/DMQtMweX96tWr3YxH/fv395YtW+Z99NFHXrVq1byHH374EFJReOU2//Vdr/x/55133NRrU6dO9Y4++mg3MxhyT9/Zms5ci6ofw4YNc///7bff3OvKe52D8OnuBg4c6MpeTYeen9PgUi+IH+oF8UW9IHnynnpBfPOfekFq1AsKZQBENG9wnTp13I21pkCaO3du4LX27du7m7xg7777rtegQQO3vabfmTx5chxSXTjzv27duu6PInzRlxDyP//DEQAp2LyfM2eOm3ZbBbSmxH3kkUfc9IPI//zfv3+/d//997ugR+nSpb20tDTvxhtv9LZu3Ur258GMGTMifpf7ea6fOgfh72nRooU7X7r+X3/99XzLe+oF8UW9IHnyPxz1goLNe+oFsUW9oPDVC4ron9g2TgEAAAAAAEgshW4MEAAAAAAAUPgQAAEAAAAAACmPAAgAAAAAAEh5BEAAAAAAAEDKIwACAAAAAABSHgEQAAAAAACQ8giAAAAAAACAlEcABAAAAAAApDwCIABQgF555RVLS0uzokWL2rPPPmv333+/tWjRgnMAAEAhRL0AKFhFPM/zCvgzASQg3YhPmjTJFi1aFJP9XXXVVbZt2za3T/yf9PR0q1Klig0bNswuuugiq1ixoh08eND27t1rRxxxBPkGAEgY1AvyH/UCoOAVj8NnAsgHGRkZVqRIEdeyID/t37/fSpQoka+fkWoUZ9b5Wb16tcu/8847z2rWrBl4/bDDDrNksW/fPitZsmS8kwEAyAb1gsRFvQCII7UAAVDw2rdv7/Xr188tFSpU8I444ghv8ODB3sGDB93re/bs8W6//XavVq1aXtmyZb1TTjnFmzFjRuD9r7/+ulexYkXvgw8+8I477jivWLFi3sqVK7P8TL3/5JNPdvvTe9u0aeOtWrXK7UtfB8GL1on+/9JLL3ldu3Z17xsyZIh34MAB75prrvHq1avnlS5d2mvQoIH37LPPBj5H24Tvz0/76tWrve7du7vPr1y5snf++eeHpHv//v3eTTfd5F4//PDDvTvvvNPr2bOn161bN/f6G2+84dYrf4Lp9SuuuCJHea/jqV+/vleiRAmX9jfffDPwWo8ePbyLL744ZPt9+/a586PPloyMDO/RRx8NHH+zZs28CRMmhOSzjvnjjz/2TjzxRPc5kfJYx628at68ebb5Fs3evXvdNVSjRg2vVKlSXp06dVzafFu3bvWuu+46r1q1au71xo0be//5z38Cr7/33nve8ccf75UsWdKrW7eu99RTT4XsX+sefPBB78orr/TKly/v9erVy63//PPPvbZt27rjr127tjtnO3fuzFH+AwAyo15AvYB6AZD/CIAAcazoHHbYYd4tt9ziLV261HvrrbdcgOGVV15xr/fu3dsFKGbPnu0tX77ce/LJJ90N7M8//+xe1w21bqy1zZdffun2sWvXrqifp8CCggp33HGH29+PP/7ojRkzxvvtt9+83bt3u2CLbo7Xr1/vFq0T3YTr5nn06NHeihUr3PYKCNx3333eN9984/3666+BtI8fP969Z8eOHS6I0Llz58D+dKOu9ylYo+DJ4sWLXRouu+wyr2HDhu51efjhh12AY+LEid5PP/3k3XDDDS5A5AdAlC4dx7vvvhs4to0bN3rFixf3Pvvss2zzXftVvg0fPtxbtmyZ9/TTT7vgkf/ejz76yCtTpow7Bp8CBlqXnp4eSGOjRo28KVOmuDzRudC5mTlzZkgARIGRqVOnuvxeu3at9+mnn7r18+bNc3miQFJwACRavmVF10VaWpq7ThTMUmBi7NixgUDNqaee6s6r0qG06lgUmJH58+d7RYsWdQEO5YWOQ8fpB7/8AIjyX4ERHYe/lCtXznvmmWfc9ajr74QTTvCuuuqqbPMfABAZ9QLqBdQLgPxHAASIY0VHwQC/xYfcddddbp2CDLop//3330Pe06FDB2/QoEHu/36LgkWLFuXo87Zs2eK292/SwwXfiAfTe2699dZs969WCBdddFHgd7UU8IMWvn/9618u2BF8zLrB1033J5984n6vXr26u6n3qTKgVg3B++rbt6/XpUuXwO8KYqhFR/B+o1HAqE+fPiHr1CLl3HPPDQSKqlSpkqlVyCWXXOL+r5YnCvbMmTMnZB/XXnut2y44ADJp0qSQbb799ttAy49o+R4p37KilhdnnXVWxGNXnirAoeBGJAo+nX322SHrBg4c6FqEBAdALrjggkzHqlYlwRR40Wf99ddfOU47AOB/qBf8H+oF1AuA/MQsMEAcnXrqqW7cDl/r1q3tl19+se+//9713W3QoIEbH8JfZs2aZStWrAhsr7EYmjVrlqPPOvzww93ApJ06dbKuXbvac889Z+vXr8/Re0866aRM64YPH24tW7a0qlWrurRpFHONcZGV7777zpYvX27ly5cPHJPStWfPHndc27dvt40bN9opp5wSeE+xYsXc5wTr06ePTZ061X7//Xf3+5gxY9yxBedlND/99JOddtppIev0u9ZL8eLF7eKLL7a3337b/b5r1y774IMP7PLLL3e/K/27d++2s88+O+TcvPnmmyHnJlq+xZqOWwPXNmzY0G6++WaXLz6tr127truOcpMXugZ1/UU7Dp1H5Xnw8eu60oCuK1eujPkxAkBhQb2AesGhol4AZI1BUIEEtHPnTnfjv2DBAvczWPCAmWXKlMnRTb/v9ddfdzfJU6ZMsfHjx9vgwYNt2rRprsKVlXLlyoX8Pm7cOLvjjjvs6aefdkEbBTSefPJJ+/rrr7M9LgUz/OBCMAVScuqEE06w5s2bu6DDOeecYz/88INNnjzZYkXBjvbt29umTZtc/iifO3fuHDgG0ecdeeSRIe8rVapUlvmWH0488UQXdPjvf/9rn376qQvedOzY0d577z2X7lgIPw7lwfXXX++upXB16tSJyWcCAEK/d6kXREe94H+oFwBZIwACxFF4wGDu3Ll27LHHuoJcT+B1A96uXbuYfqb2rWXQoEEueDF27FgXAFFrkuCn/ln58ssvrU2bNnbjjTcG1oW3foi0PxXKCrxUq1bNKlSoEHHf1atXt2+++cZOP/1097v2sXDhQmvRokXIdr1797Znn33WtQLRDX9aWlqO0n7ccce59Pfq1SvkeI4//vjA7zo27U9pVWChe/fugZlvtJ0CHWrtoiBJrOXmPPiUl5dccolb/vGPf7hgzZ9//ulaB61du9Z+/vnniK1A/LwIpt+1bXjgLfw8/vjjj3bMMcfkKp0AgKxRL8iMegH1AiCW6AIDxJFuogcMGGDLli2zd955x1544QW75ZZb3A2oWiH07NnTJk6c6J7wz5s3z4YOHZrnlg7ah4IeX331lf3222+uq4S6OugmWOrVq+e2UbeJzZs32969e6PuS0Ga+fPn2yeffOJuru+9914XtAim/S1evNgdm/an6V91TFWqVLFu3brZ559/7j5v5syZriWBbtTlpptucsepbid6r/Jj69atmVq6XHbZZe49o0aNsmuuuSbH+TBw4EDXfWPEiBHu+IcNG+byWC1awvc/cuRI1wLE7/4iau2ibW+77TZ74403XOBHARqdO/1+qCLlW1aUfl07S5cudediwoQJVqNGDatUqZIL0CiQdNFFF7nj8FuKqAWQ3H777TZ9+nR76KGH3HuV/hdffDFTXoS76667bM6cOda/f393vSgfdb70OwAg76gXUC8IR70AiLF8HWEEQJaDnd14442BWU40Jew999wTGMzSn2lFU61q1pKaNWt6F154oZs9JXga3JzasGGDG8xS+/GnPNX+NVOIP7inBjGtVKlSpmlw33///ZB9aVvN+KHP1/YalPTuu+8OGcxz06ZNboBNzXQTPJ2rZjbRtLYaaFQzp2jwUg1Kun379sAgpP379w/kiQaG1SCll156aaZj0tSskabEPZRpcH2aoUbpVj6FDzCq3zXtrwZ01T6qVq3qderUyZs1a1bIIKiagja3g6BGy7doNGtQixYt3KwsyjMNlLtw4cKQwW+vvvpqN42vpqxt0qSJm+kmfBpcHYcGmw0egFZ0/JrtJZxmsvHTqc/WjDePPPJIlmkFAERHvYB6gVAvAPJXEf0T66AKgOydccYZrluHunEgOg2sqVYqGttCLRWCdejQwRo3bmzPP/88WQgASGrUC3KGegGAQ8EYIAASit89R9031A1HXTLUdUNdUnzqEqOuM1peeumluKYXAADkH+oFAGKJMUCAFBI8LWn4ojE3kkHRokXdGB0nn3yym5JVUwJrdhN/rBLRIK6a5u3xxx93078GU4uQaHkQafaZRPfoo49GPZ4uXbrEO3kAgARGvYB6AYBQdIEBUsjy5cujvqYpW2M1LWqiPymKNnCoRpLXIKbJRLO5aIlE5zN8Kl4AAHzUC6gXAAhFAAQAAAAAAKQ8usAAAAAAAICURwAEAAAAAACkPAIgAAAAAAAg5REAAQAAAAAAKY8ACAAAAAAASHkEQAAAAAAAQMojAAIAAAAAAFIeARAAAAAAAGCp7v8BOMZ9JBf7JccAAAAASUVORK5CYII=", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "Headline comparison\n", "======================================================================\n", " honest designed selection-biased\n", " family size 12 30\n", " aggregate PBO (classical) 0.658 0.360\n", " median overfit_score (FLOW) 0.500 0.772\n", " max overfit_score 0.680 0.885\n", " # strategies \u22650.7 0 29\n", " # strategies \u22650.9 0 0\n" ] } ], "source": [ "import matplotlib\n", "import matplotlib.pyplot as plt\n", "\n", "honest_vals = list(honest_scores.values())\n", "selected_vals = list(selected_scores.values())\n", "\n", "fig, axes = plt.subplots(1, 2, figsize=(11, 4), sharey=True)\n", "\n", "axes[0].hist(honest_vals, bins=np.arange(0, 1.05, 0.1),\n", " color='#2C7BB6', edgecolor='black', alpha=0.85)\n", "axes[0].axvline(np.median(honest_vals), color='black', ls='-', lw=1,\n", " label=f'median={np.median(honest_vals):.2f}')\n", "axes[0].set_title(f'Honest designed family (n={len(honest_vals)})\\n'\n", " f'max={max(honest_vals):.2f}, # \u22650.7={sum(1 for v in honest_vals if v>=0.7)}')\n", "axes[0].set_xlabel('per_strategy_overfit_score')\n", "axes[0].set_ylabel('count')\n", "axes[0].set_xlim(0, 1.02)\n", "axes[0].legend()\n", "\n", "axes[1].hist(selected_vals, bins=np.arange(0, 1.05, 0.1),\n", " color='#D7301F', edgecolor='black', alpha=0.85)\n", "axes[1].axvline(np.median(selected_vals), color='black', ls='-', lw=1,\n", " label=f'median={np.median(selected_vals):.2f}')\n", "axes[1].set_title(f'Selection-biased lucky family (n={len(selected_vals)})\\n'\n", " f'max={max(selected_vals):.2f}, # \u22650.7={sum(1 for v in selected_vals if v>=0.7)}')\n", "axes[1].set_xlabel('per_strategy_overfit_score')\n", "axes[1].set_xlim(0, 1.02)\n", "axes[1].legend()\n", "plt.tight_layout()\n", "plt.show()\n", "\n", "print()\n", "print('Headline comparison')\n", "print('=' * 70)\n", "print(f' honest designed selection-biased')\n", "print(f' family size {len(honest_vals):>16d} {len(selected_vals):>16d}')\n", "print(f' aggregate PBO (classical) {honest_report.pbo:>16.3f} {selected_report.pbo:>16.3f}')\n", "print(f' median overfit_score (FLOW) {np.median(honest_vals):>16.3f} {np.median(selected_vals):>16.3f}')\n", "print(f' max overfit_score {max(honest_vals):>16.3f} {max(selected_vals):>16.3f}')\n", "print(f' # strategies \u22650.7 {sum(1 for v in honest_vals if v>=0.7):>16d} {sum(1 for v in selected_vals if v>=0.7):>16d}')\n", "print(f' # strategies \u22650.9 {sum(1 for v in honest_vals if v>=0.9):>16d} {sum(1 for v in selected_vals if v>=0.9):>16d}')\n" ] }, { "cell_type": "markdown", "id": "aae3abea", "metadata": {}, "source": [ "## Section 8 \u2014 Verdict + falsification\n", "\n", "**What this notebook would have looked like if FLOW were NOT catching\n", "selection bias:** the median, max, and tail-count of the lucky-30\n", "distribution would look identical to the honest-12 distribution. The\n", "overfit_score would be a uniform-around-0.5 noise, useless for triage.\n", "\n", "**What FLOW catches.** The lucky-30 strategies have *zero* underlying\n", "skill (random sign sequences) but their training Sharpe is positive by\n", "selection. FLOW's null distribution \u2014 generated from synth alternative\n", "histories that share training-period statistics \u2014 sits near zero for\n", "each of these strategies, because under FLOW's data-generating-process\n", "model, a random-position strategy has expected Sharpe \u2248 0. The real\n", "(selected, inflated) Sharpe lands in the upper tail of that null \u2192\n", "score elevates.\n", "\n", "**Practical takeaway.** After running any parameter search:\n", "1. Read each top candidate's `per_strategy_overfit_score`.\n", "2. **Score \u2265 0.7** \u2192 real Sharpe lives in the upper half of FLOW's\n", " null distribution \u2014 the strategy may have been selected for luck.\n", " Investigate before deploying.\n", "3. **Score \u2248 0.5** \u2192 real Sharpe matches FLOW's null expectation. No\n", " selection-bias flag. (Doesn't mean the strategy works, just that\n", " its training Sharpe is consistent with what FLOW says it should be.)\n", "4. **Score \u2264 0.3** \u2192 real Sharpe is below FLOW's null \u2014 the strategy\n", " underperformed. Could be hidden friction or implementation bug.\n", "\n", "**The connection to deflated Sharpe.** This is essentially the\n", "deflated-Sharpe-ratio logic from Bailey & L\u00f3pez de Prado, but computed\n", "non-parametrically using FLOW's data-driven null instead of a Gaussian\n", "extreme-value formula. No need to specify `n_trials` \u2014 the diagnostic\n", "is per-strategy." ] }, { "cell_type": "markdown", "id": "c883e080", "metadata": {}, "source": [ "## Try this on your own data" ] }, { "cell_type": "markdown", "id": "a0b99c19", "metadata": {}, "source": [ "## Try this on YOUR data\n", "\n", "```python\n", "# Try this on YOUR data ----------------------------------------------------\n", "# Same workflow on your own panel and your own search:\n", "#\n", "# your_panel = pd.read_parquet('your_universe.parquet')\n", "# your_train = your_panel.loc['2015':'2022']\n", "#\n", "# # 1. Fit FLOW\n", "# handle = sf.fit_async(your_train, features=list(your_panel.columns),\n", "# data_types={c: 'price' for c in your_panel.columns},\n", "# horizon=252, seed=42)\n", "# result = sf.fetch_result(handle, poll_timeout_s=2400)\n", "#\n", "# # 2. Run your real parameter search\n", "# your_pool = {f'cand_{i}': your_strategy_fn(params_i) for i, params_i in enumerate(your_grid)}\n", "#\n", "# # 3. Selection step \u2014 top K by in-sample metric\n", "# train_metric = {name: fn(your_train)['sharpe'] for name, fn in your_pool.items()}\n", "# top_names = [n for n, _ in sorted(train_metric.items(), key=lambda x: -x[1])[:30]]\n", "# your_selected = {n: your_pool[n] for n in top_names}\n", "#\n", "# # 4. FLOW-aware overfit detection\n", "# report = sf.evaluate_family(your_selected, your_train,\n", "# model_id=result.model_id, n_paths=200,\n", "# primary_metric='sharpe')\n", "# for name, score in sorted(report.per_strategy_overfit_score.items(),\n", "# key=lambda x: -x[1]):\n", "# print(f'{name}: overfit_score = {score:.2f}')\n", "#\n", "# Heads up:\n", "# \u2022 Use horizon \u2265 252 for the FLOW fit to keep synth-Sharpe variance\n", "# comparable to your real-Sharpe variance.\n", "# \u2022 Search pool size matters \u2014 selecting top 30 from 500 gives a\n", "# stronger selection-bias signal than top 30 from 50.\n", "# \u2022 For very large selected sets (>50), pass pbo_cscv_splits=8 to\n", "# roughly halve the CSCV-PBO compute cost.\n", "\n", "```" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.12" } }, "nbformat": 4, "nbformat_minor": 5 }