{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"id": "47de02e1",
"metadata": {},
"source": [
"# Running Tune experiments with AxSearch\n",
"In this tutorial we introduce Ax, while running a simple Ray Tune experiment. Tune’s Search Algorithms integrate with Ax and, as a result, allow you to seamlessly scale up a Ax optimization process - without sacrificing performance.\n",
"\n",
"Ax is a platform for optimizing any kind of experiment, including machine learning experiments, A/B tests, and simulations. Ax can optimize discrete configurations (e.g., variants of an A/B test) using multi-armed bandit optimization, and continuous/ordered configurations (e.g. float/int parameters) using Bayesian optimization. Results of A/B tests and simulations with reinforcement learning agents often exhibit high amounts of noise. Ax supports state-of-the-art algorithms which work better than traditional Bayesian optimization in high-noise settings. Ax also supports multi-objective and constrained optimization which are common to real-world problems (e.g. improving load time without increasing data use). Ax belongs to the domain of \"derivative-free\" and \"black-box\" optimization.\n",
"\n",
"In this example we minimize a simple objective to briefly demonstrate the usage of AxSearch with Ray Tune via `AxSearch`. It's useful to keep in mind that despite the emphasis on machine learning experiments, Ray Tune optimizes any implicit or explicit objective. Here we assume `ax-platform==0.2.4` library is installed withe python version >= 3.7. To learn more, please refer to the [Ax website](https://ax.dev/)."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "297d8b18",
"metadata": {
"tags": [
"remove-cell"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: ax-platform==0.2.4 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (0.2.4)\n",
"Requirement already satisfied: botorch==0.6.2 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from ax-platform==0.2.4) (0.6.2)\n",
"Requirement already satisfied: jinja2 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from ax-platform==0.2.4) (3.0.3)\n",
"Requirement already satisfied: pandas in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from ax-platform==0.2.4) (1.3.5)\n",
"Requirement already satisfied: scipy in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from ax-platform==0.2.4) (1.4.1)\n",
"Requirement already satisfied: plotly in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from ax-platform==0.2.4) (5.6.0)\n",
"Requirement already satisfied: scikit-learn in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from ax-platform==0.2.4) (0.24.2)\n",
"Requirement already satisfied: typeguard in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from ax-platform==0.2.4) (2.13.3)\n",
"Requirement already satisfied: gpytorch>=1.6 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from botorch==0.6.2->ax-platform==0.2.4) (1.6.0)\n",
"Requirement already satisfied: torch>=1.9 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from botorch==0.6.2->ax-platform==0.2.4) (1.9.0)\n",
"Requirement already satisfied: multipledispatch in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from botorch==0.6.2->ax-platform==0.2.4) (0.6.0)\n",
"Requirement already satisfied: MarkupSafe>=2.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from jinja2->ax-platform==0.2.4) (2.0.1)\n",
"Requirement already satisfied: pytz>=2017.3 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from pandas->ax-platform==0.2.4) (2022.1)\n",
"Requirement already satisfied: numpy>=1.17.3 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from pandas->ax-platform==0.2.4) (1.21.6)\n",
"Requirement already satisfied: python-dateutil>=2.7.3 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from pandas->ax-platform==0.2.4) (2.8.2)\n",
"Requirement already satisfied: tenacity>=6.2.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from plotly->ax-platform==0.2.4) (8.0.1)\n",
"Requirement already satisfied: six in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from plotly->ax-platform==0.2.4) (1.16.0)\n",
"Requirement already satisfied: joblib>=0.11 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from scikit-learn->ax-platform==0.2.4) (1.1.0)\n",
"Requirement already satisfied: threadpoolctl>=2.0.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from scikit-learn->ax-platform==0.2.4) (3.0.0)\n",
"Requirement already satisfied: typing-extensions in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from torch>=1.9->botorch==0.6.2->ax-platform==0.2.4) (4.1.1)\n",
"\u001b[33mWARNING: There was an error checking the latest version of pip.\u001b[0m\u001b[33m\n",
"\u001b[0m"
]
}
],
"source": [
"# !pip install ray[tune]\n",
"!pip install ax-platform==0.2.4"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "59b1e0d1",
"metadata": {},
"source": [
"Click below to see all the imports we need for this example.\n",
"You can also launch directly into a Binder instance to run this notebook yourself.\n",
"Just click on the rocket symbol at the top of the navigation."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "cbae6dbe",
"metadata": {
"tags": [
"hide-input"
]
},
"outputs": [],
"source": [
"import numpy as np\n",
"import time\n",
"\n",
"import ray\n",
"from ray import train, tune\n",
"from ray.tune.search.ax import AxSearch"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "7b2b6af7",
"metadata": {},
"source": [
"Let's start by defining a classic benchmark for global optimization.\n",
"The form here is explicit for demonstration, yet it is typically a black-box.\n",
"We artificially sleep for a bit (`0.02` seconds) to simulate a long-running ML experiment.\n",
"This setup assumes that we're running multiple `step`s of an experiment and try to tune 6-dimensions of the `x` hyperparameter."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "0f7fbe0f",
"metadata": {},
"outputs": [],
"source": [
"def landscape(x):\n",
" \"\"\"\n",
" Hartmann 6D function containing 6 local minima.\n",
" It is a classic benchmark for developing global optimization algorithms.\n",
" \"\"\"\n",
" alpha = np.array([1.0, 1.2, 3.0, 3.2])\n",
" A = np.array(\n",
" [\n",
" [10, 3, 17, 3.5, 1.7, 8],\n",
" [0.05, 10, 17, 0.1, 8, 14],\n",
" [3, 3.5, 1.7, 10, 17, 8],\n",
" [17, 8, 0.05, 10, 0.1, 14],\n",
" ]\n",
" )\n",
" P = 10 ** (-4) * np.array(\n",
" [\n",
" [1312, 1696, 5569, 124, 8283, 5886],\n",
" [2329, 4135, 8307, 3736, 1004, 9991],\n",
" [2348, 1451, 3522, 2883, 3047, 6650],\n",
" [4047, 8828, 8732, 5743, 1091, 381],\n",
" ]\n",
" )\n",
" y = 0.0\n",
" for j, alpha_j in enumerate(alpha):\n",
" t = 0\n",
" for k in range(6):\n",
" t += A[j, k] * ((x[k] - P[j, k]) ** 2)\n",
" y -= alpha_j * np.exp(-t)\n",
" return y"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "0b1ae9df",
"metadata": {},
"source": [
"Next, our `objective` function takes a Tune `config`, evaluates the `landscape` of our experiment in a training loop,\n",
"and uses `train.report` to report the `landscape` back to Tune."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "8c3f252e",
"metadata": {},
"outputs": [],
"source": [
"def objective(config):\n",
" for i in range(config[\"iterations\"]):\n",
" x = np.array([config.get(\"x{}\".format(i + 1)) for i in range(6)])\n",
" train.report(\n",
" {\"timesteps_total\": i, \"landscape\": landscape(x), \"l2norm\": np.sqrt((x ** 2).sum())}\n",
" )\n",
" time.sleep(0.02)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "d9982d95",
"metadata": {},
"source": [
"Next we define a search space. The critical assumption is that the optimal hyperparamters live within this space. Yet, if the space is very large, then those hyperparamters may be difficult to find in a short amount of time."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "30f75f5a",
"metadata": {},
"outputs": [],
"source": [
"search_space = {\n",
" \"iterations\":100,\n",
" \"x1\": tune.uniform(0.0, 1.0),\n",
" \"x2\": tune.uniform(0.0, 1.0),\n",
" \"x3\": tune.uniform(0.0, 1.0),\n",
" \"x4\": tune.uniform(0.0, 1.0),\n",
" \"x5\": tune.uniform(0.0, 1.0),\n",
" \"x6\": tune.uniform(0.0, 1.0)\n",
"}"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "106d8578",
"metadata": {
"tags": [
"remove-cell"
]
},
"outputs": [],
"source": [
"ray.init(configure_logging=False)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "932f74e6",
"metadata": {},
"source": [
"Now we define the search algorithm from `AxSearch`. If you want to constrain your parameters or even the space of outcomes, that can be easily done by passing the argumentsas below."
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "34dd5c95",
"metadata": {},
"outputs": [],
"source": [
"algo = AxSearch(\n",
" parameter_constraints=[\"x1 + x2 <= 2.0\"],\n",
" outcome_constraints=[\"l2norm <= 1.25\"],\n",
")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "f6d18a99",
"metadata": {},
"source": [
"We also use `ConcurrencyLimiter` to constrain to 4 concurrent trials. "
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "dcd905ef",
"metadata": {},
"outputs": [],
"source": [
"algo = tune.search.ConcurrencyLimiter(algo, max_concurrent=4)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "10fd5427",
"metadata": {},
"source": [
"The number of samples is the number of hyperparameter combinations that will be tried out. This Tune run is set to `1000` samples.\n",
"You can decrease this if it takes too long on your machine, or you can set a time limit easily through `stop` argument in the `train.RunConfig()` as we will show here."
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "c53349a5",
"metadata": {},
"outputs": [],
"source": [
"num_samples = 100\n",
"stop_timesteps = 200"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "6c661045",
"metadata": {
"tags": [
"remove-cell"
]
},
"outputs": [],
"source": [
"# Reducing samples for smoke tests\n",
"num_samples = 10"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "91076c5a",
"metadata": {},
"source": [
"Finally, we run the experiment to find the global minimum of the provided landscape (which contains 5 false minima). The argument to metric, `\"landscape\"`, is provided via the `objective` function's `session.report`. The experiment `\"min\"`imizes the \"mean_loss\" of the `landscape` by searching within `search_space` via `algo`, `num_samples` times or when `\"timesteps_total\": stop_timesteps`. This previous sentence is fully characterizes the search problem we aim to solve. With this in mind, notice how efficient it is to execute `tuner.fit()`."
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "2f519d63",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"[INFO 07-22 15:04:18] ax.service.ax_client: Starting optimization with verbose logging. To disable logging, set the `verbose_logging` argument to `False`. Note that float values in the logs are rounded to 6 decimal points.\n",
"[INFO 07-22 15:04:18] ax.service.utils.instantiation: Created search space: SearchSpace(parameters=[FixedParameter(name='iterations', parameter_type=INT, value=100), RangeParameter(name='x1', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x2', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x3', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x4', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x5', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x6', parameter_type=FLOAT, range=[0.0, 1.0])], parameter_constraints=[ParameterConstraint(1.0*x1 + 1.0*x2 <= 2.0)]).\n",
"[INFO 07-22 15:04:18] ax.modelbridge.dispatch_utils: Using Bayesian optimization since there are more ordered parameters than there are categories for the unordered categorical parameters.\n",
"[INFO 07-22 15:04:18] ax.modelbridge.dispatch_utils: Using Bayesian Optimization generation strategy: GenerationStrategy(name='Sobol+GPEI', steps=[Sobol for 12 trials, GPEI for subsequent trials]). Iterations after 12 will take longer to generate due to model-fitting.\n",
"Detected sequential enforcement. Be sure to use a ConcurrencyLimiter.\n"
]
},
{
"data": {
"text/html": [
"== Status ==<br>Current time: 2022-07-22 15:04:35 (running for 00:00:16.56)<br>Memory usage on this node: 9.9/16.0 GiB<br>Using FIFO scheduling algorithm.<br>Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.13 GiB heap, 0.0/2.0 GiB objects<br>Current best trial: 34b7abda with landscape=-1.6624439263544026 and parameters={'iterations': 100, 'x1': 0.26526361983269453, 'x2': 0.9248840995132923, 'x3': 0.15171580761671066, 'x4': 0.43602637108415365, 'x5': 0.8573104059323668, 'x6': 0.08981018699705601}<br>Result logdir: /Users/kai/ray_results/ax<br>Number of trials: 10/10 (10 TERMINATED)<br><table>\n",
"<thead>\n",
"<tr><th>Trial name </th><th>status </th><th>loc </th><th style=\"text-align: right;\"> iterations</th><th style=\"text-align: right;\"> x1</th><th style=\"text-align: right;\"> x2</th><th style=\"text-align: right;\"> x3</th><th style=\"text-align: right;\"> x4</th><th style=\"text-align: right;\"> x5</th><th style=\"text-align: right;\"> x6</th><th style=\"text-align: right;\"> iter</th><th style=\"text-align: right;\"> total time (s)</th><th style=\"text-align: right;\"> ts</th><th style=\"text-align: right;\"> landscape</th><th style=\"text-align: right;\"> l2norm</th></tr>\n",
"</thead>\n",
"<tbody>\n",
"<tr><td>objective_2dfbe86a</td><td>TERMINATED</td><td>127.0.0.1:44721</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\">0.0558336</td><td style=\"text-align: right;\">0.0896192</td><td style=\"text-align: right;\">0.958956 </td><td style=\"text-align: right;\">0.234474 </td><td style=\"text-align: right;\">0.174516 </td><td style=\"text-align: right;\">0.970311 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 2.57372</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\">-0.805233 </td><td style=\"text-align: right;\"> 1.39917</td></tr>\n",
"<tr><td>objective_2fa776c0</td><td>TERMINATED</td><td>127.0.0.1:44726</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\">0.744772 </td><td style=\"text-align: right;\">0.754537 </td><td style=\"text-align: right;\">0.0950125</td><td style=\"text-align: right;\">0.273877 </td><td style=\"text-align: right;\">0.0966829</td><td style=\"text-align: right;\">0.368943 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 2.6361 </td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\">-0.11286 </td><td style=\"text-align: right;\"> 1.16341</td></tr>\n",
"<tr><td>objective_2fabaa1a</td><td>TERMINATED</td><td>127.0.0.1:44727</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\">0.405704 </td><td style=\"text-align: right;\">0.374626 </td><td style=\"text-align: right;\">0.935628 </td><td style=\"text-align: right;\">0.222185 </td><td style=\"text-align: right;\">0.787212 </td><td style=\"text-align: right;\">0.00812439</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 2.62393</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\">-0.11348 </td><td style=\"text-align: right;\"> 1.35995</td></tr>\n",
"<tr><td>objective_2faee7c0</td><td>TERMINATED</td><td>127.0.0.1:44728</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\">0.664728 </td><td style=\"text-align: right;\">0.207519 </td><td style=\"text-align: right;\">0.359514 </td><td style=\"text-align: right;\">0.704578 </td><td style=\"text-align: right;\">0.755882 </td><td style=\"text-align: right;\">0.812402 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 2.62069</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\">-0.0119837 </td><td style=\"text-align: right;\"> 1.53035</td></tr>\n",
"<tr><td>objective_313d3d3a</td><td>TERMINATED</td><td>127.0.0.1:44747</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\">0.0418746</td><td style=\"text-align: right;\">0.992783 </td><td style=\"text-align: right;\">0.906027 </td><td style=\"text-align: right;\">0.594429 </td><td style=\"text-align: right;\">0.825393 </td><td style=\"text-align: right;\">0.646362 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 3.16233</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\">-0.00677976</td><td style=\"text-align: right;\"> 1.80573</td></tr>\n",
"<tr><td>objective_32c9acd8</td><td>TERMINATED</td><td>127.0.0.1:44726</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\">0.126064 </td><td style=\"text-align: right;\">0.703408 </td><td style=\"text-align: right;\">0.344681 </td><td style=\"text-align: right;\">0.337363 </td><td style=\"text-align: right;\">0.401396 </td><td style=\"text-align: right;\">0.679202 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 3.12119</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\">-0.904622 </td><td style=\"text-align: right;\"> 1.16864</td></tr>\n",
"<tr><td>objective_32cf8ca2</td><td>TERMINATED</td><td>127.0.0.1:44756</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\">0.0910936</td><td style=\"text-align: right;\">0.304138 </td><td style=\"text-align: right;\">0.869848 </td><td style=\"text-align: right;\">0.405435 </td><td style=\"text-align: right;\">0.567922 </td><td style=\"text-align: right;\">0.228608 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 2.70791</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\">-0.146532 </td><td style=\"text-align: right;\"> 1.18178</td></tr>\n",
"<tr><td>objective_32d8dd20</td><td>TERMINATED</td><td>127.0.0.1:44758</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\">0.603178 </td><td style=\"text-align: right;\">0.409057 </td><td style=\"text-align: right;\">0.729056 </td><td style=\"text-align: right;\">0.0825984</td><td style=\"text-align: right;\">0.572948 </td><td style=\"text-align: right;\">0.508304 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 2.64158</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\">-0.247223 </td><td style=\"text-align: right;\"> 1.28691</td></tr>\n",
"<tr><td>objective_34adf04a</td><td>TERMINATED</td><td>127.0.0.1:44768</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\">0.454189 </td><td style=\"text-align: right;\">0.271772 </td><td style=\"text-align: right;\">0.530871 </td><td style=\"text-align: right;\">0.991841 </td><td style=\"text-align: right;\">0.691843 </td><td style=\"text-align: right;\">0.472366 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 2.70327</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\">-0.0132915 </td><td style=\"text-align: right;\"> 1.49917</td></tr>\n",
"<tr><td>objective_34b7abda</td><td>TERMINATED</td><td>127.0.0.1:44771</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\">0.265264 </td><td style=\"text-align: right;\">0.924884 </td><td style=\"text-align: right;\">0.151716 </td><td style=\"text-align: right;\">0.436026 </td><td style=\"text-align: right;\">0.85731 </td><td style=\"text-align: right;\">0.0898102 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 2.68521</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\">-1.66244 </td><td style=\"text-align: right;\"> 1.37185</td></tr>\n",
"</tbody>\n",
"</table><br><br>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"[INFO 07-22 15:04:19] ax.service.ax_client: Generated new trial 0 with parameters {'x1': 0.055834, 'x2': 0.089619, 'x3': 0.958956, 'x4': 0.234474, 'x5': 0.174516, 'x6': 0.970311, 'iterations': 100}.\n",
"[INFO 07-22 15:04:22] ax.service.ax_client: Generated new trial 1 with parameters {'x1': 0.744772, 'x2': 0.754537, 'x3': 0.095012, 'x4': 0.273877, 'x5': 0.096683, 'x6': 0.368943, 'iterations': 100}.\n",
"[INFO 07-22 15:04:22] ax.service.ax_client: Generated new trial 2 with parameters {'x1': 0.405704, 'x2': 0.374626, 'x3': 0.935628, 'x4': 0.222185, 'x5': 0.787212, 'x6': 0.008124, 'iterations': 100}.\n",
"[INFO 07-22 15:04:22] ax.service.ax_client: Generated new trial 3 with parameters {'x1': 0.664728, 'x2': 0.207519, 'x3': 0.359514, 'x4': 0.704578, 'x5': 0.755882, 'x6': 0.812402, 'iterations': 100}.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Result for objective_2dfbe86a:\n",
" date: 2022-07-22_15-04-22\n",
" done: false\n",
" experiment_id: 4ef8a12ac94a4f4fa483ec18e347967f\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 1\n",
" l2norm: 1.3991721132671366\n",
" landscape: -0.8052333562869153\n",
" node_ip: 127.0.0.1\n",
" pid: 44721\n",
" time_since_restore: 0.00022912025451660156\n",
" time_this_iter_s: 0.00022912025451660156\n",
" time_total_s: 0.00022912025451660156\n",
" timestamp: 1658498662\n",
" timesteps_since_restore: 0\n",
" timesteps_total: 0\n",
" training_iteration: 1\n",
" trial_id: 2dfbe86a\n",
" warmup_time: 0.0035619735717773438\n",
" \n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"[INFO 07-22 15:04:24] ax.service.ax_client: Completed trial 0 with data: {'landscape': (-0.805233, None), 'l2norm': (1.399172, None)}.\n",
"[INFO 07-22 15:04:24] ax.service.ax_client: Generated new trial 4 with parameters {'x1': 0.041875, 'x2': 0.992783, 'x3': 0.906027, 'x4': 0.594429, 'x5': 0.825393, 'x6': 0.646362, 'iterations': 100}.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Result for objective_2faee7c0:\n",
" date: 2022-07-22_15-04-24\n",
" done: false\n",
" experiment_id: 3699644e85ac439cb7c1a36ed0976307\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 1\n",
" l2norm: 1.530347488145437\n",
" landscape: -0.011983676977099367\n",
" node_ip: 127.0.0.1\n",
" pid: 44728\n",
" time_since_restore: 0.00022292137145996094\n",
" time_this_iter_s: 0.00022292137145996094\n",
" time_total_s: 0.00022292137145996094\n",
" timestamp: 1658498664\n",
" timesteps_since_restore: 0\n",
" timesteps_total: 0\n",
" training_iteration: 1\n",
" trial_id: 2faee7c0\n",
" warmup_time: 0.0027179718017578125\n",
" \n",
"Result for objective_2fa776c0:\n",
" date: 2022-07-22_15-04-24\n",
" done: false\n",
" experiment_id: c555bfed13ac43e5b8c8e9f6d4b9b2f7\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 1\n",
" l2norm: 1.1634068454629019\n",
" landscape: -0.11285961764770336\n",
" node_ip: 127.0.0.1\n",
" pid: 44726\n",
" time_since_restore: 0.000225067138671875\n",
" time_this_iter_s: 0.000225067138671875\n",
" time_total_s: 0.000225067138671875\n",
" timestamp: 1658498664\n",
" timesteps_since_restore: 0\n",
" timesteps_total: 0\n",
" training_iteration: 1\n",
" trial_id: 2fa776c0\n",
" warmup_time: 0.0026290416717529297\n",
" \n",
"Result for objective_2dfbe86a:\n",
" date: 2022-07-22_15-04-24\n",
" done: true\n",
" experiment_id: 4ef8a12ac94a4f4fa483ec18e347967f\n",
" experiment_tag: 1_iterations=100,x1=0.0558,x2=0.0896,x3=0.9590,x4=0.2345,x5=0.1745,x6=0.9703\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 100\n",
" l2norm: 1.3991721132671366\n",
" landscape: -0.8052333562869153\n",
" node_ip: 127.0.0.1\n",
" pid: 44721\n",
" time_since_restore: 2.573719024658203\n",
" time_this_iter_s: 0.0251619815826416\n",
" time_total_s: 2.573719024658203\n",
" timestamp: 1658498664\n",
" timesteps_since_restore: 0\n",
" timesteps_total: 99\n",
" training_iteration: 100\n",
" trial_id: 2dfbe86a\n",
" warmup_time: 0.0035619735717773438\n",
" \n",
"Result for objective_2fabaa1a:\n",
" date: 2022-07-22_15-04-24\n",
" done: false\n",
" experiment_id: eb9287e4fe5f44c7868dc943e2642312\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 1\n",
" l2norm: 1.3599537840291782\n",
" landscape: -0.11348012497414121\n",
" node_ip: 127.0.0.1\n",
" pid: 44727\n",
" time_since_restore: 0.00022077560424804688\n",
" time_this_iter_s: 0.00022077560424804688\n",
" time_total_s: 0.00022077560424804688\n",
" timestamp: 1658498664\n",
" timesteps_since_restore: 0\n",
" timesteps_total: 0\n",
" training_iteration: 1\n",
" trial_id: 2fabaa1a\n",
" warmup_time: 0.0025510787963867188\n",
" \n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"[INFO 07-22 15:04:27] ax.service.ax_client: Completed trial 3 with data: {'landscape': (-0.011984, None), 'l2norm': (1.530347, None)}.\n",
"[INFO 07-22 15:04:27] ax.service.ax_client: Generated new trial 5 with parameters {'x1': 0.126064, 'x2': 0.703408, 'x3': 0.344681, 'x4': 0.337363, 'x5': 0.401396, 'x6': 0.679202, 'iterations': 100}.\n",
"[INFO 07-22 15:04:27] ax.service.ax_client: Completed trial 1 with data: {'landscape': (-0.11286, None), 'l2norm': (1.163407, None)}.\n",
"[INFO 07-22 15:04:27] ax.service.ax_client: Generated new trial 6 with parameters {'x1': 0.091094, 'x2': 0.304138, 'x3': 0.869848, 'x4': 0.405435, 'x5': 0.567922, 'x6': 0.228608, 'iterations': 100}.\n",
"[INFO 07-22 15:04:27] ax.service.ax_client: Completed trial 2 with data: {'landscape': (-0.11348, None), 'l2norm': (1.359954, None)}.\n",
"[INFO 07-22 15:04:27] ax.service.ax_client: Generated new trial 7 with parameters {'x1': 0.603178, 'x2': 0.409057, 'x3': 0.729056, 'x4': 0.082598, 'x5': 0.572948, 'x6': 0.508304, 'iterations': 100}.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Result for objective_313d3d3a:\n",
" date: 2022-07-22_15-04-27\n",
" done: false\n",
" experiment_id: fa7afd557e154fbebe4f54d8eedb3573\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 1\n",
" l2norm: 1.805729990121368\n",
" landscape: -0.006779757704679272\n",
" node_ip: 127.0.0.1\n",
" pid: 44747\n",
" time_since_restore: 0.00021076202392578125\n",
" time_this_iter_s: 0.00021076202392578125\n",
" time_total_s: 0.00021076202392578125\n",
" timestamp: 1658498667\n",
" timesteps_since_restore: 0\n",
" timesteps_total: 0\n",
" training_iteration: 1\n",
" trial_id: 313d3d3a\n",
" warmup_time: 0.0029790401458740234\n",
" \n",
"Result for objective_2faee7c0:\n",
" date: 2022-07-22_15-04-27\n",
" done: true\n",
" experiment_id: 3699644e85ac439cb7c1a36ed0976307\n",
" experiment_tag: 4_iterations=100,x1=0.6647,x2=0.2075,x3=0.3595,x4=0.7046,x5=0.7559,x6=0.8124\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 100\n",
" l2norm: 1.530347488145437\n",
" landscape: -0.011983676977099367\n",
" node_ip: 127.0.0.1\n",
" pid: 44728\n",
" time_since_restore: 2.6206929683685303\n",
" time_this_iter_s: 0.027359962463378906\n",
" time_total_s: 2.6206929683685303\n",
" timestamp: 1658498667\n",
" timesteps_since_restore: 0\n",
" timesteps_total: 99\n",
" training_iteration: 100\n",
" trial_id: 2faee7c0\n",
" warmup_time: 0.0027179718017578125\n",
" \n",
"Result for objective_2fa776c0:\n",
" date: 2022-07-22_15-04-27\n",
" done: true\n",
" experiment_id: c555bfed13ac43e5b8c8e9f6d4b9b2f7\n",
" experiment_tag: 2_iterations=100,x1=0.7448,x2=0.7545,x3=0.0950,x4=0.2739,x5=0.0967,x6=0.3689\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 100\n",
" l2norm: 1.1634068454629019\n",
" landscape: -0.11285961764770336\n",
" node_ip: 127.0.0.1\n",
" pid: 44726\n",
" time_since_restore: 2.6361019611358643\n",
" time_this_iter_s: 0.0264589786529541\n",
" time_total_s: 2.6361019611358643\n",
" timestamp: 1658498667\n",
" timesteps_since_restore: 0\n",
" timesteps_total: 99\n",
" training_iteration: 100\n",
" trial_id: 2fa776c0\n",
" warmup_time: 0.0026290416717529297\n",
" \n",
"Result for objective_32c9acd8:\n",
" date: 2022-07-22_15-04-27\n",
" done: false\n",
" experiment_id: c555bfed13ac43e5b8c8e9f6d4b9b2f7\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 1\n",
" l2norm: 1.1686440476629836\n",
" landscape: -0.9046216637367911\n",
" node_ip: 127.0.0.1\n",
" pid: 44726\n",
" time_since_restore: 0.00020194053649902344\n",
" time_this_iter_s: 0.00020194053649902344\n",
" time_total_s: 0.00020194053649902344\n",
" timestamp: 1658498667\n",
" timesteps_since_restore: 0\n",
" timesteps_total: 0\n",
" training_iteration: 1\n",
" trial_id: 32c9acd8\n",
" warmup_time: 0.0026290416717529297\n",
" \n",
"Result for objective_2fabaa1a:\n",
" date: 2022-07-22_15-04-27\n",
" done: true\n",
" experiment_id: eb9287e4fe5f44c7868dc943e2642312\n",
" experiment_tag: 3_iterations=100,x1=0.4057,x2=0.3746,x3=0.9356,x4=0.2222,x5=0.7872,x6=0.0081\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 100\n",
" l2norm: 1.3599537840291782\n",
" landscape: -0.11348012497414121\n",
" node_ip: 127.0.0.1\n",
" pid: 44727\n",
" time_since_restore: 2.623929977416992\n",
" time_this_iter_s: 0.032716989517211914\n",
" time_total_s: 2.623929977416992\n",
" timestamp: 1658498667\n",
" timesteps_since_restore: 0\n",
" timesteps_total: 99\n",
" training_iteration: 100\n",
" trial_id: 2fabaa1a\n",
" warmup_time: 0.0025510787963867188\n",
" \n",
"Result for objective_32d8dd20:\n",
" date: 2022-07-22_15-04-30\n",
" done: false\n",
" experiment_id: 171527593b0f4cbf941c0a03faaf0953\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 1\n",
" l2norm: 1.2869105702896437\n",
" landscape: -0.24722262157458608\n",
" node_ip: 127.0.0.1\n",
" pid: 44758\n",
" time_since_restore: 0.00021886825561523438\n",
" time_this_iter_s: 0.00021886825561523438\n",
" time_total_s: 0.00021886825561523438\n",
" timestamp: 1658498670\n",
" timesteps_since_restore: 0\n",
" timesteps_total: 0\n",
" training_iteration: 1\n",
" trial_id: 32d8dd20\n",
" warmup_time: 0.002732992172241211\n",
" \n",
"Result for objective_32cf8ca2:\n",
" date: 2022-07-22_15-04-29\n",
" done: false\n",
" experiment_id: 37610500f6df493aae4e7e46bb21bf09\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 1\n",
" l2norm: 1.1817810425508524\n",
" landscape: -0.14653248187442922\n",
" node_ip: 127.0.0.1\n",
" pid: 44756\n",
" time_since_restore: 0.00025081634521484375\n",
" time_this_iter_s: 0.00025081634521484375\n",
" time_total_s: 0.00025081634521484375\n",
" timestamp: 1658498669\n",
" timesteps_since_restore: 0\n",
" timesteps_total: 0\n",
" training_iteration: 1\n",
" trial_id: 32cf8ca2\n",
" warmup_time: 0.0032138824462890625\n",
" \n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"[INFO 07-22 15:04:30] ax.service.ax_client: Completed trial 4 with data: {'landscape': (-0.00678, None), 'l2norm': (1.80573, None)}.\n",
"[INFO 07-22 15:04:30] ax.service.ax_client: Generated new trial 8 with parameters {'x1': 0.454189, 'x2': 0.271772, 'x3': 0.530871, 'x4': 0.991841, 'x5': 0.691843, 'x6': 0.472366, 'iterations': 100}.\n",
"[INFO 07-22 15:04:30] ax.service.ax_client: Completed trial 5 with data: {'landscape': (-0.904622, None), 'l2norm': (1.168644, None)}.\n",
"[INFO 07-22 15:04:30] ax.service.ax_client: Generated new trial 9 with parameters {'x1': 0.265264, 'x2': 0.924884, 'x3': 0.151716, 'x4': 0.436026, 'x5': 0.85731, 'x6': 0.08981, 'iterations': 100}.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Result for objective_313d3d3a:\n",
" date: 2022-07-22_15-04-30\n",
" done: true\n",
" experiment_id: fa7afd557e154fbebe4f54d8eedb3573\n",
" experiment_tag: 5_iterations=100,x1=0.0419,x2=0.9928,x3=0.9060,x4=0.5944,x5=0.8254,x6=0.6464\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 100\n",
" l2norm: 1.805729990121368\n",
" landscape: -0.006779757704679272\n",
" node_ip: 127.0.0.1\n",
" pid: 44747\n",
" time_since_restore: 3.1623308658599854\n",
" time_this_iter_s: 0.02911996841430664\n",
" time_total_s: 3.1623308658599854\n",
" timestamp: 1658498670\n",
" timesteps_since_restore: 0\n",
" timesteps_total: 99\n",
" training_iteration: 100\n",
" trial_id: 313d3d3a\n",
" warmup_time: 0.0029790401458740234\n",
" \n",
"Result for objective_32c9acd8:\n",
" date: 2022-07-22_15-04-30\n",
" done: true\n",
" experiment_id: c555bfed13ac43e5b8c8e9f6d4b9b2f7\n",
" experiment_tag: 6_iterations=100,x1=0.1261,x2=0.7034,x3=0.3447,x4=0.3374,x5=0.4014,x6=0.6792\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 100\n",
" l2norm: 1.1686440476629836\n",
" landscape: -0.9046216637367911\n",
" node_ip: 127.0.0.1\n",
" pid: 44726\n",
" time_since_restore: 3.1211891174316406\n",
" time_this_iter_s: 0.02954697608947754\n",
" time_total_s: 3.1211891174316406\n",
" timestamp: 1658498670\n",
" timesteps_since_restore: 0\n",
" timesteps_total: 99\n",
" training_iteration: 100\n",
" trial_id: 32c9acd8\n",
" warmup_time: 0.0026290416717529297\n",
" \n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"[INFO 07-22 15:04:32] ax.service.ax_client: Completed trial 7 with data: {'landscape': (-0.247223, None), 'l2norm': (1.286911, None)}.\n",
"[INFO 07-22 15:04:32] ax.service.ax_client: Completed trial 6 with data: {'landscape': (-0.146532, None), 'l2norm': (1.181781, None)}.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Result for objective_32d8dd20:\n",
" date: 2022-07-22_15-04-32\n",
" done: true\n",
" experiment_id: 171527593b0f4cbf941c0a03faaf0953\n",
" experiment_tag: 8_iterations=100,x1=0.6032,x2=0.4091,x3=0.7291,x4=0.0826,x5=0.5729,x6=0.5083\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 100\n",
" l2norm: 1.2869105702896437\n",
" landscape: -0.24722262157458608\n",
" node_ip: 127.0.0.1\n",
" pid: 44758\n",
" time_since_restore: 2.6415798664093018\n",
" time_this_iter_s: 0.026781082153320312\n",
" time_total_s: 2.6415798664093018\n",
" timestamp: 1658498672\n",
" timesteps_since_restore: 0\n",
" timesteps_total: 99\n",
" training_iteration: 100\n",
" trial_id: 32d8dd20\n",
" warmup_time: 0.002732992172241211\n",
" \n",
"Result for objective_32cf8ca2:\n",
" date: 2022-07-22_15-04-32\n",
" done: true\n",
" experiment_id: 37610500f6df493aae4e7e46bb21bf09\n",
" experiment_tag: 7_iterations=100,x1=0.0911,x2=0.3041,x3=0.8698,x4=0.4054,x5=0.5679,x6=0.2286\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 100\n",
" l2norm: 1.1817810425508524\n",
" landscape: -0.14653248187442922\n",
" node_ip: 127.0.0.1\n",
" pid: 44756\n",
" time_since_restore: 2.707913875579834\n",
" time_this_iter_s: 0.027456998825073242\n",
" time_total_s: 2.707913875579834\n",
" timestamp: 1658498672\n",
" timesteps_since_restore: 0\n",
" timesteps_total: 99\n",
" training_iteration: 100\n",
" trial_id: 32cf8ca2\n",
" warmup_time: 0.0032138824462890625\n",
" \n",
"Result for objective_34adf04a:\n",
" date: 2022-07-22_15-04-33\n",
" done: false\n",
" experiment_id: 4f65c5b68f5c49d98fda388e37c83deb\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 1\n",
" l2norm: 1.4991655675380078\n",
" landscape: -0.01329150870283869\n",
" node_ip: 127.0.0.1\n",
" pid: 44768\n",
" time_since_restore: 0.00021600723266601562\n",
" time_this_iter_s: 0.00021600723266601562\n",
" time_total_s: 0.00021600723266601562\n",
" timestamp: 1658498673\n",
" timesteps_since_restore: 0\n",
" timesteps_total: 0\n",
" training_iteration: 1\n",
" trial_id: 34adf04a\n",
" warmup_time: 0.0027239322662353516\n",
" \n",
"Result for objective_34b7abda:\n",
" date: 2022-07-22_15-04-33\n",
" done: false\n",
" experiment_id: f135a2c40f5644ba9d2ae096a9dd10e0\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 1\n",
" l2norm: 1.3718451333547932\n",
" landscape: -1.6624439263544026\n",
" node_ip: 127.0.0.1\n",
" pid: 44771\n",
" time_since_restore: 0.0002338886260986328\n",
" time_this_iter_s: 0.0002338886260986328\n",
" time_total_s: 0.0002338886260986328\n",
" timestamp: 1658498673\n",
" timesteps_since_restore: 0\n",
" timesteps_total: 0\n",
" training_iteration: 1\n",
" trial_id: 34b7abda\n",
" warmup_time: 0.002721071243286133\n",
" \n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"[INFO 07-22 15:04:35] ax.service.ax_client: Completed trial 8 with data: {'landscape': (-0.013292, None), 'l2norm': (1.499166, None)}.\n",
"[INFO 07-22 15:04:35] ax.service.ax_client: Completed trial 9 with data: {'landscape': (-1.662444, None), 'l2norm': (1.371845, None)}.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Result for objective_34adf04a:\n",
" date: 2022-07-22_15-04-35\n",
" done: true\n",
" experiment_id: 4f65c5b68f5c49d98fda388e37c83deb\n",
" experiment_tag: 9_iterations=100,x1=0.4542,x2=0.2718,x3=0.5309,x4=0.9918,x5=0.6918,x6=0.4724\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 100\n",
" l2norm: 1.4991655675380078\n",
" landscape: -0.01329150870283869\n",
" node_ip: 127.0.0.1\n",
" pid: 44768\n",
" time_since_restore: 2.7032668590545654\n",
" time_this_iter_s: 0.029300928115844727\n",
" time_total_s: 2.7032668590545654\n",
" timestamp: 1658498675\n",
" timesteps_since_restore: 0\n",
" timesteps_total: 99\n",
" training_iteration: 100\n",
" trial_id: 34adf04a\n",
" warmup_time: 0.0027239322662353516\n",
" \n",
"Result for objective_34b7abda:\n",
" date: 2022-07-22_15-04-35\n",
" done: true\n",
" experiment_id: f135a2c40f5644ba9d2ae096a9dd10e0\n",
" experiment_tag: 10_iterations=100,x1=0.2653,x2=0.9249,x3=0.1517,x4=0.4360,x5=0.8573,x6=0.0898\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 100\n",
" l2norm: 1.3718451333547932\n",
" landscape: -1.6624439263544026\n",
" node_ip: 127.0.0.1\n",
" pid: 44771\n",
" time_since_restore: 2.6852078437805176\n",
" time_this_iter_s: 0.029579877853393555\n",
" time_total_s: 2.6852078437805176\n",
" timestamp: 1658498675\n",
" timesteps_since_restore: 0\n",
" timesteps_total: 99\n",
" training_iteration: 100\n",
" trial_id: 34b7abda\n",
" warmup_time: 0.002721071243286133\n",
" \n"
]
}
],
"source": [
"tuner = tune.Tuner(\n",
" objective,\n",
" tune_config=tune.TuneConfig(\n",
" metric=\"landscape\",\n",
" mode=\"min\",\n",
" search_alg=algo,\n",
" num_samples=num_samples,\n",
" ),\n",
" run_config=train.RunConfig(\n",
" name=\"ax\",\n",
" stop={\"timesteps_total\": stop_timesteps}\n",
" ),\n",
" param_space=search_space,\n",
")\n",
"results = tuner.fit()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "860b53b0",
"metadata": {},
"source": [
"And now we have the hyperparameters found to minimize the mean loss."
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "12906421",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Best hyperparameters found were: {'iterations': 100, 'x1': 0.26526361983269453, 'x2': 0.9248840995132923, 'x3': 0.15171580761671066, 'x4': 0.43602637108415365, 'x5': 0.8573104059323668, 'x6': 0.08981018699705601}\n"
]
}
],
"source": [
"print(\"Best hyperparameters found were: \", results.get_best_result().config)"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "68872424",
"metadata": {
"tags": [
"remove-cell"
]
},
"outputs": [],
"source": [
"ray.shutdown()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.13"
},
"orphan": true
},
"nbformat": 4,
"nbformat_minor": 5
}