{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"id": "6df76a1f",
"metadata": {},
"source": [
"# Using MLflow with Tune\n",
"\n",
"(tune-mlflow-ref)=\n",
"\n",
"[MLflow](https://mlflow.org/) is an open source platform to manage the ML lifecycle, including experimentation,\n",
"reproducibility, deployment, and a central model registry. It currently offers four components, including\n",
"MLflow Tracking to record and query experiments, including code, data, config, and results.\n",
"\n",
"```{image} /images/mlflow.png\n",
":align: center\n",
":alt: MLflow\n",
":height: 80px\n",
":target: https://www.mlflow.org/\n",
"```\n",
"\n",
"Ray Tune currently offers two lightweight integrations for MLflow Tracking.\n",
"One is the {ref}`MLflowLoggerCallback <tune-mlflow-logger>`, which automatically logs\n",
"metrics reported to Tune to the MLflow Tracking API.\n",
"\n",
"The other one is the {ref}`setup_mlflow <tune-mlflow-setup>` function, which can be\n",
"used with the function API. It automatically\n",
"initializes the MLflow API with Tune's training information and creates a run for each Tune trial.\n",
"Then within your training function, you can just use the\n",
"MLflow like you would normally do, e.g. using `mlflow.log_metrics()` or even `mlflow.autolog()`\n",
"to log to your training process.\n",
"\n",
"```{contents}\n",
":backlinks: none\n",
":local: true\n",
"```\n",
"\n",
"## Running an MLflow Example\n",
"\n",
"In the following example we're going to use both of the above methods, namely the `MLflowLoggerCallback` and\n",
"the `setup_mlflow` function to log metrics.\n",
"Let's start with a few crucial imports:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "b0e47339",
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"import tempfile\n",
"import time\n",
"\n",
"import mlflow\n",
"\n",
"from ray import train, tune\n",
"from ray.air.integrations.mlflow import MLflowLoggerCallback, setup_mlflow\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "618b6935",
"metadata": {
"pycharm": {
"name": "#%% md\n"
}
},
"source": [
"Next, let's define an easy training function (a Tune `Trainable`) that iteratively computes steps and evaluates\n",
"intermediate scores that we report to Tune."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "f449538e",
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"def evaluation_fn(step, width, height):\n",
" return (0.1 + width * step / 100) ** (-1) + height * 0.1\n",
"\n",
"\n",
"def train_function(config):\n",
" width, height = config[\"width\"], config[\"height\"]\n",
"\n",
" for step in range(config.get(\"steps\", 100)):\n",
" # Iterative training function - can be any arbitrary training procedure\n",
" intermediate_score = evaluation_fn(step, width, height)\n",
" # Feed the score back to Tune.\n",
" train.report({\"iterations\": step, \"mean_loss\": intermediate_score})\n",
" time.sleep(0.1)\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "722e5d2f",
"metadata": {
"pycharm": {
"name": "#%% md\n"
}
},
"source": [
"Given an MLFlow tracking URI, you can now simply use the `MLflowLoggerCallback` as a `callback` argument to\n",
"your `RunConfig()`:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "8e0b9ab7",
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"def tune_with_callback(mlflow_tracking_uri, finish_fast=False):\n",
" tuner = tune.Tuner(\n",
" train_function,\n",
" tune_config=tune.TuneConfig(num_samples=5),\n",
" run_config=train.RunConfig(\n",
" name=\"mlflow\",\n",
" callbacks=[\n",
" MLflowLoggerCallback(\n",
" tracking_uri=mlflow_tracking_uri,\n",
" experiment_name=\"mlflow_callback_example\",\n",
" save_artifact=True,\n",
" )\n",
" ],\n",
" ),\n",
" param_space={\n",
" \"width\": tune.randint(10, 100),\n",
" \"height\": tune.randint(0, 100),\n",
" \"steps\": 5 if finish_fast else 100,\n",
" },\n",
" )\n",
" results = tuner.fit()\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "e086f110",
"metadata": {},
"source": [
"To use the `setup_mlflow` utility, you simply call this function in your training function.\n",
"Note that we also use `mlflow.log_metrics(...)` to log metrics to MLflow.\n",
"Otherwise, this version of our training function is identical to its original."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "144b8f39",
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"def train_function_mlflow(config):\n",
" tracking_uri = config.pop(\"tracking_uri\", None)\n",
" setup_mlflow(\n",
" config,\n",
" experiment_name=\"setup_mlflow_example\",\n",
" tracking_uri=tracking_uri,\n",
" )\n",
"\n",
" # Hyperparameters\n",
" width, height = config[\"width\"], config[\"height\"]\n",
"\n",
" for step in range(config.get(\"steps\", 100)):\n",
" # Iterative training function - can be any arbitrary training procedure\n",
" intermediate_score = evaluation_fn(step, width, height)\n",
" # Log the metrics to mlflow\n",
" mlflow.log_metrics(dict(mean_loss=intermediate_score), step=step)\n",
" # Feed the score back to Tune.\n",
" train.report({\"iterations\": step, \"mean_loss\": intermediate_score})\n",
" time.sleep(0.1)\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "dc480366",
"metadata": {},
"source": [
"With this new objective function ready, you can now create a Tune run with it as follows:"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "4b9fe6be",
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"def tune_with_setup(mlflow_tracking_uri, finish_fast=False):\n",
" # Set the experiment, or create a new one if does not exist yet.\n",
" mlflow.set_tracking_uri(mlflow_tracking_uri)\n",
" mlflow.set_experiment(experiment_name=\"setup_mlflow_example\")\n",
"\n",
" tuner = tune.Tuner(\n",
" train_function_mlflow,\n",
" tune_config=tune.TuneConfig(num_samples=5),\n",
" run_config=train.RunConfig(\n",
" name=\"mlflow\",\n",
" ),\n",
" param_space={\n",
" \"width\": tune.randint(10, 100),\n",
" \"height\": tune.randint(0, 100),\n",
" \"steps\": 5 if finish_fast else 100,\n",
" \"tracking_uri\": mlflow.get_tracking_uri(),\n",
" },\n",
" )\n",
" results = tuner.fit()\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "915dfd30",
"metadata": {},
"source": [
"If you hapen to have an MLFlow tracking URI, you can set it below in the `mlflow_tracking_uri` variable and set\n",
"`smoke_test=False`.\n",
"Otherwise, you can just run a quick test of the `tune_function` and `tune_decorated` functions without using MLflow."
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "05d11774",
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"2022-12-22 10:37:53,580\tINFO worker.py:1542 -- Started a local Ray instance. View the dashboard at \u001b[1m\u001b[32mhttp://127.0.0.1:8265 \u001b[39m\u001b[22m\n"
]
},
{
"data": {
"text/html": [
"<div class=\"tuneStatus\">\n",
" <div style=\"display: flex;flex-direction: row\">\n",
" <div style=\"display: flex;flex-direction: column;\">\n",
" <h3>Tune Status</h3>\n",
" <table>\n",
"<tbody>\n",
"<tr><td>Current time:</td><td>2022-12-22 10:38:04</td></tr>\n",
"<tr><td>Running for: </td><td>00:00:06.73 </td></tr>\n",
"<tr><td>Memory: </td><td>10.4/16.0 GiB </td></tr>\n",
"</tbody>\n",
"</table>\n",
" </div>\n",
" <div class=\"vDivider\"></div>\n",
" <div class=\"systemInfo\">\n",
" <h3>System Info</h3>\n",
" Using FIFO scheduling algorithm.<br>Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/4.03 GiB heap, 0.0/2.0 GiB objects\n",
" </div>\n",
" \n",
" </div>\n",
" <div class=\"hDivider\"></div>\n",
" <div class=\"trialStatus\">\n",
" <h3>Trial Status</h3>\n",
" <table>\n",
"<thead>\n",
"<tr><th>Trial name </th><th>status </th><th>loc </th><th style=\"text-align: right;\"> height</th><th style=\"text-align: right;\"> width</th><th style=\"text-align: right;\"> loss</th><th style=\"text-align: right;\"> iter</th><th style=\"text-align: right;\"> total time (s)</th><th style=\"text-align: right;\"> iterations</th><th style=\"text-align: right;\"> neg_mean_loss</th></tr>\n",
"</thead>\n",
"<tbody>\n",
"<tr><td>train_function_b275b_00000</td><td>TERMINATED</td><td>127.0.0.1:801</td><td style=\"text-align: right;\"> 66</td><td style=\"text-align: right;\"> 36</td><td style=\"text-align: right;\">7.24935</td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\"> 0.587302</td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\"> -7.24935</td></tr>\n",
"<tr><td>train_function_b275b_00001</td><td>TERMINATED</td><td>127.0.0.1:813</td><td style=\"text-align: right;\"> 33</td><td style=\"text-align: right;\"> 35</td><td style=\"text-align: right;\">3.96667</td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\"> 0.507423</td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\"> -3.96667</td></tr>\n",
"<tr><td>train_function_b275b_00002</td><td>TERMINATED</td><td>127.0.0.1:814</td><td style=\"text-align: right;\"> 75</td><td style=\"text-align: right;\"> 29</td><td style=\"text-align: right;\">8.29365</td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\"> 0.518995</td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\"> -8.29365</td></tr>\n",
"<tr><td>train_function_b275b_00003</td><td>TERMINATED</td><td>127.0.0.1:815</td><td style=\"text-align: right;\"> 28</td><td style=\"text-align: right;\"> 63</td><td style=\"text-align: right;\">3.18168</td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\"> 0.567739</td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\"> -3.18168</td></tr>\n",
"<tr><td>train_function_b275b_00004</td><td>TERMINATED</td><td>127.0.0.1:816</td><td style=\"text-align: right;\"> 20</td><td style=\"text-align: right;\"> 18</td><td style=\"text-align: right;\">3.21951</td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\"> 0.526536</td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\"> -3.21951</td></tr>\n",
"</tbody>\n",
"</table>\n",
" </div>\n",
"</div>\n",
"<style>\n",
".tuneStatus {\n",
" color: var(--jp-ui-font-color1);\n",
"}\n",
".tuneStatus .systemInfo {\n",
" display: flex;\n",
" flex-direction: column;\n",
"}\n",
".tuneStatus td {\n",
" white-space: nowrap;\n",
"}\n",
".tuneStatus .trialStatus {\n",
" display: flex;\n",
" flex-direction: column;\n",
"}\n",
".tuneStatus h3 {\n",
" font-weight: bold;\n",
"}\n",
".tuneStatus .hDivider {\n",
" border-bottom-width: var(--jp-border-width);\n",
" border-bottom-color: var(--jp-border-color0);\n",
" border-bottom-style: solid;\n",
"}\n",
".tuneStatus .vDivider {\n",
" border-left-width: var(--jp-border-width);\n",
" border-left-color: var(--jp-border-color0);\n",
" border-left-style: solid;\n",
" margin: 0.5em 1em 0.5em 1em;\n",
"}\n",
"</style>\n"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<div class=\"trialProgress\">\n",
" <h3>Trial Progress</h3>\n",
" <table>\n",
"<thead>\n",
"<tr><th>Trial name </th><th>date </th><th>done </th><th>episodes_total </th><th>experiment_id </th><th>experiment_tag </th><th>hostname </th><th style=\"text-align: right;\"> iterations</th><th style=\"text-align: right;\"> iterations_since_restore</th><th style=\"text-align: right;\"> mean_loss</th><th style=\"text-align: right;\"> neg_mean_loss</th><th>node_ip </th><th style=\"text-align: right;\"> pid</th><th style=\"text-align: right;\"> time_since_restore</th><th style=\"text-align: right;\"> time_this_iter_s</th><th style=\"text-align: right;\"> time_total_s</th><th style=\"text-align: right;\"> timestamp</th><th style=\"text-align: right;\"> timesteps_since_restore</th><th>timesteps_total </th><th style=\"text-align: right;\"> training_iteration</th><th>trial_id </th><th style=\"text-align: right;\"> warmup_time</th></tr>\n",
"</thead>\n",
"<tbody>\n",
"<tr><td>train_function_b275b_00000</td><td>2022-12-22_10-38-01</td><td>True </td><td> </td><td>28feaa4dd8ab4edab810e8109e77502e</td><td>0_height=66,width=36</td><td>kais-macbook-pro.anyscale.com.beta.tailscale.net</td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\"> 7.24935</td><td style=\"text-align: right;\"> -7.24935</td><td>127.0.0.1</td><td style=\"text-align: right;\"> 801</td><td style=\"text-align: right;\"> 0.587302</td><td style=\"text-align: right;\"> 0.126818</td><td style=\"text-align: right;\"> 0.587302</td><td style=\"text-align: right;\"> 1671705481</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 5</td><td>b275b_00000</td><td style=\"text-align: right;\"> 0.00293493</td></tr>\n",
"<tr><td>train_function_b275b_00001</td><td>2022-12-22_10-38-04</td><td>True </td><td> </td><td>245010d0c3d0439ebfb664764ae9db3c</td><td>1_height=33,width=35</td><td>kais-macbook-pro.anyscale.com.beta.tailscale.net</td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\"> 3.96667</td><td style=\"text-align: right;\"> -3.96667</td><td>127.0.0.1</td><td style=\"text-align: right;\"> 813</td><td style=\"text-align: right;\"> 0.507423</td><td style=\"text-align: right;\"> 0.122086</td><td style=\"text-align: right;\"> 0.507423</td><td style=\"text-align: right;\"> 1671705484</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 5</td><td>b275b_00001</td><td style=\"text-align: right;\"> 0.00553799</td></tr>\n",
"<tr><td>train_function_b275b_00002</td><td>2022-12-22_10-38-04</td><td>True </td><td> </td><td>898afbf9b906448c980f399c72a2324c</td><td>2_height=75,width=29</td><td>kais-macbook-pro.anyscale.com.beta.tailscale.net</td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\"> 8.29365</td><td style=\"text-align: right;\"> -8.29365</td><td>127.0.0.1</td><td style=\"text-align: right;\"> 814</td><td style=\"text-align: right;\"> 0.518995</td><td style=\"text-align: right;\"> 0.123554</td><td style=\"text-align: right;\"> 0.518995</td><td style=\"text-align: right;\"> 1671705484</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 5</td><td>b275b_00002</td><td style=\"text-align: right;\"> 0.0040431 </td></tr>\n",
"<tr><td>train_function_b275b_00003</td><td>2022-12-22_10-38-04</td><td>True </td><td> </td><td>03a4476f82734642b6ab0a5040ca58f8</td><td>3_height=28,width=63</td><td>kais-macbook-pro.anyscale.com.beta.tailscale.net</td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\"> 3.18168</td><td style=\"text-align: right;\"> -3.18168</td><td>127.0.0.1</td><td style=\"text-align: right;\"> 815</td><td style=\"text-align: right;\"> 0.567739</td><td style=\"text-align: right;\"> 0.125471</td><td style=\"text-align: right;\"> 0.567739</td><td style=\"text-align: right;\"> 1671705484</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 5</td><td>b275b_00003</td><td style=\"text-align: right;\"> 0.00406194</td></tr>\n",
"<tr><td>train_function_b275b_00004</td><td>2022-12-22_10-38-04</td><td>True </td><td> </td><td>ff8c7c55ce6e404f9b0552c17f7a0c40</td><td>4_height=20,width=18</td><td>kais-macbook-pro.anyscale.com.beta.tailscale.net</td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\"> 3.21951</td><td style=\"text-align: right;\"> -3.21951</td><td>127.0.0.1</td><td style=\"text-align: right;\"> 816</td><td style=\"text-align: right;\"> 0.526536</td><td style=\"text-align: right;\"> 0.123327</td><td style=\"text-align: right;\"> 0.526536</td><td style=\"text-align: right;\"> 1671705484</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 5</td><td>b275b_00004</td><td style=\"text-align: right;\"> 0.00332022</td></tr>\n",
"</tbody>\n",
"</table>\n",
"</div>\n",
"<style>\n",
".trialProgress {\n",
" display: flex;\n",
" flex-direction: column;\n",
" color: var(--jp-ui-font-color1);\n",
"}\n",
".trialProgress h3 {\n",
" font-weight: bold;\n",
"}\n",
".trialProgress td {\n",
" white-space: nowrap;\n",
"}\n",
"</style>\n"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2022-12-22 10:38:04,477\tINFO tune.py:772 -- Total run time: 7.99 seconds (6.71 seconds for the tuning loop).\n"
]
},
{
"data": {
"text/html": [
"<div class=\"tuneStatus\">\n",
" <div style=\"display: flex;flex-direction: row\">\n",
" <div style=\"display: flex;flex-direction: column;\">\n",
" <h3>Tune Status</h3>\n",
" <table>\n",
"<tbody>\n",
"<tr><td>Current time:</td><td>2022-12-22 10:38:11</td></tr>\n",
"<tr><td>Running for: </td><td>00:00:07.00 </td></tr>\n",
"<tr><td>Memory: </td><td>10.7/16.0 GiB </td></tr>\n",
"</tbody>\n",
"</table>\n",
" </div>\n",
" <div class=\"vDivider\"></div>\n",
" <div class=\"systemInfo\">\n",
" <h3>System Info</h3>\n",
" Using FIFO scheduling algorithm.<br>Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/4.03 GiB heap, 0.0/2.0 GiB objects\n",
" </div>\n",
" \n",
" </div>\n",
" <div class=\"hDivider\"></div>\n",
" <div class=\"trialStatus\">\n",
" <h3>Trial Status</h3>\n",
" <table>\n",
"<thead>\n",
"<tr><th>Trial name </th><th>status </th><th>loc </th><th style=\"text-align: right;\"> height</th><th style=\"text-align: right;\"> width</th><th style=\"text-align: right;\"> loss</th><th style=\"text-align: right;\"> iter</th><th style=\"text-align: right;\"> total time (s)</th><th style=\"text-align: right;\"> iterations</th><th style=\"text-align: right;\"> neg_mean_loss</th></tr>\n",
"</thead>\n",
"<tbody>\n",
"<tr><td>train_function_mlflow_b73bd_00000</td><td>TERMINATED</td><td>127.0.0.1:842</td><td style=\"text-align: right;\"> 37</td><td style=\"text-align: right;\"> 68</td><td style=\"text-align: right;\">4.05461</td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\"> 0.750435</td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\"> -4.05461</td></tr>\n",
"<tr><td>train_function_mlflow_b73bd_00001</td><td>TERMINATED</td><td>127.0.0.1:853</td><td style=\"text-align: right;\"> 50</td><td style=\"text-align: right;\"> 20</td><td style=\"text-align: right;\">6.11111</td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\"> 0.652748</td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\"> -6.11111</td></tr>\n",
"<tr><td>train_function_mlflow_b73bd_00002</td><td>TERMINATED</td><td>127.0.0.1:854</td><td style=\"text-align: right;\"> 38</td><td style=\"text-align: right;\"> 83</td><td style=\"text-align: right;\">4.0924 </td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\"> 0.6513 </td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\"> -4.0924 </td></tr>\n",
"<tr><td>train_function_mlflow_b73bd_00003</td><td>TERMINATED</td><td>127.0.0.1:855</td><td style=\"text-align: right;\"> 15</td><td style=\"text-align: right;\"> 93</td><td style=\"text-align: right;\">1.76178</td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\"> 0.650586</td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\"> -1.76178</td></tr>\n",
"<tr><td>train_function_mlflow_b73bd_00004</td><td>TERMINATED</td><td>127.0.0.1:856</td><td style=\"text-align: right;\"> 75</td><td style=\"text-align: right;\"> 43</td><td style=\"text-align: right;\">8.04945</td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\"> 0.656046</td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\"> -8.04945</td></tr>\n",
"</tbody>\n",
"</table>\n",
" </div>\n",
"</div>\n",
"<style>\n",
".tuneStatus {\n",
" color: var(--jp-ui-font-color1);\n",
"}\n",
".tuneStatus .systemInfo {\n",
" display: flex;\n",
" flex-direction: column;\n",
"}\n",
".tuneStatus td {\n",
" white-space: nowrap;\n",
"}\n",
".tuneStatus .trialStatus {\n",
" display: flex;\n",
" flex-direction: column;\n",
"}\n",
".tuneStatus h3 {\n",
" font-weight: bold;\n",
"}\n",
".tuneStatus .hDivider {\n",
" border-bottom-width: var(--jp-border-width);\n",
" border-bottom-color: var(--jp-border-color0);\n",
" border-bottom-style: solid;\n",
"}\n",
".tuneStatus .vDivider {\n",
" border-left-width: var(--jp-border-width);\n",
" border-left-color: var(--jp-border-color0);\n",
" border-left-style: solid;\n",
" margin: 0.5em 1em 0.5em 1em;\n",
"}\n",
"</style>\n"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<div class=\"trialProgress\">\n",
" <h3>Trial Progress</h3>\n",
" <table>\n",
"<thead>\n",
"<tr><th>Trial name </th><th>date </th><th>done </th><th>episodes_total </th><th>experiment_id </th><th>experiment_tag </th><th>hostname </th><th style=\"text-align: right;\"> iterations</th><th style=\"text-align: right;\"> iterations_since_restore</th><th style=\"text-align: right;\"> mean_loss</th><th style=\"text-align: right;\"> neg_mean_loss</th><th>node_ip </th><th style=\"text-align: right;\"> pid</th><th style=\"text-align: right;\"> time_since_restore</th><th style=\"text-align: right;\"> time_this_iter_s</th><th style=\"text-align: right;\"> time_total_s</th><th style=\"text-align: right;\"> timestamp</th><th style=\"text-align: right;\"> timesteps_since_restore</th><th>timesteps_total </th><th style=\"text-align: right;\"> training_iteration</th><th>trial_id </th><th style=\"text-align: right;\"> warmup_time</th></tr>\n",
"</thead>\n",
"<tbody>\n",
"<tr><td>train_function_mlflow_b73bd_00000</td><td>2022-12-22_10-38-08</td><td>True </td><td> </td><td>62703cfe82e54d74972377fbb525b000</td><td>0_height=37,width=68</td><td>kais-macbook-pro.anyscale.com.beta.tailscale.net</td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\"> 4.05461</td><td style=\"text-align: right;\"> -4.05461</td><td>127.0.0.1</td><td style=\"text-align: right;\"> 842</td><td style=\"text-align: right;\"> 0.750435</td><td style=\"text-align: right;\"> 0.108625</td><td style=\"text-align: right;\"> 0.750435</td><td style=\"text-align: right;\"> 1671705488</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 5</td><td>b73bd_00000</td><td style=\"text-align: right;\"> 0.0030272 </td></tr>\n",
"<tr><td>train_function_mlflow_b73bd_00001</td><td>2022-12-22_10-38-11</td><td>True </td><td> </td><td>03ea89852115465392ed318db8021614</td><td>1_height=50,width=20</td><td>kais-macbook-pro.anyscale.com.beta.tailscale.net</td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\"> 6.11111</td><td style=\"text-align: right;\"> -6.11111</td><td>127.0.0.1</td><td style=\"text-align: right;\"> 853</td><td style=\"text-align: right;\"> 0.652748</td><td style=\"text-align: right;\"> 0.110796</td><td style=\"text-align: right;\"> 0.652748</td><td style=\"text-align: right;\"> 1671705491</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 5</td><td>b73bd_00001</td><td style=\"text-align: right;\"> 0.00303078</td></tr>\n",
"<tr><td>train_function_mlflow_b73bd_00002</td><td>2022-12-22_10-38-11</td><td>True </td><td> </td><td>3731fc2966f9453ba58c650d89035ab4</td><td>2_height=38,width=83</td><td>kais-macbook-pro.anyscale.com.beta.tailscale.net</td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\"> 4.0924 </td><td style=\"text-align: right;\"> -4.0924 </td><td>127.0.0.1</td><td style=\"text-align: right;\"> 854</td><td style=\"text-align: right;\"> 0.6513 </td><td style=\"text-align: right;\"> 0.108578</td><td style=\"text-align: right;\"> 0.6513 </td><td style=\"text-align: right;\"> 1671705491</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 5</td><td>b73bd_00002</td><td style=\"text-align: right;\"> 0.00310016</td></tr>\n",
"<tr><td>train_function_mlflow_b73bd_00003</td><td>2022-12-22_10-38-11</td><td>True </td><td> </td><td>fb35841742b348b9912d10203c730f1e</td><td>3_height=15,width=93</td><td>kais-macbook-pro.anyscale.com.beta.tailscale.net</td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\"> 1.76178</td><td style=\"text-align: right;\"> -1.76178</td><td>127.0.0.1</td><td style=\"text-align: right;\"> 855</td><td style=\"text-align: right;\"> 0.650586</td><td style=\"text-align: right;\"> 0.109097</td><td style=\"text-align: right;\"> 0.650586</td><td style=\"text-align: right;\"> 1671705491</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 5</td><td>b73bd_00003</td><td style=\"text-align: right;\"> 0.0576491 </td></tr>\n",
"<tr><td>train_function_mlflow_b73bd_00004</td><td>2022-12-22_10-38-11</td><td>True </td><td> </td><td>6d3cbf9ecc3446369e607ff78c67bc29</td><td>4_height=75,width=43</td><td>kais-macbook-pro.anyscale.com.beta.tailscale.net</td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\"> 8.04945</td><td style=\"text-align: right;\"> -8.04945</td><td>127.0.0.1</td><td style=\"text-align: right;\"> 856</td><td style=\"text-align: right;\"> 0.656046</td><td style=\"text-align: right;\"> 0.109869</td><td style=\"text-align: right;\"> 0.656046</td><td style=\"text-align: right;\"> 1671705491</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 5</td><td>b73bd_00004</td><td style=\"text-align: right;\"> 0.00265694</td></tr>\n",
"</tbody>\n",
"</table>\n",
"</div>\n",
"<style>\n",
".trialProgress {\n",
" display: flex;\n",
" flex-direction: column;\n",
" color: var(--jp-ui-font-color1);\n",
"}\n",
".trialProgress h3 {\n",
" font-weight: bold;\n",
"}\n",
".trialProgress td {\n",
" white-space: nowrap;\n",
"}\n",
"</style>\n"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2022-12-22 10:38:11,514\tINFO tune.py:772 -- Total run time: 7.01 seconds (6.98 seconds for the tuning loop).\n"
]
}
],
"source": [
"smoke_test = True\n",
"\n",
"if smoke_test:\n",
" mlflow_tracking_uri = os.path.join(tempfile.gettempdir(), \"mlruns\")\n",
"else:\n",
" mlflow_tracking_uri = \"<MLFLOW_TRACKING_URI>\"\n",
"\n",
"tune_with_callback(mlflow_tracking_uri, finish_fast=smoke_test)\n",
"if not smoke_test:\n",
" df = mlflow.search_runs(\n",
" [mlflow.get_experiment_by_name(\"mlflow_callback_example\").experiment_id]\n",
" )\n",
" print(df)\n",
"\n",
"tune_with_setup(mlflow_tracking_uri, finish_fast=smoke_test)\n",
"if not smoke_test:\n",
" df = mlflow.search_runs(\n",
" [mlflow.get_experiment_by_name(\"setup_mlflow_example\").experiment_id]\n",
" )\n",
" print(df)\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "f0df0817",
"metadata": {},
"source": [
"This completes our Tune and MLflow walk-through.\n",
"In the following sections you can find more details on the API of the Tune-MLflow integration.\n",
"\n",
"## MLflow AutoLogging\n",
"\n",
"You can also check out {doc}`here </tune/examples/includes/mlflow_ptl_example>` for an example on how you can\n",
"leverage MLflow auto-logging, in this case with Pytorch Lightning\n",
"\n",
"## MLflow Logger API\n",
"\n",
"(tune-mlflow-logger)=\n",
"\n",
"```{eval-rst}\n",
".. autoclass:: ray.air.integrations.mlflow.MLflowLoggerCallback\n",
" :noindex:\n",
"```\n",
"\n",
"## MLflow setup API\n",
"\n",
"(tune-mlflow-setup)=\n",
"\n",
"```{eval-rst}\n",
".. autofunction:: ray.air.integrations.mlflow.setup_mlflow\n",
" :noindex:\n",
"```\n",
"\n",
"## More MLflow Examples\n",
"\n",
"- {doc}`/tune/examples/includes/mlflow_ptl_example`: Example for using [MLflow](https://github.com/mlflow/mlflow/)\n",
" and [Pytorch Lightning](https://github.com/PyTorchLightning/pytorch-lightning) with Ray Tune."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.7"
},
"orphan": true
},
"nbformat": 4,
"nbformat_minor": 5
}