{
"cells": [
{
"cell_type": "markdown",
"id": "815af48c-67f4-4335-bf16-11068f7094bb",
"metadata": {},
"source": [
"# Manage Experiment Relationships\n",
"\n",
"``rubicon-ml`` experiments can be tagged with special identifiers to denote a parent/child relationship.\n",
"This can be used to track hierarchical or iterative experiments, among other things.\n",
"\n",
"First, let's create a project."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "c44f140a-0d40-4919-b058-8f986dd9bcb1",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<rubicon_ml.client.project.Project at 0x121d7af50>"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from rubicon_ml import Rubicon\n",
"\n",
"rubicon = Rubicon(persistence=\"memory\")\n",
"project = rubicon.create_project(name=\"hierarchical experiments\")\n",
"project"
]
},
{
"cell_type": "markdown",
"id": "d9aee6c2-8891-4a5b-98d6-37cce80bb40f",
"metadata": {},
"source": [
"## Hierarchical experiments\n",
"\n",
"Now we can log some experiments in a nested loop. Imagine logging an experiment for each node of a\n",
"gradient boosted tree, or something along those lines.\n",
"\n",
"We can use ``parent_experiment.add_child_experiment(child_experiment)`` to automatically add tags\n",
"to both ``parent_experiment`` and ``child_experiment`` that represent their relationship."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "af1bd79b-dd77-4ccb-affb-8abea69f581b",
"metadata": {},
"outputs": [],
"source": [
"root_experiment = project.log_experiment(name=\"root\")\n",
"\n",
"for n in range(3):\n",
" node_experiment = project.log_experiment(name=f\"node_{n}\")\n",
" root_experiment.add_child_experiment(node_experiment)\n",
"\n",
" for m in range(2):\n",
" nested_node_experiment = project.log_experiment(name=f\"node_{n}_{m}\")\n",
" node_experiment.add_child_experiment(nested_node_experiment)"
]
},
{
"cell_type": "markdown",
"id": "a2f6583c-081a-4afe-ba3a-5e6b8744f274",
"metadata": {},
"source": [
"To retrieve experiments, start at the root experiment and call ``get_child_experiments`` to return a\n",
"list of ``rubicon-ml`` objects representing each of the tagged child experiments."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "30068979-44ad-4fd5-9dab-6a2dbee66078",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"id: f8c897d7-852d-4020-8715-264452a5b8ab\n",
"tags: ['child:754dff35-aa87-4385-9bcd-af5c1f5b0b7e', 'child:fdf25d72-d1bb-47bc-8cb2-4a088ed0ba33'] \n",
"\n",
"\tid: 754dff35-aa87-4385-9bcd-af5c1f5b0b7e\n",
"\ttags: ['parent:f8c897d7-852d-4020-8715-264452a5b8ab'] \n",
"\n",
"\tid: fdf25d72-d1bb-47bc-8cb2-4a088ed0ba33\n",
"\ttags: ['parent:f8c897d7-852d-4020-8715-264452a5b8ab'] \n",
"\n",
"id: 5f8c14c1-50d9-4c49-b0b2-6a59c6f3d707\n",
"tags: ['child:d9012c99-2888-43d1-833b-78d51de75a3a', 'child:c65f8c5e-bf5b-4a82-94cb-8b669545b951'] \n",
"\n",
"\tid: d9012c99-2888-43d1-833b-78d51de75a3a\n",
"\ttags: ['parent:5f8c14c1-50d9-4c49-b0b2-6a59c6f3d707'] \n",
"\n",
"\tid: c65f8c5e-bf5b-4a82-94cb-8b669545b951\n",
"\ttags: ['parent:5f8c14c1-50d9-4c49-b0b2-6a59c6f3d707'] \n",
"\n",
"id: da33e918-96c0-4e56-9075-7941515cc18f\n",
"tags: ['child:7c5b2f4b-1e7c-40be-8cda-8f3a00067e98', 'child:cd30f3b2-bd63-4318-974a-6668648bf4ac'] \n",
"\n",
"\tid: 7c5b2f4b-1e7c-40be-8cda-8f3a00067e98\n",
"\ttags: ['parent:da33e918-96c0-4e56-9075-7941515cc18f'] \n",
"\n",
"\tid: cd30f3b2-bd63-4318-974a-6668648bf4ac\n",
"\ttags: ['parent:da33e918-96c0-4e56-9075-7941515cc18f'] \n",
"\n"
]
}
],
"source": [
"for experiment in root_experiment.get_child_experiments():\n",
" print(\"id:\", experiment.id)\n",
" print(\"tags:\", [t for t in experiment.tags if \"child\" in t], \"\\n\")\n",
"\n",
" for nested_experiment in experiment.get_child_experiments():\n",
" print(\"\\tid:\", nested_experiment.id)\n",
" print(\"\\ttags:\", nested_experiment.tags, \"\\n\")"
]
},
{
"cell_type": "markdown",
"id": "12bc18eb-0edd-4054-840b-c4b969a150fb",
"metadata": {},
"source": [
"## Iterative experiments\n",
"\n",
"We can leverage ``add_child_experiment`` to maintain iterative relationships too. This could be\n",
"used to log metadata about of each iteration of recursive feature elimination and preserve the\n",
"linear history of the model training."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "0f104792-7cbf-4508-b37c-11dfb158b608",
"metadata": {},
"outputs": [],
"source": [
"current_experiment = project.log_experiment(name=\"experiment_0\")\n",
"\n",
"for n in range(3):\n",
" next_experiment = project.log_experiment(name=f\"experiment_{n+1}\")\n",
" current_experiment.add_child_experiment(next_experiment)\n",
"\n",
" current_experiment = next_experiment\n",
"\n",
"last_experiment = current_experiment"
]
},
{
"cell_type": "markdown",
"id": "a0f42e74-8fc2-460a-9282-ed0492639d75",
"metadata": {},
"source": [
"Similarly to ``get_child_experiments``, we can use ``get_parent_experiment`` to return a ``rubicon-ml``\n",
"object representing the tagged parent experiment."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "35613e52-84f1-4d2e-8e3c-8c0f2b731d89",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"name: experiment_3\n",
"\tid: a85ba9d9-1473-44c2-b3e9-8a744534485b\n",
"\ttags: ['parent:47ecf8a7-b799-4308-994b-aa6d8698dc2b'] \n",
"\n",
"name: experiment_2\n",
"\tid: 47ecf8a7-b799-4308-994b-aa6d8698dc2b\n",
"\ttags: ['child:a85ba9d9-1473-44c2-b3e9-8a744534485b', 'parent:aea5f005-9792-442c-b98c-8d9b9e39f99b'] \n",
"\n",
"name: experiment_1\n",
"\tid: aea5f005-9792-442c-b98c-8d9b9e39f99b\n",
"\ttags: ['child:47ecf8a7-b799-4308-994b-aa6d8698dc2b', 'parent:f4a393ef-0b32-4f70-ac82-07a0877da328'] \n",
"\n",
"name: experiment_0\n",
"\tid: f4a393ef-0b32-4f70-ac82-07a0877da328\n",
"\ttags: ['child:aea5f005-9792-442c-b98c-8d9b9e39f99b'] \n",
"\n"
]
}
],
"source": [
"experiments = [last_experiment]\n",
"\n",
"while len(experiments) != 0:\n",
" experiment = experiments[0]\n",
"\n",
" print(\"name:\", experiment.name)\n",
" print(\"\\tid:\", experiment.id)\n",
" print(\"\\ttags:\", experiment.tags, \"\\n\")\n",
"\n",
" experiments = experiment.get_parent_experiments()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}