{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "\n", "# Comet.ml Python API\n", "\n", "Comet.ml has an extensive interface to all of your data using a [REST API](https://en.wikipedia.org/wiki/Representational_state_transfer) through [Comet.ml endpoints](https://www.comet.ml/docs/rest-api/getting-started/). Now, you can access this information easily through the Comet.ml Python SDK. Requires version comet_ml version 3.0.0 or greater." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Setup" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To install comet_ml:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%pip install comet_ml --quiet" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To run the following experiments, you'll need to set your COMET_API_KEY. The easiest way to to this is to use the `comet_ml.init()` function:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import comet_ml\n", "\n", "comet_ml.init()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Quick Overview\n", "\n", "To access the Python API through the Comet.ml SDK, you will need to make an API() instance. \n", "\n", "**Note: this is a new interface.**\n", "\n", "### What's new?\n", "\n", "The new API:\n", "\n", "* is faster, having more items cached\n", "* allows setting and logging more items\n", "* `APIExperiment` now works similarly to `Experiment`, `ExistingExperiment`, and `OfflineExperiment`\n", "* has a consistent interface\n", "\n", "Let's try it out! \n", "\n", "First, we import the API class and other libraries we will need. Note that this is a new interface and comes from `comet_ml.api`:" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "from comet_ml import API\n", "import comet_ml\n", "\n", "import matplotlib.pyplot as plt\n", "%matplotlib inline" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "and create the API instance:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "comet_api = API()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Using the `comet_api` instance, you can get the name of your workspaces:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['sampleworkspace', 'cometpublic']" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "comet_api.get()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you reference your workspace by name using `comet_api.get(WORKSPACE_NAME)`, you'll see your projects:" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['caffe2', 'ludwig', 'comet-sagemaker', 'fasttext', 'comet-notebooks', 'shap', 'parameter-space-exploration', 'comet-examples', 'fastai', '3d-histograms', 'pyspark-example', 'general', 'keras-example', 'home-credit']\n" ] } ], "source": [ "print(comet_api.get(\"cometpublic\"))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Or, get the projects from another user or shared workspace:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['mnist']" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "comet_api.get(\"testuser\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Using the same method, you can refer to a project by name and get all of the experiments in a project:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[,\n", " ,\n", " ,\n", " ,\n", " ,\n", " ,\n", " ,\n", " ,\n", " ,\n", " ,\n", " ,\n", " ,\n", " ,\n", " ,\n", " ,\n", " ,\n", " ,\n", " ,\n", " ,\n", " ,\n", " ]" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "comet_api.get(\"cometpublic\", \"fasttext\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Or, using the slash delimiter:" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[,\n", " ,\n", " ,\n", " ,\n", " ,\n", " ,\n", " ,\n", " ,\n", " ,\n", " ,\n", " ,\n", " ,\n", " ,\n", " ,\n", " ,\n", " ,\n", " ,\n", " ,\n", " ,\n", " ,\n", " ]" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "comet_api.get(\"cometpublic/fasttext\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And one more level, get an `APIExperiment` object using the Experiment's ID:" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "comet_api.get(\"cometpublic\", \"fasttext\", 'e64c5915920f481bab8f4cb4dbd615be')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Or, again using the slash shorthand:" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "comet_api.get(\"cometpublic/fasttext/e64c5915920f481bab8f4cb4dbd615be\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's get an experiment and save it to a variable named `exp`:" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "exp = comet_api.get(\"cometpublic/fasttext/e64c5915920f481bab8f4cb4dbd615be\")" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "exp" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There are a number of items you get and set from the APIExperiment instance. For a complete reference, see: https://www.comet.ml/docs/python-sdk/API/" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For example, we can explore the `other` property, which shows items saved with Experiment.log_other(NAME, VALUE):" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[{'name': 'Name',\n", " 'valueMax': 'last',\n", " 'valueMin': 'last',\n", " 'valueCurrent': 'last'},\n", " {'name': 'storage_size_bytes',\n", " 'valueMax': '0',\n", " 'valueMin': '0',\n", " 'valueCurrent': '0',\n", " 'timestampMax': 1574708415277,\n", " 'timestampMin': 1574708415277,\n", " 'timestampCurrent': 1574708415277},\n", " {'name': 'trainable_params',\n", " 'valueMax': '376560',\n", " 'valueMin': '376560',\n", " 'valueCurrent': '376560',\n", " 'timestampMax': 1529089066169,\n", " 'timestampMin': 1529089066169,\n", " 'timestampCurrent': 1529089066169}]" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "exp.get_others_summary()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this example, we see that the experiment has the `Name` \"last\". We can use `Name` to also look up experiments:" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "('e64c5915920f481bab8f4cb4dbd615be', 'last')" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "exp = comet_api.get(\"cometpublic/fasttext/last\")\n", "exp.id, exp.name" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Perhaps one of the most useful abilities for the Python API is to access your experiment's data in order to create a variation of a plot. To access the raw metric data, use the `.get_metrics()` method of the APIExperiment:" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "2804" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "len(exp.get_metrics())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We see here were over 2800 metrics logged during the training of this experiment. We can get the first using indexing with an integer:" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'metricName': 'val_acc',\n", " 'metricValue': '0.9832226913061168',\n", " 'timestamp': 1529089066713,\n", " 'step': 69,\n", " 'epoch': None,\n", " 'runContext': None,\n", " 'offset': 314}" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "exp.get_metrics()[0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "That shows that the \"acc\" (accuracy) metric had a value of about 0.5 at step 1 of the experiment.\n", "\n", "We can also filter on a single metric name, like so:" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [], "source": [ "acc_metrics = exp.get_metrics(\"acc\")" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "700" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "len(acc_metrics)" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'metricName': 'acc',\n", " 'metricValue': '0.5003255009651184',\n", " 'timestamp': 1529089066219,\n", " 'step': 1,\n", " 'epoch': None,\n", " 'runContext': None,\n", " 'offset': 29}" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "acc_metrics[0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Therefore, `exp.get_metrics(\"acc\")` gives us the dictionary for all \"acc\" items. We can then easily use Python's built in zip and matplotlib to plot these values:" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [], "source": [ "steps_acc = [(m[\"step\"], float(m[\"metricValue\"])) for m in acc_metrics]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This breaks up the data into (step, value) pairs:" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(1, 0.5003255009651184)" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "steps_acc[0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A little Python trick to separate the steps from the accuracies so we can easily use matplotlib:" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [], "source": [ "steps, acc = zip(*steps_acc[:100]) # just the first 100 for now" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD5CAYAAAA3Os7hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAe4UlEQVR4nO3de3Rc5Xnv8e8zMxpdbUm25ZtksA3mYsAGYwgpIRASuoAk0CY5p5C2adoE2pOS9JZzDjk5K01ZTbu6TlaTZh2aU9qkuTUQQpLGSUgot1xIAlg2xsZ3g7El+SLJ1l2a+3P+mLGrCBnJ9oy2Zs/vs5aWZ+95rXn2bOk3r9797r3N3RERkfIXCboAEREpDgW6iEhIKNBFREJCgS4iEhIKdBGRkFCgi4iERGyqBmb2ReAdQLe7XzrJ8wb8A3ArMAq83903T/V9FyxY4MuXLz/tgkVEKtmmTZt63b1lsuemDHTgS8D/Bb5yiudvAVYVvt4AfL7w7+tavnw57e3t03h5ERE5wcwOnOq5KYdc3P2nwPHXaXI78BXPexZoMrMlp1+miIicjWKMobcCHeOWOwvrRERkBs3oQVEzu9vM2s2svaenZyZfWkQk9IoR6F3AsnHLbYV1r+HuD7j7endf39Iy6Zi+iIicoWIE+gbgfZZ3DTDg7oeL8H1FROQ0TGfa4oPADcACM+sE/hKoAnD3/wc8Sn7K4j7y0xZ/v1TFiojIqU0Z6O5+5xTPO/DHRatIRETOyHTmoYuURDKTpW8kzcBYmv7RFEOJDIlMlkQ6RzaXY159NS1zqmmuq+L4SIqjg0l6h5PUVEVprK2isbaKeCxCxCBiRjKTZTiZZSSZIRoxmmqraKyrojoWJZvLkc462ZyTcyfnkMnmGE1lGU1lSWVzxKNGPBYhFomQzuZIZXKkc04sYsSjEWJRYziZoW8kRf9omoaaGK1NtSxtqmV+Q5yG6hj18RiRiL3udnvh9aNTtBM5XQp0KZrhZIbeoSTDyQwjyQwjqczJgO0eTNLRN0rH8VGODiY4NpxiKJkJuuSSiEcjWOFDJhY1qmNRaqry64YSGYYSGbI5Z359nEVza2iur2I0lWUokWE4kcHJ33TGMJrqqmiZU82ChmpS2RyDY/kPQHeoqYpQUxUlFjEiZpgZEct/UETMyOac0XSWsVSGVCZ3sj4HTtzXpqE6xv2/vY65NTH+z3/s5thwimzOyeTyDU58WA4lMhwbSXJsOEXEoLk+zry6ODVV0cIHZL59LJL/4MvmnIGxNINjaYaTGWKRCNFI/v0Y/zEWj+W3oToWJRrJ124Y6WyORCZHIp3FCu1OvK+ZXP6D2QsfilVRwx2ShfaZnFNbFaUuHqUqGmEklWEwkf+ZPHFDHzOjpaGa1uZaljbWMJzMcnQoQfdggmzOqYpGiMciVMciJx+7QyKdJZnJkfNCm2j+MORQMs3gWIZEOjtum/LbfGK/5Dz/YZ7NOX9w7QretnpR0X/2FOhyWkaSGV442E/EoKEmRiwS4bn9x3h8x1Ge33/8ZBBMZtHcapY113FpayMLGqpZ0BCnuT5OU22cxtoq5tTEqI1HqYlFiUTg+EiK7sEk/WNp5tVXsXBODS1zqkmmc/le/ViKzLhedzwWyfeSq2MnA2VgLE0yk80HTcSIRIyoWT70IkZdPEptVT5wUxknnc2RyeWIRfK/xFVRI5PzfG89m6Ohuorm+iqaauMMJdJ09Y/R1TdG32iakWSG4WSGVDb/C+8OqUyOZCZHMp0l586cmirm1ubft+6hJN2DCfpGUzRUx1jSWEN9PHay555zp280Tc9Qkv29I8SjERrrqmiuixMxSKRzDCczZLJ+8vVOhGvOwSC/ffEoTXVxbFySGnB8NM0vXznGyz3DVMci/NNPXmF+fZz66lghWCFb+H4N1VUsaIhzzrw6cg59IykODyRIZXMnQ98dMrkc2ZxjZsytraKxLk5rcy25XP658T8fJ96f4WSG3uEUuZN/PeXDsroQik6+s3DiQykWsZPv0fgPnxMhWhsxEukshwfSpLM56qtjNNZW0dpUgxXehFzO6R5K8szeXo4OJaitirJ4bv7nq6Y6SjKTYyiR4fiJv9Syuf98jaooEYNMNv9z4eT36/yGOLVVUVKZHIlMlmQ6d3JfuPvJYI+YkS3RneIU6DKl3uEkP9x2mMd3dvPsy8dIZXOvabNqYQN3vXkl57c0UF8do6E6RkNNjIbqKPXVMZoLvbnT0dZcV6xNKIn5DdXMb6hmTVtT0KWckWf29vI7X3gOgMMDCQC+9PtXc1lbY5BlzbhszokYJ8O+nCnQZVIDo2me2dfLtzd38uM9PWRzzvL5dfzuG8/lzRe0EI9GGElmGEtnubS1kRUL6oMuWc7C0cF8oC9qrA64kpkXpmMZCvQKls05n//xPp7a1U1TXZzmujjZXI6tnQO80jsC5IdJPnjdCt51RRsXLGoIRS9GXuvwQIJYxFhQX3mBHiYK9ArVN5LiT7+xhZ/s6WFtWyNHBxPsOjxIzuGytkbefWUbV57bzFXL54WqByOTOzqQYNHcmiln6MjspkCvQC91DfCHX91Ez1CST/3mpbz36nPU865wRwYTLJqr3nm5U6BXmG9t6uRj39nGgvo4D//RG7l8WXke0JPiOjKQ4OIlc4MuQ86SAr1CpLM5PvWDnXzpF69yzcp53P/edcxvUI9M8tMHjwwmuOHChUGXImdJgR5iL3b08/OXe9lxaJAtHf109o3xwTet4N5bLiIW1e1kJW8okWY0lWVxBc5wCRsFegiNJDP87Q938rVnDwLQ1lzL6iVz+fitF3PLZbqZlPyqI4Upi4sbawOuRM6WAj1knt9/nI9+80U6+kb54JtWcM+N59NUFw+6LJnFjhZOKlo8tybgSuRsKdBDIp3N8dkn9vCPP36ZZc11fOPuN3L1inlBlyVl4MRZoksaFejlToEeAgeOjfCRh7bwYkc/v7V+GZ9452rqq7VrZXpODLks1LTFsqff+jLm7jyyqZNPbthONGLc/951vH2Nxsjl9BwdTDCvPk517PSutSOzjwK9TPWNpPjYt7fxo+1HeMOKefz9b11Oa5MOasnpOzyQYNksvxCaTI8CvQx1HB/l3Z//BX2jKe695SLuum6lTs+XMzaUyLBY4+ehoEAvQ3/9gx0MJzN850PXcmlrZV3qVEpjkWa4hILOLikzv3i5l8e2H+VDN5ynMJei0QyXcFCgl5FMNsd939tBa1MtH7xuZdDlSIhoDno4KNDLyEMbO9h1ZIiPv/3i0777j8jrWaQeeihoDL0MJNJZtnUN8PeP7+HqFfO45dLFQZckIaMhl3BQoM9i2zoH+KvvbWdr5wCpbI6aqgh/+c7Vuna5FJ0OioaDAn2WGklm+NDXN5FM53j/tcu58txm1p/brEveStHVxaPMrVEUhIH24iz1N4/upLNvjIf/8I1ctVzXZJHSWTy3Rn/1hYQOis5CP93Tw789d5C7rlupMJeS03BLeCjQZ5mBsTT/81tbOX9hA39+0wVBlyMVQAdEw0NDLrPMpx/bTfdQkm//zpWamigzQlMWw0M99Fmk4/goD208yB1XLWOtbt4sM0QnFYWHAn0W+dyTezEzPnzjqqBLkQqiC3OFhwJ9lni5Z5hvbe7kd685V79gMiOWL6hj1cIG1rTpmkBhMa1AN7ObzWy3me0zs3snef5cM3vSzLaa2Y/NrK34pYbbZ5/YS01VlP92w3lBlyIVoq25jsf//HqW6ObQoTFloJtZFLgfuAVYDdxpZqsnNPs08BV3XwPcB/xtsQsNs52HB/nei4f4/WuXs0AnDonIGZpOD/1qYJ+7v+LuKeAh4PYJbVYDTxUePz3J83IKx4aTfPSbLzKnJsbd16l3LiJnbjqB3gp0jFvuLKwb70XgXYXHvwnMMbP5Z19euHX1j/Ff/umX7Ose5nN3XEFjXVXQJYlIGSvWQdGPAteb2QvA9UAXkJ3YyMzuNrN2M2vv6ekp0kuXp33dw7zn87+gZzDJVz/wBt5y0cKgSxKRMjedE4u6gGXjltsK605y90MUeuhm1gC82937J34jd38AeABg/fr1foY1lz13556vbyadzfHQH17DJUs1y0BEzt50eugbgVVmtsLM4sAdwIbxDcxsgZmd+F4fA75Y3DLDZfPBfnYdGeKjv36hwlxEimbKQHf3DHAP8BiwE3jY3beb2X1mdluh2Q3AbjPbAywCPlWiekPhwecPUh+P8s61S4MuRURCZFrXcnH3R4FHJ6z7xLjHjwCPFLe0cBoYS/P9rYd417o26qt1KR0RKR6dKTrDvruli0Q6x3uvPifoUkQkZBToM8jd+fpzB7mstZFLWzV2LiLFpUCfQVs68gdD71TvXERKQIE+gx58/iB18Si3Xa6DoSJSfAr0GdJxfJR/f+EQv3lFKw06GCoiJaBAnyGf/o/dRCJwz43nB12KiISUAn0GbO3s57tbDvGBN63QpUpFpGQU6CXm7nzqBzuZXx/nj67X1RRFpHQU6CX25M5untt/nD952yrm1OhqiiJSOgr0EsrlnL/70S5WLqjXVEURKTkFegn9/OVe9nYP8+G3nk9VVG+1iJSWUqaEvv7cQZrrqrj1siVBlyIiFUCBXiLdQwke33GU91zZRnUsGnQ5IlIBFOgl8s32TjI519i5iMwYBXoJZHPOg88f5NfOm8/KloagyxGRCqFAL4Gf7u2hs2+M975BvXMRmTkK9BL4+nMHWdAQ59dXLw66FBGpIAr0IuseSvDUrm7ec+Uy4jG9vSIyc5Q4RfbDbUfI5px3rWsNuhQRqTAK9CL7/tZDXLCogQsWzQm6FBGpMAr0Ijo8MMbGV/t4xxrdwEJEZp4CvYh+sPUwAO9YozNDRWTmKdCL6PtbD7N6yVzNPReRQCjQi6Tj+ChbOvp5x1r1zkUkGAr0Inl0W2G45TKNn4tIMBToRfL9rYdZ29bIOfPrgi5FRCqUAr0IXu4ZZlvXgGa3iEigFOhF8K8/3088GuH2KxToIhIcBfpZ6htJ8cimTn7jiqUsnFMTdDkiUsEU6Gfpa88eIJHO8cHrVgZdiohUOAX6WUiks3z5l69y/QUtOtVfRAKnQD8L393SRe9wirvUOxeRWWBagW5mN5vZbjPbZ2b3TvL8OWb2tJm9YGZbzezW4pc6u7g7//Kz/Vy0eA7Xnj8/6HJERKYOdDOLAvcDtwCrgTvNbPWEZv8beNjdrwDuAP6x2IXONj/Z08Pe7mHuum4lZhZ0OSIi0+qhXw3sc/dX3D0FPATcPqGNA3MLjxuBQ8UrcXb6wjP7WTinmneu1VRFEZkdphPorUDHuOXOwrrxPgn8jpl1Ao8CHy5KdbPU7iND/GxvL7/3a8t1VyIRmTWKlUZ3Al9y9zbgVuCrZvaa721md5tZu5m19/T0FOmlZ94Xn9lPTVWE916tm0CLyOwxnUDvApaNW24rrBvvA8DDAO7+S6AGWDDxG7n7A+6+3t3Xt7S0nFnFAesdTvKdLV28e10bzfXxoMsRETlpOoG+EVhlZivMLE7+oOeGCW0OAm8FMLOLyQd6+XbBX8fXnj1AKpPjD960IuhSRER+xZSB7u4Z4B7gMWAn+dks283sPjO7rdDsL4C7zOxF4EHg/e7upSo6KIl0lq89e4C3XNjCebqJhYjMMrHpNHL3R8kf7By/7hPjHu8Ari1uabPPj146Qu9wig+8SScSicjsoykap+GJnUdpmVOtE4lEZFZSoE9TNuf8bG8vb17VohOJRGRWUqBP04ud/QyMpbnhwvKcnSMi4adAn6Yf7+4hYnDdqtfMxhQRmRUU6NP0kz09XL6siaY6zT0XkdlJgT4Nx4aTbO3s5/oLFgZdiojIKSnQp+GZfb24w/UaPxeRWUyBPg0/2d3DvPo4a1obgy5FROSUFOhTyOWcn+zp4bpVC4hENF1RRGYvBfoUth8a5NhIStMVRWTWU6BP4end3QBct0qBLiKzmwJ9Co9tP8K6c5pY0FAddCkiIq9Lgf46Dh4bZfuhQW65dEnQpYiITEmB/joe234EgJsvXRxwJSIiU1Ogv44fvnSYS5bOZdm8uqBLERGZkgL9FI4OJth8sJ+bL1HvXETKgwL9FP6jMNxyy2UKdBEpDwr0U/jhS0c4r6We8xfOCboUEZFpUaBP4vhIiuf2H9fBUBEpKwr0STyx8yjZnGu6ooiUFQX6JH68u5sljTVcsnRu0KWIiEybAn0Smw/0c9Xyebp3qIiUFQX6BIf6xzgymGDdOU1BlyIicloU6BNsPtgHwLpzmwOuRETk9CjQJ9h8oJ/qWISLFmv8XETKiwJ9ghc6+ljT1kg8prdGRMqLUmucZCbL9q5B1p2j4RYRKT8K9HFe6hoklc1xhQJdRMqQAn2cF04eENUMFxEpPwr0cTYf7KOtuZaFc2qCLkVE5LQp0Md54WC/xs9FpGwp0AsOD4xxeCDBFTqhSETK1LQC3cxuNrPdZrbPzO6d5PnPmNmWwtceM+svfqmltflAvmT10EWkXMWmamBmUeB+4CagE9hoZhvcfceJNu7+Z+Pafxi4ogS1ltTmg31UxyJcvEQnFIlIeZpOD/1qYJ+7v+LuKeAh4PbXaX8n8GAxipsp/aMpvrvlEFctn6cTikSkbE0nvVqBjnHLnYV1r2Fm5wIrgKdO8fzdZtZuZu09PT2nW2vJ/PUPdtI3muJjt14UdCkiImes2N3RO4BH3D072ZPu/oC7r3f39S0tLUV+6TPz0z09PLKpkz+6fiWXLG0MuhwRkTM2nUDvApaNW24rrJvMHZTRcMtIMsP/+s42VrbU8+EbVwVdjojIWZlOoG8EVpnZCjOLkw/tDRMbmdlFQDPwy+KWWDqfeXwPnX1j/N2711BTFQ26HBGRszJloLt7BrgHeAzYCTzs7tvN7D4zu21c0zuAh9zdS1Nqcbk733mhi3esWcJVy+cFXY6IyFmbctoigLs/Cjw6Yd0nJix/snhlld6hgQTHRlK8YeX8oEsRESmKip2jt60zfyLRmlYdCBWRcKjYQN/aOUBV1LhoyZygSxERKYqKDfRtXQNcuHgO1TEdDBWRcKjIQHd3tnYOcFmrLsQlIuFRkYHecXyMgbE0a9o0fi4i4VGRgf5i4YDoZTogKiIhUpGBvq1rgHgswgWLdEBURMKjIgN9a2c/Fy+ZqysrikioVFyi5XLOS12Dmn8uIqFTcYG+/9gIw8kMl+mAqIiETMUF+rbOAQDNcBGR0Km4QN/aOUBNVYTzWxqCLkVEpKgqLtC3dfVzydJGYtGK23QRCbmKSrVs4YCo5p+LSBhVVKDv7R5iLJ3l8mU65V9EwqeiAv3FjsIlc3VAVERCqKICfUvHAHNrYiyfXx90KSIiRVdRgf5iRz9rlzURiVjQpYiIFF3FBPpYKsvuo0OsbdP4uYiEU8UE+o7DA2RzzlodEBWRkKqYQN/SkT9DdK0OiIpISFVMoL/Y0c/SxhoWzq0JuhQRkZKonEDv7GeNxs9FJMQqItD7RlIcODaq8XMRCbWKCPQTt5xbu0zj5yISXpUR6B0DmOkeoiISbpUR6J39nN/SwJyaqqBLEREpmdAHuruztbNf4+ciEnqhD/Q9R4fpHU5x5bnNQZciIlJSoQ/0J3YeBeDGixYGXImISGmFPtAf33GUtW2NLNIJRSIScqEO9O7BBFs6+rlp9aKgSxERKblpBbqZ3Wxmu81sn5nde4o2/9XMdpjZdjP7enHLPDNP7uoG4G0KdBGpALGpGphZFLgfuAnoBDaa2QZ33zGuzSrgY8C17t5nZrNiwPqJHUdpa67lwkVzgi5FRKTkptNDvxrY5+6vuHsKeAi4fUKbu4D73b0PwN27i1vm6RtNZXhmXy83rV6EmW5oISLhN51AbwU6xi13FtaNdwFwgZn93MyeNbObJ/tGZna3mbWbWXtPT8+ZVTxNz+ztJZnJcdPFGm4RkcpQrIOiMWAVcANwJ/DPZvaaM3nc/QF3X+/u61taWor00pN7fMdR5tbEuGrFvJK+jojIbDGdQO8Clo1bbiusG68T2ODuaXffD+whH/CByOacp3Z185aLFlIVDfVEHhGRk6aTdhuBVWa2wsziwB3Ahglt/p187xwzW0B+COaVItZ5WnYeHuTYSEonE4lIRZky0N09A9wDPAbsBB529+1mdp+Z3VZo9hhwzMx2AE8D/93dj5Wq6KlsOtAHwFXLNdwiIpVjymmLAO7+KPDohHWfGPfYgT8vfAVu04E+ljTWsLSpNuhSRERmTCgHmDcd6GOdLsYlIhUmdIF+ZCBBV/8YV56jQBeRyhK6QN98MD9+rsvlikilCV2gbzrQR01VhNVL5wZdiojIjAploK9pbdL8cxGpOKFKvUQ6y/ZDAzogKiIVKVSBvq1rgHTWNX4uIhUpVIF+4oSidefohtAiUnlCF+grFtQzv6E66FJERGZcaALd3XnhYB/rNP9cRCpUaAL9wLFReodTGj8XkYoVmkB/Zl8vAFevUKCLSGUKTaA/vaubZfNqOa+lIehSREQCEYpAT6Sz/PzlXm68cKHuHyoiFSsUgf7sK8dIpHPcoBtaiEgFC0WgP72rm5qqCG9cOT/oUkREAlP2ge7uPLW7m2vPW0BNVTTockREAlP2gf5yzzAdx8d4i4ZbRKTClX2gP7WrG0CBLiIVr+wD/eldPVy4aA6tun+oiFS4sg70wUSaja8eV+9cRIQyD/Rf7Oslk3PecmFL0KWIiASurAP9uf3HqamKcIUuyCUiUt6B3v5qH5cvayIeK+vNEBEpirJNwuFkhu2HBrhq+bygSxERmRXKNtC3HOwn57BegS4iApRxoD//6nEiptvNiYicULaB3v7qcS5eMpc5NVVBlyIiMiuUZaCnszleONiv8XMRkXHKMtB3HBpkLJ1l/XJNVxQROaEsA33jq8cB1EMXERlnWoFuZjeb2W4z22dm907y/PvNrMfMthS+Plj8Uv9T+6t9nDOvjkVza0r5MiIiZSU2VQMziwL3AzcBncBGM9vg7jsmNP2Gu99Tghp/hbuz8dXjXK/T/UVEfsV0euhXA/vc/RV3TwEPAbeXtqxT2987wrGRlIZbREQmmE6gtwId45Y7C+smereZbTWzR8xsWVGqm0T7q30AXKUDoiIiv6JYB0W/Byx39zXA48CXJ2tkZnebWbuZtff09JzRCzXVVXHT6kWc19Jw5tWKiITQdAK9Cxjf424rrDvJ3Y+5e7Kw+C/AlZN9I3d/wN3Xu/v6lpYzGwP/9UsW88/vW4+ZndH/FxEJq+kE+kZglZmtMLM4cAewYXwDM1sybvE2YGfxShQRkemYcpaLu2fM7B7gMSAKfNHdt5vZfUC7u28APmJmtwEZ4Djw/hLWLCIikzB3D+SF169f7+3t7YG8tohIuTKzTe6+frLnyvJMUREReS0FuohISCjQRURCQoEuIhISCnQRkZAIbJaLmfUAB07jvywAektUzmym7a48lbrt2u7pOdfdJz0zM7BAP11m1n6qqTphpu2uPJW67drus6chFxGRkFCgi4iERDkF+gNBFxAQbXflqdRt13afpbIZQxcRkddXTj10ERF5HWUR6FPdpDoszGyZmT1tZjvMbLuZ/Ulh/Twze9zM9hb+DeXtmswsamYvmNn3C8srzOy5wn7/RuHyzaFiZk2Fu3ztMrOdZvbGStjfZvZnhZ/xl8zsQTOrCeP+NrMvmlm3mb00bt2k+9fyPlfY/q1mtu50X2/WB/q4m1TfAqwG7jSz1cFWVTIZ4C/cfTVwDfDHhW29F3jS3VcBTxaWw+hP+NVr6f8d8Bl3Px/oAz4QSFWl9Q/Aj9z9ImAt+e0P9f42s1bgI8B6d7+U/GW57yCc+/tLwM0T1p1q/94CrCp83Q18/nRfbNYHOrPsJtWl5O6H3X1z4fEQ+V/uVvLbe+K2fl8GfiOYCkvHzNqAt5O/4xWWvyXVjcAjhSah224zawTeDHwBwN1T7t5PBexv8vdiqDWzGFAHHCaE+9vdf0r+HhHjnWr/3g58xfOeBZom3DxoSuUQ6NO9SXWomNly4ArgOWCRux8uPHUEWBRQWaX0WeB/ALnC8nyg390zheUw7vcVQA/wr4Whpn8xs3pCvr/dvQv4NHCQfJAPAJsI//4+4VT796yzrhwCveKYWQPwLeBP3X1w/HOen5YUqqlJZvYOoNvdNwVdywyLAeuAz7v7FcAIE4ZXQrq/m8n3RlcAS4F6XjssURGKvX/LIdCnvEl1mJhZFfkw/zd3/3Zh9dETf3oV/u0Oqr4SuRa4zcxeJT+kdiP5seWmwp/kEM793gl0uvtzheVHyAd82Pf324D97t7j7mng2+R/BsK+v0841f4966wrh0Cf8ibVYVEYN/4CsNPd/37cUxuA3ys8/j3guzNdWym5+8fcvc3dl5Pfv0+5+28DTwPvKTQL43YfATrM7MLCqrcCOwj5/iY/1HKNmdUVfuZPbHeo9/c4p9q/G4D3FWa7XAMMjBuamR53n/VfwK3AHuBl4ONB11PC7XwT+T+/tgJbCl+3kh9PfhLYCzwBzAu61hK+BzcA3y88Xgk8D+wDvglUB11fCbb3cqC9sM//HWiuhP0N/BWwC3gJ+CpQHcb9DTxI/jhBmvxfZB841f4FjPyMvpeBbeRnAZ3W6+lMURGRkCiHIRcREZkGBbqISEgo0EVEQkKBLiISEgp0EZGQUKCLiISEAl1EJCQU6CIiIfH/AYDU7d+v19hvAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "plt.plot(steps, acc);" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "That's it for a quick overview. Now let's look in detail at each component." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Workspaces\n", "\n", "`comet_api.get()` reports your workspace names:" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['sampleworkspace', 'cometpublic']" ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "comet_api.get()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can also interate over those names:" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "sampleworkspace\n", "cometpublic\n" ] } ], "source": [ "for workspace in comet_api.get():\n", " print(workspace)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As we saw above, you can also access other public workspaces as well:" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['mnist']" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "comet_api.get(\"testuser\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Projects\n", "\n", "Under get(WORKSPACE_NAME), you'll find the projects:" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['caffe2',\n", " 'ludwig',\n", " 'comet-sagemaker',\n", " 'fasttext',\n", " 'comet-notebooks',\n", " 'shap',\n", " 'parameter-space-exploration',\n", " 'comet-examples',\n", " 'fastai',\n", " '3d-histograms',\n", " 'pyspark-example',\n", " 'general',\n", " 'keras-example',\n", " 'home-credit']" ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "comet_api.get(\"cometpublic\")" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [], "source": [ "project = comet_api.get(\"cometpublic\", \"comet-notebooks\")\n", "## OR:\n", "#project = comet_api.get(\"cometpublic/comet-notebooks\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you just print out, or iterate over a project, you get access to the experiments:" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[,\n", " ]" ] }, "execution_count": 28, "metadata": {}, "output_type": "execute_result" } ], "source": [ "project" ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "('7092a5e4c362453fb0b3f06785a1d30c', None)" ] }, "execution_count": 29, "metadata": {}, "output_type": "execute_result" } ], "source": [ "project[0].id, project[0].name" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "('d21f94a1c71841d2961da1e6ddb5ab20', 'example 001')" ] }, "execution_count": 30, "metadata": {}, "output_type": "execute_result" } ], "source": [ "project[1].id, project[1].name" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Experiments\n", "\n", "Continuing with the dictionary-like access, you can see and iterate over the experiment ids:" ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[,\n", " ]" ] }, "execution_count": 31, "metadata": {}, "output_type": "execute_result" } ], "source": [ "comet_api.get(\"cometpublic\", \"comet-notebooks\")" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 32, "metadata": {}, "output_type": "execute_result" } ], "source": [ "exp = comet_api.get(\"cometpublic\", \"comet-notebooks\", 'd21f94a1c71841d2961da1e6ddb5ab20')\n", "## OR\n", "# exp = comet_api.get(\"cometpublic/comet-notebooks/d21f94a1c71841d2961da1e6ddb5ab20\")\n", "exp" ] }, { "cell_type": "code", "execution_count": 33, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 33, "metadata": {}, "output_type": "execute_result" } ], "source": [ "exp = comet_api.get(\"cometpublic\", \"comet-notebooks\", 'example 001')\n", "## OR\n", "## exp = comet_api.get(\"cometpublic/comet-notebooks/example 001\")\n", "exp" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Regular Expression Experiment Name Matching\n", "\n", "You can also use regular expressions as the name for the experiment:" ] }, { "cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[]" ] }, "execution_count": 34, "metadata": {}, "output_type": "execute_result" } ], "source": [ "comet_api.get_experiments(\"cometpublic\", \"comet-notebooks\", \"example.*\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Query API\n", "\n", "The Python API provides programmatic access to the same query system in the web UI. There are 5 types of items that you can query:\n", "\n", "* Metric: items logged with `log_metric()`\n", "* Metadata: items logged by the system\n", "* Other: items logged with `log_other()`\n", "* Parameter: items logged with `log_parameter()`\n", "* Tag: items logged with `add_tags()`\n", "\n", "To use these, you first need to import them:" ] }, { "cell_type": "code", "execution_count": 36, "metadata": {}, "outputs": [], "source": [ "from comet_ml.api import Metric, Metadata, Other, Parameter, Tag" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can then use these to build a query expression, like so:" ] }, { "cell_type": "code", "execution_count": 37, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[,\n", " ]" ] }, "execution_count": 37, "metadata": {}, "output_type": "execute_result" } ], "source": [ "comet_api.query(\"cometpublic\", \"general\", Parameter(\"max_iter\") == 100)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The API.query() method takes the following args:\n", "\n", "* workspace: String, the name of the workspace\n", "* project_name: String, the name of the project\n", "* query: a query expression (see below)\n", "* archived: (optional boolean), query the archived experiments if True\n", "\n", "A query is of the form:\n", "\n", "```python\n", "((QUERY-VARIABLE OPERATOR VALUE) & ...)\n", "\n", "# or:\n", "\n", "(QUERY-VARIABLE.METHOD(VALUE) & ...)\n", "```\n", "\n", "where:\n", "\n", "* `QUERY-VARIABLE` is `Metric(NAME)`, `Parameter(NAME)`, `Other(NAME)`, `Metadata(NAME)`, or `Tag(VALUE)`.\n", "* `OPERATOR` is any of the standard mathematical operators `==`, `<=`, `>=`, `!=`, `<`, `>`.\n", "* `METHOD` is `between()`, `contains()`, `startswith()`, or `endswith()`.\n", "\n", "You may also place the bitwise `~` not operator in front of an expression\n", "which means to invert the expression. Use `&` to combine additional\n", "criteria. Currently, `|` (bitwise or) is not supported.\n", "\n", "`VALUE` can be any query type, includeing `string`, `boolean`, `double`,\n", "`datetime`, or `timenumber` (number of seconds). `None` and `\"\"` are special\n", "values that mean `NULL` and `EMPTY`, respectively. Use\n", "`API.get_project_variables(WORKSPACE, PROJECT_NAME)` to see query variables\n", "and types for a project.\n", "\n", "When using `datetime`, be aware that the backend is using UTC datetimes. If you\n", "do not receive the correct experiments via a datetime query, please check with\n", "the web UI query builder to verify timezone of the server.\n", "\n", "`query()` returns a list of matching `APIExperiments()`.\n", "\n", "Examples:\n", "\n", "```python\n", "# Find all experiments that have an acc metric value > .98:\n", ">>> api.query(\"workspace\", \"project\", Metric(\"acc\") > .98)\n", "[APIExperiment(), ...]\n", "\n", "# Find all experiments that have a loss metric < .1 and\n", "# a learning_rate parameter value >= 0.3:\n", ">>> loss = Metric(\"loss\")\n", ">>> lr = Parameter(\"learning_rate\")\n", ">>> query = ((loss < .1) & (lr >= 0.3))\n", ">>> api.query(\"workspace\", \"project\", query)\n", "[APIExperiment(), ...]\n", "\n", "# Find all of the experiments tagged \"My simple tag\":\n", ">>> tagged = Metric(\"My simple tag\")\n", ">>> api.query(\"workspace\", \"project\", tagged)\n", "[APIExperiment(), ...]\n", "\n", "# Find all experiments started before Sept 24, 2019 at 5:00am:\n", ">>> q = Metadata(\"start_server_timestamp\") < datetime(2019, 9, 24, 5)\n", ">>> api.query(\"workspace\", \"project\", q)\n", "[APIExperiment(), ...]\n", "\n", "# Find all experiments lasting more that 2 minutes (in seconds):\n", ">>> q = Metadata(\"duration\") > (2 * 60)\n", ">>> api.query(\"workspace\", \"project\", q)\n", "[APIExperiment(), ...]\n", "```\n", "\n", "Notes:\n", "\n", "* Use `~` for `not` on any expression\n", "* Use `~QUERY-VARIABLE.between(2,3)` for values not between 2 and 3\n", "* Use `(QUERY-VARIABLE == True)` for truth\n", "* Use `(QUERY-VARIABLE == False)` for not true\n", "* Use `(QUERY-VARIABLE == None)` for testing null\n", "* Use `(QUERY-VARIABLE != None)` or `~(QUERY-VARIABLE == None)` for testing not null\n", "* Use `(QUERY-VARIABLE == \"\")` for testing empty\n", "* Use `(QUERY-VARIABLE != \"\")` or `~(QUERY-VARIABLE == \"\")` for testing not empty\n", "* Use Python's datetime(YEAR, MONTH, DAY, HOUR, MINUTE, SECONDS) for comparing datetimes, like\n", " `Metadata(\"start_server_timestamp\")` or `Metadata(\"end_server_timestamp\")`\n", "* Use seconds for comparing timenumbers, like `Metadata(\"duration\")`\n", "* Use `API.get_project_variables(WORKSPACE, PROJECT_NAME)` to see query variables\n", " and types.\n", "\n", "Do not use 'and', 'or', 'not', 'is', or 'in'. These are logical operators and you must use mathematical operators for queries. For example, always use '==' where you might usually use 'is'.\n", "\n", "How can you know what query variables are available for a project? Use API.get_query_variables(WORKSPACE, PROJECT_NAME):" ] }, { "cell_type": "code", "execution_count": 38, "metadata": {}, "outputs": [], "source": [ "vars = comet_api.get_query_variables(\"cometpublic\", \"general\")" ] }, { "cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[Metadata('hasNotes'),\n", " Metadata('duration'),\n", " Metric('sys.cpu.percent.04'),\n", " Parameter('warm_start'),\n", " Metric('sys.cpu.percent.avg'),\n", " Parameter('learning_rate'),\n", " Metric('train_reg_mean_loss'),\n", " Metadata('file_name'),\n", " Metric('sys.ram.total'),\n", " Parameter('beta2'),\n", " Metadata('experimentKey'),\n", " Parameter('run_arg_2'),\n", " Metric('sys.ram.used'),\n", " Parameter('intercept_scaling'),\n", " Metadata('end_server_timestamp'),\n", " Metric('sys.cpu.percent.08'),\n", " Metadata('file_path'),\n", " Metadata('runActive'),\n", " Parameter('C'),\n", " Metric('class/train_batch_mean_hits_at_k_class'),\n", " Parameter('run_arg_4'),\n", " Parameter('max_iter'),\n", " Metadata('runId'),\n", " Metric('sys.cpu.percent.03'),\n", " Parameter('curr_step'),\n", " Parameter('class_weight'),\n", " Parameter('multi_class'),\n", " Metric('sys.cpu.percent.01'),\n", " Parameter('fit_intercept'),\n", " Metadata('optimizationId'),\n", " Parameter('run_arg_3'),\n", " Metric('sys.cpu.percent.02'),\n", " Parameter('penalty'),\n", " Parameter('beta1'),\n", " Metadata('codeSHA'),\n", " Parameter('epsilon'),\n", " Parameter('solver'),\n", " Metric('class/train_mean_loss_class'),\n", " Metadata('installed_packages'),\n", " Metadata('last_message_time'),\n", " Metadata('hasImages'),\n", " Metadata('user_name'),\n", " Metadata('start_server_timestamp'),\n", " Parameter('tol'),\n", " Parameter('Optimizer'),\n", " Other('throttled'),\n", " Parameter('verbose'),\n", " Metric('sys.cpu.percent.05'),\n", " Metric('class/train_batch_accuracy_class'),\n", " Parameter('run_arg_1'),\n", " Metadata('hasHtml'),\n", " Parameter('dual'),\n", " Tag('ludwig'),\n", " Metric('sys.cpu.percent.07'),\n", " Parameter('n_jobs'),\n", " Parameter('run_arg_0'),\n", " Other('storage_size_bytes'),\n", " Metric('sys.cpu.percent.06'),\n", " Parameter('random_state'),\n", " Metric('sys.load.avg')]" ] }, "execution_count": 39, "metadata": {}, "output_type": "execute_result" } ], "source": [ "vars" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Experiment Properties" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this brief dictionary representation, you will see that `get_others_summary()`, `get_metrics_summary()` and `get_parameters_summary()` give summary data for each item:" ] }, { "cell_type": "code", "execution_count": 40, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[{'name': 'f',\n", " 'valueMax': '/run/user/1000/jupyter/kernel-b1c4403c-c673-4e7f-90dc-853616d231e2.json',\n", " 'valueMin': '/run/user/1000/jupyter/kernel-b1c4403c-c673-4e7f-90dc-853616d231e2.json',\n", " 'valueCurrent': '/run/user/1000/jupyter/kernel-b1c4403c-c673-4e7f-90dc-853616d231e2.json',\n", " 'timestampMax': 1542823692798,\n", " 'timestampMin': 1542823692798,\n", " 'timestampCurrent': 1542823692798}]" ] }, "execution_count": 40, "metadata": {}, "output_type": "execute_result" } ], "source": [ "exp.get_parameters_summary()" ] }, { "cell_type": "code", "execution_count": 41, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "('Name', 'example 001')" ] }, "execution_count": 41, "metadata": {}, "output_type": "execute_result" } ], "source": [ "exp.get_others_summary()[0][\"name\"], exp.get_others_summary()[0][\"valueCurrent\"]" ] }, { "cell_type": "code", "execution_count": 42, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'name': 'train_loss',\n", " 'valueMax': '2.3872790336608887',\n", " 'valueMin': '2.3872790336608887',\n", " 'valueCurrent': '2.3872790336608887',\n", " 'timestampMax': 1542823695820,\n", " 'timestampMin': 1542823695820,\n", " 'timestampCurrent': 1542823695820,\n", " 'runContextMax': 'train',\n", " 'runContextMin': 'train',\n", " 'runContextCurrent': 'train',\n", " 'stepMax': 0,\n", " 'stepMin': 0,\n", " 'stepCurrent': 0}" ] }, "execution_count": 42, "metadata": {}, "output_type": "execute_result" } ], "source": [ "exp.get_metrics_summary(\"train_loss\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can see more information on the methods and propeties on an APIExperiment instance here: https://www.comet.ml/docs/python-sdk/APIExperiment/" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Just like when creating and logging data, you can also use the `.display()` method to show the Comet.ml page for that experiment right in the notebook:" ] }, { "cell_type": "code", "execution_count": 43, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "exp.display()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Logging data\n", "\n", "With this version of the API, you can now easily log, set, and add data to an experiment, either one that you just created, or one that existed previously.\n", "\n", "To create a new APIExperiment, you can (assuming that your API key, workspace, and project name have been set in a Comet configure file):\n", "\n", "```python\n", "from comet_ml.api import APIExperiment\n", "api_experiment = APIExperiment()\n", "```\n", "\n", "If you have not set your API key, workspace, and project name, you can pass any and all to the constructor:\n", "\n", "```python\n", "from comet_ml.api import APIExperiment\n", "api_experiment = APIExperiment(api_key=\"MY-KEY\", \n", " workspace=\"MY-WORKSPACE\",\n", " project_name=\"MY-PROJECT\")\n", "```\n", "\n", "To create an APIExperiment from a previously-made experiment, you can (assuming that your keys have been set):\n", "\n", "```python\n", "from comet_ml.api import APIExperiment\n", "api_experiment = APIExperiment(previous_experiment=\"7364746746743746\") # previous experiment key or name\n", "```\n", "\n", "Once you have an APIExperiment, you can log, set, and add items to the experiment. You can use any of the following methods:\n", "\n", "* api_experiment.log_output(lines, context=None, stderr=False)\n", "* api_experiment.log_other(key, value)\n", "* api_experiment.log_parameter(parameter, value, step=None)\n", "* api_experiment.log_metric(metric, value, step=None)\n", "* api_experiment.log_html(html, overwrite=False)\n", "* api_experiment.log_asset(filename, step=None, overwrite=None, context=None)\n", "* api_experiment.log_image(filename, image_name=None, step=None, overwrite=None, context=None)\n", "* api_experiment.add_tags(tags)\n", "* api_experiment.set_code(code)\n", "* api_experiment.set_model_graph(graph)\n", "* api_experiment.set_os_packages(os_packages)\n", "* api_experiment.set_start_time(start_server_timestamp)\n", "* api_experiment.set_end_time(end_server_timestamp)\n", "\n", "### Existing Experiments\n", "\n", "For many aspects of an experiment, you can log them with the Python API. However, to get the full power of the streaming `Experiment` interface, you can create an `ExistingExperiment`.\n", "\n", "First, we can look up an experiment using any of the methods outlined here:" ] }, { "cell_type": "code", "execution_count": 44, "metadata": {}, "outputs": [], "source": [ "api_experiment = comet_api.get(\"cometpublic\", \"comet-notebooks\", 'example 001')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Then, we can use the APIExperiment.id property to make an `ExistingExperiment`:" ] }, { "cell_type": "code", "execution_count": 45, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "COMET INFO: Experiment is live on comet.ml https://www.comet.ml/cometpublic/comet-notebooks/d21f94a1c71841d2961da1e6ddb5ab20\n", "\n" ] } ], "source": [ "existing_experiment = comet_ml.ExistingExperiment(previous_experiment=api_experiment.id)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can make changes to the saved data using the existing experiment:\n", "\n", "https://www.comet.ml/docs/python-sdk/ExistingExperiment/" ] }, { "cell_type": "code", "execution_count": 46, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "COMET INFO: ----------------------------\n", "COMET INFO: Comet.ml Experiment Summary:\n", "COMET INFO: Data:\n", "COMET INFO: url: https://www.comet.ml/cometpublic/comet-notebooks/d21f94a1c71841d2961da1e6ddb5ab20\n", "COMET INFO: ----------------------------\n", "COMET INFO: Uploading stats to Comet before program termination (may take several seconds)\n" ] } ], "source": [ "existing_experiment.end()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Examples\n", "\n", "As seen above, you can use the Query API that allows highly efficient queries of your data. However, you can also write your own query of sorts if you need to go beyond what the Query API provides.\n", "\n", "Here is some code that prints out the names of experiments that have associated HTML (this can take a long time if you have many experiments):" ] }, { "cell_type": "code", "execution_count": 47, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " processing project release-2-0-12c ...\n", " processing experiment d21f94a1c71841d2961da1e6ddb5ab20.\n", "Found html in https://www.comet.ml/dsblank/release-2-0-12c/0f22eef674174082b8bcb2b78b116343!\n", "\n", "CPU times: user 45.1 ms, sys: 1.01 ms, total: 46.1 ms\n", "Wall time: 1 s\n" ] } ], "source": [ "%%time\n", "workspace = \"dsblank\"\n", "found = False\n", "for project in comet_api.get(workspace):\n", " if found:\n", " break\n", " print(\" processing project\", project, \"...\")\n", " print(\" processing experiment\", exp.id, end=\"\")\n", " for exp in comet_api.get(workspace, project):\n", " print(\".\", end=\"\")\n", " if exp.get_html() != None:\n", " print(\"\\nFound html in %s!\" % exp.url)\n", " found = True\n", " break\n", " print()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here is a function that will find the first experiment that has associated images:" ] }, { "cell_type": "code", "execution_count": 48, "metadata": {}, "outputs": [], "source": [ "def find_image():\n", " for workspace in comet_api.get():\n", " for project in comet_api.get(workspace):\n", " for exp in comet_api.get(workspace, project):\n", " if exp.get_asset_list(asset_type=\"image\") != []:\n", " return exp" ] }, { "cell_type": "code", "execution_count": 49, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 49, "metadata": {}, "output_type": "execute_result" } ], "source": [ "find_image()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, we get the experiment API and explore the `APIExperiment.get_asset_list()` method:" ] }, { "cell_type": "code", "execution_count": 50, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[{'fileName': 'Figure(640x480).svg',\n", " 'fileSize': 35241,\n", " 'runContext': None,\n", " 'step': None,\n", " 'link': 'https://s3.amazonaws.com/comet.ml/yLB5azBCky96p5Q5IJJJA4est.svg',\n", " 'compressedAssetLink': None,\n", " 'createdAt': 1555703549582,\n", " 'dir': 'media',\n", " 'canView': True,\n", " 'audio': False,\n", " 'video': False,\n", " 'histogram': False,\n", " 'image': True,\n", " 'type': 'image',\n", " 'metadata': None,\n", " 'assetId': '5b6a596de48041c08c3395d80cc436f6'},\n", " {'fileName': 'Figure(640x480) (1).svg',\n", " 'fileSize': 34144,\n", " 'runContext': None,\n", " 'step': None,\n", " 'link': 'https://s3.amazonaws.com/comet.ml/kkn1Ca1jRnYCFa8SRCy2RMGUT.svg',\n", " 'compressedAssetLink': None,\n", " 'createdAt': 1555703600815,\n", " 'dir': 'media',\n", " 'canView': True,\n", " 'audio': False,\n", " 'video': False,\n", " 'histogram': False,\n", " 'image': True,\n", " 'type': 'image',\n", " 'metadata': None,\n", " 'assetId': '2d45468babff4bb08befcbc03f3af2fb'},\n", " {'fileName': 'Figure(640x480) (2).svg',\n", " 'fileSize': 34700,\n", " 'runContext': None,\n", " 'step': None,\n", " 'link': 'https://s3.amazonaws.com/comet.ml/7eisOw4COHuLwtEYMNELdVURi.svg',\n", " 'compressedAssetLink': None,\n", " 'createdAt': 1555703605506,\n", " 'dir': 'media',\n", " 'canView': True,\n", " 'audio': False,\n", " 'video': False,\n", " 'histogram': False,\n", " 'image': True,\n", " 'type': 'image',\n", " 'metadata': None,\n", " 'assetId': '0bb15be9522644b88fe36528ae15fa6f'},\n", " {'fileName': 'Figure(640x480) (3).svg',\n", " 'fileSize': 35032,\n", " 'runContext': None,\n", " 'step': None,\n", " 'link': 'https://s3.amazonaws.com/comet.ml/zCWecfRiaIw3EydI9L0BfN3y8.svg',\n", " 'compressedAssetLink': None,\n", " 'createdAt': 1555703613885,\n", " 'dir': 'media',\n", " 'canView': True,\n", " 'audio': False,\n", " 'video': False,\n", " 'histogram': False,\n", " 'image': True,\n", " 'type': 'image',\n", " 'metadata': None,\n", " 'assetId': '7e7e685298f848aea0d16f9dca76435e'},\n", " {'fileName': 'Figure(640x480) (4).svg',\n", " 'fileSize': 35956,\n", " 'runContext': None,\n", " 'step': None,\n", " 'link': 'https://s3.amazonaws.com/comet.ml/TQ8EKEUBjYtY1XNhSmp0LBFa0.svg',\n", " 'compressedAssetLink': None,\n", " 'createdAt': 1555703619030,\n", " 'dir': 'media',\n", " 'canView': True,\n", " 'audio': False,\n", " 'video': False,\n", " 'histogram': False,\n", " 'image': True,\n", " 'type': 'image',\n", " 'metadata': None,\n", " 'assetId': '95ad3b25b2c145b8be69035a7819495f'}]" ] }, "execution_count": 50, "metadata": {}, "output_type": "execute_result" } ], "source": [ "comet_api.get('cometpublic/ludwig/02a0ed902ce2481fb6e2fc9009ee593c').get_asset_list(asset_type=\"image\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can get a URL for the image, and display it in the notebook:" ] }, { "cell_type": "code", "execution_count": 51, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'https://s3.amazonaws.com/comet.ml/yLB5azBCky96p5Q5IJJJA4est.svg'" ] }, "execution_count": 51, "metadata": {}, "output_type": "execute_result" } ], "source": [ "asset_list = comet_api.get('cometpublic/ludwig/02a0ed902ce2481fb6e2fc9009ee593c').get_asset_list(asset_type=\"image\")\n", "url = asset_list[0][\"link\"]\n", "url" ] }, { "cell_type": "code", "execution_count": 52, "metadata": {}, "outputs": [], "source": [ "from IPython.display import Image" ] }, { "cell_type": "code", "execution_count": 53, "metadata": {}, "outputs": [ { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "execution_count": 53, "metadata": {}, "output_type": "execute_result" } ], "source": [ "Image(url=url)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, let's write a short program that will find the run with the best accuracy given a workspace/project string:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Can we get all of the `hidden_size` parameter values for the experiments in dsblank/pytorch?" ] }, { "cell_type": "code", "execution_count": 54, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[['100', '128', '28', '0.01', '10', '2', '2', '28'],\n", " ['100', 'true', '128', '28', '0.01', '10', '2', '2', '28'],\n", " ['100', '128', '28', '0.01', '10', '2', '2', '28'],\n", " ['100', '128', '28', '0.01', '10', '2', '2', '28'],\n", " ['100', '128', '28', '0.01', '10', '2', '2', '28'],\n", " ['100', '128', '28', '0.01', '10', '2', '2', '28'],\n", " ['100', '128', '28', '0.01', '10', '2', '2', '28'],\n", " ['100', '128', '28', '0.01', '10', '2', '2', '28'],\n", " ['100', '128', '28', '0.01', '10', '2', '2', '28'],\n", " ['100', '128', '28', '0.01', '10', '2', '2', '28'],\n", " ['100', '128', '28', '0.01', '10', '2', '2', '28'],\n", " ['100', '128', '28', '0.01', '10', '2', '2', '28'],\n", " ['100', '128', '28', '0.01', '10', '2', '2', '28'],\n", " ['100', '128', '28', '0.01', '10', '2', '2', '28'],\n", " ['100', '128', '28', '0.01', '10', '2', '2', '28'],\n", " ['100', '128', '28', '0.01', '10', '2', '2', '28'],\n", " ['100', '128', '28', '0.01', '10', '2', '2', '28'],\n", " ['100', '128', '28', '0.01', '10', '2', '2', '28'],\n", " ['100', '128', '28', '0.01', '10', '2', '2', '28'],\n", " ['100', '128', '28', '0.01', '10', '2', '2', '28'],\n", " ['100', '128', '28', '0.01', '10', '2', '2', '28'],\n", " ['100', '128', '28', '0.01', '10', '2', '2', '28'],\n", " ['100', '128', '28', '0.01', '10', '2', '2', '28'],\n", " ['100', '128', '28', '0.01', '10', '2', '2', '28'],\n", " ['100', '128', '28', '0.01', '10', '2', '2', '28'],\n", " ['100', '128', '28', '0.01', '10', '2', '2', '28'],\n", " ['100', '128', '28', '0.01', '10', '2', '2', '28'],\n", " ['100', '128', '28', '0.01', '10', '2', '2', '28'],\n", " ['100', '128', '28', '0.01', '10', '2', '2', '28'],\n", " ['100', '128', '28', '0.01', '10', '2', '2', '28'],\n", " ['100', '128', '28', '0.01', '10', '3', '2', '28'],\n", " ['100', '128', '28', '0.01', '10', '3', '2', '28'],\n", " ['100', '128', '28', '0.01', '10', '3', '2', '28'],\n", " ['100', '128', '28', '0.01', '10', '3', '2', '28'],\n", " ['100', '128', '28', '0.01', '10', '2', '2', '28', '0'],\n", " ['100', '128', '28', '0.01', '10', '2', '2', '28', '599'],\n", " ['100', '128', '28', '0.01', '10', '2', '2', '28', '599'],\n", " ['1199'],\n", " ['100', '128', '28', '0.01', '10', '2', '2', '28', '599'],\n", " ['100',\n", " '361fadf1c712',\n", " '/run/user/1000/jupyter/kernel-db3ca856-5af7-46ea-85f6-6fe462cae76b.json',\n", " '128',\n", " '28',\n", " '0.01',\n", " '10',\n", " '3',\n", " '2',\n", " '28'],\n", " ['100', '128', '28', '0.01', '10', '2', '2', '28', '599']]" ] }, "execution_count": 54, "metadata": {}, "output_type": "execute_result" } ], "source": [ "[[p[\"valueCurrent\"] for p in exp.get_parameters_summary()] for exp in comet_api.get(\"dsblank/pytorch\")]" ] }, { "cell_type": "code", "execution_count": 55, "metadata": {}, "outputs": [], "source": [ "experiments = [[(exp, \"hidden_size\", int(param[\"valueCurrent\"])) \n", " for param in exp.get_parameters_summary() \n", " if param[\"name\"] == \"hidden_size\"]\n", " for exp in comet_api.get(\"dsblank/pytorch\")]\n", "experiments = [e[0] for e in experiments if len(e) > 0]" ] }, { "cell_type": "code", "execution_count": 56, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(,\n", " 'hidden_size',\n", " 128)" ] }, "execution_count": 56, "metadata": {}, "output_type": "execute_result" } ], "source": [ "experiments[0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Assets\n", "\n", "To get an asset, you need to get the asset_id. You can see all of the assets related to a project using the `APIExperiment.get_asset_list()`:" ] }, { "cell_type": "code", "execution_count": 57, "metadata": {}, "outputs": [], "source": [ "def find_asset(workspaces):\n", " for ws in workspaces or comet_api.get():\n", " for pj in comet_api.get(ws):\n", " for exp in comet_api.get(ws, pj):\n", " if exp.get_asset_list() != []:\n", " return (exp, exp.get_asset_list())\n", "exp, elist = find_asset([\"cometpublic\"])" ] }, { "cell_type": "code", "execution_count": 58, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 58, "metadata": {}, "output_type": "execute_result" } ], "source": [ "exp" ] }, { "cell_type": "code", "execution_count": 59, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "15" ] }, "execution_count": 59, "metadata": {}, "output_type": "execute_result" } ], "source": [ "len(elist)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "From there, you can use the `APIExperiment.get_asset(asset_id)` method to get the asset." ] }, { "cell_type": "code", "execution_count": 60, "metadata": {}, "outputs": [], "source": [ "description = exp.get_asset(\"0b256dc8858f4fbeb09228bc96074341\", return_type=\"json\")" ] }, { "cell_type": "code", "execution_count": 61, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'command': '/Users/ceceliashao/.local/bin/ludwig experiment --data_csv reuters-allcats.csv --model_definition_file model_definition.yaml', 'commit_hash': '5fbec5d311db', 'dataset_type': 'generic', 'input_data': 'reuters-allcats.csv', 'ludwig_version': '0.1.1', 'model_definition': {'combiner': {'type': 'concat'}, 'input_features': [{'encoder': 'parallel_cnn', 'level': 'word', 'name': 'text', 'tied_weights': None, 'type': 'text'}], 'output_features': [{'dependencies': [], 'loss': {'class_similarities_temperature': 0, 'class_weights': 1, 'confidence_penalty': 0, 'distortion': 1, 'labels_smoothing': 0, 'negative_samples': 0, 'robust_lambda': 0, 'sampler': None, 'type': 'softmax_cross_entropy', 'unique': False, 'weight': 1}, 'name': 'class', 'reduce_dependencies': 'sum', 'reduce_input': 'sum', 'top_k': 3, 'type': 'category'}], 'preprocessing': {'bag': {'fill_value': '', 'format': 'space', 'lowercase': False, 'missing_value_strategy': 'fill_with_const', 'most_common': 10000}, 'binary': {'fill_value': 0, 'missing_value_strategy': 'fill_with_const'}, 'category': {'fill_value': '', 'lowercase': False, 'missing_value_strategy': 'fill_with_const', 'most_common': 10000}, 'force_split': False, 'image': {'in_memory': True, 'missing_value_strategy': 'backfill', 'resize_method': 'crop_or_pad'}, 'numerical': {'fill_value': 0, 'missing_value_strategy': 'fill_with_const'}, 'sequence': {'fill_value': '', 'format': 'space', 'lowercase': False, 'missing_value_strategy': 'fill_with_const', 'most_common': 20000, 'padding': 'right', 'padding_symbol': '', 'sequence_length_limit': 256, 'unknown_symbol': ''}, 'set': {'fill_value': '', 'format': 'space', 'lowercase': False, 'missing_value_strategy': 'fill_with_const', 'most_common': 10000}, 'split_probabilities': [0.7, 0.1, 0.2], 'stratify': None, 'text': {'char_format': 'characters', 'char_most_common': 70, 'char_sequence_length_limit': 1024, 'fill_value': '', 'lowercase': True, 'missing_value_strategy': 'fill_with_const', 'padding': 'right', 'padding_symbol': '', 'unknown_symbol': '', 'word_format': 'space_punct', 'word_most_common': 20000, 'word_sequence_length_limit': 256}, 'timeseries': {'fill_value': '', 'format': 'space', 'missing_value_strategy': 'fill_with_const', 'padding': 'right', 'padding_value': 0, 'timeseries_length_limit': 256}}, 'training': {'batch_size': 128, 'bucketing_field': None, 'decay': False, 'decay_rate': 0.96, 'decay_steps': 10000, 'dropout_rate': 0.0, 'early_stop': 5, 'epochs': 1, 'eval_batch_size': 0, 'gradient_clipping': None, 'increase_batch_size_on_plateau': 0, 'increase_batch_size_on_plateau_max': 512, 'increase_batch_size_on_plateau_patience': 5, 'increase_batch_size_on_plateau_rate': 2, 'learning_rate': 0.001, 'learning_rate_warmup_epochs': 5, 'optimizer': {'beta1': 0.9, 'beta2': 0.999, 'epsilon': 1e-08, 'type': 'adam'}, 'reduce_learning_rate_on_plateau': 0, 'reduce_learning_rate_on_plateau_patience': 5, 'reduce_learning_rate_on_plateau_rate': 0.5, 'regularization_lambda': 0, 'regularizer': 'l2', 'staircase': False, 'validation_field': 'combined', 'validation_measure': 'loss'}}, 'random_seed': 42}\n" ] } ], "source": [ "print(description)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We hope that this gives you some ideas of how you can use the Comet Python API!" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.12" } }, "nbformat": 4, "nbformat_minor": 4 }