{ "cells": [ { "cell_type": "markdown", "id": "179a08d8-98be-4d68-9fd3-d24cf6108502", "metadata": {}, "source": [ "This notebook explores how to expires assets in an experiment after a certain date.\n", "\n", "First, we will make sure we have all of the necessary Python modules:" ] }, { "cell_type": "code", "execution_count": 1, "id": "9a1a5b37-c9fe-49bd-95ae-8998b62dee32", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Note: you may need to restart the kernel to use updated packages.\n" ] } ], "source": [ "%pip install --quiet comet_ml aitk.utils Pillow" ] }, { "cell_type": "markdown", "id": "2603ed4f-6765-4449-a91b-14ae66aa0097", "metadata": {}, "source": [ "Next, we import all of the modules we will need:" ] }, { "cell_type": "code", "execution_count": 2, "id": "844add68-4875-4a4c-9f77-93a6ac311461", "metadata": {}, "outputs": [], "source": [ "# Import Python modules:\n", "import random\n", "import os\n", "import glob\n", "import urllib\n", "import zipfile\n", "import datetime\n", "import random\n", "import json\n", "\n", "# Additional modules:\n", "from PIL import Image\n", "from aitk.utils import gallery\n", "import comet_ml" ] }, { "cell_type": "markdown", "id": "6f5168a9-34ba-4465-be7f-5e2fb178d23c", "metadata": {}, "source": [ "For this demo, we'll create some fun images. First we need to download them, so we write a general download and unzip function:" ] }, { "cell_type": "code", "execution_count": 3, "id": "03393a6a-a63e-4b8f-976d-0ffd73669660", "metadata": {}, "outputs": [], "source": [ "def download(url, filename=None):\n", " filename = filename if filename is not None else os.path.basename(url)\n", " basename, ext = os.path.splitext(filename)\n", " g = urllib.request.urlopen(url, timeout=5)\n", " with open(filename, \"wb\") as f:\n", " f.write(g.read())\n", " if ext == \".zip\":\n", " with zipfile.ZipFile(filename, 'r') as zip_ref:\n", " zip_ref.extractall(basename)" ] }, { "cell_type": "markdown", "id": "6d209600-eb4a-494e-b82f-329dba5f01cc", "metadata": {}, "source": [ "And we download a dataset of image parts. These are based on the images and code from:\n", "\n", "https://github.com/pixegami/pixel-punk-avatars" ] }, { "cell_type": "code", "execution_count": 23, "id": "6f49c907-d181-453d-827d-9e5f606a617d", "metadata": {}, "outputs": [], "source": [ "download(\"https://github.com/dsblank/pixel-punk-avatars/raw/main/images/images.zip\")" ] }, { "cell_type": "markdown", "id": "247596ac-4aab-4b8f-98a3-342024a7395b", "metadata": {}, "source": [ "We should now have a directory called \"images\" that contains faces, eyes, hair, and accessories.\n", "\n", "Now, we make sure that we have our Comet API key set:" ] }, { "cell_type": "code", "execution_count": 5, "id": "062f4304-b5bb-4c1e-a22e-4e94772c92d3", "metadata": {}, "outputs": [], "source": [ "comet_ml.init()" ] }, { "cell_type": "markdown", "id": "fbd16789-e8e8-4b15-95b6-cabb2c9ccaca", "metadata": {}, "source": [ "We write a simple image generation function:" ] }, { "cell_type": "code", "execution_count": 24, "id": "526ba1f8-2b7f-41a3-a8a7-d9b6dd5d355c", "metadata": {}, "outputs": [], "source": [ "def generate_image():\n", " bg_color = random.choice([(120, 150, 180), (255, 225, 150)])\n", " image = Image.new(\"RGBA\", (24, 24), bg_color)\n", " for layer, probability in [\n", " (\"0_face\", 1.0), (\"1_eye\", 1.0), (\"2_hair\", 0.8), (\"3_accessory\", 0.15)\n", " ]:\n", " if random.random() > probability:\n", " continue\n", " items = glob.glob(f\"images/{layer}/*.png\")\n", " layer_image = Image.open(random.choice(items))\n", " image = Image.alpha_composite(image, layer_image)\n", " return image" ] }, { "cell_type": "markdown", "id": "4cf8d644-d73c-439f-9724-0ee1fd6f9ae6", "metadata": {}, "source": [ "Every time we call it, we get a different image:" ] }, { "cell_type": "code", "execution_count": 25, "id": "9b7711d7-bc11-4c70-8e9e-5c7e394ee493", "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfQAAAH0CAYAAADL1t+KAAAY9ElEQVR4Ae3XMW4lSxkF4Ae6CQEpq5iYLZCMRORNjJyP5cSaxJrJLW/C0ZO8CuLZACmbAA0QvJv07dfqqvnr/B8R1rS763yn4Mh/eHh9//cv/kOAAAECBAgsLfDHpU/v8AQIECBAgMB/BQy6i0CAAAECBAIEDHpAiSIQIECAAAGD7g4QIECAAIEAAYMeUKIIBAgQIEDAoLsDBAgQIEAgQMCgB5QoAgECBAgQMOjuAAECBAgQCBAw6AElikCAAAECBAy6O0CAAAECBAIEDHpAiSIQIECAAAGD7g4QIECAAIEAAYMeUKIIBAgQIEDAoLsDBAgQIEAgQMCgB5QoAgECBAgQMOjuAAECBAgQCBAw6AElikCAAAECBAy6O0CAAAECBAIEDHpAiSIQIECAAAGD7g4QIECAAIEAAYMeUKIIBAgQIEDAoLsDBAgQIEAgQMCgB5QoAgECBAgQMOjuAAECBAgQCBAw6AElikCAAAECBAy6O0CAAAECBAIEDHpAiSIQIECAAAGD7g4QIECAAIEAAYMeUKIIBAgQIEDAoLsDBAgQIEAgQMCgB5QoAgECBAgQMOjuAAECBAgQCBAw6AElikCAAAECBAy6O0CAAAECBAIEDHpAiSIQIECAAAGD7g4QIECAAIEAAYMeUKIIBAgQIEDAoLsDBAgQIEAgQMCgB5QoAgECBAgQMOjuAAECBAgQCBAw6AElikCAAAECBAy6O0CAAAECBAIEDHpAiSIQIECAAAGD7g4QIECAAIEAAYMeUKIIBAgQIEDAoLsDBAgQIEAgQMCgB5QoAgECBAgQMOjuAAECBAgQCBAw6AElikCAAAECBAy6O0CAAAECBAIEDHpAiSIQIECAAAGD7g4QIECAAIEAAYMeUKIIBAgQIEDAoLsDBAgQIEAgQMCgB5QoAgECBAgQMOjuAAECBAgQCBAw6AElikCAAAECBAy6O0CAAAECBAIEDHpAiSIQIECAAAGD7g4QIECAAIEAAYMeUKIIBAgQIEDAoLsDBAgQIEAgQMCgB5QoAgECBAgQMOjuAAECBAgQCBAw6AElikCAAAECBAy6O0CAAAECBAIEDHpAiSIQIECAAAGD7g4QIECAAIEAAYMeUKIIBAgQIEDAoLsDBAgQIEAgQMCgB5QoAgECBAgQMOjuAAECBAgQCBAw6AElikCAAAECBAy6O0CAAAECBAIEDHpAiSIQIECAAAGD7g4QIECAAIEAAYMeUKIIBAgQIEDAoLsDBAgQIEAgQMCgB5QoAgECBAgQMOjuAAECBAgQCBAw6AElikCAAAECBAy6O0CAAAECBAIEDHpAiSIQIECAAAGD7g4QIECAAIEAAYMeUKIIBAgQIEDAoLsDBAgQIEAgQMCgB5QoAgECBAgQMOjuAAECBAgQCBAw6AElikCAAAECBAy6O0CAAAECBAIEDHpAiSIQIECAAAGD7g4QIECAAIEAAYMeUKIIBAgQIEDAoLsDBAgQIEAgQMCgB5QoAgECBAgQMOjuAAECBAgQCBAw6AElikCAAAECBAy6O0CAAAECBAIEDHpAiSIQIECAAAGD7g4QIECAAIEAAYMeUKIIBAgQIEDAoLsDBAgQIEAgQMCgB5QoAgECBAgQuCAg0EXgy4fPXaLKOVDg6fu3gW/3agLHBfyFftzObxIgQIAAgTICBr1MFQ5CgAABAgSOCxj043Z+kwABAgQIlBEw6GWqcBACBAgQIHBcwKAft/ObBAgQIECgjIBBL1OFgxAgQIAAgeMCBv24nd8kQIAAAQJlBAx6mSochAABAgQIHBcw6Mft/CYBAgQIECgjYNDLVOEgBAgQIEDguIBBP27nNwkQIECAQBkBg16mCgchQIAAAQLHBQz6cTu/SYAAAQIEyggY9DJVOAgBAgQIEDguYNCP2/lNAgQIECBQRsCgl6nCQQgQIECAwHEBg37czm8SIECAAIEyAga9TBUOQoAAAQIEjgsY9ON2fpMAAQIECJQRMOhlqnAQAgQIECBwXOBy/Ff9JoGxAl8+fB77AW8ncEBgxL18+v7twEn8CoFrAX+hX3v4iQABAgQILClg0JeszaEJECBAgMC1gEG/9vATAQIECBBYUsCgL1mbQxMgQIAAgWsBg37t4ScCBAgQILCkgEFfsjaHJkCAAAEC1wIG/drDTwQIECBAYEkBg75kbQ5NgAABAgSuBQz6tYefCBAgQIDAkgIGfcnaHJoAAQIECFwLGPRrDz8RIECAAIElBQz6krU5NAECBAgQuBYw6NcefiJAgAABAksKGPQla3NoAgQIECBwLWDQrz38RIAAAQIElhQw6EvW5tAECBAgQOBawKBfe/iJAAECBAgsKWDQl6zNoQkQIECAwLWAQb/28BMBAgQIEFhSwKAvWZtDEyBAgACBa4HL9Y9+InBM4MuHz8d+0W8RIPDLiP/9PH3/RraZgL/QmxUuLgECBAhkChj0zF6lIkCAAIFmAga9WeHiEiBAgECmgEHP7FUqAgQIEGgmYNCbFS4uAQIECGQKGPTMXqUiQIAAgWYCBr1Z4eISIECAQKaAQc/sVSoCBAgQaCZg0JsVLi4BAgQIZAoY9MxepSJAgACBZgIGvVnh4hIgQIBApoBBz+xVKgIECBBoJmDQmxUuLgECBAhkChj0zF6lIkCAAIFmAga9WeHiEiBAgECmgEHP7FUqAgQIEGgmYNCbFS4uAQIECGQKGPTMXqUiQIAAgWYCBr1Z4eISIECAQKaAQc/sVSoCBAgQaCZg0JsVLi4BAgQIZAoY9MxepSJAgACBZgIGvVnh4hIgQIBApoBBz+xVKgIECBBoJmDQmxUuLgECBAhkChj0zF6lIkCAAIFmAga9WeHiEiBAgECmgEHP7FUqAgQIEGgmYNCbFS4uAQIECGQKGPTMXqUiQIAAgWYCBr1Z4eISIECAQKaAQc/sVSoCBAgQaCZg0JsVLi4BAgQIZAoY9MxepSJAgACBZgIGvVnh4hIgQIBApoBBz+xVKgIECBBoJmDQmxUuLgECBAhkChj0zF6lIkCAAIFmAga9WeHiEiBAgECmgEHP7FUqAgQIEGgmcGmWd7m4Xz58Xu7MDkyAwM8XWOX/O56+f/v5WCEn8Bd6SJFiECBAgEBvAYPeu3/pCRAgQCBEwKCHFCkGAQIECPQWMOi9+5eeAAECBEIEDHpIkWIQIECAQG8Bg967f+kJECBAIETAoIcUKQYBAgQI9BYw6L37l54AAQIEQgQMekiRYhAgQIBAbwGD3rt/6QkQIEAgRMCghxQpBgECBAj0FjDovfuXngABAgRCBAx6SJFiECBAgEBvAYPeu3/pCRAgQCBEwKCHFCkGAQIECPQWMOi9+5eeAAECBEIEDHpIkWIQIECAQG8Bg967f+kJECBAIETAoIcUKQYBAgQI9BYw6L37l54AAQIEQgQuITliYzx9/7ZEtq+fPi5xTock0EXg4fW9S1Q5/y/gL3RXgQABAgQIBAgY9IASRSBAgAABAgbdHSBAgAABAgECBj2gRBEIECBAgIBBdwcIECBAgECAgEEPKFEEAgQIECBg0N0BAgQIECAQIGDQA0oUgQABAgQIGHR3gAABAgQIBAgY9IASRSBAgAABAgbdHSBAgAABAgECBj2gRBEIECBAgIBBdwcIECBAgECAgEEPKFEEAgQIECBg0N0BAgQIECAQIGDQA0oUgQABAgQIGHR3gAABAgQIBAgY9IASRSBAgAABAgbdHSBAgAABAgECBj2gRBEIECBAgMAFAYGqAm8vj6ce7e7++dT3/XjZv/7+19PfOeKFf/n1H6e/9ux+fhzw7I4693N64V5YXsBf6OUrckACBAgQIHBbwKDfNvIEAQIECBAoL2DQy1fkgAQIECBA4LaAQb9t5AkCBAgQIFBewKCXr8gBCRAgQIDAbQGDftvIEwQIECBAoLyAQS9fkQMSIECAAIHbAgb9tpEnCBAgQIBAeQGDXr4iByRAgAABArcFDPptI08QIECAAIHyAga9fEUOSIAAAQIEbgsY9NtGniBAgAABAuUFDHr5ihyQAAECBAjcFjDot408QYAAAQIEygsY9PIVOSABAgQIELgtYNBvG3mCAAECBAiUFzDo5StyQAIECBAgcFvAoN828gQBAgQIECgvYNDLV+SABAgQIEDgtoBBv23kCQIECBAgUF7gUv6EDni6wNdPH09/Z9cX/vlv/1wj+q9rHPPsU+rnbFHvqyzgL/TK7TgbAQIECBDYKWDQd0J5jAABAgQIVBYw6JXbcTYCBAgQILBTwKDvhPIYAQIECBCoLGDQK7fjbAQIECBAYKeAQd8J5TECBAgQIFBZwKBXbsfZCBAgQIDATgGDvhPKYwQIECBAoLKAQa/cjrMRIECAAIGdAgZ9J5THCBAgQIBAZQGDXrkdZyNAgAABAjsFDPpOKI8RIECAAIHKAga9cjvORoAAAQIEdgoY9J1QHiNAgAABApUFDHrldpyNAAECBAjsFDDoO6E8RoAAAQIEKgsY9MrtOBsBAgQIENgpYNB3QnmMAAECBAhUFjDoldtxNgIECBAgsFPAoO+E8hgBAgQIEKgscKl8OGdbR+Dt5bH8YUec8U+fnsvn/nHAEdlHBD/7nKv0M8LSO/sJ+Au9X+cSEyBAgECggEEPLFUkAgQIEOgnYND7dS4xAQIECAQKGPTAUkUiQIAAgX4CBr1f5xITIECAQKCAQQ8sVSQCBAgQ6Cdg0Pt1LjEBAgQIBAoY9MBSRSJAgACBfgIGvV/nEhMgQIBAoIBBDyxVJAIECBDoJ2DQ+3UuMQECBAgEChj0wFJFIkCAAIF+Aga9X+cSEyBAgECggEEPLFUkAgQIEOgnYND7dS4xAQIECAQKGPTAUkUiQIAAgX4CBr1f5xITIECAQKCAQQ8sVSQCBAgQ6Cdg0Pt1LjEBAgQIBAoY9MBSRSJAgACBfgKXfpElJnCewNvL43kv86bTBVbp5+7++fTsXthPwF/o/TqXmAABAgQCBQx6YKkiESBAgEA/AYPer3OJCRAgQCBQwKAHlioSAQIECPQTMOj9OpeYAAECBAIFDHpgqSIRIECAQD8Bg96vc4kJECBAIFDAoAeWKhIBAgQI9BMw6P06l5gAAQIEAgUMemCpIhEgQIBAPwGD3q9ziQkQIEAgUMCgB5YqEgECBAj0EzDo/TqXmAABAgQCBQx6YKkiESBAgEA/AYPer3OJCRAgQCBQwKAHlioSAQIECPQTMOj9OpeYAAECBAIFDHpgqSIRIECAQD8Bg96vc4kJECBAIFDgEphJpJ8gcHf/fPpX314eT3+nF54nMKLz8073vze5Q2eLel9lAX+hV27H2QgQIECAwE4Bg74TymMECBAgQKCygEGv3I6zESBAgACBnQIGfSeUxwgQIECAQGUBg165HWcjQIAAAQI7BQz6TiiPESBAgACBygIGvXI7zkaAAAECBHYKGPSdUB4jQIAAAQKVBQx65XacjQABAgQI7BQw6DuhPEaAAAECBCoLGPTK7TgbAQIECBDYKWDQd0J5jAABAgQIVBYw6JXbcTYCBAgQILBTwKDvhPIYAQIECBCoLGDQK7fjbAQIECBAYKeAQd8J5TECBAgQIFBZwKBXbsfZCBAgQIDATgGDvhPKYwQIECBAoLKAQa/cjrMRIECAAIGdAgZ9J5THCBAgQIBAZYFL5cM52xiBh9f301/89dPH09/phbUF3l4eax/Q6Qg0E/AXerPCxSVAgACBTAGDntmrVAQIECDQTMCgNytcXAIECBDIFDDomb1KRYAAAQLNBAx6s8LFJUCAAIFMAYOe2atUBAgQINBMwKA3K1xcAgQIEMgUMOiZvUpFgAABAs0EDHqzwsUlQIAAgUwBg57Zq1QECBAg0EzAoDcrXFwCBAgQyBQw6Jm9SkWAAAECzQQMerPCxSVAgACBTAGDntmrVAQIECDQTMCgNytcXAIECBDIFDDomb1KRYAAAQLNBAx6s8LFJUCAAIFMAYOe2atUBAgQINBMwKA3K1xcAgQIEMgUMOiZvUpFgAABAs0EDHqzwsUlQIAAgUwBg57Zq1QECBAg0EzAoDcrXFwCBAgQyBQw6Jm9SkWAAAECzQQMerPCxSVAgACBTAGDntmrVAQIECDQTMCgNytcXAIECBDIFDDomb1KRYAAAQLNBAx6s8LFJUCAAIFMAYOe2atUBAgQINBMwKA3K1xcAgQIEMgUMOiZvUpFgAABAs0EDHqzwsUlQIAAgUwBg57Zq1QECBAg0EzAoDcrXFwCBAgQyBQw6Jm9SkWAAAECzQQMerPCxSVAgACBTAGDntmrVAQIECDQTMCgNytcXAIECBDIFDDomb1KRYAAAQLNBAx6s8LFJUCAAIFMgUtmLKkIEPitwN39829/bPPf314e22QVlIC/0N0BAgQIECAQIGDQA0oUgQABAgQIGHR3gAABAgQIBAgY9IASRSBAgAABAgbdHSBAgAABAgECBj2gRBEIECBAgIBBdwcIECBAgECAgEEPKFEEAgQIECBg0N0BAgQIECAQIGDQA0oUgQABAgQIGHR3gAABAgQIBAgY9IASRSBAgAABAgbdHSBAgAABAgECBj2gRBEIECBAgIBBdwcIECBAgECAgEEPKFEEAgQIECBg0N0BAgQIECAQIGDQA0oUgQABAgQIGHR3gAABAgQIBAgY9IASRSBAgAABAhcEBAjkC7y9POaHlJBAcwF/oTe/AOITIECAQIaAQc/oUQoCBAgQaC5g0JtfAPEJECBAIEPAoGf0KAUBAgQINBcw6M0vgPgECBAgkCFg0DN6lIIAAQIEmgsY9OYXQHwCBAgQyBAw6Bk9SkGAAAECzQUMevMLID4BAgQIZAgY9IwepSBAgACB5gIGvfkFEJ8AAQIEMgQMekaPUhAgQIBAcwGD3vwCiE+AAAECGQIGPaNHKQgQIECguYBBb34BxCdAgACBDAGDntGjFAQIECDQXMCgN78A4hMgQIBAhoBBz+hRCgIECBBoLmDQm18A8QkQIEAgQ8CgZ/QoBQECBAg0F7g0zy8+AQIEfpfA3f3z73p+z8MPr+97HvMMgU0Bf6Fv8vhHAgQIECCwhoBBX6MnpyRAgAABApsCBn2Txz8SIECAAIE1BAz6Gj05JQECBAgQ2BQw6Js8/pEAAQIECKwhYNDX6MkpCRAgQIDApoBB3+TxjwQIECBAYA0Bg75GT05JgAABAgQ2BQz6Jo9/JECAAAECawgY9DV6ckoCBAgQILApYNA3efwjAQIECBBYQ8Cgr9GTUxIgQIAAgU0Bg77J4x8JECBAgMAaAgZ9jZ6ckgABAgQIbAoY9E0e/0iAAAECBNYQMOhr9OSUBAgQIEBgU8Cgb/L4RwIECBAgsIaAQV+jJ6ckQIAAAQKbAgZ9k8c/EiBAgACBNQQM+ho9OSUBAgQIENgUuGz+q38kQGC6wN398/Rv+uB+gYfX9/0Pe5LARAF/oU/E9ikCBAgQIDBKwKCPkvVeAgQIECAwUcCgT8T2KQIECBAgMErAoI+S9V4CBAgQIDBRwKBPxPYpAgQIECAwSsCgj5L1XgIECBAgMFHAoE/E9ikCBAgQIDBKwKCPkvVeAgQIECAwUcCgT8T2KQIECBAgMErAoI+S9V4CBAgQIDBRwKBPxPYpAgQIECAwSsCgj5L1XgIECBAgMFHAoE/E9ikCBAgQIDBKwKCPkvVeAgQIECAwUcCgT8T2KQIECBAgMErAoI+S9V4CBAgQIDBRwKBPxPYpAgQIECAwSsCgj5L1XgIECBAgMFHAoE/E9ikCBAgQIDBK4DLqxd5LgEAdgYfX9zqHcRICBIYI+At9CKuXEiBAgACBuQIGfa63rxEgQIAAgSECBn0Iq5cSIECAAIG5AgZ9rrevESBAgACBIQIGfQirlxIgQIAAgbkCBn2ut68RIECAAIEhAgZ9CKuXEiBAgACBuQIGfa63rxEgQIAAgSECBn0Iq5cSIECAAIG5AgZ9rrevESBAgACBIQIGfQirlxIgQIAAgbkCBn2ut68RIECAAIEhAgZ9CKuXEiBAgACBuQIGfa63rxEgQIAAgSECBn0Iq5cSIECAAIG5AgZ9rrevESBAgACBIQIGfQirlxIgQIAAgbkCBn2ut68RIECAAIEhAgZ9CKuXEiBAgACBuQIGfa63rxEgQIAAgSECBn0Iq5cSIECAAIG5AgZ9rrevESBAgACBIQIGfQirlxIgQIAAgbkCBn2ut68RIECAAIEhAgZ9CKuXEiBAgACBuQIGfa63rxEgQIAAgSECBn0Iq5cSIECAAIG5AgZ9rrevESBAgACBIQIGfQirlxIgQIAAgbkCBn2ut68RIECAAIEhAgZ9CKuXEiBAgACBuQIGfa63rxEgQIAAgSECBn0Iq5cSIECAAIG5AgZ9rrevESBAgACBIQIGfQirlxIgQIAAgbkCBn2ut68RIECAAIEhAgZ9CKuXEiBAgACBuQIGfa63rxEgQIAAgSECBn0Iq5cSIECAAIG5AgZ9rrevESBAgACBIQIGfQirlxIgQIAAgbkC/wGc0VDwYQhNQAAAAABJRU5ErkJggg==", "text/plain": [ "" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "image = generate_image()\n", "image.resize((500, 500), resample=Image.Resampling.NEAREST)" ] }, { "cell_type": "markdown", "id": "5aba8a82-2f73-4a10-8f97-417dbba762ba", "metadata": {}, "source": [ "Now, let's call it 625 times to generate a nice dataset:" ] }, { "cell_type": "code", "execution_count": 26, "id": "497f7fb4-2aa1-48a0-919c-00eafb5fb6af", "metadata": {}, "outputs": [], "source": [ "images = [generate_image() for i in range(625)]" ] }, { "cell_type": "markdown", "id": "5693e76b-35c6-43f2-9a0f-431c8dbc5bca", "metadata": {}, "source": [ "A nice function from `aitk.utils` is the `gallery()` function that generates a gallery of images from a list of images:" ] }, { "cell_type": "code", "execution_count": 27, "id": "525445cd-a899-48d1-be12-9e397171184a", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
0
\"0\"
1
\"1\"
2
\"2\"
3
\"3\"
4
\"4\"
5
\"5\"
6
\"6\"
7
\"7\"
8
\"8\"
9
\"9\"
10
\"10\"
11
\"11\"
12
\"12\"
13
\"13\"
14
\"14\"
15
\"15\"
16
\"16\"
17
\"17\"
18
\"18\"
19
\"19\"
20
\"20\"
21
\"21\"
22
\"22\"
23
\"23\"
24
\"24\"
25
\"25\"
26
\"26\"
27
\"27\"
28
\"28\"
29
\"29\"
30
\"30\"
31
\"31\"
32
\"32\"
33
\"33\"
34
\"34\"
35
\"35\"
36
\"36\"
37
\"37\"
38
\"38\"
39
\"39\"
40
\"40\"
41
\"41\"
42
\"42\"
43
\"43\"
44
\"44\"
45
\"45\"
46
\"46\"
47
\"47\"
48
\"48\"
49
\"49\"
50
\"50\"
51
\"51\"
52
\"52\"
53
\"53\"
54
\"54\"
55
\"55\"
56
\"56\"
57
\"57\"
58
\"58\"
59
\"59\"
60
\"60\"
61
\"61\"
62
\"62\"
63
\"63\"
64
\"64\"
65
\"65\"
66
\"66\"
67
\"67\"
68
\"68\"
69
\"69\"
70
\"70\"
71
\"71\"
72
\"72\"
73
\"73\"
74
\"74\"
75
\"75\"
76
\"76\"
77
\"77\"
78
\"78\"
79
\"79\"
80
\"80\"
81
\"81\"
82
\"82\"
83
\"83\"
84
\"84\"
85
\"85\"
86
\"86\"
87
\"87\"
88
\"88\"
89
\"89\"
90
\"90\"
91
\"91\"
92
\"92\"
93
\"93\"
94
\"94\"
95
\"95\"
96
\"96\"
97
\"97\"
98
\"98\"
99
\"99\"
100
\"100\"
101
\"101\"
102
\"102\"
103
\"103\"
104
\"104\"
105
\"105\"
106
\"106\"
107
\"107\"
108
\"108\"
109
\"109\"
110
\"110\"
111
\"111\"
112
\"112\"
113
\"113\"
114
\"114\"
115
\"115\"
116
\"116\"
117
\"117\"
118
\"118\"
119
\"119\"
120
\"120\"
121
\"121\"
122
\"122\"
123
\"123\"
124
\"124\"
125
\"125\"
126
\"126\"
127
\"127\"
128
\"128\"
129
\"129\"
130
\"130\"
131
\"131\"
132
\"132\"
133
\"133\"
134
\"134\"
135
\"135\"
136
\"136\"
137
\"137\"
138
\"138\"
139
\"139\"
140
\"140\"
141
\"141\"
142
\"142\"
143
\"143\"
144
\"144\"
145
\"145\"
146
\"146\"
147
\"147\"
148
\"148\"
149
\"149\"
150
\"150\"
151
\"151\"
152
\"152\"
153
\"153\"
154
\"154\"
155
\"155\"
156
\"156\"
157
\"157\"
158
\"158\"
159
\"159\"
160
\"160\"
161
\"161\"
162
\"162\"
163
\"163\"
164
\"164\"
165
\"165\"
166
\"166\"
167
\"167\"
168
\"168\"
169
\"169\"
170
\"170\"
171
\"171\"
172
\"172\"
173
\"173\"
174
\"174\"
175
\"175\"
176
\"176\"
177
\"177\"
178
\"178\"
179
\"179\"
180
\"180\"
181
\"181\"
182
\"182\"
183
\"183\"
184
\"184\"
185
\"185\"
186
\"186\"
187
\"187\"
188
\"188\"
189
\"189\"
190
\"190\"
191
\"191\"
192
\"192\"
193
\"193\"
194
\"194\"
195
\"195\"
196
\"196\"
197
\"197\"
198
\"198\"
199
\"199\"
200
\"200\"
201
\"201\"
202
\"202\"
203
\"203\"
204
\"204\"
205
\"205\"
206
\"206\"
207
\"207\"
208
\"208\"
209
\"209\"
210
\"210\"
211
\"211\"
212
\"212\"
213
\"213\"
214
\"214\"
215
\"215\"
216
\"216\"
217
\"217\"
218
\"218\"
219
\"219\"
220
\"220\"
221
\"221\"
222
\"222\"
223
\"223\"
224
\"224\"
225
\"225\"
226
\"226\"
227
\"227\"
228
\"228\"
229
\"229\"
230
\"230\"
231
\"231\"
232
\"232\"
233
\"233\"
234
\"234\"
235
\"235\"
236
\"236\"
237
\"237\"
238
\"238\"
239
\"239\"
240
\"240\"
241
\"241\"
242
\"242\"
243
\"243\"
244
\"244\"
245
\"245\"
246
\"246\"
247
\"247\"
248
\"248\"
249
\"249\"
250
\"250\"
251
\"251\"
252
\"252\"
253
\"253\"
254
\"254\"
255
\"255\"
256
\"256\"
257
\"257\"
258
\"258\"
259
\"259\"
260
\"260\"
261
\"261\"
262
\"262\"
263
\"263\"
264
\"264\"
265
\"265\"
266
\"266\"
267
\"267\"
268
\"268\"
269
\"269\"
270
\"270\"
271
\"271\"
272
\"272\"
273
\"273\"
274
\"274\"
275
\"275\"
276
\"276\"
277
\"277\"
278
\"278\"
279
\"279\"
280
\"280\"
281
\"281\"
282
\"282\"
283
\"283\"
284
\"284\"
285
\"285\"
286
\"286\"
287
\"287\"
288
\"288\"
289
\"289\"
290
\"290\"
291
\"291\"
292
\"292\"
293
\"293\"
294
\"294\"
295
\"295\"
296
\"296\"
297
\"297\"
298
\"298\"
299
\"299\"
300
\"300\"
301
\"301\"
302
\"302\"
303
\"303\"
304
\"304\"
305
\"305\"
306
\"306\"
307
\"307\"
308
\"308\"
309
\"309\"
310
\"310\"
311
\"311\"
312
\"312\"
313
\"313\"
314
\"314\"
315
\"315\"
316
\"316\"
317
\"317\"
318
\"318\"
319
\"319\"
320
\"320\"
321
\"321\"
322
\"322\"
323
\"323\"
324
\"324\"
325
\"325\"
326
\"326\"
327
\"327\"
328
\"328\"
329
\"329\"
330
\"330\"
331
\"331\"
332
\"332\"
333
\"333\"
334
\"334\"
335
\"335\"
336
\"336\"
337
\"337\"
338
\"338\"
339
\"339\"
340
\"340\"
341
\"341\"
342
\"342\"
343
\"343\"
344
\"344\"
345
\"345\"
346
\"346\"
347
\"347\"
348
\"348\"
349
\"349\"
350
\"350\"
351
\"351\"
352
\"352\"
353
\"353\"
354
\"354\"
355
\"355\"
356
\"356\"
357
\"357\"
358
\"358\"
359
\"359\"
360
\"360\"
361
\"361\"
362
\"362\"
363
\"363\"
364
\"364\"
365
\"365\"
366
\"366\"
367
\"367\"
368
\"368\"
369
\"369\"
370
\"370\"
371
\"371\"
372
\"372\"
373
\"373\"
374
\"374\"
375
\"375\"
376
\"376\"
377
\"377\"
378
\"378\"
379
\"379\"
380
\"380\"
381
\"381\"
382
\"382\"
383
\"383\"
384
\"384\"
385
\"385\"
386
\"386\"
387
\"387\"
388
\"388\"
389
\"389\"
390
\"390\"
391
\"391\"
392
\"392\"
393
\"393\"
394
\"394\"
395
\"395\"
396
\"396\"
397
\"397\"
398
\"398\"
399
\"399\"
400
\"400\"
401
\"401\"
402
\"402\"
403
\"403\"
404
\"404\"
405
\"405\"
406
\"406\"
407
\"407\"
408
\"408\"
409
\"409\"
410
\"410\"
411
\"411\"
412
\"412\"
413
\"413\"
414
\"414\"
415
\"415\"
416
\"416\"
417
\"417\"
418
\"418\"
419
\"419\"
420
\"420\"
421
\"421\"
422
\"422\"
423
\"423\"
424
\"424\"
425
\"425\"
426
\"426\"
427
\"427\"
428
\"428\"
429
\"429\"
430
\"430\"
431
\"431\"
432
\"432\"
433
\"433\"
434
\"434\"
435
\"435\"
436
\"436\"
437
\"437\"
438
\"438\"
439
\"439\"
440
\"440\"
441
\"441\"
442
\"442\"
443
\"443\"
444
\"444\"
445
\"445\"
446
\"446\"
447
\"447\"
448
\"448\"
449
\"449\"
450
\"450\"
451
\"451\"
452
\"452\"
453
\"453\"
454
\"454\"
455
\"455\"
456
\"456\"
457
\"457\"
458
\"458\"
459
\"459\"
460
\"460\"
461
\"461\"
462
\"462\"
463
\"463\"
464
\"464\"
465
\"465\"
466
\"466\"
467
\"467\"
468
\"468\"
469
\"469\"
470
\"470\"
471
\"471\"
472
\"472\"
473
\"473\"
474
\"474\"
475
\"475\"
476
\"476\"
477
\"477\"
478
\"478\"
479
\"479\"
480
\"480\"
481
\"481\"
482
\"482\"
483
\"483\"
484
\"484\"
485
\"485\"
486
\"486\"
487
\"487\"
488
\"488\"
489
\"489\"
490
\"490\"
491
\"491\"
492
\"492\"
493
\"493\"
494
\"494\"
495
\"495\"
496
\"496\"
497
\"497\"
498
\"498\"
499
\"499\"
500
\"500\"
501
\"501\"
502
\"502\"
503
\"503\"
504
\"504\"
505
\"505\"
506
\"506\"
507
\"507\"
508
\"508\"
509
\"509\"
510
\"510\"
511
\"511\"
512
\"512\"
513
\"513\"
514
\"514\"
515
\"515\"
516
\"516\"
517
\"517\"
518
\"518\"
519
\"519\"
520
\"520\"
521
\"521\"
522
\"522\"
523
\"523\"
524
\"524\"
525
\"525\"
526
\"526\"
527
\"527\"
528
\"528\"
529
\"529\"
530
\"530\"
531
\"531\"
532
\"532\"
533
\"533\"
534
\"534\"
535
\"535\"
536
\"536\"
537
\"537\"
538
\"538\"
539
\"539\"
540
\"540\"
541
\"541\"
542
\"542\"
543
\"543\"
544
\"544\"
545
\"545\"
546
\"546\"
547
\"547\"
548
\"548\"
549
\"549\"
550
\"550\"
551
\"551\"
552
\"552\"
553
\"553\"
554
\"554\"
555
\"555\"
556
\"556\"
557
\"557\"
558
\"558\"
559
\"559\"
560
\"560\"
561
\"561\"
562
\"562\"
563
\"563\"
564
\"564\"
565
\"565\"
566
\"566\"
567
\"567\"
568
\"568\"
569
\"569\"
570
\"570\"
571
\"571\"
572
\"572\"
573
\"573\"
574
\"574\"
575
\"575\"
576
\"576\"
577
\"577\"
578
\"578\"
579
\"579\"
580
\"580\"
581
\"581\"
582
\"582\"
583
\"583\"
584
\"584\"
585
\"585\"
586
\"586\"
587
\"587\"
588
\"588\"
589
\"589\"
590
\"590\"
591
\"591\"
592
\"592\"
593
\"593\"
594
\"594\"
595
\"595\"
596
\"596\"
597
\"597\"
598
\"598\"
599
\"599\"
600
\"600\"
601
\"601\"
602
\"602\"
603
\"603\"
604
\"604\"
605
\"605\"
606
\"606\"
607
\"607\"
608
\"608\"
609
\"609\"
610
\"610\"
611
\"611\"
612
\"612\"
613
\"613\"
614
\"614\"
615
\"615\"
616
\"616\"
617
\"617\"
618
\"618\"
619
\"619\"
620
\"620\"
621
\"621\"
622
\"622\"
623
\"623\"
624
\"624\"
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "gallery(images)" ] }, { "cell_type": "markdown", "id": "4944d6a1-12e1-4109-b26d-72be0eb5c135", "metadata": {}, "source": [ "Now, we create an experiment to log all of the images." ] }, { "cell_type": "code", "execution_count": 10, "id": "efee7b83-b746-4801-a5c1-d7a8be6fdf5f", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\u001b[1;38;5;214mCOMET WARNING:\u001b[0m As you are running in a Jupyter environment, you will need to call `experiment.end()` when finished to ensure all metrics and code are logged before exiting.\n", "\u001b[1;38;5;39mCOMET INFO:\u001b[0m Couldn't find a Git repository in '/home/dsblank/comet/examples' nor in any parent directory. Set `COMET_GIT_DIRECTORY` if your Git Repository is elsewhere.\n", "\u001b[1;38;5;39mCOMET INFO:\u001b[0m Experiment is live on comet.com https://www.comet.com/dsblank/pixel-punk-avatars/49d4079aa31243d69e7d3166b3255c79\n", "\n" ] } ], "source": [ "experiment = comet_ml.Experiment(project_name=\"pixel-punk-avatars\")" ] }, { "cell_type": "markdown", "id": "8124e7d2-ed7b-4ac6-a799-400b512dc275", "metadata": {}, "source": [ "We'll need some support functions to help us:" ] }, { "cell_type": "code", "execution_count": 11, "id": "8f0611f7-d0fd-46a3-ae01-92986fac93da", "metadata": {}, "outputs": [], "source": [ "def random_day(n=100):\n", " \"\"\"\n", " Return a random day n days in the future.\n", " \"\"\"\n", " today = datetime.date.today()\n", " return add_days(today, random.randint(1, n))\n", "\n", "def add_days(day, n):\n", " \"\"\"\n", " Method to do some date math.\n", " \"\"\"\n", " return day + datetime.timedelta(days=n)\n", " \n", "def day_to_timestamp(day):\n", " \"\"\"\n", " Turn a day into an integer timestamp\n", " \"\"\"\n", " date = datetime.datetime(day.year, day.month, day.day)\n", " return date.timestamp()\n", "\n", "def timestamp_to_day(timestamp):\n", " \"\"\"\n", " Turn a timestamp into a day\n", " \"\"\"\n", " return datetime.datetime.fromtimestamp(timestamp)" ] }, { "cell_type": "markdown", "id": "ec8ba4e2-529d-4ad8-bbc8-354c7f219e81", "metadata": {}, "source": [ "The reason for the timestamp is that we can't log an experation day directly, but need to encode it as a number.\n", "\n", "Now we log each image, storing the date to expire in the metadata of the image. Note that for this demo we create random dates between 1 and 100 days in the future." ] }, { "cell_type": "code", "execution_count": 12, "id": "3ef66d01-8f3b-4a6e-bb9a-7e9c24883e0f", "metadata": {}, "outputs": [], "source": [ "for image in images:\n", " experiment.log_image(\n", " image, \n", " metadata={\n", " \"expires_on\": day_to_timestamp(random_day())\n", " }\n", " )" ] }, { "cell_type": "markdown", "id": "e035f407-c428-4f1a-bbff-5b91d9406539", "metadata": {}, "source": [ "and call `experiment.end()` because we are in a Jupyter Notebook." ] }, { "cell_type": "code", "execution_count": 13, "id": "c08dae9a-a019-470c-b8b3-44033efb6446", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\u001b[1;38;5;39mCOMET INFO:\u001b[0m ---------------------------------------------------------------------------------------\n", "\u001b[1;38;5;39mCOMET INFO:\u001b[0m Comet.ml Experiment Summary\n", "\u001b[1;38;5;39mCOMET INFO:\u001b[0m ---------------------------------------------------------------------------------------\n", "\u001b[1;38;5;39mCOMET INFO:\u001b[0m Data:\n", "\u001b[1;38;5;39mCOMET INFO:\u001b[0m display_summary_level : 1\n", "\u001b[1;38;5;39mCOMET INFO:\u001b[0m url : https://www.comet.com/dsblank/pixel-punk-avatars/49d4079aa31243d69e7d3166b3255c79\n", "\u001b[1;38;5;39mCOMET INFO:\u001b[0m Uploads:\n", "\u001b[1;38;5;39mCOMET INFO:\u001b[0m conda-environment-definition : 1\n", "\u001b[1;38;5;39mCOMET INFO:\u001b[0m conda-info : 1\n", "\u001b[1;38;5;39mCOMET INFO:\u001b[0m conda-specification : 1\n", "\u001b[1;38;5;39mCOMET INFO:\u001b[0m environment details : 1\n", "\u001b[1;38;5;39mCOMET INFO:\u001b[0m filename : 1\n", "\u001b[1;38;5;39mCOMET INFO:\u001b[0m images : 625\n", "\u001b[1;38;5;39mCOMET INFO:\u001b[0m installed packages : 1\n", "\u001b[1;38;5;39mCOMET INFO:\u001b[0m notebook : 1\n", "\u001b[1;38;5;39mCOMET INFO:\u001b[0m os packages : 1\n", "\u001b[1;38;5;39mCOMET INFO:\u001b[0m source_code : 1\n", "\u001b[1;38;5;39mCOMET INFO:\u001b[0m \n", "\u001b[1;38;5;39mCOMET INFO:\u001b[0m Please wait for assets to finish uploading (timeout is 10800 seconds)\n", "\u001b[1;38;5;39mCOMET INFO:\u001b[0m Still uploading 98 file(s), remaining 18.76 KB/36.83 KB\n" ] } ], "source": [ "experiment.end()" ] }, { "cell_type": "markdown", "id": "7457855d-c890-4df8-8fd3-8eab5ecccb65", "metadata": {}, "source": [ "We can check to see that the images were loaded by examining the Graphics tab in the Come UI:" ] }, { "cell_type": "code", "execution_count": 30, "id": "c935ef25-2aa6-418b-ac64-73d33140d477", "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "experiment.display(tab=\"graphics\")" ] }, { "cell_type": "markdown", "id": "a6c15235-39b1-4a0e-8d21-1926fb65dd6a", "metadata": {}, "source": [ "Now we are ready to expire those images past a certain date. We'll use the Comet API to get the assets." ] }, { "cell_type": "code", "execution_count": 14, "id": "d811f7d7-88e0-4fb8-a128-e44246f33cc1", "metadata": {}, "outputs": [], "source": [ "api = comet_ml.API()" ] }, { "cell_type": "markdown", "id": "83c72449-e840-45e4-b3b3-c3f2025895ef", "metadata": {}, "source": [ "We write a function to delete the image assets given an experiment key (id) and date:" ] }, { "cell_type": "code", "execution_count": 31, "id": "27611c9e-7b7a-4a06-bd6d-69cd7f1f054e", "metadata": { "scrolled": true }, "outputs": [], "source": [ "def delete_experiment_assets(experiment_key, expires_on):\n", " api_experiment = api.get_experiment_by_key(experiment_key)\n", " count = 0\n", " for asset_data in api_experiment.get_asset_list():\n", " if asset_data[\"type\"] == \"image\":\n", " asset_metadata = json.loads(asset_data[\"metadata\"])\n", " asset_id = asset_data[\"assetId\"]\n", " timestamp = asset_metadata[\"expires_on\"]\n", " if timestamp > expires_on:\n", " print(f\"Deleting {asset_id}\")\n", " api_experiment.delete_asset(asset_id)\n", " count += 1\n", " print(f\"{count} assets expired\")" ] }, { "cell_type": "markdown", "id": "326e465f-caa7-432d-838c-715401b85290", "metadata": {}, "source": [ "For this demo, let's pick a day 90 days in the future:" ] }, { "cell_type": "code", "execution_count": 21, "id": "3ca9e7d1-a158-4749-9c17-a0c4e50dc7f2", "metadata": {}, "outputs": [], "source": [ "# A date 90 days in the future:\n", "expires_on = day_to_timestamp(\n", " add_days(datetime.date.today(), 90)\n", ")" ] }, { "cell_type": "markdown", "id": "fe00dc83-b76b-4b0e-9560-cf1cd598b8de", "metadata": {}, "source": [ "And now, we simply call the function given the experiment.id from above, and the date 90 days from now:" ] }, { "cell_type": "code", "execution_count": 22, "id": "45e8d6cc-afdc-496f-90a3-2b697f6d07db", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "70 assets expired\n" ] } ], "source": [ "delete_experiment_assets(experiment.id, expires_on)" ] }, { "cell_type": "markdown", "id": "d11f0eb5-e052-4eb3-bb93-3408e93af03a", "metadata": {}, "source": [ "That should have deleted about 10% of the 625 images, or about 62 images give or take." ] }, { "cell_type": "markdown", "id": "5d6829b4-a50a-4e01-85e7-29ae72a06245", "metadata": {}, "source": [ "If you run the `delete_experiment_assets()` again with the same arguments, it shouldn't delete any assets." ] }, { "cell_type": "code", "execution_count": null, "id": "b677506e-8996-4ab0-b565-cdc5fecd6692", "metadata": {}, "outputs": [], "source": [ "delete_experiment_assets(experiment.id, expires_on)" ] }, { "cell_type": "markdown", "id": "fb9d68b7-64ef-46f1-a0a6-f0e416d87294", "metadata": {}, "source": [ "Now, every once in a while you can call `delete_experiment_assets()` with a paticular date, and you'll delete all of the assets that should be \"expired\"." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.13" } }, "nbformat": 4, "nbformat_minor": 5 }