{ "cells": [ { "cell_type": "markdown", "id": "1b40e08f-8442-43ec-b50e-c755b3550f54", "metadata": { "tags": [] }, "source": [ "## LoRAs of the World Unite - Training SOTA DreamBooth LoRA with Pivotal Tuning 🧨\n", "\n", "In this notebook, we show how to fine-tune [Stable Diffusion XL (SDXL)](https://huggingface.co/docs/diffusers/main/en/api/pipelines/stable_diffusion/stable_diffusion_xl) with [DreamBooth](https://huggingface.co/docs/diffusers/main/en/training/dreambooth) and [LoRA](https://huggingface.co/docs/diffusers/main/en/training/lora) using some of the most popular SOTA methods.\n", "\n", "Learn more about the techniques used in this exmaple [here](linke to blogpost)\n", "\n", "Let's get started 🧪" ] }, { "cell_type": "markdown", "id": "43111af5-c952-47e9-b9c4-fd6bf6a3e137", "metadata": { "tags": [] }, "source": [ "## Setup 🪓" ] }, { "cell_type": "code", "execution_count": 1, "id": "3c1b6b6a-3d7a-4ff3-a7aa-6340fb09fbca", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001B[33mWARNING: Ignoring invalid distribution -etworkx (/usr/lib/python3/dist-packages)\u001B[0m\u001B[33m\n", "\u001B[0m\u001B[33mWARNING: Ignoring invalid distribution -etworkx (/usr/lib/python3/dist-packages)\u001B[0m\u001B[33m\n", "\u001B[0m\u001B[33mDEPRECATION: distro-info 0.23ubuntu1 has a non-standard version number. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of distro-info or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063\u001B[0m\u001B[33m\n", "\u001B[0m\u001B[33mDEPRECATION: python-debian 0.1.36ubuntu1 has a non-standard version number. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of python-debian or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063\u001B[0m\u001B[33m\n", "\u001B[0m\n", "\u001B[1m[\u001B[0m\u001B[34;49mnotice\u001B[0m\u001B[1;39;49m]\u001B[0m\u001B[39;49m A new release of pip is available: \u001B[0m\u001B[31;49m23.2\u001B[0m\u001B[39;49m -> \u001B[0m\u001B[32;49m23.3.1\u001B[0m\n", "\u001B[1m[\u001B[0m\u001B[34;49mnotice\u001B[0m\u001B[1;39;49m]\u001B[0m\u001B[39;49m To update, run: \u001B[0m\u001B[32;49mpython3.10 -m pip install --upgrade pip\u001B[0m\n" ] } ], "source": [ "# Install dependencies.\n", "!pip install xformers bitsandbytes transformers accelerate wandb dadaptation prodigyopt -q" ] }, { "cell_type": "code", "execution_count": 2, "id": "23d6bb49-3469-4e23-baf5-25b2344b599d", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001B[33mWARNING: Ignoring invalid distribution -etworkx (/usr/lib/python3/dist-packages)\u001B[0m\u001B[33m\n", "\u001B[0m\u001B[33mWARNING: Ignoring invalid distribution -etworkx (/usr/lib/python3/dist-packages)\u001B[0m\u001B[33m\n", "\u001B[0m\u001B[33mDEPRECATION: distro-info 0.23ubuntu1 has a non-standard version number. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of distro-info or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063\u001B[0m\u001B[33m\n", "\u001B[0m\u001B[33mDEPRECATION: python-debian 0.1.36ubuntu1 has a non-standard version number. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of python-debian or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063\u001B[0m\u001B[33m\n", "\u001B[0m\n", "\u001B[1m[\u001B[0m\u001B[34;49mnotice\u001B[0m\u001B[1;39;49m]\u001B[0m\u001B[39;49m A new release of pip is available: \u001B[0m\u001B[31;49m23.2\u001B[0m\u001B[39;49m -> \u001B[0m\u001B[32;49m23.3.1\u001B[0m\n", "\u001B[1m[\u001B[0m\u001B[34;49mnotice\u001B[0m\u001B[1;39;49m]\u001B[0m\u001B[39;49m To update, run: \u001B[0m\u001B[32;49mpython3.10 -m pip install --upgrade pip\u001B[0m\n" ] } ], "source": [ "!pip install peft -q" ] }, { "cell_type": "markdown", "id": "7951c6b4-7d33-47e9-9d62-8d4f6b2d47fd", "metadata": {}, "source": [ "Make sure to install `diffusers` from `main`." ] }, { "cell_type": "code", "execution_count": 1, "id": "3dd2fbe0-001d-4156-ace8-0e591bff58e7", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001B[33mWARNING: Ignoring invalid distribution -etworkx (/usr/lib/python3/dist-packages)\u001B[0m\u001B[33m\n", "\u001B[0m\u001B[33mWARNING: Ignoring invalid distribution -etworkx (/usr/lib/python3/dist-packages)\u001B[0m\u001B[33m\n", "\u001B[0m\u001B[33mDEPRECATION: distro-info 0.23ubuntu1 has a non-standard version number. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of distro-info or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063\u001B[0m\u001B[33m\n", "\u001B[0m\u001B[33mDEPRECATION: python-debian 0.1.36ubuntu1 has a non-standard version number. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of python-debian or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063\u001B[0m\u001B[33m\n", "\u001B[0m\n", "\u001B[1m[\u001B[0m\u001B[34;49mnotice\u001B[0m\u001B[1;39;49m]\u001B[0m\u001B[39;49m A new release of pip is available: \u001B[0m\u001B[31;49m23.2\u001B[0m\u001B[39;49m -> \u001B[0m\u001B[32;49m23.3.1\u001B[0m\n", "\u001B[1m[\u001B[0m\u001B[34;49mnotice\u001B[0m\u001B[1;39;49m]\u001B[0m\u001B[39;49m To update, run: \u001B[0m\u001B[32;49mpython3.10 -m pip install --upgrade pip\u001B[0m\n" ] } ], "source": [ "!pip install git+https://github.com/huggingface/diffusers.git -q" ] }, { "cell_type": "markdown", "id": "062701bb-6627-407a-81a4-31034f80e5e6", "metadata": {}, "source": [ "Download diffusers SDXL DreamBooth training script." ] }, { "cell_type": "code", "execution_count": 2, "id": "cfa6ab4e-e7ef-4ee7-b3b5-f2e4d5104e75", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "--2023-12-04 08:52:41-- https://raw.githubusercontent.com/huggingface/diffusers/main/examples/advanced_diffusion_training/train_dreambooth_lora_sdxl_advanced.py\n", "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.111.133, 185.199.109.133, ...\n", "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 86471 (84K) [text/plain]\n", "Saving to: ‘train_dreambooth_lora_sdxl_advanced.py.1’\n", "\n", "train_dreambooth_lo 100%[===================>] 84.44K --.-KB/s in 0.001s \n", "\n", "2023-12-04 08:52:41 (111 MB/s) - ‘train_dreambooth_lora_sdxl_advanced.py.1’ saved [86471/86471]\n" ] } ], "source": [ "!wget https://raw.githubusercontent.com/huggingface/diffusers/main/examples/advanced_diffusion_training/train_dreambooth_lora_sdxl_advanced.py" ] }, { "cell_type": "markdown", "id": "0b38876e-940e-401a-a542-05b3262d10eb", "metadata": { "id": "CRBcvYAyS8bU", "tags": [] }, "source": [ "## Dataset 🐶" ] }, { "cell_type": "markdown", "id": "a1e31e52-5383-4c77-abcf-e9744302cbf9", "metadata": { "id": "8UNkbXwj126Q" }, "source": [ "**Let's get our training data!**\n", "For this example, we'll download some images from the hub.\n", "\n", "If you already have a dataset on the hub you wish to use, you can skip this part and go straight to: \"Prep for\n", "training 💻\" section, where you'll simply specify the dataset name.\n", "\n", "If your images are saved locally, and/or you want to add BLIP generated captions,\n", "pick option 1 or 2 below." ] }, { "cell_type": "markdown", "id": "1a2596e6-f961-4824-b85d-2b8f25266fdd", "metadata": { "tags": [] }, "source": [ "**Option 1:** upload example images from your local files:" ] }, { "cell_type": "code", "execution_count": null, "id": "d4b78e16-384e-42fc-91ad-dd179356b6fe", "metadata": { "tags": [] }, "outputs": [], "source": [ "import os\n", "from google.colab import files\n", "\n", "# pick a name for the image folder\n", "local_dir = \"./my_folder\" #@param\n", "os.makedirs(local_dir)\n", "os.chdir(local_dir)\n", "\n", "# choose and upload local images into the newly created directory\n", "uploaded_images = files.upload()\n", "os.chdir(\"/content\") # back to parent directory" ] }, { "cell_type": "markdown", "id": "7d3894a5-1b56-4e4b-8e70-4cfb55093c33", "metadata": { "tags": [] }, "source": [ "**Option 2:** download example images from the hub - " ] }, { "cell_type": "code", "execution_count": 6, "id": "2f52ed6e-e2d2-4f2d-a2f9-4b6764597c4b", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 227, "referenced_widgets": [ "68ce05ca273947078ae7942354c95202", "a88eb760b5c64ce7a9aec82fa6b6bbe0", "161844ed3a8248c1922b9f17d22621c0", "7aa43d9582b645a5a02599383fa44d36", "55f7a2bc00ea4f458587babfebff5126", "6c1f5753924a41f7af89ca3ae798e77c", "8760f4ca9e16482781a589febf6909e7", "7b8dd551b1284da88042fd491994cbb9", "0cf44cf6a69443379dccaa79cec5e780", "2f149593cbca470eaf06bcc31f9d0667", "e1a61c1d18da4e03944d2053a2aa8651", "47da6fbbe5604d3fbd7902edf4cc6844", "dc7933bcfc62458e8bc88de7d4420f95", "f3e19dcf24f84ffca09ed8f30317fc56", "11895a605b094c338813daf94916e8ee", "0d97c96fa719432f8b208465bd3e3379", "2524317240ed44bf9fb24ee57aa2865c", "262dd2ba7cb845b7b663ba61f2c202d4", "9fd75c13c85042ff9a76ce1902578938", "0ffcb0d9084a48ccab436c96c06ee75f", "ad1d396bbcdb4c20a09d77396a05e72b", "82a18681fa9f4987896dac9eff481d2d", "8476735881ab4347afb17434fa2d5701", "b5627f61f8c84eefa022b39ab5be1fdd", "e2f16884010c45a2930de1c15200f5fd", "7d2662898dee4ab697b9691f700c476b", "06ab7fe22c3043a392ab774a5098c511", "f6549c46d2ff4092b3fc118692e46192", "5582978320bb4fd99f183a8aba5b8a92", "aca2097e3b6e4423ba988a346df2aaa0", "6e9a69addfc546b5a44ff788c7f338a7", "743683a87c0c458bb3f102ea6360c41d", "53d4c9cfaa1b4d14af3e87c2b1d32c86", "eb239061297046d1b90f2cef1a6a8c0b", "45262888159a49a2ac742f3aff1ea7b4", "96667878120345689fba6699754502cb", "d24e837755464ef1985f7651a8a6beec", "9aab71180e7945aaa495b24030f79887", "df0d0f760b554755b9fdc9fb8b79e409", "091bc4da1a2c4ef1bbc1b2c94fafcb1e", "8946c6414f5640de9809f8c7ec30c2ae", "ffc57fb137c849ea9d7edcaee9147a77", "83cd936fdce340729acc336cd8ceb496", "92cb855c250740c4951531bc6668b52b", "bac3a0570aa8408298920b027fc43f0a", "10f99bb4b70c4910be225cbd99a7f04a", "19987326cce5428bbc295c993e7770dc", "992ea999ffb342b9ace2d0469977645d", "63691269589d4a0aaf0c7374f2522fa9", "2bb0e2151fe948579c01faa7cdc24e22", "af41fe4a68d74172803daa14ac8528a8", "1c670206a7f546ef8cc0ed9bbb0c8c04", "94d5cbc7699341a78947ac9981167544", "91b2d632a8cb44e0b062cc4307a4e2c3", "47f316f445b8469186654570f843a58d", "bc2992bac933473a8f89f36d100e73ac", "dbdbb7736fa04406ac62d3fd55e5d21a", "dbe5a0ca62684dadab7b02718ece0e48", "22f10e02b0324cbd923a9bd2fe6a8f4a", "8ff68997ec824a58b2d23912278e51ac", "0af92ccc7ad74d818da1ddfd7dbb9f5b", "24d8e5af00774a5aa089b521d41f05c3", "a285360d05d349a28f66b177863d9a69", "6f75129a67204285b653d36ee8076021", "4c73e1a8622b4e6890780e6acd9b984d", "0250f4a5a02d406ead7824d58e5c781c" ] }, "id": "-BC36MOBdIWO", "outputId": "893332b6-b95c-419f-c72b-77a243d0bf63", "tags": [] }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "a3ee5b44c03b4af3acac8afe9d7aa907", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Fetching 24 files: 0%| | 0/24 [00:00" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import glob\n", "local_dir = \"./3d_icon\"\n", "img_paths = f\"{local_dir}/*.jpg\"\n", "imgs = [Image.open(path) for path in glob.glob(img_paths)]\n", "\n", "num_imgs_to_preview = 5\n", "image_grid(imgs[:num_imgs_to_preview], 1, num_imgs_to_preview)" ] }, { "cell_type": "markdown", "id": "0c321597-e4e2-43ee-a817-ccd20ac11365", "metadata": {}, "source": [ "### Generate custom captions with BLIP" ] }, { "cell_type": "markdown", "id": "46c4baf0-bdc1-4b9f-85c2-70ec67158a1a", "metadata": {}, "source": [ "Load BLIP2 to auto caption your images: \n", "\n", "\n", "**Note:** if you downloaded the `LinoyTsaban/3d_icon dataset` from the hub, you would find it already contains captions (generated with BLIP and prefixed with a token identifier) in the `metadata.jsonl` file\n", "You can skip this part if you wish to train on that dataset using the existing captions. " ] }, { "cell_type": "code", "execution_count": 9, "id": "c1dd96ce-328a-43e1-a707-64fef13b4512", "metadata": { "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2023-11-26 18:54:29.686931: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n", "To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n", "2023-11-26 18:54:30.425056: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "a681e100ce9d4cbab79feca16b881b85", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Loading checkpoint shards: 0%| | 0/2 [00:00" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import json\n", "from IPython.display import display, Markdown\n", "\n", "caption_prefix = \"3d icon in the style of TOK, \" #@param\n", "\n", "# saves each caption and corresponding image to a metadata.jsonl file\n", "with open(f'{local_dir}metadata.jsonl', 'w') as outfile:\n", " for img in imgs_and_paths:\n", " caption = caption_prefix + caption_images(img[1]).split(\"\\n\")[0]\n", " entry = {\"file_name\":img[0].split(\"/\")[-1], \"prompt\": caption}\n", " json.dump(entry, outfile)\n", " outfile.write('\\n')\n", " \n", "display(Markdown(f\"Your image captions are ready here: {local_dir}metadata.jsonl\"))" ] }, { "cell_type": "markdown", "id": "e0164ae3-0600-4611-88ab-3875aee93ef5", "metadata": {}, "source": [ "Free some memory:" ] }, { "cell_type": "code", "execution_count": 13, "id": "c6b90e89-176e-4ced-846c-ae47314719fd", "metadata": { "tags": [] }, "outputs": [], "source": [ "import gc\n", "\n", "# delete the BLIP2 pipelines and clear up some memory\n", "del blip_processor, blip_model\n", "gc.collect()\n", "torch.cuda.empty_cache()" ] }, { "cell_type": "markdown", "id": "5d5c2445-3ba1-4daf-9e35-84cf5b69af45", "metadata": { "id": "SqrxnDMUTADH", "tags": [] }, "source": [ "## Prep for training 💻" ] }, { "cell_type": "markdown", "id": "ae89b699-2fed-4a08-ac15-f57d4451e44b", "metadata": { "id": "U7OzF0i716Kf" }, "source": [ "Initialize `accelerate`:" ] }, { "cell_type": "code", "execution_count": 14, "id": "15c833e9-14d6-4e1d-8a79-24e5ef667290", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Configuration already exists at /home/ubuntu/.cache/huggingface/accelerate/default_config.yaml, will not override. Run `accelerate config` manually or pass a different `save_location`.\n" ] } ], "source": [ "!accelerate config default" ] }, { "cell_type": "markdown", "id": "46d4e562-22e1-4387-b7d7-c12018d9a5b4", "metadata": { "id": "Hc0bxTr918es" }, "source": [ "### Log into your Hugging Face account\n", "Pass [your **write** access token](https://huggingface.co/settings/tokens) so that we can push the trained checkpoints to the Hugging Face Hub:" ] }, { "cell_type": "code", "execution_count": 3, "id": "dac88815-fa14-4ef5-aa3f-0cfe1f66b79f", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "3Hf07sapecFY", "outputId": "b9440fd0-b0a1-4ce3-ba0d-9270484f72ba", "tags": [] }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "4b2aed906e0342d68090c171630e83ba", "version_major": 2, "version_minor": 0 }, "text/plain": [ "VBox(children=(HTML(value='
\u001B[0m\u001B[32;49m23.3.1\u001B[0m\n", "\u001B[1m[\u001B[0m\u001B[34;49mnotice\u001B[0m\u001B[1;39;49m]\u001B[0m\u001B[39;49m To update, run: \u001B[0m\u001B[32;49mpython3.10 -m pip install --upgrade pip\u001B[0m\n" ] } ], "source": [ "# makes sure we install datasets from main\n", "!pip install git+https://github.com/huggingface/datasets.git -q" ] }, { "cell_type": "markdown", "id": "7d4b79a1", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "#### 🤗Pick a name for your Dreambooth LoRA fine-tuned model:🤗\n", "This name will be used to save your model, so pick an informative name based on your chosen concept💡" ] }, { "cell_type": "code", "execution_count": 4, "id": "a24ad098", "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Defaulting to user installation because normal site-packages is not writeable\n", "\u001B[33mWARNING: Ignoring invalid distribution -etworkx (/usr/lib/python3/dist-packages)\u001B[0m\u001B[33m\n", "\u001B[0mLooking in indexes: https://pypi.org/simple/\n", "Requirement already satisfied: python-slugify in /home/ubuntu/.local/lib/python3.10/site-packages (8.0.1)\n", "Requirement already satisfied: text-unidecode>=1.3 in /home/ubuntu/.local/lib/python3.10/site-packages (from python-slugify) (1.3)\n", "\u001B[33mWARNING: Ignoring invalid distribution -etworkx (/usr/lib/python3/dist-packages)\u001B[0m\u001B[33m\n", "\u001B[0m\u001B[33mDEPRECATION: distro-info 0.23ubuntu1 has a non-standard version number. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of distro-info or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063\u001B[0m\u001B[33m\n", "\u001B[0m\u001B[33mDEPRECATION: python-debian 0.1.36ubuntu1 has a non-standard version number. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of python-debian or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063\u001B[0m\u001B[33m\n", "\u001B[0m\n", "\u001B[1m[\u001B[0m\u001B[34;49mnotice\u001B[0m\u001B[1;39;49m]\u001B[0m\u001B[39;49m A new release of pip is available: \u001B[0m\u001B[31;49m23.2\u001B[0m\u001B[39;49m -> \u001B[0m\u001B[32;49m23.3.1\u001B[0m\n", "\u001B[1m[\u001B[0m\u001B[34;49mnotice\u001B[0m\u001B[1;39;49m]\u001B[0m\u001B[39;49m To update, run: \u001B[0m\u001B[32;49mpython3.10 -m pip install --upgrade pip\u001B[0m\n" ] } ], "source": [ "!pip install python-slugify\n", "from slugify import slugify\n", "\n", "model_name = \"3d icon SDXL LoRA\" # @param\n", "output_dir = slugify(model_name)" ] }, { "cell_type": "markdown", "id": "94f7fcd1-dd0b-4aa7-8a0d-9cb388daf386", "metadata": {}, "source": [ "**Instance & Validation Prompt**\n", "* `instance_prompt` - \n", " * when custom captions are enabled this prompt is still used in case there are missing captions, as well as in the model's readme. \n", " * If custom captions are not used, this prompt will be used as the caption for all training images. \n", "* `validation_prompt` - \n", " * this prompt is used to generate images throught the training process, this way you can see the models learning curve during training. \n", " * you can also change `num_validation_images` (4 by default) and `validation_epochs` (50 by default) to control the amount images generated with the validation prompt, and the number of ephochs between each dreambooth validation. " ] }, { "cell_type": "code", "execution_count": 2, "id": "34b46540-fe25-4308-a486-760c526024bd", "metadata": { "tags": [] }, "outputs": [], "source": [ "instance_prompt = \"3d icon in the style of TOK\" # @param\n", "validation_prompt = \"a TOK icon of an astronaut riding a horse, in the style of TOK\" # @param" ] }, { "cell_type": "markdown", "id": "6dbd7ec5-d4e2-470f-97f2-011a30d04e63", "metadata": {}, "source": [ "**Set your LoRA rank**\n", "The rank of your LoRA is linked to its expressiveness.\n", "The bigger the rank the closer we are to regular dreambooth, and in theory we have more expressive power (and heavier weights). \n", "\n", "For a very simple concept that you have a good high quality image set for (e.g. a pet, a generic object), a rank as low as 4 can be enough to get great results. We reccomend going between 8 and 64 depending on your concept and how much of a priortiy it is for you to keep the LoRA small or not. " ] }, { "cell_type": "code", "execution_count": 3, "id": "a13c4644-7b6e-4500-8db0-6331e09366ef", "metadata": { "tags": [] }, "outputs": [], "source": [ "rank = 8 # @param" ] }, { "cell_type": "code", "execution_count": 6, "id": "7a0f20c3-f1e8-4ab1-9dc5-bb71cadea215", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2023-12-04 08:53:29.479966: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n", "To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n", "2023-12-04 08:53:30.305590: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n", "12/04/2023 08:53:32 - INFO - __main__ - Distributed environment: NO\n", "Num processes: 1\n", "Process index: 0\n", "Local process index: 0\n", "Device: cuda\n", "\n", "Mixed precision type: bf16\n", "\n", "You are using a model of type clip_text_model to instantiate a model of type . This is not supported for all configurations of models and can yield errors.\n", "You are using a model of type clip_text_model to instantiate a model of type . This is not supported for all configurations of models and can yield errors.\n", "{'dynamic_thresholding_ratio', 'variance_type', 'thresholding', 'clip_sample_range'} was not found in config. Values will be initialized to default values.\n", "{'attention_type', 'dropout', 'reverse_transformer_layers_per_block'} was not found in config. Values will be initialized to default values.\n", "0 text encodedr's std_token_embedding: 0.015381863340735435\n", "torch.Size([49410])\n", "1 text encodedr's std_token_embedding: 0.014391135424375534\n", "torch.Size([49410])\n", "Using decoupled weight decay\n", "Resolving data files: 100%|█████████████████| 24/24 [00:00<00:00, 169466.83it/s]\n", "12/04/2023 08:53:47 - WARNING - datasets.builder - Found cached dataset imagefolder (/home/ubuntu/.cache/huggingface/datasets/imagefolder/3d_icon-1d8dc1619b3c57b9/0.0.0/37fbb85cc714a338bea574ac6c7d0b5be5aff46c1862c1989b20e0771199e93f)\n", "100%|████████████████████████████████████████████| 1/1 [00:00<00:00, 759.70it/s]\n", "/home/ubuntu/.local/lib/python3.10/site-packages/PIL/Image.py:3182: DecompressionBombWarning: Image size (122880000 pixels) exceeds limit of 89478485 pixels, could be decompression bomb DOS attack.\n", " warnings.warn(\n", "/home/ubuntu/.local/lib/python3.10/site-packages/PIL/Image.py:3182: DecompressionBombWarning: Image size (132710400 pixels) exceeds limit of 89478485 pixels, could be decompression bomb DOS attack.\n", " warnings.warn(\n", "validation prompt: a icon of an astronaut riding a horse, in the style of \n", "\u001B[34m\u001B[1mwandb\u001B[0m: Currently logged in as: \u001B[33mlinoy\u001B[0m. Use \u001B[1m`wandb login --relogin`\u001B[0m to force relogin\n", "\u001B[34m\u001B[1mwandb\u001B[0m: wandb version 0.16.0 is available! To upgrade, please run:\n", "\u001B[34m\u001B[1mwandb\u001B[0m: $ pip install wandb --upgrade\n", "\u001B[34m\u001B[1mwandb\u001B[0m: Tracking run with wandb version 0.15.0\n", "\u001B[34m\u001B[1mwandb\u001B[0m: Run data is saved locally in \u001B[35m\u001B[1m/home/ubuntu/testing v2/wandb/run-20231204_085353-so9s09mr\u001B[0m\n", "\u001B[34m\u001B[1mwandb\u001B[0m: Run \u001B[1m`wandb offline`\u001B[0m to turn off syncing.\n", "\u001B[34m\u001B[1mwandb\u001B[0m: Syncing run \u001B[33mresilient-energy-474\u001B[0m\n", "\u001B[34m\u001B[1mwandb\u001B[0m: ⭐️ View project at \u001B[34m\u001B[4mhttps://wandb.ai/linoy/dreambooth-lora-sd-xl\u001B[0m\n", "\u001B[34m\u001B[1mwandb\u001B[0m: 🚀 View run at \u001B[34m\u001B[4mhttps://wandb.ai/linoy/dreambooth-lora-sd-xl/runs/so9s09mr\u001B[0m\n", "12/04/2023 08:53:53 - INFO - __main__ - ***** Running training *****\n", "12/04/2023 08:53:53 - INFO - __main__ - Num examples = 23\n", "12/04/2023 08:53:53 - INFO - __main__ - Num batches each epoch = 8\n", "12/04/2023 08:53:53 - INFO - __main__ - Num Epochs = 125\n", "12/04/2023 08:53:53 - INFO - __main__ - Instantaneous batch size per device = 3\n", "12/04/2023 08:53:53 - INFO - __main__ - Total train batch size (w. parallel, distributed & accumulation) = 3\n", "12/04/2023 08:53:53 - INFO - __main__ - Gradient Accumulation steps = 1\n", "12/04/2023 08:53:53 - INFO - __main__ - Total optimization steps = 1000\n", "Steps: 1%|▏ | 8/1000 [00:20<37:50, 2.29s/it, loss=0.094, lr=1]12/04/2023 08:54:14 - INFO - __main__ - Running validation... \n", " Generating 4 images with prompt: a icon of an astronaut riding a horse, in the style of .\n", "{'feature_extractor', 'image_encoder'} was not found in config. Values will be initialized to default values.\n", "\n", "Loading pipeline components...: 0%| | 0/7 [00:00 icon of an astronaut riding a horse, in the style of .\n", "{'feature_extractor', 'image_encoder'} was not found in config. Values will be initialized to default values.\n", "\n", "Loading pipeline components...: 0%| | 0/7 [00:00 icon of an astronaut riding a horse, in the style of .\n", "{'feature_extractor', 'image_encoder'} was not found in config. Values will be initialized to default values.\n", "\n", "Loading pipeline components...: 0%| | 0/7 [00:00" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from huggingface_hub import whoami\n", "from pathlib import Path\n", "from IPython.display import display, Markdown\n", "\n", "username = whoami(token=Path(\"/root/.cache/huggingface/\"))[\"name\"]\n", "repo_id = f\"{username}/{output_dir}\"\n", "\n", "link_to_model = f\"https://huggingface.co/{repo_id}\"\n", "\n", "display(Markdown(\"### Your model has finished training.\\nAccess it here: {}\".format(link_to_model)))" ] }, { "cell_type": "markdown", "id": "34fdd785-ad51-4792-ac0a-6b0aefffc0c1", "metadata": { "id": "bH7-YJwMcyra" }, "source": [ "## Inference 🐕" ] }, { "cell_type": "code", "execution_count": 8, "id": "844c968c-7f9d-4b83-8544-343979c982d0", "metadata": { "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2023-12-04 09:37:49.371395: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n", "To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n", "2023-12-04 09:37:50.221671: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "25dafb1a8a9c47bdbba334251c67a2aa", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Loading pipeline components...: 0%| | 0/7 [00:00\", \"\"], text_encoder=pipe.text_encoder, tokenizer=pipe.tokenizer)\n", "# load embeddings of text_encoder 2 (CLIP ViT-G/14)\n", "pipe.load_textual_inversion(state_dict[\"clip_g\"], token=[\"\", \"\"], text_encoder=pipe.text_encoder_2, tokenizer=pipe.tokenizer_2)" ] }, { "cell_type": "code", "execution_count": 21, "id": "8c6adf2f-3a7b-46a0-b52a-33409a2e6ce6", "metadata": { "pycharm": { "name": "#%%\n" }, "tags": [] }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "c7ac69bd381641dcb2a2d8386c0c09a8", "version_major": 2, "version_minor": 0 }, "text/plain": [ " 0%| | 0/25 [00:00" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "instance_token = \"\"\n", "prompt = f\"a {instance_token} icon of an orange llama eating ramen, in the style of {instance_token}\" \n", "\n", "image = pipe(prompt=prompt, num_inference_steps=25, cross_attention_kwargs={\"scale\": 1.0}).images[0]\n", "image\n", "\n", "\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.12" } }, "nbformat": 4, "nbformat_minor": 5 }