{ "cells": [ { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "# Image generation with Sana and OpenVINO\n", "\n", "Sana is a text-to-image framework that can efficiently generate images up to 4096 × 4096 resolution developed by NVLabs. Sana can synthesize high-resolution, high-quality images with strong text-image alignment at a remarkably fast speed, deployable on laptop GPU. \n", "Core designs include: \n", "* **Deep compression autoencoder**: unlike traditional AEs, which compress images only 8×, we trained an AE that can compress images 32×, effectively reducing the number of latent tokens.\n", "* **Linear DiT**: authors replaced all vanilla attention in DiT with linear attention, which is more efficient at high resolutions without sacrificing quality.\n", "* **Decoder-only text encoder***: T5 replaced by modern decoder-only small LLM as the text encoder and designed complex human instruction with in-context learning to enhance the image-text alignment.\n", "* **Efficient training and sampling**: Proposed Flow-DPM-Solver to reduce sampling steps, with efficient caption labeling and selection to accelerate convergence.\n", "\n", "More details about model can be found in [paper](https://arxiv.org/abs/2410.10629), [model page](https://nvlabs.github.io/Sana/) and [original repo](https://github.com/NVlabs/Sana).\n", "In this tutorial, we consider how to optimize and run Sana model using OpenVINO.\n", "#### Table of contents:\n", "\n", "- [Prerequisites](#Prerequisites)\n", "- [Select model variant](#Select-model-variant)\n", "- [Convert and Optimize model with OpenVINO](#Convert-and-Optimize-model-with-OpenVINO)\n", " - [Convert model using Optimum Intel](#Convert-model-using-Optimum-Intel)\n", " - [Compress model weights](#compress-model-weights)\n", "- [Run OpenVINO model inference](#Run-OpenVINO-model-inference)\n", "- [Interactive demo](#Interactive-demo)\n", "\n", "\n", "### Installation Instructions\n", "\n", "This is a self-contained example that relies solely on its own code.\n", "\n", "We recommend running the notebook in a virtual environment. You only need a Jupyter server to start.\n", "For details, please refer to [Installation Guide](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/README.md#-installation-guide).\n", "\n", "\n" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Prerequisites\n", "[back to top ⬆️](#Table-of-contents:)" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import platform\n", "\n", "%pip install -q \"gradio>=4.19\" \"torch>=2.1\" \"transformers\" \"nncf>=2.14.0\" \"diffusers>=0.32.0\" \"opencv-python\" \"pillow\" \"peft>=0.7.0\" --extra-index-url https://download.pytorch.org/whl/cpu\n", "%pip install -q \"sentencepiece\" \"protobuf\"\n", "%pip install -q \"git+https://github.com/huggingface/optimum-intel.git\" --extra-index-url https://download.pytorch.org/whl/cpu\n", "%pip install -qU \"openvino>=2024.6.0\"\n", "\n", "if platform.system() == \"Darwin\":\n", " %pip install \"numpy<2.0\"" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "from pathlib import Path\n", "import requests\n", "\n", "helpers = [\"notebook_utils.py\", \"cmd_helper.py\"]\n", "base_url = \"https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils\"\n", "\n", "for helper in helpers:\n", " if not Path(helper).exists():\n", " r = requests.get(f\"{base_url}/{helper}\")\n", " with open(helper, \"w\") as f:\n", " f.write(r.text)\n", "\n", "if not Path(\"gradio_helper.py\").exists():\n", " r = requests.get(url=\"https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/notebooks/sana-image-generation/gradio_helper.py\")\n", " open(\"gradio_helper.py\", \"w\").write(r.text)\n", "\n", "# Read more about telemetry collection at https://github.com/openvinotoolkit/openvino_notebooks?tab=readme-ov-file#-telemetry\n", "from notebook_utils import collect_telemetry\n", "\n", "collect_telemetry(\"sana-image-generation.ipynb\")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Select model variant\n", "[back to top ⬆️](#Table-of-contents:)" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "test_replace": { "Efficient-Large-Model/Sana_600M_512px_diffusers": "katuni4ka/tiny-random-sana" } }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "8023a00720e44af7aecf96114b790e6f", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Dropdown(description='Model:', options=('Efficient-Large-Model/Sana_600M_512px_diffusers', 'Efficient-Large-Mo…" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import ipywidgets as widgets\n", "\n", "model_ids = [\n", " \"Efficient-Large-Model/Sana_600M_512px_diffusers\",\n", " \"Efficient-Large-Model/Sana_600M_1024px_diffusers\",\n", " \"Efficient-Large-Model/Sana_1600M_1024px_BF16_diffusers\",\n", " \"Efficient-Large-Model/Sana_1600M_2Kpx_BF16_diffusers\",\n", " \"Efficient-Large-Model/Sana_1600M_4Kpx_BF16_diffusers\",\n", "]\n", "\n", "model_selector = widgets.Dropdown(\n", " options=model_ids,\n", " default=model_ids[0],\n", " description=\"Model:\",\n", ")\n", "\n", "\n", "model_selector" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Convert and Optimize model with OpenVINO\n", "[back to top ⬆️](#Table-of-contents:)\n", "\n", "Starting from 2023.0 release, OpenVINO supports PyTorch models directly via Model Conversion API. `ov.convert_model` function accepts instance of PyTorch model and example inputs for tracing and returns object of `ov.Model` class, ready to use or save on disk using `ov.save_model` function. \n", "\n", "\n", "The pipeline consists of four important parts:\n", "\n", "* Gemma Text Encoder to create condition to generate an image from a text prompt.\n", "* Transformer for step-by-step denoising latent image representation.\n", "* Deep Compression Autoencoder (DCAE) for decoding latent space to image.\n", " \n", "### Convert model using Optimum Intel\n", "[back to top ⬆️](#Table-of-contents:)\n", "\n", "For convenience, we will use OpenVINO integration with HuggingFace Optimum. 🤗 [Optimum Intel](https://huggingface.co/docs/optimum/intel/index) is the interface between the 🤗 Transformers and Diffusers libraries and the different tools and libraries provided by Intel to accelerate end-to-end pipelines on Intel architectures.\n", "\n", "Among other use cases, Optimum Intel provides a simple interface to optimize your Transformers and Diffusers models, convert them to the OpenVINO Intermediate Representation (IR) format and run inference using OpenVINO Runtime. `optimum-cli` provides command line interface for model conversion and optimization. \n", "\n", "General command format:\n", "\n", "```bash\n", "optimum-cli export openvino --model --task \n", "```\n", "\n", "where task is task to export the model for, if not specified, the task will be auto-inferred based on the model (in case of image generation, **text-to-image** should be selected). You can find a mapping between tasks and model classes in Optimum TaskManager [documentation](https://huggingface.co/docs/optimum/exporters/task_manager). Additionally, you can specify weights compression using `--weight-format` argument with one of following options: `fp32`, `fp16`, `int8` and `int4`. Fro int8 and int4 [nncf](https://github.com/openvinotoolkit/nncf) will be used for weight compression. More details about model export provided in [Optimum Intel documentation](https://huggingface.co/docs/optimum/intel/openvino/export#export-your-model)." ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "test_replace": { "additional_args = {\"variant\": variant, \"weight-format\": \"fp16\"}": "additional_args = {\"weight-format\": \"fp16\"}" } }, "outputs": [], "source": [ "from pathlib import Path\n", "\n", "model_id = model_selector.value\n", "variant = \"fp16\" if \"BF16\" not in model_id else \"bf16\"\n", "\n", "model_dir = Path(model_id.split(\"/\")[-1])\n", "\n", "additional_args = {\"variant\": variant, \"weight-format\": \"fp16\"}" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "from cmd_helper import optimum_cli\n", "\n", "if not model_dir.exists():\n", " optimum_cli(model_id, model_dir, additional_args=additional_args)" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "### Compress model weights\n", "[back to top ⬆️](#Table-of-contents:)\n", "\n", "For reducing model memory consumption we will use weights compression. The [Weights Compression](https://docs.openvino.ai/2024/openvino-workflow/model-optimization-guide/weight-compression.html) algorithm is aimed at compressing the weights of the models and can be used to optimize the model footprint and performance of large models where the size of weights is relatively larger than the size of activations, for example, Large Language Models (LLM). Compared to INT8 compression, INT4 compression improves performance even more, but introduces a minor drop in prediction quality. We will use [NNCF](https://github.com/openvinotoolkit/nncf) for transformer weight compression." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "affae3655f5441dfb72427546204d2f2", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Checkbox(value=True, description='Weight compression')" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "to_compress = widgets.Checkbox(\n", " value=True,\n", " description=\"Weight compression\",\n", " disabled=False,\n", ")\n", "\n", "to_compress" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "test_replace": { "group_size=64": "group_size=-1" } }, "outputs": [ { "data": { "text/html": [ "
Applying Weight Compression ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% • 0:00:15 • 0:00:00\n",
       "
\n" ], "text/plain": [ "Applying Weight Compression \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[35m100%\u001b[0m • \u001b[38;2;0;104;181m0:00:15\u001b[0m • \u001b[38;2;0;104;181m0:00:00\u001b[0m\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import openvino as ov\n",
    "import nncf\n",
    "import gc\n",
    "\n",
    "compressed_transformer = Path(model_dir) / \"transformer/openvino_model_i4.xml\"\n",
    "\n",
    "if to_compress.value and not compressed_transformer.exists():\n",
    "    core = ov.Core()\n",
    "\n",
    "    ov_model = core.read_model(model_dir / \"transformer/openvino_model.xml\")\n",
    "\n",
    "    compressed_model = nncf.compress_weights(ov_model, mode=nncf.CompressWeightsMode.INT4_SYM, group_size=64, ratio=1.0)\n",
    "    ov.save_model(compressed_model, compressed_transformer)\n",
    "    del compressed_model\n",
    "    del ov_model\n",
    "\n",
    "    gc.collect();"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run OpenVINO model inference\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "`OVDiffusionPipeline` from Optimum Intel provides ready-to-use interface for running Diffusers models using OpenVINO. It supports various models including Stable Diffusion, Stable Diffusion XL, LCM, Stable Diffusion v3 and Flux. Similar to original Diffusers pipeline, for initialization, we should use `from_preptrained` method providing model id from HuggingFace hub or local directory (both original PyTorch and OpenVINO models formats supported, in the first case model class additionally will trigger model conversion)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6901a172b9204e569d83e14cdee2b1e6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Dropdown(description='Device:', options=('CPU', 'AUTO'), value='CPU')"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from notebook_utils import device_widget\n",
    "\n",
    "device = device_widget(default=\"CPU\", exclude=[\"NPU\"])\n",
    "device"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from optimum.intel.openvino import OVDiffusionPipeline\n",
    "\n",
    "ov_pipe = OVDiffusionPipeline.from_pretrained(model_dir, device=device.value, transformer_file_name=compressed_transformer.name if to_compress.value else None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c628edbecf33450caccd8cb5f09173f8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/20 [00:00"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "prompt = \"Cute 🐶 Wearing 🕶 flying on the 🌈\"\n",
    "\n",
    "image = ov_pipe(\n",
    "    prompt,\n",
    "    generator=torch.Generator(\"cpu\").manual_seed(1234563),\n",
    ").images[0]\n",
    "\n",
    "image"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Interactive demo\n",
    "[back to top ⬆️](#Table-of-contents:)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gradio_helper import make_demo\n",
    "\n",
    "demo = make_demo(ov_pipe)\n",
    "\n",
    "# if you are launching remotely, specify server_name and server_port\n",
    "#  demo.launch(server_name='your server name', server_port='server port in int')\n",
    "# if you have any issue to launch on your platform, you can pass share=True to launch method:\n",
    "# demo.launch(share=True)\n",
    "# it creates a publicly shareable link for the interface. Read more in the docs: https://gradio.app/docs/\n",
    "try:\n",
    "    demo.launch(debug=True)\n",
    "except Exception:\n",
    "    demo.launch(debug=True, share=True)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  },
  "openvino_notebooks": {
   "imageUrl": "https://github.com/user-attachments/assets/bacfcd2a-ac36-4421-9d1b-4e34aa0a9f62",
   "tags": {
    "categories": [
     "Model Demos",
     "AI Trends"
    ],
    "libraries": [],
    "other": [
     "Stable Diffusion"
    ],
    "tasks": [
     "Text-to-Image"
    ]
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}