{ "cells": [ { "cell_type": "markdown", "id": "318c88c4-3b67-4a8a-9200-7e88ff9f08b2", "metadata": {}, "source": [ "# Warning\n", "\n", "[⚠️ For cost-efficiency, we no longer host a DALL-E Flow demo server. Click here to see how you can deploy it for yourself!](https://github.com/jina-ai/dalle-flow#server)\n" ] }, { "cell_type": "markdown", "id": "e13ed1c4", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "

\n", "\"DALL·E\n", "
\n", "

\n", "\n", "\n", "A Human-in-the-loop? workflow for creating HD images from text" ] }, { "cell_type": "markdown", "id": "0836667a", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "[![GitHub Repo stars](https://img.shields.io/github/stars/jina-ai/dalle-flow?style=social)](https://github.com/jina-ai/dalle-flow) [![Google Colab](https://img.shields.io/badge/Slack-2.8k-blueviolet?logo=slack&logoColor=white&style=flat-square)](https://slack.jina.ai) [![GitHub last commit (branch)](https://img.shields.io/github/last-commit/jina-ai/dalle-flow/main)](https://colab.research.google.com/github/jina-ai/dalle-flow/blob/main/client.ipynb)" ] }, { "cell_type": "markdown", "id": "f11f924b", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "\n", "\n", "🚧 If you find your request fails, it is possible the server is occasionally down for maintaince. Please give it a try in 5 minutes.\n", "\n", "- ⚠️ **2022/8/8** Started using CLIP-as-service as an [external executor](https://docs.jina.ai/fundamentals/flow/add-executors/#external-executors). Now you can easily [deploy your own CLIP executor](#run-your-own-clip) if you want. There is [a small breaking change](https://github.com/jina-ai/dalle-flow/pull/74/files#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5R103) as a result of this improvement, so [please _reopen_ the notebook in Google Colab](https://colab.research.google.com/github/jina-ai/dalle-flow/blob/main/client.ipynb).\n", "- ⚠️ **2022/7/6** Due to server migration to AWS EKS, server url changed to `grpcs://dalle-flow.dev.jina.ai`. All connections are now with TLS again, [Please _reopen_ the notebook in Google Colab!](https://colab.research.google.com/github/jina-ai/dalle-flow/blob/main/client.ipynb)\n", "- ⚠️ **2022/6/25** Unexpected downtime between 6/25 0:00 - 12:00 CET due to out of GPU quotas. The new server now has 2 GPUs, add healthcheck.\n", "- 2022/6/3 Reduce default number of images to 2 per pathway, 4 for diffusion.\n", "- ⚠️ 2022/5/23 Fix an upstream bug in CLIP-as-service. This bug makes the 2nd diffusion step irrelevant to the given texts. New Dockerfile proved to be reproducible on a AWS EC2 `p2.x8large` instance.\n", "- 2022/5/13b Removing TLS as Cloudflare gives 100s timeout, making DALLE Flow in usable [Please _reopen_ the notebook in Google Colab!](https://colab.research.google.com/github/jina-ai/dalle-flow/blob/main/client.ipynb).\n", "- 🔐 2022/5/13 New Mega checkpoint! All connections are now with TLS, [Please _reopen_ the notebook in Google Colab!](https://colab.research.google.com/github/jina-ai/dalle-flow/blob/main/client.ipynb).\n", "- 🌟 2022/5/10 [A Dockerfile is added! Now you can easily deploy your own DALL·E Flow](https://github.com/jina-ai/dalle-flow). New Mega checkpoint! Smaller memory-footprint, the whole Flow can now fit into **one GPU with 21GB memory**.\n", "- 🌟 2022/5/9 Less memory-footprint overall, the whole Flow can now fit into one GPU with 18GB memory!\n", "- 🌟 2022/5/7 DALL·E Flow just got updated!\n", " - New DALL·E Mega checkpoint\n", " - Improved GLID3 memory-efficiency and parameters\n", "- 🌟 2022/5/6 DALL·E Flow just got updated!\n", " - The first step will generate 16 candidates: **8 from DALL·E Mega, 8 from GLID3-XL**; ranked by CLIP-as-service.\n", " - Optimized the flow efficiency, diffusion and upscaling is much faster now!\n", "- ~~⚠️ 2022/5/3 **The number of images is restrict to 9 for DALL·E Mega, and 16 for GLID3-XL**~~\n", "- ⚠️ 2022/5/2 **Due to the massive requests now, the server is super busy.** You can deploy your own server by following [the instruction here](https://github.com/jina-ai/dalle-flow#server).\n" ] }, { "cell_type": "markdown", "id": "e6af8cfd", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "Using client is super easy. The following steps are best run in Jupyter notebook or [Google Colab](https://colab.research.google.com/github/jina-ai/dalle-flow/blob/main/client.ipynb). \n", "\n", "The only dependency you will need are [DocArray](https://github.com/jina-ai/docarray) and [Jina](https://github.com/jina-ai/jina), as DocArray is already included in Jina you only need to install `jina`.\n", "\n", "> On Google Colab, you will be asked to restart the kernel. Go ahead and restart." ] }, { "cell_type": "code", "execution_count": null, "id": "c95e266f", "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "!pip install jina" ] }, { "cell_type": "markdown", "id": "3059e59c", "metadata": { "pycharm": { "name": "#%%\n" } }, "source": [ "We have provided a demo server for you to play:" ] }, { "cell_type": "code", "execution_count": null, "id": "3f1bb189", "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "server_url = 'grpcs://dalle-flow.dev.jina.ai'" ] }, { "cell_type": "markdown", "id": "72aacbed", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "### Step 1: Generate via DALL·E Mega\n", "\n", "Now let's define the prompt:" ] }, { "cell_type": "code", "execution_count": null, "id": "ac1ec094", "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "prompt = 'an oil painting of a humanoid robot playing chess in the style of Matisse'" ] }, { "cell_type": "markdown", "id": "de6d7244", "metadata": {}, "source": [ "Do you need some hints and suggestions on the prompt? Check out those tricks:\n", " - [A Guide to Writing Prompts for Text-to-image AI](https://docs.google.com/document/d/17VPu3U2qXthOpt2zWczFvf-AH6z37hxUbvEe1rJTsEc/edit?usp=sharing)\n", " - [CLIP Templates](https://docs.google.com/document/d/1j2IAumYz4iZopOTAAOcCUKbFXP0jHK8mRgD4NLFKkaw/edit?usp=sharing)" ] }, { "cell_type": "markdown", "id": "69d0b10b", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "Let's submit it to the server and visualize the results:" ] }, { "cell_type": "code", "execution_count": null, "id": "da672930", "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "%%time\n", "\n", "from docarray import Document\n", "\n", "doc = Document(text=prompt).post(server_url, parameters={'num_images': 2})\n", "da = doc.matches\n", "\n", "da.plot_image_sprites(fig_size=(10,10), show_index=True)" ] }, { "cell_type": "markdown", "id": "9298f59f", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "Here we generate 4 candidates, 2 from DALLE-mega and 2 from GLID3 XL, this is as defined in `num_images`, which takes about ~2 minutes. You can use a smaller value if it is too long for you. The results are sorted by [CLIP-as-service](https://github.com/jina-ai/clip-as-service), with index-`0` as the best candidate judged by CLIP. \n", "\n", "> You could generate up to 8 images per pathway via `num_images`, resulting 16 candidate images in total. But it will be much slower." ] }, { "cell_type": "markdown", "id": "3bf3a62b", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "### Step 2: Select and refinement via GLID3 XL\n", "\n", "Of course, you may think differently. Notice the number in the top-left corner? Select the one you like the most and get a better view:" ] }, { "cell_type": "code", "execution_count": null, "id": "8e106718", "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "fav_id = 3\n", "\n", "fav = da[fav_id]\n", "fav.embedding = doc.embedding\n", "\n", "fav.display()" ] }, { "cell_type": "markdown", "id": "eec3e4ea", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "Now let's submit the selected candidates to the server for diffusion." ] }, { "cell_type": "code", "execution_count": null, "id": "13cf87ce", "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "%%time\n", "\n", "diffused = fav.post(f'{server_url}', parameters={'skip_rate': 0.6, 'num_images': 4}, target_executor='diffusion').matches\n", "\n", "diffused.plot_image_sprites(fig_size=(10,10), show_index=True)" ] }, { "cell_type": "markdown", "id": "fbb54ed9", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "This will give 4 images based on the given image. You may allow the model to improvise more by giving `skip_rate` a near-zero value, or a near-one value to force its closeness to the given image. The whole procedure takes about ~1 minutes.\n", "\n", "> You could generate upto 8 images via `num_images`. But it will be slower." ] }, { "cell_type": "markdown", "id": "978ccf66", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "### Step 3: Select and upscale via SwinIR\n", "\n", "Select the image you like the most, and give it a closer look:\n" ] }, { "cell_type": "code", "execution_count": null, "id": "7f00f44e", "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "dfav_id = 2\n", "\n", "fav = diffused[dfav_id]\n", "\n", "fav.display()" ] }, { "cell_type": "markdown", "id": "ab54a790", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "If not satisfied, you can rerun the last cell in step 2 with this new `fav`.\n", "\n", "Finally, submit to the server for the last step: upscaling to 1024 x 1024px.\n", "\n", "> This step should be much faster (~10s) as the Flow is designed in unblocked manner." ] }, { "cell_type": "code", "execution_count": null, "id": "b60e8bfc", "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "%%time\n", "\n", "fav = fav.post(f'{server_url}/upscale')\n", "fav.display()" ] }, { "cell_type": "markdown", "id": "e910e8aa", "metadata": { "pycharm": { "name": "#%%\n" } }, "source": [ "> 💁‍♂️ On Google colab this image may render exactly the same size as before. But it is in 1024x1024 already. Right click on the image and copy/save it. You will see.\n", "\n", "That's it! It is _the one_.\n", "\n", "\n", "Btw, [DocArray is a powerful and easy-to-use data structure for unstructured data](https://github.com/jina-ai/docarray), that makes all of these possible. It is super productive for data scientists who work in cross-/multi-modal domain. To learn more about DocArray, [please check out the docs](https://docarray.jina.ai)." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.8" } }, "nbformat": 4, "nbformat_minor": 5 }