# Supported Model List
## Image Generation Models
- SDXL
- SD 1.5
- SD 2.0
- SD 2.1
- SD 3 Medium
- SD 3.5 Medium
- SD 3.5 Large
- Flux.1 Dev
- Flux.1 Schnell
- Flux.1 Kontext
- Flux.1 Krea
- Flux.2 Klein 4B
- Flux.2 Klein 9B
- Flux.3 Klein (Base + Distilled)
- CogView 4
- LongCat Image
- Z-Image Turbo
- Z-Image Base
- Krea 2 Turbo
- Krea 2 Raw
- Qwen Image
- Qwen Image 2.1
- Ideogram 4
- ERNIE-Image
- ERNIE-Image-Turbo
- Lumina Image 2.0
- Hidream I1 Full
- Auraflow
- MOSS-Image
- Fibo 2.0 Dev
## Image Edit Models
- SDXL Img2Img
- SD 1.5 Img2Img
- SD 2.1 Img2Img
- SD 3 Medium Img2Img
- SD 3.5 Medium Img2Img
- SD 3.5 Large Img2Img
- Flux.1 Kontext
- Flux.1 Fill Dev
- Flux.2 Klein 4B
- Flux.2 Klein 9B
- Flux.3 Klein (Base + Distilled)
- LongCat Image Edit
- Qwen Image Edit
- Qwen Image 2.1
- HiDream E1 Full
- JoyAI Image Edit
- Fibo 2.0 Edit Turbo
## Video Generation Models
- MiniMax-H3 (FL2VA)
- LTX 2.5 (Text To Video + Image To Video)
- Wan2.2-TI2V-5B (Text To Video + Image To Video)
---
## TODOs
- [ ] **ControlNet / T2I-Adapter Integration:** Enable structural conditioning (canny, depth, pose) while maintaining strict VRAM streaming budgets.
---
## Quick Start
### Local Setup
```bash
git clone https://github.com/Jit-Roy/weellm.git
cd weellm
pip install -r requirements.txt
```
### Kaggle / Colab Setup
```bash
!git clone https://github.com/Jit-Roy/weellm.git
%cd weellm
!pip install -r requirements.txt
```
### Kaggle / Colab Setup (via pip, no git clone needed)
```bash
!pip install -q git+https://github.com/Jit-Roy/weellm
```
### Running the CLI
Run the `main.py` script and explicitly pass your VRAM and RAM budgets (in GB) to ensure you stay within your hardware limits.
```bash
# Run directly from a Hugging Face repository ID!
python main.py \
--model "black-forest-labs/FLUX.1-dev" \
--prompt "A majestic lion at golden hour" \
--negative_prompt "blurry, distorted, low quality" \
--height 1024 \
--width 1024 \
--steps 20 \
--guidance_scale 3.5 \
--seed 42 \
--dtype bfloat16 \
--vram_budget 4 \
--ram_budget 4 \
--output "flux_lion.png" \
--verbose
# Run Text-to-Video from a Hugging Face repository ID!
python main.py \
--model "Lightricks/LTX-Video" \
--prompt "A drone flying over a snowy mountain peak at sunrise." \
--height 512 \
--width 704 \
--steps 10 \
--seed 12345 \
--dtype bfloat16 \
--vram_budget 4 \
--ram_budget 4 \
--output "output_video.mp4" \
--verbose
```
---
## ⚠️ Cloud vs. Local Execution Warning
> [!WARNING]
> **This streaming architecture is specifically optimized for local execution on modern PCs equipped with NVMe SSDs and modern GPUs. You will experience slowdowns on standard cloud instances due to restricted disk I/O speeds.**
---
## ⚠️ Autoregressive & Vision-Language Model Limitations
> Dense autoregressive text-to-image models (like GLM-Image or other Vision-Language Models) are **not practically supported** by SSD streaming without quantization.
> Unlike diffusion models which process latents in a few discrete steps (e.g., 4 to 20 steps), autoregressive models must generate tokens sequentially. A 1024x1024 image requires generating over 1000 tokens. Generating 1000 tokens means doing 1000 full forward passes through the massive text encoder.
> Streaming a 20GB model from an SSD 1000 times requires transferring ~20 Terabytes of data, which mathematically takes **several hours** to generate a single image. While the code technically executes without crashing under 4GB VRAM, the wait time is completely impractical. If you wish to run these autoregressive models efficiently, you must use 4-bit/8-bit quantization so the weights can sit entirely in System RAM or VRAM.
---
## Python API
### Text to Image
```python
from weellm import WeeTextToImagePipeline
import torch
pipe = WeeTextToImagePipeline.from_pretrained(
"Tongyi-MAI/Z-Image-Turbo",
device="cuda",
torch_dtype=torch.bfloat16,
vram_budget=4,
ram_budget=4
)
image = pipe.generate(
prompt="A serene Japanese zen garden at sunrise",
negative_prompt="blurry, low quality",
height=512,
width=512,
num_inference_steps=4,
guidance_scale=1.5,
seed=42,
)
image.save("output.png")
```
### Image to Image
```python
from weellm import WeeImageToImagePipeline
from PIL import Image
import torch
pipe = WeeImageToImagePipeline.from_pretrained(
"black-forest-labs/FLUX.1-dev",
device="cuda",
torch_dtype=torch.bfloat16,
vram_budget=4,
ram_budget=4
)
init_image = Image.open("input.jpg").convert("RGB")
image = pipe.generate(
prompt="A futuristic cyberpunk city",
negative_prompt="blurry, distorted",
image=init_image,
height=1024,
width=1024,
num_inference_steps=20,
guidance_scale=3.5,
seed=42,
)
image.save("output_i2i.png")
```
### Text to Video
```python
from weellm import WeeVideoPipeline
import torch
pipe = WeeVideoPipeline.from_pretrained(
"MiniMaxAI/MiniMax-H3",
device="cuda",
torch_dtype=torch.bfloat16,
text_encoder_path="unsloth/MiniMax-H3-GGUF/qwen3vl_32b_minimax_h3-Q4_K_M.gguf",
transformer_path="unsloth/MiniMax-H3-GGUF/minimax_h3_fl2va_pruned-Q5_0.gguf",
lora_weights="lightx2v/Minimax-h3-Turbo/minimax_h3_ref2v_turbo_4step_v0.1_bf16.safetensors"
)
video_frames = pipe(
prompt=(
"A fast sports car driving through a futuristic neon city at night, "
"rain reflecting the lights, 4K, cinematic lighting."
),
height=512,
width=512,
num_frames=56,
num_inference_steps=6,
seed=42,
)
video_frames.save("minimax_output.mp4", fps=24)
```
### Image to Video
```python
from weellm import WeeVideoPipeline
from diffusers.pipelines.ltx2.utils import DISTILLED_SIGMA_VALUES
from PIL import Image
import torch
pipe = WeeVideoPipeline.from_pretrained(
r"Lightricks/LTX-2.5-Diffusers",
device="cuda",
torch_dtype=torch.float16,
text_encoder_path="elix3r/gemma4-12b-with-proj-ltx-2.5-GGUF/gemma4-12b-with-proj-ltx-2.5-Q5_K_M.gguf",
transformer_path="Abiray/LTX-2.5-Distilled-GGUF/LTX-2.5-Distilled-Q4_K_M.gguf"
)
first_frame = Image.open(r"first_frame.png").convert("RGB")
last_frame = Image.open(r"last_frame.png").convert("RGB")
video_frames = pipe(
prompt=(
'The person naturally stands up from the seated position and calmly looks toward the mountains.'
'Smooth, realistic human movement, natural posture, stable camera, no sudden movements.'
),
first_frame=first_frame,
last_frame=last_frame,
num_frames=121,
frame_rate=24.0,
sigmas=DISTILLED_SIGMA_VALUES,
guidance_scale=1.0,
seed=42,
audio_guidance_scale=1.0,
stg_scale=0.0,
audio_stg_scale=0.0,
modality_scale=1.0,
audio_modality_scale=1.0,
)
output_path = "video.mp4"
video_frames.save(output_path, fps=24)
```
### Using GGUF Models
You can use quantized `.gguf` weights for the transformer or text encoders to save disk space and RAM. The weights will be streamed and dequantized on the fly.
```python
from weellm import WeeTextToImagePipeline
import torch
pipe = WeeTextToImagePipeline.from_pretrained(
"black-forest-labs/FLUX.2-klein-4B",
transformer_path="unsloth/FLUX.2-klein-4B-GGUF/flux-2-klein-4b-Q4_K_M.gguf",
text_encoder_path="unsloth/Qwen3-4B-GGUF/Qwen3-4B-Q5_K_M.gguf",
# If using SD/SDXL models, use `unet_path` instead of `transformer_path`
# unet_path="path/to/juggernaut-xl-v9-Q8_0.gguf",
device="cuda",
torch_dtype=torch.bfloat16,
vram_budget=4,
ram_budget=4
)
image = pipe.generate(
prompt="A majestic lion at golden hour",
height=1024,
width=1024,
num_inference_steps=4,
)
image.save("flux_klein_gguf.png")
```
---
## Star History
