# 🦅 Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Models
Eagle 2.5 is a family of frontier vision-language models (VLMs) designed for long-context multimodal learning. While most existing VLMs focus on short-context tasks, Eagle 2.5 addresses the challenges of long video comprehension and high-resolution image understanding, providing a generalist framework for both. Eagle 2.5 supports up to 512 video frames and is trained jointly on image + video data.
This repository provides the end-to-end guidance and scripts for the environment setup, data preparation, training, and inference of the Eagle VLM.
## Highlight
### 🚀 Strong Results Across The Board
- SOTA on 6 out of 10 long video benchmarks
- Outperforms GPT-4o (0806) on 3/5 video tasks
- Outperforms Gemini 1.5 Pro on 4/6 video tasks
- Matches or outperforms Qwen2.5-VL-72B on multiple key datasets
- 72.4% on Video-MME with 512 input frames
- Strong image understanding with consistent improvement over Eagle 2, matching Qwen2.5-VL.
### 🎯 Key Innovations
- **Information-First Sampling**:
- *Image Area Preservation (IAP)*: Optimizes image tiling to retain most of the original image area and aspect ratio, preserving fine-grained details.
- *Automatic Degrade Sampling (ADS)*: Dynamically balances visual and textual input, ensuring complete text retention while maximizing visual content within context length constraints.
- **Progressive Mixed Post-Training**:
- Gradually increases context length during training, enhancing the model's ability to process varying input sizes and improving information density over static sampling.
- **Diversity-Driven Data Recipe**:
- Combines open-source data (human-annotated and synthetic) with the self-curated Eagle-Video-110K dataset, collected via a diversity-driven strategy and annotated with both story-level and clip-level QA pairs.
### ⚡ Efficiency & Framework Optimization
- **GPU Memory Optimization**:
- Integrate Triton-based fused operators replacing PyTorch’s MLP, RMSNorm, and RoPE implementations.
- Reduced GPU memory with fused linear layers + cross-entropy loss (removes intermediate logit storage) and CPU-offloading of hidden states.
- Sufficient to fit up to 32K context length with an 8B model on a single GPU.
- **Distributed Context Parallelism**:
- Adopts a two-layer communication group based on Ulysses and Ring/Context Parallelism building on USP.
- Implements ZigZag Llama3-style Context Parallelism with all-gather KV to reduce communication latency.
- **Video Decoding Acceleration**:
- Optimized sparse video frame sampling with rapid video metadata parsing, improved long video decoding and reduced memory consumption.
- **Inference Acceleration**:
- Supports vLLM deployment with reduced memory and accelerated inference.
## Model Details
- **Model Type**: Long-context vision-language model
- **Architecture**:
- Vision encoder: Siglip2-So400m-Patch16-512
- Language model: Qwen2.5-7B-Instruct
- Multimodal base architecture: LLaVA with tiling-based vision input
- **Supported Inputs**:
- Long video sequences (up to 512 frames)
- High-resolution images (up to 4K HD input size)
- Multi-page documents
- Long text
- **Training Strategy**:
- Progressive mixed post-training, expanding from 32K to 128K context length
- Information-first sampling for optimal visual and textual information retention
- **Training Data**:
- Open-source video and document datasets
- Eagle-Video-110K (110K long videos with dual-level annotation)
## Getting Started
### 📚 Onboarding
Recommended order:
1) Set environment variables → 2) Install → 3) Prepare data → 4) Train → 5) Demo → 6) Inference
- Onboarding overview: see `./document/0.onboarding.md`
### ⚙️ Installation & Environment
- Detailed steps and dependencies: `./document/1.installing.md`
- Conda environment (Python 3.10)
- PyTorch and FlashAttention (match your CUDA)
- Install this repo with `pip install -e .`
- Troubleshooting notes (specific Transformers version, OpenCV dependencies, etc.)
### 📂 Data Preparation (Playground)
- Directory structure and JSONL/LMDB examples: `./document/2.preparing_playground.md`
- `playground/sft_recipe` (data recipe)
- `playground/sft_jsonl` and `playground/sft_data` (annotations and raw data)
- Example parquet→LMDB conversion scripts are not included in this repo
- Use `shell/prepare.sh` to normalize and generate `.prepare.json` (internal `submit_prepare_job.sh` is not included)
- LMDB reading example and tips: `./document/how_to_use_lmdb_to_read_images.md`
### 💪 Training (Stage-2 / Finetuning)
- Full training entry points and multinode/multigpu options: `./document/3.training.md`
- Single-node example: `GPUS=8 bash shell/train_stage2.sh 1 work_dirs/eagle2.5_debug`
- Multi-node example (srun/internal submit_job): `PARTITION=xxx GPUS=16 bash shell/train_stage2.sh 2 work_dirs/eagle2.5_multinode`
### ✨ Launching Streamlit Demo
- Interactive testing of the VLM with UI. Refer to document for more details: `./document/4.streamlit_demo.md`
### 🔮 Inference
- End-to-end usage and multimodal examples (single/multiple images, single/multiple videos, streaming, batch): `./document/5.inference.md`
- Load with `transformers` `AutoModel`/`AutoProcessor`: `"nvidia/Eagle-2.5-8B"`
- Recommended `torch_dtype=torch.bfloat16`; run `model.generate(...)` on GPU
## Benchmark Results
### 🎥 Video Benchmarks
| Benchmark | GPT-4o | Gemini-1.5 Pro | InternVL2.5-8B | Qwen2.5-VL-8B | **Eagle2.5-8B** |
|--------------------------------------------|--------------------|-------------------|---------------------|---------------------|---------------------|
| MVBench