Web-Shepherd: Advancing Process Reward Models for Web Agents
[](assets/WebShepherd.pdf)
[](https://huggingface.co/collections/LangAGI-Lab/web-shepherd-advancing-prms-for-reinforcing-web-agents-682b4f4ad607fc27c4dc49e8)
[](https://huggingface.co/datasets/LangAGI-Lab/WebPRMCollection_preference_pair)
[](https://demo.web-shepherd.io/)
Web-Shepherd is the **first process reward model (PRM)** designed specifically for web agents. It evaluates trajectories at the step level to provide interpretable and cost-efficient feedback for both learning and inference-time decision making in web navigation tasks.
---
## ๐ Overview
Recent multimodal language models (MLLMs) have made progress in web automation but struggle with long-horizon planning and cost efficiency. To tackle this, Web-Shepherd introduces:
- **WebPRM Collection**: A dataset with 40K+ step-level preference annotations and structured checklists.
- **WebRewardBench**: A benchmark to test PRM effectiveness across diverse web tasks.
- **Web-Shepherd**: A PRM trained to provide step-wise feedback and reward using structured subgoal checklists.
---
## ๐ง Key Features
- โ Step-level trajectory evaluation for web agents
- ๐งพ Checklist-guided reward modeling for interpretability and progress awareness
- ๐ฐ 100ร cost reduction compared to prompting GPT-4o
- ๐ Outperforms GPT-4o-mini by **10.9 points** on WebArena-lite
- ๐ Various use cases, including RL, Trajectory Search, and Reflexion
---
## ๐ Assets
| Resource | Description | Size | Link |
|----------|-------------|------|------|
| WebPRM Collection | Step-level preference annotations with structured checklists | 40K+ annotations | [๐ค huggingface](https://huggingface.co/datasets/LangAGI-Lab/WebPRMCollection_preference_pair) |
| WebRewardBench | Comprehensive benchmark for evaluating PRM effectiveness | 1000+ test cases | [๐ค huggingface](https://huggingface.co/datasets/LangAGI-Lab/WebRewardBench) |
| Web-Shepherd (3B, text-only) | Trained process reward model for web navigation | 3B parameters | [๐ค huggingface](https://huggingface.co/WebShepherd/web-shepherd-base) |
| Web-Shepherd (3B, ๐ผ๏ธ multimodal) | Trained process reward model for web navigation | 3B parameters | [Coming Soon!]() |
| Web-Shepherd (8B, text-only) | Large version of Web-Shepherd | 8B parameters | [๐ค huggingface](https://huggingface.co/WebShepherd/web-shepherd-large) |
## WebPRM Collection
The WebPRM Collection contains over 40K step-level preference annotations across diverse web tasks. Each annotation includes:
- **Structured Checklists**: Task-specific criteria for evaluating step quality
- **Step-level Preferences**: Human annotations comparing trajectory steps
- **Rich Metadata**: Task context, action types, and rationales
The dataset enables training process reward models that can provide interpretable feedback for web navigation.
## Web-Shepherd
Web-Shepherd is a process reward model trained on the WebPRM Collection to provide step-level feedback for web navigation tasks. Key components include:
- **Checklist Generation**: Processes text and visual inputs from web pages
- **Reward Prediction with Checklist**: Incorporates structured evaluation criteria
The model achieves strong performance while being significantly more cost-efficient than alternatives like GPT-4o.
## ๐ Project Structure
```bash
.
โโโ webshepherd/ # Source code for the model
โ โโโ models/ # Model architecture & training
โ โโโ data/ # Dataset loading and processing
โ โโโ inference/ # Inference utilities (e.g., Best-of-n, feedback)
โโโ scripts/ # Training & evaluation scripts
โโโ configs/ # Model configs and experiment settings
โโโ data/ # Downloaded or generated datasets
โ โโโ webprm_collection/ # Annotated instructions, checklists, and actions
โ โโโ webrewardbench/ # Meta-evaluation benchmark
โโโ demo/ # Hugging Face Space setup (optional)
โโโ results/ # Logs and evaluation outputs
โโโ README.md
```
