OpenClaw-RL

Empowering OpenClaw with RL — Train a personalized agent simply by talking to it.

Scalable RL in real-world settings — Agentic RL for terminal, GUI, SWE, and tool-call settings.

## 📰 News - **[2026/4/15]** 🙌 We sincerely thank [Fireworks AI](https://fireworks.ai) for its generous support of this project, which has enabled more experiments and faster iteration. - **[2026/4/11]** ✨ Qwen3.5-4B/9B/27B is supported now, both text and multi-modal! - **[2026/4/4]** 👨‍👦‍👦 We support optimizing a single model based on feedback from a group of people. - **[2026/3/25]** 🙌 We sincerely thank [Tinker](https://thinkingmachines.ai/tinker/) for its generous support of this project, which has enabled more experiments and faster iteration. - **[2026/3/20]** 💻 You can use your own openclaw now, simply install [this extension](https://github.com/Gen-Verse/OpenClaw-RL/tree/main/extensions/rl-training-headers). - **[2026/3/13]** ☁️ OpenClaw-RL now supports both local GPU and cloud ([Tinker](https://thinkingmachines.ai/tinker/)) deployment. Launch with [**one line of code**](#combinemethod) — Hybrid RL, OPD, and Binary RL all supported! - **[2026/3/12]** ⚡ We support LoRA training now! - **[2026/3/10]** 📃 We have released our [**Technical Report**](https://arxiv.org/abs/2603.10165)! 🏆 Ranked **#1** on [HuggingFace Daily Papers](https://huggingface.co/papers/2603.10165)! - **[2026/3/10]** 🔥 Huge updates today! We released a [new combination method](./openclaw-combine), along with an [interesting evaluation](./openclaw-test) of these OpenClaw-RL methods. Track 2 is released too, featuring scalable RL implementations for general agent settings across [terminal](./terminal-rl), [GUI](./gui-rl), [SWE](./swe-rl), and [tool-call](./toolcall-rl) scenarios. We only focus on real-world settings! - **[2026/3/3]** 🙌 Working with the authors of [SDFT](https://arxiv.org/abs/2601.19897) and [SDPO](https://arxiv.org/abs/2601.20802), we have integrated their methods into [openclaw-opd](./openclaw-opd). We welcome the integration of novel and effective methods! - **[2026/3/3]** 📺 Check out these community tutorial videos on OpenClaw-RL: [Video 1](https://www.youtube.com/watch?v=5xnm1vB7G64) | [Video 2](https://www.youtube.com/watch?v=ZtN6Gg_bdJE) - **[2026/2/26]** 🔥 We release **OpenClaw-RL v1** — a fully asynchronous RL framework for training personalized AI agents from natural conversation feedback. --- ## 💡 TL;DR > **OpenClaw-RL** is a fully asynchronous reinforcement learning framework that turns everyday conversations into training signals for personalized AI agents, and supports training general agents with large-scale environment parallelization. Most RL-for-LLM systems assume centralized, batch-mode training with pre-collected datasets. **OpenClaw-RL** takes a fundamentally different approach: it wraps your self-hosted model in [OpenClaw](https://openclaw.ai) as an OpenAI-compatible API, intercepts live multi-turn conversations, and continuously optimizes the policy in the background — all without interrupting your usage.

Overview

> **Highlights:** Fully async 4-component loop · Self-hosted & private · Zero manual labeling · Three learning paradigms (Binary RL / OPD / Combine) · Personal + General agent support

🌈 Features

### Fully Asynchronous 4-Component Architecture OpenClaw-RL decouples **agent serving**, **rollout collection**, **PRM/judge evaluation**, and **policy training** into independent async loops. None of them block one another: the model continues serving requests while training runs in the background, and judging happens concurrently with new interactions. ### Self-Hosted & Private by Design The entire stack, including the **policy model**, **judge/PRM**, and **trainer**, runs on **your own infrastructure**. Conversation data stays within your system, and no third-party model API is required. ### From Feedback to Gradient — Automatically You do not need to manually label data. The system automatically: - Organizes multi-turn interactions into session-aware training trajectories - Classifies API messages into **main-line** (trainable) vs. **side** (non-trainable) turns - Uses the next user, environment, or tool feedback as a natural "next-state" signal - Runs PRM/judge evaluation asynchronously, with majority voting when needed for more robust scoring - Submits ready samples to the trainer as they become available ### Three Optimization Methods in One Framework **Binary RL (GRPO):** A Process Reward Model scores each turn based on next-state feedback. The scalar reward is then used with GRPO advantage estimation and a PPO-style clipped surrogate loss. **On-Policy Distillation (OPD):** When the next state reveals useful hindsight, a judge model extracts a textual hint. This hint augments the original prompt to create an enhanced teacher, whose token-level log-probability gap with the student becomes a directional advantage signal richer than any scalar reward. **Hybrid Method:** OpenClaw-RL further combines Binary RL and OPD in a unified training recipe, leveraging the dense scalar supervision of Binary RL together with the richer token-level directional signal from OPD. This combination achieves stronger and more robust optimization than either method alone. ### From Personal Agents to Real-World Agentic RL The same framework supports both personalized OpenClaw optimization and scalable RL for **terminal**, **GUI**, **SWE**, and **tool-call** agents in real-world settings.

--- ## 🎯 Roadmap Our long-term goal is to **advance personalized, practically useful agents with reinforcement learning**. The roadmap has two tracks: #### Track 1 — [Personal Agent Optimization](#personalagent) (Small-Scale but Personal) ✅ **Release Track 1:** Fully async OpenClaw-RL framework with Binary RL + OPD ✅ Best recipe discovery via demonstration experiments ✅ Support LoRA Training ✅ Deploy training on [Tinker](https://thinkingmachines.ai/tinker/) ✅ Deploy training on [Fireworks AI](https://fireworks.ai) #### Track 2 — [General Agents Optimization](#generalagent) (Scalable Infra) ✅ **Release Track 2:** Scalable agentic RL infra for general agents ✅ Support Qwen3.5 ⬜ Support more cloud services ## 📝 Contents - [Personal OpenClaw Optimization](#personalagent) - [Bybrid RL](#combinemethod) - [Method Evaluation](#evalmethod) - [Agentic RL in Real World Settings](#agentrl) - [Terminal Agent](#terminal) - [GUI Agent](#gui) - [SWE Agent](#swe) - [Tool-call Agent](#toolcall) --- ## 🔧 Personal Agent Optimization Quick Start ### 1. Deployment Requirements - **Hardware:** 8× GPUs (default; configurable via `NUM_GPUS`, `ACTOR_GPUS`, `ROLLOUT_GPUS`, `PRM_GPUS`) - **Software:** CUDA 12.9, Python 3.12 - **Framework:** [Slime](https://github.com/THUDM/slime) (our base RL framework) For detailed environment setup, see [Slime](https://github.com/THUDM/slime) or [`./instructions/README.md`](./instructions/README.md). ### 2. Start the RL Server ```bash cd slime bash ../openclaw-combine/run_qwen3_4b_openclaw_topk_select.sh ``` This method combines binary RL and OPD to achieve the best optimization. See [`./openclaw-combine/README.md`](./openclaw-combine/README.md) for algorithm details. Once running, the model is served as an OpenAI-compatible API at: ``` http://:30000/v1 ``` where `` is the **IP address** of the machine running the RL server (e.g. `115.190.98.251`). The port `30000` is the default and can be changed via the `PORT` environment variable. **Take note of this endpoint** — you will need it when configuring OpenClaw in the next step. We also provide an interesting case for evaluation. A student who uses OpenClaw to do homework, does not want to be found using AI. A teacher who also uses OpenClaw to grade student's homework, wants the comments to be specific and friendly.

Evaluation Setting — Both student, TA and teacher use AI!

See [`./openclaw-test/README.md`](./openclaw-test/README.md) for setup and algorithm details. Example of evaluation [results](https://github.com/Gen-Verse/OpenClaw-RL/blob/main/openclaw-test/results.txt).

### 3. OpenClaw Setup You can use your own openclaw, just install [this extension](https://github.com/Gen-Verse/OpenClaw-RL/tree/main/extensions/rl-training-headers).

Then configure OpenClaw to route requests to your RL server.

Open your `openclaw.json` (or the equivalent settings file) and add a provider entry under `"models"` → `"providers"`: Example of Slime-based RL server: ```json { "models": { "providers": { "qwen": { "baseUrl": "http://:30000/v1", "apiKey": "apiKey", "api": "openai-completions", "models": [ { "id": "qwen3-4b", "name": "Qwen3 4B", "reasoning": true, "input": ["text"], "cost": { "input": 0, "output": 0, "cacheRead": 0, "cacheWrite": 0 }, "contextWindow": 32768, "maxTokens": 8192 } ] } } } } ``` Replace `` with the IP address of your RL server machine. The `apiKey` should match the `SGLANG_API_KEY` you set when starting the server. Example of Tinker-based RL server: ```json { "models": { "providers": { "openclaw-rl": { "baseUrl": "http://localhost:30000/v1", "apiKey": "no-auth-needed", "api": "openai-completions", "models": [ { "id": "qwen3-4b-lora", "name": "Qwen3 4B (OpenClaw-RL LoRA)", "reasoning": true, "input": ["text"], "cost": { "input": 0, "output": 0, "cacheRead": 0, "cacheWrite": 0 }, "contextWindow": 32768, "maxTokens": 8192 } ] } } } } ``` That's it — start chatting with your OpenClaw agent. The RL server will automatically collect conversation trajectories, compute rewards, and train the model. Your agent gets better the more you use it.

--- ## 🔧 Agentic RL in Real-world Settings The same asynchronous RL backbone that powers our personal-agent setting can also support large-scale optimization for these broader real-world environments. | Setting | Environment | Next-state signal | Horizon | |---|---|---|---| | Terminal | Shell execution sandbox | stdout/stderr, exit code | Long | | GUI | Screen state + accessibility tree | Visual state diff, task progress | Long | | SWE | Code repository + test suite | Test verdicts, diff, lint output | Long | | Tool-call | API/function execution | Return values, error traces | Medium | ### 🖥️ Terminal Agent — the most widely used computer-use agent ```bash cd slime bash ../terminal-rl/terminal_qwen3_8b_rl.sh ``` See [`./terminal-rl/README.md`](./terminal-rl/README.md) for setup details. ### 📟 GUI Agent — the most general computer-use agent ```bash cd slime bash ../gui-rl/gui_qwen3vl_8b_rl.sh ``` See [`./gui-rl/README.md`](./gui-rl/README.md) for setup details. ### 👨‍💻 SWE Agent — software engineering agent ```bash cd slime bash ../swe-rl/run_swe_rl_32b_remote_8nodes.sh ``` See [`./swe-rl/README.md`](./swe-rl/README.md) for setup details. ### 🛠️ Tool-call Agent — the most practical agent ```bash cd slime bash ../toolcall-rl/retool_qwen3_4b_rl.sh ``` See [`./toolcall-rl/README.md`](./toolcall-rl/README.md) for setup details. ## 📖 Citation ``` @article{wang2026openclawrl, title={OpenClaw-RL: Train Any Agent Simply by Talking}, author={Wang, Yinjie and Chen, Xuyang and Jin, Xiaolong and Wang, Mengdi and Yang, Ling}, journal={arXiv preprint arXiv:2603.10165}, year={2026} } @article{wang2026rlanything, title={RLAnything: Forge Environment, Policy, and Reward Model in Completely Dynamic RL System}, author={Wang, Yinjie and Xie, Tianbao and Shen, Ke and Wang, Mengdi and Yang, Ling}, journal={arXiv preprint arXiv:2602.02488}, year={2026} } ``` ## 🙏 Acknowledgements This work aims to explore more effective paradigms for Agentic RL. Our implementation builds upon the excellent codebases of [slime](https://github.com/THUDM/slime), [OpenClaw](https://github.com/openclaw/openclaw), [Tinker](https://thinkingmachines.ai/tinker/) and [Open-AgentRL](https://github.com/Gen-Verse/Open-AgentRL). We also build terminal RL using [SETA](https://github.com/camel-ai/seta)'s dataset and agent framework, GUI RL using [OSWorld](https://github.com/xlang-ai/OSWorld)'s evaluation scripts, SWE RL using [mini-swe-agent](https://github.com/SWE-agent/mini-swe-agent)'s evaluation scripts, and tool-call RL based on the work of [Retool](https://github.com/ReTool-RL/ReTool). We sincerely thank these projects for their valuable insights and high-quality implementations, which have greatly facilitated our research. ## ⚠️ Reminder When using OpenClaw-RL, please do not provide sensitive personal information during conversations with the model. Also, make sure to keep your API keys secure and never expose them in prompts, logs, or shared files. ---