OpenRaft
Advanced Raft in 🦀 Rust on any async runtime. Please ⭐ on GitHub!
[](https://crates.io/crates/openraft)
[](https://docs.rs/openraft)
[](https://deepwiki.com/databendlabs/openraft)
[](https://docs.rs/openraft/0.10.0-alpha.32/openraft/docs/index.html)
[](https://discord.gg/ZKw3WG7FQ9)
[](https://github.com/databendlabs/openraft/actions/workflows/ci.yaml)
[](https://coveralls.io/github/databendlabs/openraft?branch=main)
This project intends to improve raft as the next-generation consensus protocol for distributed data storage systems (SQL, NoSQL, KV, Streaming, Graph ... or maybe something more exotic).
Currently, openraft is the consensus engine of meta-service cluster in [databend](https://github.com/databendlabs/databend).
- 🚀 **Get started**:
- [OpenRaft guide](https://docs.rs/openraft/0.10.0-alpha.32/openraft/docs/getting_started/index.html) is the best place to get started,
- [OpenRaft docs](https://docs.rs/openraft/0.10.0-alpha.32/openraft/docs/index.html) for more in-depth details,
- [OpenRaft FAQ](https://docs.rs/openraft/0.10.0-alpha.32/openraft/docs/faq/index.html) explains some common questions.
- [OpenRaft on DeepWiki](https://deepwiki.com/databendlabs/openraft) provides detailed architectural documentation to help understand OpenRaft internals.
- 💡 **Example Applications**:
- [Examples with OpenRaft 0.10](https://github.com/databendlabs/openraft/tree/release-0.10/examples) require OpenRaft `0.10.0-alpha.32` (alpha) on [crates.io/openraft](https://crates.io/crates/openraft);
- [Examples with OpenRaft 0.9](https://github.com/databendlabs/openraft/tree/release-0.9/examples) require OpenRaft 0.9 on [crates.io/openraft](https://crates.io/crates/openraft).
- 🙌 **Questions**?
- Why not take a peek at our [FAQ](https://docs.rs/openraft/0.10.0-alpha.32/openraft/docs/faq/index.html)? You might find just what you need.
- Wanna chat? Come hang out with us on [Discord](https://discord.gg/ZKw3WG7FQ9)!
- Or start a new discussion over on [GitHub](https://github.com/databendlabs/openraft/discussions/new).
- Or join our [Feishu group](https://applink.feishu.cn/client/chat/chatter/add_by_link?link_token=d20l9084-6d36-4470-bac5-4bad7378d003).
- And hey, if you're on WeChat, add us: `drmingdrmer`. Let's get the conversation started!
Whatever your style, we're here to support you. 🚀 Let's make something awesome together!
- OpenRaft is derived from [async-raft](https://docs.rs/crate/async-raft/latest) with several bugs fixed: [Fixed bugs](https://github.com/databendlabs/openraft/blob/main/derived-from-async-raft.md).
# Status
- The features are almost complete for building an application.
- Performance: Supports 33,000 writes/sec for a single writer and 5,615,000 writes/sec with batch writes. See: [Performance](#performance)
- Unit test coverage stands at 92%.
- The chaos test has not yet been completed, and further testing is needed to ensure the application's robustness and reliability.
## API status
- **OpenRaft API is not stable yet**. Before `1.0.0`, an upgrade may contain incompatible changes.
Check our [change-log](https://github.com/databendlabs/openraft/blob/main/change-log.md). A commit message starts with a keyword to indicate the modification type of the commit:
- `DataChange:` on-disk data types changes, which may require manual upgrade.
- `Change:` if it introduces incompatible changes.
- `Feature:` if it introduces compatible non-breaking new features.
- `Fix:` if it just fixes a bug.
## Versions
- **Branch main** has been under active development.
The main branch is for the [release-0.10](https://github.com/databendlabs/openraft/tree/release-0.10).
Latest alpha on crates.io: [v0.10.0-alpha.32](https://crates.io/crates/openraft/0.10.0-alpha.32).
The `0.10` line is in **alpha** — the API may still change before the final `0.10.0` release.
- **Branch [release-0.9](https://github.com/databendlabs/openraft/tree/release-0.9)**:
Latest: ( [v0.9.0](https://github.com/databendlabs/openraft/tree/v0.9.0) | [Change log](https://github.com/databendlabs/openraft/blob/release-0.9/change-log.md#v090) );
Upgrade guide: ⬆️ [0.8 to 0.9](https://docs.rs/openraft/0.9.0/openraft/docs/upgrade_guide/upgrade_08_09/index.html);
`release-0.9` **Won't** accept new features but only bug fixes.
- **Branch [release-0.8](https://github.com/databendlabs/openraft/tree/release-0.8)**:
Latest: ( [v0.8.8](https://github.com/databendlabs/openraft/tree/v0.8.8) | [Change log](https://github.com/databendlabs/openraft/blob/release-0.8/change-log.md#v088) );
Upgrade guide: ⬆️ [0.7 to 0.8](https://docs.rs/openraft/0.8.4/openraft/docs/upgrade_guide/upgrade_07_08/index.html), ⬆️ [0.8.3 to 0.8.4](https://docs.rs/openraft/0.8.4/openraft/docs/upgrade_guide/upgrade_083_084/index.html);
`release-0.8` **Won't** accept new features but only bug fixes.
- **Branch [release-0.7](https://github.com/databendlabs/openraft/tree/release-0.7)**:
Latest: ( [v0.7.6](https://github.com/databendlabs/openraft/tree/v0.7.6) | [Change log](https://github.com/databendlabs/openraft/blob/release-0.7/change-log.md#v076) );
Upgrade guide: ⬆️ [0.6 to 0.7](https://docs.rs/openraft/0.8.4/openraft/docs/upgrade_guide/upgrade_06_07/index.html);
`release-0.7` **Won't** accept new features but only bug fixes.
- **Branch [release-0.6](https://github.com/databendlabs/openraft/tree/release-0.6)**:
Latest: ( [v0.6.8](https://github.com/databendlabs/openraft/tree/v0.6.8) | [Change log](https://github.com/databendlabs/openraft/blob/release-0.6/change-log.md) );
`release-0.6` **won't** accept new features but only bug fixes.
# Roadmap
- [x] **2022-10-31** [Extended joint membership](https://docs.rs/openraft/0.10.0-alpha.32/openraft/docs/data/extended_membership/index.html)
- [x] **2023-02-14** Minimize confliction rate when electing;
See: [OpenRaft Vote design](https://docs.rs/openraft/0.10.0-alpha.32/openraft/docs/data/vote/index.html);
Or use [standard Raft leader-ID mode](https://docs.rs/openraft/0.10.0-alpha.32/openraft/docs/data/leader_id/index.html).
- [x] **2023-04-26** Goal performance is 1,000,000 put/sec.
- [ ] Reduce the complexity of vote and pre-vote: [get rid of pre-vote RPC](https://github.com/databendlabs/openraft/discussions/15);
- [ ] Support flexible quorum, e.g.: [Hierarchical Quorums](https://zookeeper.apache.org/doc/r3.5.9/zookeeperHierarchicalQuorums.html)
- [ ] Consider introducing read-quorum and write-quorum,
improve efficiency with a cluster with an even number of nodes.
# Performance
The benchmark is focused on the OpenRaft framework itself and is run on a
minimized store and network. This is **NOT a real world** application benchmark!!!
Single writes:
| clients | put/s |
| --: | --: |
| 4096 | 3,548,000 |
| 1024 | 3,006,000 |
| 256 | 1,808,000 |
| 64 | 912,000 |
| 1 | 33,000 |
Batch writes (4 entries per batch):
| clients | put/s |
| --: | --: |
| 4096 | 5,615,000 |
For benchmark detail, go to the [./benchmarks/minimal](./benchmarks/minimal) folder.
# Features
What sets OpenRaft apart from a standard Raft implementation:
- **Generalized membership change**: [extended joint membership](https://docs.rs/openraft/0.10.0-alpha.32/openraft/docs/data/extended_membership/index.html) changes an arbitrary set of nodes in a single operation; standard Raft's one-node-at-a-time change is a restricted special case.
- **Fewer election conflicts**: the redesigned [Vote](https://docs.rs/openraft/0.10.0-alpha.32/openraft/docs/data/vote/index.html) minimizes election conflict rate — a split vote does not force a new term; the [standard Raft leader-ID mode](https://docs.rs/openraft/0.10.0-alpha.32/openraft/docs/data/leader_id/index.html) is also supported.
- **Async and event-driven**: state transitions are driven by Raft events without periodic ticks; messages are batched, reaching millions of writes/sec in the [framework benchmark](#performance).
- **Fully pluggable**: storage (`RaftLogStorage`, `RaftStateMachine`), networking (`RaftNetworkV2`, including application-defined snapshot transport), and every core type — node ID, node, term, vote, log entry — via `RaftTypeConfig`.
- **Runtime-agnostic**: tokio by default, [compio](./rt-compio) and [monoio](./rt-monoio) via the `AsyncRuntime` trait, and a `single-threaded` mode that removes `Send` bounds.
- **Deterministically tested**: a [turmoil-based simulation fuzzer](./tests-turmoil) steps the cluster tick by tick and verifies Raft invariants from the paper and the TLA+ spec after every tick.
- **Observable**: streaming metrics via watch channels, and [tracing](https://docs.rs/tracing/) instrumentation with [compile-time verbosity filtering](https://docs.rs/tracing/latest/tracing/level_filters/index.html).
## Functionality:
- ✅ **Leader election**: by policy or manually ([`Trigger::elect()`][]).
- ✅ **Leader transfer**: [`Trigger::transfer_leader()`][].
- ✅ **Pre-vote**: avoid unnecessary term increments by enabling [`Config::enable_pre_vote`][].
- ✅ **Non-voter(learner) Role**: refer to [`add_learner()`][].
- ✅ **Log Compaction**(snapshot of state machine): by policy or manually ([`Trigger::snapshot()`][]).
- ✅ **Snapshot replication**.
- ✅ **Dynamic Membership**: using joint membership config change. Refer to [dynamic membership](https://docs.rs/openraft/0.10.0-alpha.32/openraft/docs/cluster_control/dynamic_membership/index.html)
- ✅ **Linearizable read**: [`ensure_linearizable()`][].
- ✅ **Metrics**: [`Raft::metrics()`][], [`Raft::data_metrics()`][], and [`Raft::server_metrics()`][].
- ⛔️ **Won't support**: Single-step config change. Single-step membership change is a restricted subset of joint consensus that only allows changing one node at a time. Openraft uses the more general [joint consensus](https://docs.rs/openraft/0.10.0-alpha.32/openraft/docs/cluster_control/dynamic_membership/index.html) approach which supports arbitrary membership changes in a single operation.
- ✅ Toggle heartbeat / election: [`RuntimeConfigHandle::heartbeat()`][] / [`RuntimeConfigHandle::elect()`][].
- ✅ Trigger snapshot / election manually: [`Trigger::snapshot()`][] / [`Trigger::elect()`][].
- ✅ Purge log by policy or manually: [`Trigger::purge_log()`][].
# Who uses it
- [Databend](https://github.com/databendlabs/databend) - The Next-Gen Cloud [Data+AI] Analytics
- [Walrus](https://github.com/nubskr/walrus) - Distributed message streaming engine with Raft-based metadata coordination.
- [CnosDB](https://github.com/cnosdb/cnosdb) - A cloud-native open source distributed time series database.
- [RobustMQ](https://github.com/robustmq/robustmq) - Next generation cloud-native converged message queue.
- [RocketMQ-rust](https://github.com/mxsm/rocketmq-rust) - Apache RocketMQ reimplemented in Rust.
- [Hiqlite](https://github.com/sebadob/hiqlite) - Highly-available, embeddable, Raft-based SQLite with caching.
- [Octopii](https://github.com/octopii-rs/octopii) - A distributed systems kernel for building replicated, fault-tolerant services.
- [Renegade](https://github.com/renegade-fi/renegade) - On-chain dark pool for anonymous trading.
- [Helyim](https://github.com/helyim/helyim) - [SeaweedFS](https://github.com/seaweedfs/seaweedfs) implemented in pure Rust.
- [Ahnlich](https://github.com/deven96/ahnlich) - In-memory vector database with an AI proxy.
- [raymondshe/matchengine-raft](https://github.com/raymondshe/matchengine-raft) - A example to demonstrate how openraft persists snapshots/logs to disk.
- [tsoracle](https://github.com/prisma-risk/tsoracle) - Distributed timestamp and sequence oracle serving strictly monotonic, gapless IDs.
- [yuyang0/rrqlite](https://github.com/yuyang0/rrqlite) - A rust implementation of [rqlite](https://github.com/rqlite/rqlite).
📣 Using openraft in your project? We'd love to feature it here — [open an issue](https://github.com/databendlabs/openraft/issues/new) to let us know, or skip the wait and [submit a pull request directly](https://github.com/databendlabs/openraft/edit/main/README.md).
# Contributing
Check out the [CONTRIBUTING.md](https://github.com/databendlabs/openraft/blob/main/CONTRIBUTING.md)
guide for more details on getting started with contributing to this project.
## Contributors
Thanks to everyone who contributes to OpenRaft:
[
](https://github.com/ariesdevil)
[
](https://github.com/AustenSchunk)
[
](https://github.com/bohutang)
[
](https://github.com/byronwasti)
[
](https://github.com/caicancai)
[
](https://github.com/CentRa-Linux)
[
](https://github.com/claude)
[
](https://github.com/cliff0412)
[
](https://github.com/ClSlaid)
[
](https://github.com/danthegoodman1)
[
](https://github.com/devillove084)
[
](https://github.com/Diggsey)
[
](https://github.com/diptanu)
[
](https://github.com/drmingdrmer)
[
](https://github.com/eliasyaoyc)
[
](https://github.com/emmanuel-ferdman)
[
](https://github.com/emon100)
[
](https://github.com/ethe)
[
](https://github.com/evanj)
[
](https://github.com/George-Miao)
[
](https://github.com/getong)
[
](https://github.com/GrapeBaBa)
[
](https://github.com/gtema)
[
](https://github.com/guojidan)
[
](https://github.com/hadronzoo)
[
](https://github.com/HaHa421)
[
](https://github.com/homa31)
[
](https://github.com/iamazy)
[
](https://github.com/iMashtak)
[
](https://github.com/Jason5Lee)
[
](https://github.com/jdockerty)
[
](https://github.com/jgrund)
[
](https://github.com/jkosh44)
[
](https://github.com/josedab)
[
](https://github.com/justinlatimer)
[
](https://github.com/kper)
[
](https://github.com/leiysky)
[
](https://github.com/Licenser)
[
](https://github.com/lichuang)
[
](https://github.com/linsinn)
[
](https://github.com/Lymah123)
[
](https://github.com/maackle)
[
](https://github.com/maciejkow)
[
](https://github.com/manuelarte)
[
](https://github.com/manuelgdlvh)
[
](https://github.com/MarinPostma)
[
](https://github.com/markus-behrens)
[
](https://github.com/marvin-j97)
[
](https://github.com/mengxuroblox)
[
](https://github.com/mfelsche)
[
](https://github.com/Miaxos)
[
](https://github.com/monoid)
[
](https://github.com/MrCroxx)
[
](https://github.com/nasuiyile)
[
](https://github.com/Nicholas-Ball)
[
](https://github.com/penghs520)
[
](https://github.com/PeterKnego)
[
](https://github.com/pfwang80s)
[
](https://github.com/Phoenix500526)
[
](https://github.com/pinylin)
[
](https://github.com/ppamorim)
[
](https://github.com/PsiACE)
[
](https://github.com/Qian-Cheng-nju)
[
](https://github.com/rvielma)
[
](https://github.com/sainad2222)
[
](https://github.com/schreter)
[
](https://github.com/SebastianThiebaud)
[
](https://github.com/shuoli84)
[
](https://github.com/socutes)
[
](https://github.com/sollhui)
[
](https://github.com/SteveLauC)
[
](https://github.com/sunli829)
[
](https://github.com/sunsided)
[
](https://github.com/svix-jplatte)
[
](https://github.com/tangwz)
[
](https://github.com/thedodd)
[
](https://github.com/tisonkun)
[
](https://github.com/tobodner)
[
](https://github.com/trungda)
[
](https://github.com/tvsfx)
[
](https://github.com/undecidedapollo)
[
](https://github.com/Veeupup)
[
](https://github.com/vigith)
[
](https://github.com/wvwwvwwv)
[
](https://github.com/xu-cheng)
[
](https://github.com/Xuanwo)
[
](https://github.com/YangKian)
[
](https://github.com/ygf11)
[
](https://github.com/youichi-uda)
[
](https://github.com/zach-schoenberger)
[
](https://github.com/zhixinwen)
# License
OpenRaft is licensed under the terms of the [MIT License](https://en.wikipedia.org/wiki/MIT_License#License_terms)
or the [Apache License 2.0](http://www.apache.org/licenses/LICENSE-2.0), at your choosing.
[`change_membership()`]: https://docs.rs/openraft/0.10.0-alpha.32/openraft/raft/struct.Raft.html#method.change_membership
[`add_learner()`]: https://docs.rs/openraft/0.10.0-alpha.32/openraft/raft/struct.Raft.html#method.add_learner
[`Trigger::purge_log()`]: https://docs.rs/openraft/0.10.0-alpha.32/openraft/raft/trigger/struct.Trigger.html#method.purge_log
[`RuntimeConfigHandle::heartbeat()`]: https://docs.rs/openraft/0.10.0-alpha.32/openraft/raft/struct.RuntimeConfigHandle.html#method.heartbeat
[`RuntimeConfigHandle::elect()`]: https://docs.rs/openraft/0.10.0-alpha.32/openraft/raft/struct.RuntimeConfigHandle.html#method.elect
[`Trigger::elect()`]: https://docs.rs/openraft/0.10.0-alpha.32/openraft/raft/trigger/struct.Trigger.html#method.elect
[`Trigger::snapshot()`]: https://docs.rs/openraft/0.10.0-alpha.32/openraft/raft/trigger/struct.Trigger.html#method.snapshot
[`Trigger::transfer_leader()`]: https://docs.rs/openraft/0.10.0-alpha.32/openraft/raft/trigger/struct.Trigger.html#method.transfer_leader
[`Config::enable_pre_vote`]: https://docs.rs/openraft/0.10.0-alpha.32/openraft/struct.Config.html#structfield.enable_pre_vote
[`Raft::metrics()`]: https://docs.rs/openraft/0.10.0-alpha.32/openraft/raft/struct.Raft.html#method.metrics
[`Raft::data_metrics()`]: https://docs.rs/openraft/0.10.0-alpha.32/openraft/raft/struct.Raft.html#method.data_metrics
[`Raft::server_metrics()`]: https://docs.rs/openraft/0.10.0-alpha.32/openraft/raft/struct.Raft.html#method.server_metrics
[`ensure_linearizable()`]: https://docs.rs/openraft/0.10.0-alpha.32/openraft/raft/struct.Raft.html#method.ensure_linearizable