# Cactus Logo [![Docs][docs-shield]][docs-url] [![Website][website-shield]][website-url] [![GitHub][github-shield]][github-url] [![HuggingFace][hf-shield]][hf-url] [![Reddit][reddit-shield]][reddit-url] [![Blog][blog-shield]][blog-url] A hybrid edge-cloud AI engine for mobile devices & wearables. ``` ┌─────────────────┐ │ Cactus Engine │ ←── OpenAI-compatible APIs for text, speech, and vision. └─────────────────┘ │ ┌─────────────────┐ │ Cactus Graph │ ←── Zero-copy computation graph └─────────────────┘ │ ┌─────────────────┐ │ Cactus Kernels │ ←── CPU/GPU kernels for (Apple, Samsung, Pixel, etc.) └─────────────────┘ │ ┌─────────────────┐ │ Cactus Quants │ ←── Custom rotation-based quantization technique └─────────────────┘ ``` ## Quick Demo (Mac) - Step 1: `brew install cactus-compute/cactus/cactus` - Step 2: `cactus run` ## Cactus Engine ```cpp #include "cactus_engine.h" cactus_model_t model = cactus_init( "path/to/weight/folder", "path to txt or dir of txts for auto-rag", false ); const char* messages = R"([ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "My name is Henry Ndubuaku"} ])"; const char* options = R"({ "max_tokens": 50, "stop_sequences": ["<|im_end|>"] })"; char response[4096]; int result = cactus_complete( model, // model handle messages, // JSON chat messages response, // response buffer sizeof(response), // buffer size options, // generation options nullptr, // tools JSON nullptr, // streaming callback nullptr, // user data nullptr, // pcm audio buffer 0 // pcm buffer size ); ``` Example response from Gemma4-E2B ```json { "success": true, // generation succeeded "error": null, // error details if failed "cloud_handoff": false, // true if cloud model used "response": "Hi there!", "function_calls": [], // parsed tool calls "segments": [], // transcription segments (empty for chat) "confidence": 0.8193, // model confidence "confidence_threshold": 0.7, // resolved handoff threshold (model-dependent) "time_to_first_token_ms": 45.23, "total_time_ms": 163.67, "prefill_tps": 1621.89, "decode_tps": 168.42, "ram_usage_mb": 245.67, "prefill_tokens": 28, "decode_tokens": 50, "total_tokens": 78 } ``` ## Cactus Graph ```cpp #include "cactus_graph.h" CactusGraph graph; auto a = graph.input({2, 3}, Precision::FP16); auto b = graph.input({3, 4}, Precision::INT8); auto x1 = graph.matmul(a, b, false); auto x2 = graph.transpose(x1); auto result = graph.matmul(b, x2, true); float a_data[6] = {1.1f, 2.3f, 3.4f, 4.2f, 5.7f, 6.8f}; float b_data[12] = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12}; graph.set_input(a, a_data, Precision::FP16); graph.set_input(b, b_data, Precision::INT8); graph.execute(); void* output_data = graph.get_output(result); graph.hard_reset(); ``` ## Inference Speed - LLM: Gemma-4-E2B-CQ4 (1k-context prefill / decode for 100 tokens) - VLM: Gemma-4-E2B-CQ4 (256px image encode time / decode) - Transcribe: Parakeet-TDT-0.6B-CQ4 (20s audio end-to-end transcribe time) - 1k-Context RAM: peak MB during the LLM benchmark - No speculative decode or MTP, pure decode Command: `cactus benchmark` [optional `--ios` or `--android`] | Device | LLM | VLM | Transcribe | RAM | |--------|-----|-----|------------|---------------| | Mac M5 Max | 2964tps / 154tps | 0.09s / 168tps | 0.15s | 1348MB | | Mac M4 Pro | 1963tps / 101tps | 0.25s / 112tps | 0.21s | 1225MB | | Mac M3 Pro | 1294tps / 64tps | 0.40s / 72tps | 0.37s | 735MB | | iPad/Vision Pro M5 | 1336tps / 71tps | 0.25s / 80tps | 0.27s | 703MB | | iPhone 17 Pro | 729tps / 37tps | 0.5s / 39tps | 0.51s | 644MB | | iPhone 15 Pro | 517tps / 26tps | 1.15s / 27tps | 0.82s | 633MB | N/B: With 1k-context prefill and decode for 100 runs on M5 Max - `LFM2.5-VL-1.6B` = 289toks/sec - `Qwen3-1.7B` = 155toks/sec - `LFM2.5-VL-450m` = 472toks/sec, image encodes in 43ms - `LFM22.5-VL-230m` = 555toks/sec ## Output Quality - Gemma-4-E2B-it accuracy across bit widths, averaged over 3 seeds. - CQ3.26 and CQ2.54 are mixed-precision, CQ2/CQ3/CQ4 are uniformly quantized. - Full results in [docs/cactus_quants.md](/docs/cactus_quants.md): | Task | F16 (Original) | CQ4 | CQ3.26 | CQ2.54 | CQ2 | |-----------|-----|-----|--------|--------|-----| | ARC-E | 73.80 | 73.73 | 74.20 | 68.20 | 50.80 | | ARC-C | 56.47 | 52.47 | 51.53 | 37.20 | 24.73 | | HellaSwag | 46.93 | 47.07 | 45.20 | 40.73 | 35.87 | | WinoGrande | 61.00 | 61.13 | 59.60 | 60.13 | 51.27 | | MMLU | 62.33 | 59.45 | 57.63 | 47.19 | 33.18 | | GPQA | 34.34 | 34.34 | 31.82 | 30.81 | 23.23 | | GSM8K | 73.67 | 71.20 | 66.20 | 22.00 | 0.40 | | HumanEval | 54.88 | 57.11 | 53.66 | 15.24 | 1.02 | | BFCL Simple | 92.00 | 92.42 | 91.50 | 82.25 | 18.75 | | BFCL Multi | 89.00 | 88.33 | 89.00 | 52.50 | 13.67 | | BFCL Parallel | 84.00 | 83.67 | 82.50 | 30.00 | 3.33 | | BFCL Parallel-Multi | 78.00 | 83.33 | 82.00 | 37.00 | 1.33 | ## Supported Models - Any HuggingFace model can be converted using `cactus convert [HF-Name]`, though experimental. - Liquid, Gemma. whisper. parakeet and Qwen model families are especially tested. - Some models have been pre-uploaded [here](https://huggingface.co/Cactus-Compute), just run `cactus download [HF-Name]`. - `cactus run [HF-Name]` albeit first downloads or convert the model if not found. ## Needle [Needle](https://github.com/cactus-compute/needle) is a 26m parameter model for on-device tool calling: ```bash cactus run Cactus-Compute/needle [--tools my_tools.json] # OpenAI function-calling format; demo toolset by default ``` ## Learn More | Reference | Language | Description | |-----------|----------|-------------| | [Cactus Engine](/docs/cactus_engine.md) | C | Chat completion, streaming, tool calling, transcription, embeddings, RAG, vision, vector index, cloud handoff | | [Cactus Graph](/docs/cactus_graph.md) | C++ | Tensor operations, matrix multiplication, attention, normalization, activation functions | | [Cactus Kernels](/docs/cactus_kernels.md) | C++ | ARM NEON SIMD kernels for matmul, attention, convolution, quantization, DSP, image processing | | [Cactus Quants](/docs/cactus_quants.md) | C++ | Rotation-and-codebook quantization from 4-bit to 1-bit for all weight tensors | | [Cactus Hybrid](/docs/cactus_hybrid.md) | C/Python | Route hard queries to the cloud automatically based on local model confidence | | [Python Package](/python/) | Python | Python package and CLI | ## Bindings - [Swift](/bindings/swift/) - [Kotlin](/bindings/kotlin/) - [Flutter](/bindings/flutter/) - [React Native](/bindings/react-native/) - [Python](/bindings/python/) - [Rust](/bindings/rust/) ## Using this repo ``` ┌────────────────────────────────────────────────────────────────────────────────┐ │ │ │ Step 0: if on Linux (Ubuntu/Debian) │ │ sudo apt-get install python3.12 python3.12-venv python3-pip cmake │ │ build-essential libcurl4-openssl-dev │ │ │ │ Step 1: clone and setup │ │ git clone https://github.com/cactus-compute/cactus && cd cactus │ │ source ./setup │ │ │ │ Step 2: use the commands │ │────────────────────────────────────────────────────────────────────────────────│ │ │ │ cactus auth manage cloud API key │ │ --status show key status │ │ --clear remove saved key │ │ │ │ cactus run [model|path] run a model (downloads if needed) │ │ --bits 1|2|3|4|2.54|3.26 CQ quantization (default: 4) │ │ --backend cpu|metal inference backend (default: auto) │ │ --image image file for VLM inference │ │ --audio audio file for audio chat │ │ --system system prompt │ │ --prompt send prompt immediately │ │ --tools tool definitions for tool calling │ │ --thinking enable thinking/reasoning mode │ │ --token HuggingFace token (gated models) │ │ --reconvert force local rebuild from source │ │ │ │ cactus transcribe [model] live microphone transcription with a model│ │ --file audio file to transcribe (WAV) │ │ --language language code (default: en) │ │ --bits 1|2|3|4|2.54|3.26 CQ quantization (default: 4) │ │ --token HuggingFace token (gated models) │ │ --reconvert force local rebuild from source │ │ │ │ cactus download [model] get a bundle (prebuilt, else build) │ │ --bits 1|2|3|4|2.54|3.26 CQ quantization (default: 4) │ │ --token HuggingFace token (gated models) │ │ --reconvert refresh cached bundle │ │ │ │ cactus convert [dir] HuggingFace -> Cactus CQ weights │ │ --bits 1|2|3|4 CQ quantization (default: 4) │ │ --token HuggingFace token (gated models) │ │ --reconvert force weight conversion from source │ │ --lora merge a LoRA adapter before converting │ │ --weights-only stop after CQ weights (skip the graph) │ │ --artifact-dir bundle output (default: weights/) │ │ │ │ cactus serve [model] OpenAI-compatible local HTTP server │ │ --host bind address (default: 127.0.0.1) │ │ --port port (default: 8080) │ │ --bits 1|2|3|4|2.54|3.26 CQ quantization (default: 4) │ │ --backend cpu|metal inference backend (default: auto) │ │ --token HuggingFace token (gated models) │ │ --reconvert force local rebuild from source │ │ --no-cloud-handoff disable automatic cloud handoff │ │ --confidence-threshold <0..1> handoff to cloud below this confidence │ │ --cloud-timeout-ms max wait for cloud handoff │ │ │ │ cactus code run the AI coding agent (TUI / print) │ │ --serve-model auto-start a server with this model │ │ --bits 1|2|3|4|2.54|3.26 CQ quantization (default: 4) │ │ --backend cpu|metal inference backend (default: auto) │ │ --token HuggingFace token (gated models) │ │ --reconvert force local rebuild from source │ │ --host server address (default: 127.0.0.1) │ │ --port server port (default: 8080) │ │ --no-serve require a running server (no auto-start) │ │ --no-cloud-handoff disable automatic cloud handoff │ │ --confidence-threshold <0..1> handoff to cloud below this confidence │ │ --cloud-timeout-ms max wait for cloud handoff │ │ -- pass remaining args to the agent │ │ │ │ cactus list list downloaded models │ │ │ │ cactus build build cactus libraries │ │ --apple Apple (iOS/macOS) │ │ --android Android │ │ --python shared lib for Python FFI │ │ │ │ cactus test run the test suite │ │ --component kernels | graph | engine | all │ │ (default: all) │ │ --model default: google/gemma-4-E2B-it │ │ --transcription-model default: nvidia/parakeet-tdt-0.6b-v3 │ │ --bits 1|2|3|4|2.54|3.26 CQ quantization (default: 4) │ │ --backend cpu|metal inference backend (default: auto) │ │ --token HuggingFace token (gated models) │ │ --reconvert force local rebuild of test models │ │ --suite run a single test suite by name │ │ (resolved across components, │ │ e.g. llm → engine) │ │ --list list components and suites │ │ --ios run on connected iPhone │ │ --android run on connected Android │ │ --enable-telemetry send cloud telemetry (off by default) │ │ │ │ cactus benchmark run the engine benchmark suite │ │ --model default: google/gemma-4-E2B-it │ │ --transcription-model default: nvidia/parakeet-tdt-0.6b-v3 │ │ --bits 1|2|3|4|2.54|3.26 CQ quantization (default: 4) │ │ --backend cpu|metal inference backend (default: auto) │ │ --ios run on connected iPhone │ │ --android run on connected Android │ │ │ │ cactus clean delete build artifacts, weights, venv │ │ cactus --help show this help │ │ │ └────────────────────────────────────────────────────────────────────────────────┘ ``` ## Citation If you use Cactus in your research, please cite it as follows: ```bibtex @software{cactus, title = {Cactus: AI Inference Engine for Phones & Wearables}, author = {Ndubuaku, Henry and Cactus Team}, url = {https://github.com/cactus-compute/cactus}, year = {2025} } ``` **N/B:** Scroll all the way up and click the shields link for resources! [docs-shield]: https://img.shields.io/badge/Docs-555?style=for-the-badge&logo=readthedocs&logoColor=white [docs-url]: https://cactus-compute.github.io/cactus/ [website-shield]: https://img.shields.io/badge/Website-555?style=for-the-badge&logo=safari&logoColor=white [website-url]: https://cactuscompute.com/ [github-shield]: https://img.shields.io/badge/GitHub-555?style=for-the-badge&logo=github&logoColor=white [github-url]: https://github.com/cactus-compute/cactus [hf-shield]: https://img.shields.io/badge/HuggingFace-555?style=for-the-badge&logo=huggingface&logoColor=white [hf-url]: https://huggingface.co/Cactus-Compute [reddit-shield]: https://img.shields.io/badge/Reddit-555?style=for-the-badge&logo=reddit&logoColor=white [reddit-url]: https://www.reddit.com/r/cactuscompute/ [blog-shield]: https://img.shields.io/badge/Blog-555?style=for-the-badge&logo=hashnode&logoColor=white [blog-url]: https://cactuscompute.com/blog