## NOTE This document is for those who want to try our benchmark without docker/podman, in that case some of the tasks are not runnable. If you have docker/podman installed, please refer to [README.md](README.md). ## Quick Start ### Installation Dependencies Make sure you have `uv` installed, otherwise please install it: ``` # this is for macOS and linux command # by default it will install uv to $HOME/.local/bin # you probably need to add it to your $PATH curl -LsSf https://astral.sh/uv/install.sh | sh # check whether uv can be found which uv ``` We provide one command to install everything, we maintain the environment with `uv`. Just run: ``` bash global_preparation/install_env.sh [true|false] # `true` if you have sudo. ``` This command will automatically process some config files and install the dependencies **directly on your host machine**, including node.js, kind (k8s in docker), kubectl, playwright, npm packages, uv packages, uv tools, for you (except docker/podman). But if you encounter some troubles in later steps, please first check if these are properly installed (e.g. whether your playwright and kind works properly). ### Configure Global Configs Simply set these two env variables, note that `TOOLATHLON_OPENAI_BASE_URL` must be an OpenAI SDK compatible one, as our agent scaffold relies on this: ``` export TOOLATHLON_OPENAI_API_KEY="your-custom-api-key" export TOOLATHLON_OPENAI_BASE_URL="https://your-custom-endpoint.com" # e.g. "https://openrouter.ai/api/v1" for OpenRouter, "https://api.anthropic.com/v1/" for Anthropic ``` This will use our unified model provider. You can also use any model deployed on your own machine, like via [vLLM](https://github.com/vllm-project/vllm) or [SGLang](https://github.com/sgl-project/sglang), in that case you do not need to set the api key. (Optional) We also provide some pre-configurated options for you in `configs/global_configs.py` to manage all LLM APIs. You may open this file and fill in the api keys in it, and specify which provider you want to use later. You can find details about model providers in `utils/api_model/model_provider.py`. ### Quick Example After the above two steps, we provide a very quick example here. We use *claude-sonnet-4-5* via **openrouter** in this example, so make sure you have configured TOOLATHLON_OPENAI_API_KEY and TOOLATHLON_OPENAI_BASE_URL accordingly if you want to run this script without any modification. ``` bash scripts/quick_start/quick_start_run.sh ``` You can find the resulted logs, trajectories, and agent workspace all in `dumps_quick_start/anthropic_claude-sonnet-4-5/finalpool/SingleUserTurn-find-alita-paper`. ## Full Preparation ### Choose a Proper Machine To run our benchmark, we strongly suggest you deploy it on a Linux machine with docker installed that can directly access the Internet. ### Configure App-Aware Tokens, Keys and Credentials Please read carefully through [how2register_accounts.md](global_preparation/how2register_accounts.md) and follow the guides. You need to register some accounts and configure some tokens/api keys/secrets in `configs/token_key_session.py`. ### Misc Configuration Simply run the following: ``` bash global_preparation/misc_configuartion.sh ``` ### Run Single Task We use the same script `scripts/quick_start/quick_start_run.sh` to run any task, just simply edit the `task` variable in this script: ``` bash scripts/quick_start/quick_start_run.sh ``` Note: some tasks may fail since you do not have docker/podman and thus cannot deploy the needed apps or launch a k8s cluster via kind.