# Agent-E **Free Trial: Managed Web Agent & Orchestrator**
Try our Web Agent (Agent-E with enterprise enhancements) and multi-agent orchestrator. Access features such as advanced logging, role-based access, and cloud-hosted scalable infrastructure, backed by expert support. Sign up [here](https://www.emergence.ai/orchestrator) [Discord](https://discord.gg/wgNfmFuqJF)    [Cite paper](https://arxiv.org/abs/2407.13032) _Note: The WebVoyager validation used [nested_chat_for_hierarchial_planning branch](https://github.com/EmergenceAI/Agent-E/tree/nested_chat_for_hierarchial_planning) and GPT4-Turbo_ Agent-E is an agent based system that aims to automate actions on the user's computer. At the moment it focuses on automation within the browser. The system is based on on [AG2 agent framework](https://docs.ag2.ai/docs/Home). This provides a natural language way to interacting with a web browser: - Fill out forms (web forms not PDF yet) using information about you or from another site - Search and sort products on e-commerce sites like Amazon based on various criteria, such as bestsellers or price. - Locate specific content and details on websites, from sports scores on ESPN to contact information on university pages. - Navigate to and interact with web-based media, including playing YouTube videos and managing playback settings like full-screen and mute. - Perform comprehensive web searches to gather information on a wide array of topics, from historical sites to top local restaurants. - Manage and automate tasks on project management platforms (like JIRA) by filtering issues, easing the workflow for users. - Provide personal shopping assistance, suggesting products based on the user's needs, such as storage options for game cards. While Agent-E is growing, it is already equipped to handle a versatile range of tasks, but the best task is the one that you come up with. So, take it for a spin and tell us what you were able to do with it. For more information see our [blog article](https://www.emergence.ai/blog/distilling-the-web-for-multi-agent-automation). ## Quick Start Using Scripts To get started with Agent-E, follow the steps below to install dependencies and configure your environment. #### 1. Run the Installation Script - **macOS/Linux**: - From the project root, run the following command to set up the environment and install all dependencies: ```bash ./install.sh ``` - For **Playwright support**, you can pass the `-p` flag to install Playwright without further prompting: ```bash ./install.sh -p ``` - **Windows**: - From the project root, execute the following command in PowerShell: ```powershell .\win_install.ps1 ``` - To install Playwright without further prompting, add the `-p` flag: ```powershell .\win_install.ps1 -p ``` #### 2. Configure Environment Variables - Go to the newly created `.env` and `agents_llm_config.json` and follow the instructions to set the fields #### 3. Run Agent-E Once you have set up the environment and installed all the dependencies, you can run Agent-E using the following command: ```bash python -m ae.main ``` **For macOS Users** ```bash python -u -m ae.main ``` ## Manual Setup ### 1. Install `uv` Agent-E uses `uv` to manage the Python virtual environment and package dependencies. - **macOS/Linux**: ```bash curl -LsSf https://astral.sh/uv/install.sh | sh - **Windows**: ```bash powershell -c "irm https://astral.sh/uv/install.ps1 | iex" ``` - Alternatively, you can install `uv` using `pip`: `pip install uv` ### 2. Set up the virtual environment Use `uv` to create and activate a virtual environment for the project. ```bash uv venv --python 3.11 # 3.10+ should also work source .venv/bin/activate # On Windows: .venv\Scripts\activate ``` ### 3. Install dependencies Generate the `requirements.txt` file from the `pyproject.toml` and install dependencies. ```bash uv pip compile pyproject.toml -o requirements.txt uv pip install -r requirements.txt ``` To install extras for development, run: ```bash uv pip install -r pyproject.toml --extra dev ``` ### 4.(Optional) Install Playwright Drivers If you do not have Google Chrome installed locally and don’t want to install it, you can use Playwright for browser automation. ```bash playwright install ``` ### 5. Configure the environment Create a `.env` file by copying the provided example file. ```bash cp .env-example .env ``` - Edit the `.env` file and set the following variables: - `AUTOGEN_MODEL_NAME` (e.g., `gpt-4-turbo` for optimal performance). - `AUTOGEN_MODEL_API_KEY` LLM API key. - If using a model other than OpenAI, configure `AUTOGEN_MODEL_BASE_URL` (url where the completion endpoint is hosted, but don't put `/completion` in it), `AUTOGEN_MODEL_API_TYPE`, and `AUTOGEN_MODEL_API_VERSION`. - Optionally configure `AUTOGEN_LLM_TEMPERATURE` and `AUTOGEN_LLM_TOP_P`. - If you want to use local chrome browser over playwright browser, go to chrome://version/ in chrome, find the path to your profile and set `BROWSER_STORAGE_DIR` to the path value ## Environment Variables Agent-E relies on several environment variables for its configuration. You need to define these in a `.env` file in the project root. A sample `.env-example` file is provided for convenience. ### Key Variables: - **`AUTOGEN_MODEL_NAME`** Name of the LLM model you want to use (e.g., `gpt-4-turbo`). This is required for most setups. - **`AUTOGEN_MODEL_API_KEY`** Your API key for accessing the LLM model (e.g., OpenAI API key). - **`AUTOGEN_MODEL_BASE_URL`** *(optional)* Base URL for the model if it's hosted on a service other than OpenAI (e.g., Azure OpenAI services). Example: `https://api.groq.com/openai/v1` or `https://.openai.azure.com` - **`AUTOGEN_MODEL_API_TYPE`** *(optional)* Type of model API (e.g., `azure` for Azure-hosted models). - **`AUTOGEN_MODEL_API_VERSION`** *(optional)* Version of the model API to use, typically needed for Azure models (e.g., `2023-03-15-preview`). - **`AUTOGEN_LLM_TEMPERATURE`** *(optional)* Sets the temperature for the LLM. Controls randomness in output. Defaults to `0.0` for `gpt-*` models. - **`AUTOGEN_LLM_TOP_P`** *(optional)* Sets the top-p value, which controls the diversity of token sampling. Defaults to `0.001` for `gpt-*` models. - **`BROWSER_STORAGE_DIR`** *(optional)* Path to your local Chrome browser profile, required if using a local Chrome instance instead of Playwright. - **`SAVE_CHAT_LOGS_TO_FILE`** Set to `true` or `false` (Default: `true`). Indicates whether to save chat logs to a file or print them to stdout. - **`LOG_MESSAGES_FORMAT`** Set to `json` or `text` (Default: `text`). Specifies the format for logging messages. - **`ADDITIONAL_SKILL_DIRS`** *(optional)* A comma-separated list of directories or `.py` files where additional skills can be loaded from. This is used to dynamically load skills from specified directories or files. Example: `ADDITIONAL_SKILL_DIRS="./private_skills,./extra_skills/my_custom_skill.py"` would be added to the `.env` file (or equivalent) - **`PLANNER_USER_INPUT_SKILL_ENABLED`** *(optional)* Set to `true` or `false` (Default: `false`). Specifies whether to allow the planner agent to get user input or not. ## Running the Code Once you have set up the environment and installed all the dependencies, you can run Agent-E using `./run.sh` script or using the following command: ```bash python -m ae.main ``` ### For macOS Users If you encounter `BlockingIOError` (Errno 35) when running the program on macOS, execute the following command to avoid the issue: ```bash python -u -m ae.main ``` ### Expected Behavior Once Agent-E is running, you should see an icon in the browser interface. Clicking on this icon will open a chat-like interface where you can input natural language commands. Example commands you can try: - `open youtube and search for funny cat videos` - `find iPhone 14 on Amazon and sort by best seller` ## Advanced Usage ### Launch via Web Endpoint Agent-E provides a FastAPI wrapper, allowing you to send commands via HTTP and receive streaming results. This feature is useful for programmatic task automation or integrating Agent-E into larger systems. #### To launch the FastAPI server: 1. On Linux/macOS, run the following command: ```bash uvicorn ae.server.api_routes:app --reload --loop asyncio ``` 2. On Windows, run the same command but without ```--reload``` (Python still has different async implementations across OSes, removing --reload helping finding a workaround, see this [answer on StackOverflow](https://stackoverflow.com/a/78795990)): ```cmd uvicorn ae.server.api_routes:app --loop asyncio ``` 2. Send POST requests to execute tasks. For example, to execute a task using cURL: ```bash curl --location 'http://127.0.0.1:8000/execute_task' \ --header 'Content-Type: application/json' \ --data '{ "command": "go to espn, look for soccer news, report the names of the most recent soccer champs" }' ``` Optionally, the API request can include an llm_config object if you want to apply a different configuration during API request execution. The llm_config object should have configuration seperately for planner_agent and browser_nav_agent. See `agents_llm_config-example.json` for an exmaple. ```bash curl --location 'http://127.0.0.1:8000/execute_task' \ --header 'Content-Type: application/json' \ --data '{ "command": "go to espn, look for soccer news, report the names of the most recent soccer champs", "llm_config":{"planner_agent":{...}, "browser_nav_agent":{...}} }' ``` ### Customizing LLM Parameters Agent-E supports advanced LLM configurations using environment variables or JSON-based configuration files. This allows users to customize how the underlying model behaves, such as setting temperature, top-p, and model API base URLs. To configure Agent-E using a JSON file, add the following to your `.env` file: ```makefile AGENTS_LLM_CONFIG_FILE=agents_llm_config.json AGENTS_LLM_CONFIG_FILE_REF_KEY=openai_gpt ``` A sample JSON config file is provided in the project root: `agents_llm_config-example.json`. #### Default Values for LLM Parameters If you do not set `temperature`, `top_p`, or `seed` in your `.env` file or JSON configuration, Agent-E will use the following default values: - For `gpt-*` models: - `"temperature": 0.0` - `"top_p": 0.001` - `"seed": 12345` - For other models: - `"temperature": 0.1` - `"top_p": 0.1` ## Open-source Models Agent-E supports the use of open-source models through LiteLLM and Ollama. This allows users to run language models locally on their machines, with LiteLLM translating OpenAI-format inputs to local models' endpoints. ### Steps to Use Open-source Models: 1. **Install LiteLLM**: ```bash pip install 'litellm[proxy]' ``` 2. **Install Ollama**: - For Mac and Windows, download [Ollama](https://ollama.com/download). - For Linux: ```bash curl -fsSL https://ollama.com/install.sh | sh ``` 3. **Pull Ollama Models**: Before using a model, download it from the library. The list of available models is [here](https://ollama.com/library). For example, to pull the Mistral v0.3 model: ```bash ollama pull mistral:v0.3 ``` 4. **Run LiteLLM**: Start the LiteLLM proxy using the downloaded model: ```bash litellm --model ollama_chat/mistral:v0.3 ``` 5. **Configure Model in AutoGen**: Modify your `.env` file as follows. No model name or API keys are required since the model is running locally. ```bash AUTOGEN_MODEL_NAME=NotRequired AUTOGEN_MODEL_API_KEY=NotRequired AUTOGEN_MODEL_BASE_URL=http://0.0.0.0:400 ``` ### Notes: - Running local Large Language Models (LLMs) with Agent-E is possible, but has not been thoroughly tested. Use this feature with caution. ## Troubleshooting Below are some common issues you may encounter when setting up or running Agent-E, along with steps to resolve them. ### 1. pip not installed in the virtual environment If you encounter an issue where `pip` is not installed in the virtual environment after setup, follow these steps: 1. Activate the virtual environment: ```bash source .venv/bin/activate # On Windows: .venv\Scripts\activate 2. Install `pip`: ```bash python -m ensurepip --upgrade ``` 3. Deactivate the virtual environment: ```bash deactivate ``` 4. Reactivate the virtual environment: ```bash source .venv/bin/activate # On Windows: .venv\Scripts\activate ``` 5. Check for `pip` in the `.venv/bin` directory. You should now have `pip` installed. ### 2. BlockingIOError on macOS If you are on `macOS` and encounter the following error: ``` BlockingIOError: [Errno 35] write could not complete without blocking ``` This happens when AutoGen tries to print large amounts of text to the terminal. To fix this, run the following command with the `-u` flag to make output unbuffered: ```bash python -u -m ae.main ``` Note: Using unbuffered output may result in some output not appearing in the terminal. ### 3. Playwright driver issues If you do not have Google Chrome installed locally and run into issues with browser automation, install the Playwright drivers: ```bash playwright install ``` Playwright will install the necessary browser binaries to run the automation tasks without needing Chrome locally installed. ### 4. Chrome profile not found If you want to use your local Chrome browser instead of Playwright and encounter issues finding the browser profile path, follow these steps: 1. Open Chrome and go to `chrome://version/`. 2. Locate the `Profile Path`. 3. Set the `BROWSER_STORAGE_DIR` environment variable in the `.env` file to this path: ``` BROWSER_STORAGE_DIR=/path/to/your/chrome/profile ``` If you encounter other issues, please refer to the project’s [GitHub issues](https://github.com/EmergenceAI/Agent-E/issues) or reach out on [Discord](https://discord.gg/wgNfmFuqJF) for assistance. ## Demos | Video | Command | Description | |-----------|-------------|-------------| | [![Oppenheimer Video](docs/images/play-video-on-youtube-thumbnail.png)](https://www.youtube.com/embed/v4BgYiDHNZs) | There is an Oppenheimer video on youtube by Veritasium, can you find it and play it? | | | [![Example 2: Use information to fill forms](docs/images/form-filling-thumbnail.png)](https://www.youtube.com/embed/uyE7tfKkB0E) | Can you do this task? Wait for me to review before submitting. | Takes the highlighted text from the email as part of the instruction.