# Reproducing Evaluation Results This guide provides instructions to reproduce evaluation results and running experiments for **DroidTask** and **LlamaTouch** using our system and selected baselines. --- ## 1. Reproducing Evaluation Results for DroidTask ### 📌 Existing Results Overview To see an overview of all examined tasks you can preview the evaluation results of our system and the compared baselines. The results are in the following directory: ``` evaluation/droidtask/data/evaluated_results ``` ### 📌 **Mandatory Files** To evaluate the DroidTask experiments, you need to download the experimental data from the following link: 🔗 **[Download Link](https://cloud.tsinghua.edu.cn/f/cbd4606d5d4749c38c1c/)** Once downloaded, extract the results for each agent into the following directory: ``` evaluation/droidtask/data/ ``` The code that was generated by the model is in the **code.txt** file inside the folder for each evaluated task. (eg. autodroidv2/applauncher/task1.yaml/code.txt ) --- ### 🚀 **Running the Evaluation** To evaluate a specific agent for DroidTask, run the following command: ```sh sh scripts/evaluate_droidtask.sh ``` To run evaluation for different models, please change the **-a** argument with some of the following options: "autodroidv2", "autodroid", "seeclick", "cogagent" ## 📂 Results Output The evaluation results for DroidTask will be saved in: ``` evaluation/droidtask/evaluation_output ``` Additionally, the script will print the results directly in the command line for immediate review. ## 2. Reproducing Evaluation Results for LlamaTouch ### 📌 Existing Results Overview To see an overview of all examined tasks you can preview the evaluation results of our system and the compared baselines. The results are in the following directory: ``` evaluation/llama_touch/data/evaluated_results ``` ### 📌 Mandatory Files To evaluate the LlamaTouch experiments, download the required data from the following link: 🔗 **[Download Link](https://cloud.tsinghua.edu.cn/f/e7022742d06b47388a31/)** Extract the results for each agent into the following directory: ``` evaluation/llama_touch/data/ ``` ### 🚀 Running the Evaluation To evaluate a specific agent for LlamaTouch, use the command: ``` sh scripts/evaluate_llama_touch.sh ``` To run evaluation for different models, please change the **-a** argument with some of the following options: "autodroidv2", "autodroid", "seeclick", "cogagent" ## 📂 Results Output The evaluation results for LlamaTouch will be saved in: ``` evaluation/llama_touch/evaluation_output/ ``` Additionally, the script will print the results directly in the command line for immediate review. ## 3. Additional Notes - Ensure that all required files are correctly extracted into their respective directories before running the evaluation. - The evaluation scripts will process the results for each agent and generate performance metrics. - If you encounter any issues, verify the downloaded data and directory structure. For any questions or troubleshooting, feel free to open an issue in the repository. 🚀 # Running Experiments Setup To run experiments with the fine-tuned model we provide, please download the model [here](https://huggingface.co/BlitherBoom/AutoDroid-V2). Save the model files in the following directory: **model/autodroidv2** # Running LLama Touch Experiment ## Overview This guide explains how to set up and run an experiment using our system over the LLama Touch-selected applications and tasks. --- ## 1. Mandatory Setup Before running an experiment, you need to download and set up the required data files. ### 📌 **Required Downloads:** - **📦 APK Files:** - The APK for each application must be referenced for the system to run. - Download the APKs for the used **LlamaTouch applications** from [here](https://cloud.tsinghua.edu.cn/d/cfdbaef17a7a462fa7e8/). - Extract the downloaded data into the following directory: ``` evaluation/llama_touch/apks/ ``` - **📲 Emulator Setup: ** - Follow the guide for setting up the android emulator according to the expected LlamaTouch benchmark environment. You can find the guide in: ``` evaluation/llama_touch/emulator_guide.md ``` ``` evaluation/llama_touch/emulator_transfer.md ``` --- ## 2. Running an Experiment ### ✅ **Setup Emulator Configuration** Before running the experiment, ensure that: - The **emulator is correctly set up** and running. - All required **APKs are downloaded**. - The **emulator name and port** are correctly specified in the configuration file: ``` evaluation/llama_touch/config/config.py ``` Inside this file, update the `EMULATOR_CONTROLLER_AGRS` and `AVD_NAME` variables according to your emulator settings. ### 🚀 **Run the Experiment** Once everything is set up, execute the following command to start the experiment: ```sh sh scripts/run_llama_touch.sh ``` This will automatically process the specified tasks and applications. ## 3. Experiment Task Configuration The experiment will run for all tasks defined in the following file: ``` evaluation/llama_touch/instructions/setup_test.tsv ``` ## 4. Additional Notes - Verify that your emulator is running before executing the script. - If you encounter issues, check the emulator settings and the config.py file. - Keep your APKs properly referenced inside the evaluation/llama_touch/apks/ directory. # Running DroidTask Experiment ## Overview This guide explains how to set up and run an experiment using our system over the DroidTask-selected applications and tasks. --- ## 1. Mandatory Files Before running an experiment, you need to download and set up the required data files. ### 📌 **Required Downloads:** - **📦 APK Files:** - The APK for each application must be referenced for the system to run. - Download the APKs for all **DroidTask applications** from [here](https://cloud.tsinghua.edu.cn/d/cb5817c334ee4eb08abc/). - Extract the downloaded data into the following directory: ``` evaluation/droidtask/apks/ ``` - **📲 APK Installation:** - Install all the downloaded APKs on your virtual device before proceeding. --- ## 2. Running an Experiment ### ✅ **Setup Emulator Configuration** Before running the experiment, ensure that: - The **emulator is correctly set up** and running. - All required **APKs are downloaded and installed** on the emulator. - The **emulator name and port** are correctly specified in the configuration file: ``` evaluation/droidtask/experiment/config.py ``` Inside this file, update the `EMULATOR_ARGS` variable according to your emulator settings. ### 🚀 **Run the Experiment** Once everything is set up, execute the following command to start the experiment: ```sh sh scripts/run_droidtask.sh ``` This will automatically process the specified tasks and applications. ## 3. Experiment Task Configuration The experiment will run for all tasks defined in the following JSON file: ``` evaluation/droidtask/experiment/tasks/tasks-gt.json ``` ## 4. Additional Notes - Verify that your emulator is running before executing the script. - If you encounter issues, check the emulator settings and the config.py file. - Keep your APKs properly referenced inside the evaluation/droidtask/apks/ directory.