# [ICLR2025] ToolDial: Multi-turn Dialogue Generation Method for Tool-Augmented Language Models ## [News] - **[2025.05.01]** 🤗 We have released the fine-tuned "td-llama-general". This model can be used for a wide range of tasks related to tool agents. **We encourage you to use it for various purposes.** See [HuggingFace Link](https://huggingface.co/HOLILAB/td-llama-general). - **[2025.05.01]** 🤗 We have released the fine-tuned "td-llama-op" for Overall Performance task. We encourage using this model solely for reproducing the overall performance results. See [HuggingFace Link](https://huggingface.co/HOLILAB/td-llama-op). We have write the instruction to use at the **Experiments** section of this page. ## TL;DR ToolDial is a method that generates multi-turn dialogue data between a user and a tool-augmented language model based on GPT, utilizing an API graph. We release the code for the API graph-based multi-turn dialogue generation method devised in the ToolDial work. We provide the API graph construction, dialogue data generation, and the evaluation codes used in the experiments. Additionally, we release all the datasets used in the experiments. # 0. Environment Setting Create new conda environment and install packages. ``` conda create -n tooldial python=3.11 ## We encourage you to set python version in 3.11 conda activate tooldial pip install torch==2.4.0 torchvision==0.19.0 torchaudio==2.4.0 --index-url https://download.pytorch.org/whl/cu118 pip install -r requirements.txt ``` # 1. Graph Construction In this stage, we will construct API Graph. To construct the graph, we need the embedding matrix and its quite large so we provide it through our Google Drive: https://drive.google.com/drive/folders/1txuVDgUQw1rDA1ChoUGC4-FuYAOxD0S9?hl=ko **1. Download all three files in `graph_construction` and place it in the base directory (ToolDial folder).** **2. Run the below script** ``` python graph_generation.py ${LCS} ${keyword_similarity} ${description_similarity} ``` LCS, keyword_similarity, description_similarity stands for threshold of each criteria. For example, if you want to use LCS:0.83, keyword_similarity:0.5, description_similarity:0.77 **(this is the threshold used in our work)**, you should run the following: ``` python graph_generation.py 0.83 0.5 0.77 ``` # 2. Graph Evaluation (Removing invalid edges) In this stage, we filter all invalid edges from previously constructed API graph. In our work, we performed many tasks using tmux across multiple sessions in parallel. **0. Move the directory** ``` cd graph_evaluation ``` **1. Open the Stable ToolBench server** - 1.1 First, insert your openai API key in `graph_evaluation/config.yml` - 1.2 Then run the below script ``` python server/main.py ``` or ``` tmux new-session -d -s "server" "python server/main.py" ``` We encourage you to use **tmux sesseion**, because if the stable toolbench server is run in a regular terminal, a new terminal must be opened to execute other code. **2. Graph Edge Evaluation** After you open the StableToolBench Server, initiate the python code and bash script sequentially. ``` python stable_graph_metric/grapheval/file_split.py 20 083_05_077 ./start_tmux.sh 20 083_05_077 ``` After all 20 session is finished, run ``` python stable_graph_metric/grapheval/file_merge.py 20 083_05_077 python stable_graph_metric/grapheval/edge2pair.py 083_05_077 ``` The values provided as arguments are examples. `20` represents the number of tmux sessions, and the specified number of sessions will be created to enable parallel processing. `083_05_077` is the threshold provided during Graph Construction. **Caution about arg** - When inputting, it must be provided in a format where decimal points are removed and concatenated with _. - When inputting values like 0.5 or 0.6, they must be entered as **05 or 06.** Entering them as 050 or 060 may result in an error. - When running the four scripts for Graph Edge Evaluation, the number of tmux sessions and the threshold initially provided must remain consistent, as shown in the example above. If you used other threshold, you may change the input threshold `083_05_077` to other value (074_06_054 etc). Once the evaluation is done, make sure to turn off the StableToolBench server with ``` Ctrl-C ## If you turn on the server with python server/main.py ``` or ``` tmux kill-session -t server ## If you turn on the server with tmux ``` # 3. Dialogue Generation (Scenario Instruction Generation + Dialogue Generation) First, switch the directory with ``` cd ../dialogue_generation ``` If you have completed up to step 2, the Folder `data_generation/pair_list` should contain the following 8 files: ``` aug_desc.json all_edge_list_graph.json normal_graph2idx.json normal_idx2igraph.json normal_idx2igraph.json normal_graph.npy pair_list.json pair_list_original.json ``` The `pair_list.json` file contains valid edges created through **Graph Construction** and **Graph Evaluation.** In step **Dialogue Generation**, conversation data will be generated using `pair_list.json`. We also release `pair_list_original.json` includes the edges we used when creating the previously distributed dialogue data. - Currently, in the third line of `pair_split.py`, the script is set to load the `pair_list_original.json` file. If you want to create dialogue data using the edges you created, replace `pair_list_original.json` with `pair_list.json` in `pair_split.py` and then execute the code. The code for Dialogue Data generation also can executed using parallel processing through tmux. First, fill the "api_key" in `config.yml` in `dialogue_generation` with your openai api key. ``` api_key: device: "cuda" is_sampling: True ## ----------Very Important------------- ## Default value of 'is_sampling' is True, which only generates the dialogue with sampled edges. ## If you set it False, whole edges will be used and it will cost you about $2,500~$3,000. ## ----------Very Important------------- ``` Then, run the below script in order ``` python pair_split.py {n_tmux_session_for_split} # As the number of splits increases, GPU memory usage rises sharply in later stages. A split of 4 to 5 is recommended. ./start_tmux.sh {n_tmux_session_for_split} {category} cd {category} ./start_tmux.sh {n_tmux_session_for_split} {category} python dial_post.py {n_tmux_session_for_split} {category} ``` We have categorized the edges into four types ![edge_type](subgraph_type.png) Dialogue generation code is executed differently depending on each edge type. Fill in {category} with one of yesyes, yeszero, zeroyes, or zerozero, and execute the above code. Set {n_tmux_session_for_split} to the same number used when running pair_split.py. After running the script for all categories, you will see below four json files in `dialogue_generation` directory ``` sample_dialogue_yesyes.json sample_dialogue_yeszero.json sample_dialogue_zeroyes.json sample_dialogue_zerozero.json ``` Next, merge all category's dialogues ``` cd .. python merge_cat.py ``` And if file `total_dialogue.json` is in the current directory (dialogue_generation), the dialogue data has been completed! # 4. Experiments **1. Dataset Download** First, we need to download the dataset. Change your directory into `dialogue_data` ``` cd ../dialogue_data ``` We release the data used in our experiments in our Google Drive. https://drive.google.com/drive/folders/1txuVDgUQw1rDA1ChoUGC4-FuYAOxD0S9?hl=ko Download all 9 json files in `dataset` into `dialogue_data`. - Base File: `train.json`, `val.json`, `test.json` - File for "DST", "Action Prediction", "Faithfulness": `train_dialogue.json`, `val_dialogue.json`, `test_dialogue.json` - File for "Overall Performance": `train_dialogue_overall_obs.json`, `val_dialogue_overall_obs.json`, `test_dialogue_overall_obs.json` We also provide two conversion code. - `dialogue_preprocess.py`: Input: Base file. Output: `train_dialogue.json`, `val_dialogue.json`, `test_dialogue.json` - `dialogue_generation_overall.py`: Input: Base file. Output: `train_dialogue_overall_obs.json`, `val_dialogue_overall_obs.json`, `test_dialogue_overall_obs.json` All experiments were conducted using **testset**.

**2. Initiate Experiment with the dataset** To run the experiment, change the directory with ``` cd ../experiments # from ToolDial/dialogue_data to ToolDial/experiments ```
2-0. Hardware settings We used PyTorch's Distributed Data Parallel (DDP) for training and inference of the open-source model. If you see the `run_train.sh`, ``` #!/bin/bash export MASTER_ADDR=localhost export MASTER_PORT=29500 # You can change the port number if needed CUDA_VISIBLE_DEVICES=0,1,2,3 python {experiment_script}.py ``` `CUDA_VISIBLE_DEVICES`: Specify the GPUs to be used for training and inference. It is also possible to use only a single GPU. `experiment_script`: Name of the script to run for the experiment. Script for each experiment is as follows: - DST: dst_train_dist_normal - Action Prediction: action_train_dist_normal - Faithfulness: hall_train_dist_normal - Overall Performance: overall_train_dist_normal Fill `CUDA_VISIBLE_DEVICES` and `experiment_script` and run `run_train.sh`. Before executing `run_train.sh`, see the description about the arguments for each evaluation task code.
2-1. Dialogue State Tracking (DST) (1) Select the GPU and fill CUDA_VISIBLE_DEVICES (2) Fill the experiment_script with `dst_train_dist_normal` (3) Check the arguments of `dst_train.yaml` - `train_file`, `test_file`: The dataset to be used for the experiments is currently set to use the data we have released. - `model_id`: The name of the open-source model to be used. It should be entered in the format `{repo}/{model_name}` as provided by HuggingFace (e.g., meta-llama/Meta-Llama-3-8B-Instruct). - `is_van`: `True` or `False`. If `True`, training is skipped, and inference is performed directly using the model specified in `model_id`. - `is_give_label`: `True` or `False`. If `True`, the experiment is conducted in the "With GT" setting; if False, it is conducted in the "W/O GT" setting. (4) Run `./run_train.sh`. We encourage you to use tmux with ``` tmux new-session -d -s "dst_train" "./run_train.sh" ``` (5) Once the code execution is complete, the experimental results are stored under `dst_result/{model_name}/{withgt or wogt}`, divided by the number of GPUs used. (6) To view the experimental results and error analysis: Navigate to the `dst_result` directory: ``` cd dst_result ``` Then, configure the `dst_error_analysis.yml` file with the appropriate values for model_name (only the `model_name` from `{repo}/{model_name}`), for example, if the model was `meta-llama/Meta-Llama-3-8B-Instruct`, then model name will be `Meta-Llama-3-8B-Instruct`, and is_give_label (`True` or `False`). Finally, run the ``` python dst_error_analysis.py ``` This will provide the experimental results and error analysis based on the arguments set in `dst_error_analysis.yml`.
2-2. Action Prediction (1) Select the GPU and fill CUDA_VISIBLE_DEVICES (2) Fill the experiment_script with `action_train_dist_normal` (3) Check the arguments of `action_train.yaml` - `train_file`, `test_file`: The dataset to be used for the experiments is currently set to use the data we have released. - `model_id`: The name of the open-source model to be used. It should be entered in the format `{repo}/{model_name}` as provided by HuggingFace. - `is_van`: `True` or `False`. If `True`, training is skipped, and inference is performed directly using the model specified in `model_id`. - `is_give_label`: `True` or `False`. If `True`, the experiment is conducted in the "With GT" setting; if False, it is conducted in the "W/O GT" setting. (4) Run `./run_train.sh`. We encourage you to use tmux with ``` tmux new-session -d -s "action_train" "./run_train.sh" ``` (5) Once the code execution is complete, the experimental results are stored under `action_prediction_result/{model_name}/{withgt or wogt}`, divided by the number of GPUs used. (6) To view the experimental results and error analysis: Navigate to the `action_prediction_result` directory: ``` cd action_prediction_result ``` Then, configure the `action_error_analysis.yml` file with the appropriate values for model_name (only the `model_name` from `{repo}/{model_name}`) and is_give_label (`True` or `False`). Finally, run the ``` python action_error_analysis.py ``` This will provide the experimental results and error analysis based on the arguments set in `action_error_analysis.yml`.
2-3. Faithfulness (1) Select the GPU and fill CUDA_VISIBLE_DEVICES (2) Fill the experiment_script with `hall_train_dist_normal` (3) Check the arguments of `hall_train.yaml` - `train_file`, `test_file`: The dataset to be used for the experiments is currently set to use the data we have released. - `model_id`: The name of the open-source model to be used. It should be entered in the format `{repo}/{model_name}` as provided by HuggingFace. - `api_key`: Your openai API key - `is_van`: `True` or `False`. If `True`, training is skipped, and inference is performed directly using the model specified in `model_id`. (4) Run `./run_train.sh`. We encourage you to use tmux with ``` tmux new-session -d -s "hall_train" "./run_train.sh" ``` (5) Once the code execution is complete, the experimental results are stored under `faithfulness_result/{model_name}`, divided by the number of GPUs used. (6) To view the experimental results and error analysis: Navigate to the `faithfulness_result` directory: ``` cd faithfulness_result ``` Then, configure the `hall_result.yml` file with the appropriate values for model_name (only the `model_name` from `{repo}/{model_name}`). Finally, run the ``` python hall_result.py ``` This will provide the experimental resultsbased on the arguments set in `hall_result.yml`.
2-4. Overall Performance (1) Select the GPU and fill CUDA_VISIBLE_DEVICES (2) Fill the experiment_script with "overall_train_dist_normal" (3) Check the arguments of `overall_train.yaml` - `train_file`, `test_file`: The dataset to be used for the experiments is currently set to use the data we have released. - `model_id`: The name of the open-source model to be used. It should be entered in the format `{repo}/{model_name}` as provided by HuggingFace. - `is_van`: `True` or `False`. If `True`, training is skipped, and inference is performed directly using the model specified in `model_id`. (4) Run `./run_train.sh`. We encourage you to use tmux with ``` tmux new-session -d -s "overall_train" "./run_train.sh" ``` (5) Once the code execution is complete, the experimental results are stored under `overall_result/{model_name}`, divided by the number of GPUs used. (6) To view the experimental results and error analysis: Navigate to the `overall_result` directory: ``` cd overall_result ``` Then, configure the `overall_result.yml` file with the appropriate values for model_name (only the `model_name` from `{repo}/{model_name}`). Finally, run the ``` python overall_result.py ``` This will provide the experimental resultsbased on the arguments set in `overall_result.yml`.
#### [Additional] We have released the overall performance model we used. Please to run the `overall_infer.py` with yaml setting: ``` train_file: ../dialogue_data/train_dialogue_overall_obs.json test_file: ../dialogue_data/test_dialogue_overall_obs.json model_id: "HOLILAB/td-llama-op" is_van: True ``` and `run_train.sh` as below: ``` #!/bin/bash # # Set the MASTER_ADDR and MASTER_PORT environment variables export MASTER_ADDR=localhost export MASTER_PORT=29500 # You can change the port number if needed # Run the Python training script CUDA_VISIBLE_DEVICES=0,1,2,3 python overall_infer.py ## You can change the CUDA_VISIBLE_DEVICES ``` 2-5. GPT baslines inference results - We also release the inference results used to present the GPT baseline experiment results in our paper. - We also provide error analysis code for DST and Action Prediction. - Inside the `dst_result` and `action_prediction_result` folders, there is a `gpt_baselines` folder containing the inference results and error analysis code. - By filling in each `yml` file appropriately and running the error analysis code, you can view the experimental results and error analysis outcomes. # Citation ``` @article{shim2025tooldial, title={Tooldial: Multi-turn dialogue generation method for tool-augmented language models}, author={Shim, Jeonghoon and Seo, Gyuhyeon and Lim, Cheongsu and Jo, Yohan}, journal={arXiv preprint arXiv:2503.00564}, year={2025} } ```