# Attribute Tagging

Implementation of the attribute tagging pipeline for the paper:

**[Towards an Enhanced, Faithful, and Adaptable Web Interaction Environment](https://openreview.net/pdf?id=y-F1kab2Its)**  
[John Yang](https://john-b-yang.github.io/), [Howard Chen](https://howard50b.github.io/), [Karthik Narasimhan](https://www.cs.princeton.edu/~karthikn/)

This repository contains code for reproducing the paper's results.

* For the implementations of the `reviews` section and V2 reward function, please refer to the original [WebShop](https://github.com/princeton-nlp/WebShop) repository (specifically PRs [#8](https://github.com/princeton-nlp/WebShop/pull/8) and [#10](https://github.com/princeton-nlp/WebShop/pull/10))
* The survey for quantifying task difficulty and soliciting feedback for qualitative, semantic improvements can be found at this [link](https://docs.google.com/forms/d/e/1FAIpQLSd0q7SZjCUNxIhtmMgVHByMJan2tX60vWN1IX8c3HfDQJnPrw/viewform).

## 🏗️ Set Up
Follow these initial steps to install the necessary dependencies and required data to run the summarization model used for attribute tagging.
1. `pip install -r requirements.txt`
2. `pip install git+https://github.com/huggingface/transformers` (Model requires latest version, docs [here](https://huggingface.co/docs/transformers/installation#installing-from-source))
3. Create a `data` folder at the root of this repository
4. Download the following datasets from this google drive [link](https://drive.google.com/drive/u/2/folders/1iIqUIfSOl1UW0M9PeDaQlRW6zn_Jx194) and place them in the `data` folder
    * `items_shuffle.json`
    * `reviews.json`
    * `items_ins_v2.json`

## 🛠️ Usage
### Attribute Tagging Dataset
To generate the synthetic dataset consisting of `(product information, attributes)` pairs, run:
```
python syn-data-gen/gen-data.py
```
This generates the following files in the `data/attr-gen-data/` folder.
* `attr_gen_dataset.json`: Entire 10000 dataset of [product info -> attributes]
* `attr_gen_dataset_train_{size}.csv`: Training data splits of size [50, 200, 500, 1000, 2000, 3000, 4000, 5000, 6500, 8000]
* `attr_gen_dataset_valid.json`: Validation data (1000 rows)
* `attr_gen_dataset.test.json`: Test data (1000 rows)

### T5 Summarization Model
The `run_summarization.py` and `run_summarization_no_trainer.py` files are directly taken from [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/summarization). Each file contains the pipeline for preprocessing the target data, training the specified model, then evaluating the model on the validation set. The primary difference is that the `_no_trainer` file does not use the [Trainer](https://huggingface.co/docs/transformers/main_classes/trainer) class.

The only addition made to these files is logging to record the raw predictions for purposes of computing accuracy.

Within the `model/scripts` directory...
* *To run the model on GPU enabled devices* refer to the `bash` subdirectory
* *To submit a model run as a Slurm job* refer to the `slurm` subdirectory

Within the above folder, the file names refer to the following:
* `run`: Runs `run_summarization.py`. Used for `t5-small` and `t5-base` models.
* `run_no`: Runs `run_summarization_no_trainer.py`. Used for `t5-large` model.
* `run_acc`: Runs `run_summarization_no_trainer.py` with the [accelerate](https://huggingface.co/docs/accelerate) library. Used for `t5-3b` model.

The `accelerate` configurations are specified in `model/default_config.yml`.

**Run Output**:
A single run of these files will generate a corresponding log file under `model/logs/{model name}/predictions/log_{train set size}.txt`

To calculate a model's accuracy, run the following script:
```
python compute_accuracy.py \
    --models "t5-small t5-base" \
    --sizes "50 200 500 1000 2000 3000 4000 5000 6500 8000" \
    --debug \
    --plot
```