# NTREX
NTREX -- News Test References for MT Evaluation from English into a total of 128 target languages with document-level information.
## Data structure
`NTREX-128/newstest2019-src.eng.txt` - original source data
`NTREX-128/newstest2019-ref.*` - human translated references for the original 128 languages
`NTREX-additional/newstest2019-ref.*` - human translated references for additional languages
`LANGUAGES.tsv` - mapping between ISO codes and language names
`DOCUMENT_IDS.tsv` - line aligned document boundaries
## Updates
References for two Swiss German dialects (Bern and Zurich German) collected as part of [this work](https://aclanthology.org/2023.wmt-1.99/) have been added to the `NTREX-additional` folder.
Data package has been released on 11/24/2022. Enjoy!
## License
The NTREX-128 data set is released under the [CC BY-SA 4.0 license](https://github.com/MicrosoftTranslator/NTREX/blob/main/LICENSE.md).
## How to Cite
For the original NTREX-128 dataset, please cite:
```
@inproceedings{federmann-etal-2022-ntrex,
title = "{NTREX}-128 {--} News Test References for {MT} Evaluation of 128 Languages",
author = "Federmann, Christian and Kocmi, Tom and Xin, Ying",
booktitle = "Proceedings of the First Workshop on Scaling Up Multilingual Evaluation",
month = "nov",
year = "2022",
address = "Online",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2022.sumeval-1.4",
pages = "21--24",
}
```
as well as the WMT 2019 paper that provided the English source data NTREX-128 is based on:
```
@inproceedings{barrault-etal-2019-findings,
title = "Findings of the 2019 Conference on Machine Translation ({WMT}19)",
author = {Barrault, Lo{\"\i}c and
Bojar, Ond{\v{r}}ej and
Costa-juss{\`a}, Marta R. and
Federmann, Christian and
Fishel, Mark and
Graham, Yvette and
Haddow, Barry and
Huck, Matthias and
Koehn, Philipp and
Malmasi, Shervin and
Monz, Christof and
M{\"u}ller, Mathias and
Pal, Santanu and
Post, Matt and
Zampieri, Marcos},
editor = "Bojar, Ond{\v{r}}ej and
Chatterjee, Rajen and
Federmann, Christian and
Fishel, Mark and
Graham, Yvette and
Haddow, Barry and
Huck, Matthias and
Yepes, Antonio Jimeno and
Koehn, Philipp and
Martins, Andr{\'e} and
Monz, Christof and
Negri, Matteo and
N{\'e}v{\'e}ol, Aur{\'e}lie and
Neves, Mariana and
Post, Matt and
Turchi, Marco and
Verspoor, Karin",
booktitle = "Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)",
month = aug,
year = "2019",
address = "Florence, Italy",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/W19-5301",
doi = "10.18653/v1/W19-5301",
pages = "1--61",
}
```
For the additional Swiss German dialect references, please cite:
```
@inproceedings{aepli-etal-2023-benchmark,
title = "A Benchmark for Evaluating Machine Translation Metrics on Dialects without Standard Orthography",
author = {Aepli, No{\"e}mi and
Amrhein, Chantal and
Schottmann, Florian and
Sennrich, Rico},
editor = "Koehn, Philipp and
Haddow, Barry and
Kocmi, Tom and
Monz, Christof",
booktitle = "Proceedings of the Eighth Conference on Machine Translation",
month = dec,
year = "2023",
address = "Singapore",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2023.wmt-1.99",
doi = "10.18653/v1/2023.wmt-1.99",
pages = "1045--1065",
}
```