go-mf | Classification of the molecular function of a transcript's product as defined by the GO Resource. | `multilabel` | [website](https://geneontology.org/) |
| GO Biological Process | go-bp | Classification of the biological process a transcript's product participates in as defined by the GO Resource. | `multilabel` | [website](https://geneontology.org/) |
| GO Cellular Component | go-cc | Classification of the cellular component where a transcript's product is localized as defined by the GO Resource. | `multilabel` | [website](https://geneontology.org/) |
### Translation Regulation
| Dataset Name | Catalogue Identifier | Description | Tasks | Citation |
|---|---|---|---|---|
| Mean Ribosome Load (Sugimoto) | mrl‑sugimoto | Mean ribosome load (MRL) per transcript isoform as measured in human cells using isoform-resolved ribosome profiling. | `regression` | [paper](https://www.nature.com/articles/s41594-022-00819-2) |
| Mean Ribosome Load (Sample) | mrl‑sample‑egfp mrl‑sample‑mcherrymrl‑sample‑designedmrl‑sample‑varying | Mean ribosome load (MRL) measured in an MPRA of randomized and designed 5'UTR regions attached to eGFP or mCherry reporters. Includes various RNA modifications and UTR lengths. | `regression` | [paper](https://pubmed.ncbi.nlm.nih.gov/31267113/)|
| Mean Ribosome Load & Half-life | mrl‑hl‑lbkwk | Joint prediction of ribosome load and RNA half-life from synthetic mRNA sequences in the Leppek et al. dataset. | `regression` | [paper](https://pubmed.ncbi.nlm.nih.gov/33821271/) |
| Translation Efficiency (Human) | translation‑efficiency‑human | Translation efficiency of human transcripts measured using ribosome profiling. | `regression` | [paper](https://www.nature.com/articles/s41587-025-02712-x) |
| Translation Efficiency (Mouse) | translation‑efficiency‑mouse | Translation efficiency of mouse transcripts measured using ribosome profiling. | `regression` | [paper](https://www.nature.com/articles/s41587-025-02712-x) |
### RNA Stability
| Dataset Name | Catalogue Identifier | Description | Tasks | Citation |
|---|---|---|---|---|
| RNA Half-life (Human) | rnahl‑human | RNA half-life of human transcripts measured using time-course RNA-seq after transcription inhibition. | `regression` | [paper](https://genomebiology.biomedcentral.com/articles/10.1186/s13059-022-02811-x) |
| RNA Half-life (Mouse) | rnahl‑mouse | RNA half-life of mouse transcripts measured using time-course RNA-seq after transcription inhibition. | `regression` | [paper](https://genomebiology.biomedcentral.com/articles/10.1186/s13059-022-02811-x) |
### Protein-RNA Interactions
| Dataset Name | Catalogue Identifier | Description | Tasks | Citation |
|---|---|---|---|---|
| eCLIP RBP Binding (K562) | eclip‑binding‑k562 | RNA-binding protein (RBP) binding sites on mRNA sequences identified using eCLIP-seq in K562 cells. Covers ~80 different RBPs. | `multilabel` | [paper](https://www.nature.com/articles/s41586-020-2077-3) |
| eCLIP RBP Binding (HepG2) | eclip‑binding‑hepg2 | RNA-binding protein (RBP) binding sites on mRNA sequences identified using eCLIP-seq in HepG2 cells. Covers ~70 different RBPs. | `multilabel` | [paper](https://www.nature.com/articles/s41586-020-2077-3) |
### Subcellular Localization
| Dataset Name | Catalogue Identifier | Description | Tasks | Citation |
|---|---|---|---|---|
| Protein Subcellular Localization | prot‑loc | Subcellular localization of transcript protein products based on experimental evidence from the Human Protein Atlas. | `multilabel` | [website](https://www.proteinatlas.org/) |
| RNA Subcellular Localization (Fazal) | rna‑loc‑fazal | Subcellular localization of mRNA molecules measured using APEX-seq (proximity labeling + RNA-seq) in human cells. | `multilabel` | [paper](https://doi.org/10.1016/j.cell.2019.05.027) |
### RNA Lifecycle
| Dataset Name | Catalogue Identifier | Description | Tasks | Citation |
|---|---|---|---|---|
| RNA Lifecycle (Ietswaart) | rna‑lifecycle‑ietswaart | RNA lifecycle properties including synthesis, processing, and degradation rates in human cells. | `multilabel` | [paper](https://pubmed.ncbi.nlm.nih.gov/38964322/) |
### miRNA Target Prediction
| Dataset Name | Catalogue Identifier | Description | Tasks | Citation |
|---|---|---|---|---|
| miRNA Target (MirTarClash) | mirna‑target | Experimentally validated miRNA target sites on human mRNAs from CLASH-based experiments. Binary classification for top 20 most frequent miRNAs. | `multilabel` | [paper](https://academic.oup.com/database/article/doi/10.1093/database/baaf023/8106627) |
### Variant Effect Prediction
| Dataset Name | Catalogue Identifier | Description | Tasks | Citation |
|---|---|---|---|---|
| VEP TraitGym (Mendelian) | vep‑traitgym‑mendelian | Pathogenicity prediction for genetic variants in 3'UTR and 5'UTR regions associated with Mendelian diseases. | `classification` | [paper](https://www.biorxiv.org/content/10.1101/2025.02.11.637758v1) |
| VEP TraitGym (Complex) | vep‑traitgym‑complex | Pathogenicity prediction for genetic variants in 3'UTR and 5'UTR regions associated with complex traits. | `classification` | [paper](https://www.biorxiv.org/content/10.1101/2025.02.11.637758v1) |
| UTR Variants (Bohn) | utr‑variants‑bohn‑utr5utr‑variants‑bohn‑utr3 | Variant effect prediction for 5'UTR and 3'UTR variants from Bohn et al. | `classification` | [paper](https://www.frontiersin.org/journals/molecular-biosciences/articles/10.3389/fmolb.2023.1257550/full) |
### Adding a new dataset
New datasets should inherit from `BenchmarkDataset`. Dataset names cannot contain underscores. Each new dataset should download raw data and process it into a dataframe by overriding `process_raw_data`. This dataframe should store transcript as rows, using string encoding in the `sequence` column. If homology splitting is required, a column `gene` containing gene names is required. Six track embedding also requires columns `cds` and `splice`. The target column can have any name, as it is specified at time of probing. New datasets should be added to `DATASET_CATALOG`.
## Citation
If you use mRNABench in your research, please cite:
```bibtex
@article{shi_dalal_fradkin_2025_mrnabench,
author = {Shi, Ruian and Dalal, Taykhoom and Fradkin, Philip and Koyyalagunta, Divya and Chhabria, Simran and Jung, Andrew and Tam, Cyrus and Ceyhan, Defne and Lin, Jessica and Laverty, Kaitlin U. and Baali, Ilyes and Wang, Bo and Morris, Quaid},
title = {mRNABench: A curated benchmark for mature mRNA property and function prediction},
elocation-id = {2025.07.05.662870},
year = {2025},
doi = {10.1101/2025.07.05.662870},
publisher = {Cold Spring Harbor Laboratory},
URL = {https://www.biorxiv.org/content/early/2025/07/08/2025.07.05.662870},
eprint = {https://www.biorxiv.org/content/early/2025/07/08/2025.07.05.662870.full.pdf},
journal = {bioRxiv}
}
```
The original sources for each dataset and model should be cited if used, and can be found above. A substantial number of model implementations use the the `multimolecule` package: https://huggingface.co/multimolecule; citation information can be found on their HuggingFace.
## Evo2 Setup
Inference using Evo2 requires installing the following in its own environment. Note: There may be an issue where the evo_40b models, when downloaded, have their merged checkpoints stored one directory above the HuggingFace hub cache. You may need to manually move the checkpoint into its corresponding snapshot directory: `/hub/models--arcinstitute-evo2_40b*/snapshots/snapshot_name/`
**Hardware Requirements:** Evo2 can only be run on H100 GPUs.
```bash
conda create --name evo_bench -c conda-forge python=3.11 gxx=12.2.0 -y
conda activate evo_bench
pip install torch==2.6.0+cu124 --index-url https://download.pytorch.org/whl/cu124
pip install vtx==1.0.4
pip install evo2==0.2.0
pip install flash-attn==2.7.4.post1
cd path/to/mRNA/bench
pip install -e .
```
## Dev Mode Setup
Dev mode requires additional dependencies for generating datasets from scratch and accessing certain datasets.
```bash
conda create --name mrna_bench_dev python=3.10
conda activate mrna_bench_dev
pip install torch==2.2.2 --index-url https://download.pytorch.org/whl/cu121
pip install mrna-bench[base_models, dev]
```