# Enveda-180: High throughput generation of a large open multimodal small molecule LC-MS/MS and ion mobility library

<p>
  <a href="https://doi.org/10.5281/zenodo.20436851"><img alt="Zenodo" src="https://zenodo.org/badge/DOI/10.5281/zenodo.20436851.svg" height="22px"></a>
  <a href="LICENSE"><img alt="License" src="https://img.shields.io/badge/license-MIT-green.svg" height="22px"></a>
  <a href="https://www.python.org/"><img alt="Python: >=3.11" src="https://img.shields.io/badge/python-%3E=3.11-blue?logo=python" height="22px"></a>
  <a href="https://github.com/astral-sh/ruff"><img alt="Linting: Ruff" src="https://img.shields.io/badge/linting-ruff-yellow?logo=ruff" height="22px"></a>
<p>

<p align="center">
  <img src="https://raw.githubusercontent.com/enveda/enveda-180/main/docs/workflow.png" width="50%"/>
</p>

This repository contains the code and notebooks for the Enveda-180 project,
which generates a large open multimodal MS/MS + ion mobility spectral library
for drug-like small molecules from the Enamine REAL space. The project is a
collaboration between Enveda and the scientific community, and we welcome
contributions and feedback.

## Citation

> Krettler, C., Kind, T., DeBloois, E., Killian, M., Prince, J., Taylor, J.,
> Gadiya, Y., Simpson, P., Domingo-Fernández, D., Allen, A., Colluru, V.,
> Healey, D. (2026). Enveda-180: generation of a large open multimodal MS/MS
> and ion mobility spectral library for drug-like small molecules.
> *ChemRxiv*. https://doi.org/10.26434/chemrxiv.15004319/v1

## Repository layout

```
enveda-180/
  data_processing/        # Python package: ms_prepper + extraction + curation
  scripts/                # CLI runners (build manifest, run ms_prepper, run pipeline)
  notebooks/              # process.ipynb — papermill-driven pipeline entry point
  tests/                  # Unit tests
  docs/                   # Design / agent notes
  pyproject.toml          # Hatchling build; the package installs as `data_processing`
  README.md, LICENSE
```

The package lives at `data_processing/` and is installed under the import
name `data_processing`.

## Install

```bash
uv sync
# or, with pip:
pip install -e .
```

Python 3.11–3.13 is supported.

## Pipeline overview

The `data_processing` package has two halves:

1. **`data_processing.ms_prepper`** — read a Bruker `.d` folder and write
   four per-msrun parquets (`ms1`, `ms2`, `ms2_event`, `frame`).
2. **`data_processing.{extraction, curation, pipeline}`** — targeted
   extraction of MS/MS spectra for each compound-pool member across all common
   adducts, plus quality-curation helpers.

### Adducts targeted by the extraction pipeline

The default adduct list is surfaced at the package root:

```python
from data_processing import COMMON_ADDUCTS, CATION_ADDUCTS
```

Positive mode: `[M+H]+`, `[M+Na]+`, `[M+NH4]+`, `[M+K]+`, `[2M+Na]+`,
`[2M+H]+`, `[M-H2O+H]+`, `[M-H4O2+H]+`

Negative mode: `[M-H]-`, `[M+Cl]-`, `[M+FA-H]-`, `[M+Br]-`, `[M+Hac-H]-`,
`[2M-H]-`, `[2M+FA-H]-`, `[2M+Hac-H]-`, `[M-H2O-H]-`

Cation (natively charged) mode: `[M]+`

All exact monoisotopic offsets are computed from the adduct name strings via
`molmass` in `data_processing.extraction.load.compute_adduct_metadata`. To
add an adduct, append its name to `COMMON_ADDUCTS` (or `CATION_ADDUCTS`) in
[data_processing/extraction/load.py](data_processing/extraction/load.py).

## Quick start

### 1. Convert a Bruker `.d` to parquets

```bash
create-signal-tables --input /path/to/sample.d --output-dir /data/enamine_prep
```

or from Python:

```python
from data_processing.ms_prepper import pipeline
pipeline.run(input_path="/path/to/sample.d", output_dir="/data/enamine_prep")
```

This writes four parquets named `{msrun_id}.{ms1,ms2,ms2_event,frame}.parquet`.

### 2. Configure where the inputs and outputs live

Set environment variables (a `.env` file at the repo root is auto-loaded):

```bash
ENAMINE_PROCESSING_DATA_PATH=/data/data_processing
ENAMINE_PREP_DATA_PATH=/data/data_processing/input/enamine_prep
ENAMINE_PROCESSING_LOG_LEVEL=INFO
```

Expected input layout:

```
${ENAMINE_PROCESSING_DATA_PATH}/
  input/
    enamine_prep/
      {msrun_id}.ms1.parquet
      {msrun_id}.ms2.parquet
      {msrun_id}.ms2_event.parquet
      {msrun_id}.frame.parquet
    preprocessed/
      <my-batch>/
        compound_metadata.parquet
        compound_pool_id_to_msrun_ids_df.parquet
  output/
    <my-batch>/
      <version>/
        extracted/...
        annotated/...
        annotated_matched/...
        final/...
```

### 3. Run the extraction + curation pipeline

```python
import asyncio
from data_processing import pipeline

asyncio.run(pipeline.run(
    name="my-batch",
    version="v0.1",
    log_path="/tmp/enamine-processing-run",
))
```

Or call the headline extraction function directly:

```python
from pathlib import Path
from data_processing.pipeline import extract_spectra

spectra = extract_spectra(
    compound_pool_id_to_msrun_ids_df_path=Path(".../compound_pool_id_to_msrun_ids_df.parquet"),
    compound_metadata_df_path=Path(".../compound_metadata.parquet"),
    export_folder=Path(".../output/my-batch/v0.1"),
    version="v0.1",
)
```

## Running the 14 enamine batches end-to-end

`scripts/` contains the end-to-end runners used to produce the Enveda-180
library:

```bash
# 1. Build a manifest mapping MSB ids -> raw .d filenames for every batch.
#    (Enveda-internal: loads data from Databricks.)
python scripts/build_run_manifest.py --all

# 2. Download the .d folders listed in the manifest to a local directory.
#    (Use whatever tool your environment provides — azcopy, gsutil, rclone, ...)

# 3. Run ms_prepper on every .d folder for one batch.
python scripts/run_ms_prepper.py \
    --batch enamine-part1 \
    --raw-dir /scratch/enamine/raw \
    --output-dir $ENAMINE_PREP_DATA_PATH \
    --skip-existing

# 4. Run the extraction + curation pipeline.
python scripts/run_pipeline.py --batch enamine-part1 --version v0.1
```

External users without Enveda Databricks access should skip step 1 and
hand-author `raw_file_manifest.parquet` with at least the columns
`msrun_id, raw_file`.

## Notebooks

[notebooks/process.ipynb](notebooks/process.ipynb) is the canonical
papermill-driven entry point for running the extraction + curation
pipeline against one preprocessed batch. It is the notebook-flavoured
equivalent of `scripts/run_pipeline.py` — pick whichever you prefer.

```bash
name="my-batch"
version=$(date +%b-%d-%Y)
folder_path="$ENAMINE_PROCESSING_DATA_PATH/logs/${name}"
log_path="${folder_path}/${version}"
mkdir -p "$folder_path"

papermill notebooks/process.ipynb "${log_path}.ipynb" \
    -p name "$name" \
    -p version "$version" \
    -p log_path "$log_path"
```

## Public configuration surface

Every tunable knob is exposed in `data_processing.config` and re-exported
from the package root:

| Name | Source | Description |
| --- | --- | --- |
| `COMMON_ADDUCTS` | `extraction.load` | Adducts searched per compound. |
| `CATION_ADDUCTS` | `extraction.load` | Adducts for natively charged species. |
| `DATA_PATH` | `extraction.constants` | Root data directory (`$ENAMINE_PROCESSING_DATA_PATH`). |
| `ENAMINE_PREP_DATA_PATH` | `extraction.constants` | ms_prepper parquet directory. |
| `ENAMINE_PREP_VERSION` | `extraction.constants` | Stamped on every extracted spectrum. |
| `INDEX_COLUMNS` | `curation.constants` | `["msrun_id", "msms_aggregate_id"]`. |
| `HASH_PRECISION` | `curation.constants` | Spectrum-hash mz rounding (default `1`). |
| `TOP_N`, `TOLERANCE_DA` | `curation.constants` | Library-match knobs. |
| `EXPORT_KEYS`, `DS8_EXPORT_KEYS` | `curation.constants` | Columns written to disk. |

## SIRIUS annotation

SIRIUS annotation is not invoked automatically. Run SIRIUS yourself
(<https://bio.informatik.uni-jena.de/software/sirius/>) on the `extracted`
jsonl output and merge the resulting columns into the dataframe before
continuing. All downstream curation code reads SIRIUS columns defensively, so
the pipeline still completes without them.

## License

MIT. See [LICENSE](LICENSE).