# Enveda CASMI 2026 — Molecule ID from Mass Spectra Kaggle competition: https://www.kaggle.com/competitions/enveda-CASMI26-molecule-id-mass-spectra Full roadmap (phases, technical approach comparison, risks): see the project doc shared in chat (Claude Artifact "Enveda CASMI 2026 Roadmap"). ## Task Predict up to 25 ranked SMILES candidates per `molecule_id` from LC-MS/MS spectra. Scored with Mean Reciprocal Rank @ 25 (MRR@25), matched via RDKit tautomer canonicalization + InChIKey14 (connectivity only, stereochemistry ignored). - Start: 2026-09-14 - Entry / team-merge deadline: 2026-12-07 - Final submission deadline: 2026-12-14 - Max 5 submissions/day, 2 final submissions selected ## If the machine restarts mid-run Nothing here needs to be redone from scratch. Everything expensive is written to disk as it is produced, so a shutdown costs at most the single epoch that was in progress. **Safe across a restart** — all of `data/processed/`: the candidate pool and its fingerprints (`candidate_fingerprints.parquet`, `pool_multi_fp.npy`, `pool_fp2.npy`), the training targets (`train_multi_fp.npz`, `train_fp2.npz`), every model checkpoint, and the 28 GB PubChem pool under `data/pubchem/`. Also safe: everything pushed to git, and everything already on Kaggle (datasets, kernel versions, submissions). **Resume the only long job — training.** One command, safe to run at any time; it reports and does nothing if training is already alive: ```bash cd ~/Desktop/isaack/casmi-2026 bash scripts/resume_training.sh # v2 stack; pass v1 for the other ``` Do not count on the process itself surviving. Backgrounded training here has *not* reliably outlived a session ending — the v1 run came through several, the v2 run died mid-epoch and sat dead for six days looking alive to a careless `pgrep` (the pattern matched the checking command's own command line; `pgrep -f "train_multi[_]fingerprint"` avoids that). Per-epoch checkpoints plus `--resume` are the mechanism that actually works, and a restart costs at most the epoch in progress. To check for real, compare the log's mtime against the clock rather than trusting that a process exists: ```bash ls -la --time-style=+%F_%T logs/train_fp2*.log | tail -2; date +%F_%T ``` It prints which checkpoint it resumed from and which epoch it continues at. Checkpoints written from now on also carry the optimizer and scheduler state, so a resume is exact; older ones hold weights only, in which case the learning-rate schedule is fast-forwarded to the right step and the optimizer moments are lost (stated in the log when it happens). **Everything else is stateless — just re-run it.** The sweeps and selectors (`scripts/sweep_paired.py`, `scripts/select_multi_checkpoint.py`) hold no state and take minutes. The `until ...; do sleep; done` shells that wait on jobs are disposable; nothing depends on them. **Check what is still running** before assuming anything was lost: ```bash ps -eo pid,etime,cmd | grep "python3 -u scripts" | grep -v grep ls -t data/processed/*_e*.pt | head ``` ## Project layout ``` casmi-2026/ ├── data/ │ ├── raw/ # untouched train.parquet, test.parquet, sample_submission.csv │ └── processed/ # cleaned/curated spectra, cached features ├── notebooks/ # exploration notebooks ├── src/ # reusable pipeline code (loading, similarity, scoring, submission) ├── submissions/ # generated submission.csv files, timestamped ├── requirements.txt └── README.md ``` ## Setup 1. Create a virtualenv / conda env and install dependencies: ```bash pip install -r requirements.txt ``` 2. Set up the Kaggle API (one-time, manual — see below), then download the data: ```bash bash scripts/download_data.sh ``` ### Kaggle API setup (manual step — do this yourself) 1. Go to https://www.kaggle.com/settings → **API** section → **Create New Token**. This downloads a `kaggle.json` file containing your username and API key. 2. Move it into place and lock down permissions: ```bash mkdir -p ~/.kaggle mv ~/Downloads/kaggle.json ~/.kaggle/kaggle.json chmod 600 ~/.kaggle/kaggle.json ``` 3. On the competition page, click **Join Competition** and accept the rules (required before the API will let you download data or submit — this has to be done by you, logged in, in a real browser). 4. Verify it works: ```bash kaggle competitions list -s casmi ``` ## Current status **Public leaderboard: 0.245** (MRR@25), from 0.063 at the first submission. | submission | public | what changed | |---|---|---| | Phase 1 | 0.063 | library search over deduplicated reference spectra | | Phase 2 | 0.194 | COCONUT candidate pool, mass window tightened 150 Da → 1 mDa | | Phase 3 | 0.230 | learned spectrum→fingerprint model, fused with analog propagation | | Phase 4 | 0.232 | tiered PubChem pool (94M structures, memory-mapped) | | Phase 5 | 0.236 | large peak transformer in the ensemble | | Phase 5 | 0.240 | its best-Tanimoto checkpoint | | Phase 6 | **0.245** | multi-fingerprint scoring: Morgan r3 + MACCS alongside Morgan r2 | The last two came from changing the *measurement*, not the search: a paired per-molecule test (`scripts/sweep_paired.py`) resolves the ~0.01 MRR effects this pipeline actually turns on, which n=150 with unpaired means cannot. It found two real gains in two sweeps and predicted both leaderboard moves correctly (+0.005 predicted / +0.004 actual, then +0.003 / +0.004) — after proxy-driven estimates of +15% delivered +0.002, and +1.2% delivered −0.003. Anything that harness reports as `ns` is genuinely not worth chasing. - [x] Kaggle API configured, rules accepted - [x] Data downloaded (train: 2,539,608 spectra; test: 1,213 spectra / 400 molecules) - [x] Baseline pipeline reproduced — see `src/baseline.py`. Coarse sparse-cosine pre-filter over a deduplicated library (~500k unique structure/adduct spectra from `enveda-180`, `enveda-np-examples`, `gnps`, `riken`, `pluskal_ms2`) + exact modified-cosine rescoring of the top candidates, max-pooled across a molecule's spectra, deduplicated on the tautomer-canonical InChIKey14 the competition actually scores against. Full test set runs in ~1-2 minutes (well under the 9h Kaggle limit). **Local validation MRR@25: 0.4454** (held out `enveda-np-examples`, the library closest to the test distribution, as a pseudo-test set with known ground truth — optimistic vs. the real hidden test set, since these are common compounds that also appear elsewhere in training, but confirms the pipeline mechanics are correct). First real submission written to `submissions/baseline_v1.csv`, validated against `sample_submission.csv`. **Real public leaderboard score: 0.063** — badly under even the ~0.15-0.23 "library-only ceiling" public notebooks reported. A harder local validation (structures entirely absent from the library, so the real Class 2/3 failure mode) confirmed why: **0.0000 MRR@25** — the pipeline never abstains, it confidently guesses wrong every time a structure isn't literally already in the library. - [x] Candidate-pool expansion (Phase 2, in progress) — see `src/candidates.py`, `src/propagation.py`, `src/pipeline_v2.py`. Expands guesses beyond the training library to COCONUT (~729k combined candidates, natural products + training structures, fingerprinted and cached), ranking candidates via `max_a sim(anchor)^4 * Tanimoto(candidate, anchor)` against spectrally-similar library analogs (the recipe public notebooks converged on). Two validated fixes on the same hard, structure-absent-from-library validation set: - Candidate expansion alone: 0.0000 -> 0.0145 MRR@25. - + entropy-weighted similarity (Li et al. 2021, replacing plain cosine — diagnosed directly here: plain cosine gave "0.99 similar" anchors that were chemically unrelated, Tanimoto ~0.1, because a single shared dominant fragment peak dominates raw cosine regardless of the rest of the spectrum): 0.0145 -> **0.0211**. First submission (window=150 Da, exponent=4, no spectrum merging): real public leaderboard score **0.075** (up from Phase 1's 0.063) — a real, positive move in the direction the local hard validation predicted, though smaller in absolute terms (local: +0.0211, real: +0.012). Encouraging as a sanity check on the validation methodology, but still well under research-suggested ceilings. **Follow-up tuning found the mass window was badly miscalibrated.** A local parameter sweep on the same hard-validation set found scores *monotonically improving* as the window shrank — the "generous analog window" was diluting every ranking with tens of thousands of same-mass-region-but-chemically-irrelevant candidates. Combined with merging a molecule's multi-collision-energy spectra into one consensus spectrum before anchor search (`pipeline_v2.merge_spectra`, grouped by adduct), the tuned defaults (`MASS_WINDOW_DA=0.3`, `PROPAGATION_EXPONENT=3.0`, both in `src/propagation.py`) score **0.1725 MRR@25** on the full 150-molecule hard validation set — up from 0.0211, an 8x improvement — and run in 56s instead of 1120s (20x faster, since far fewer candidates get scored per query). Resubmitted (kernel v5) 2026-09-18: **real public leaderboard score 0.138** — more than double the original 0.063 baseline, and ~1.8x the previous Phase 2 submission (0.075). Real-world gain (0.075 -> 0.138, +0.063) is smaller in relative terms than the local validation jump (0.0211 -> 0.1725, ~8x) but large and unambiguous either way — strong confirmation the tuned mass window and spectrum merging generalize to the real hidden test set. **Kept sweeping past 0.3 Da — the window isn't an analog-hop tolerance.** Candidates are always matched against the query's own measured neutral mass, so tightening the window just means trusting the instrument's mass accuracy more. Score kept improving monotonically all the way to ~1 mDa (`MASS_WINDOW_DA=0.001`, `PROPAGATION_EXPONENT=2.5`: **0.4392 MRR@25**), the real accuracy floor — tighter than that and real measurement noise starts excluding true answers. Added adaptive widening for spectra that land with zero candidates at that width (retry at 10x wider, up to a 1 Da cap, before the frequency fallback): cut zero-hit spectra from 16/150 to 6/150 and pushed the validation score to **0.4586** — within reach of the 0.4454 measured on the much easier "common compounds" holdout from Phase 1, but on a validation set built specifically to be hard. **Kernel v6 (these settings) crashed on submit**: ran cleanly in manual testing (public test set) but threw an unhandled exception when Kaggle re-ran it against the actual hidden test set. Most likely cause: `build_submission`'s `sample_submission.csv` coverage check used to hard-`raise` on any molecule_id mismatch, which only passed locally because the public test set's ids happen to match the public sample file. Fixed in three layers (see commit `3a25a55`): the coverage check now warns instead of raising, `predict_test_set` wraps each molecule's prediction in try/except with a frequency-fallback on any failure, and the notebook's outermost cell has one more try/except around the whole prediction step as a last resort. Resubmitted (kernel v7): ran without error and scored **real public leaderboard score 0.194** — our best score yet, confirming both that the tight-window tuning generalizes to the real hidden test set and that the defensive fixes resolved whatever crashed v6. **Score trajectory this session: 0.063 → 0.075 → 0.138 → (crash) → 0.194** — a 3x improvement from the original baseline. **Kernel v8 (all 11 training libraries as anchors): 0.193 — flat.** Expanding the anchor library from 5 to all 11 sources lifted the local hard validation from 0.4586 to 0.4878–0.4914 (validated with two independent held-out pairs), but that did *not* transfer to the real hidden test set (0.194 → 0.193, noise). Instructive: the local validation tracked the big structural changes well (mass-window tuning: huge local gain → big real gain) but not this subtler one. Likely reason: the held-out compounds come from *other public libraries*, so adding more public libraries naturally helps find their analogs — but the real test set is Enveda's own timsTOF measurements of specific natural products, where extra public- library coverage adds little. This is the instrument/chemistry domain shift flagged in the original roadmap, showing up as a local-vs-real gap (~0.49 local vs ~0.19 real) that library-count tuning can't close. Two real submissions now plateaued at ~0.19, suggesting this is near the ceiling of the pure "propagate-from-spectral-analogs" approach. - [x] Learned spectrum -> fingerprint model (Phase 3) — see `src/fingerprint_model.py`, `src/pipeline_v3.py`, `scripts/build_fp_training_data.py`, `scripts/train_fingerprint_model.py`. An MLP (61.8M params) over 0.2 Da fragment bins + neutral-loss bins + precursor/ion-mode features, predicting the 2048-bit Morgan fingerprint with BCE; trained on 2.49M spectra split by structure (hard-validation structures excluded), 11 min on an RTX A4000, held-out Tanimoto@0.5 ~0.31. Candidates ranked by fingerprint log-likelihood under the predicted bit probabilities (one matrix multiply), fused with propagation at alpha=0.3. Hard validation: propagation-only 0.4456, model-only 0.4836, fused **0.4932**. **Real public leaderboard score: 0.214** (up from 0.194) — broke the ~0.19 plateau two Phase 2 configurations hit, confirming the model transfers to the timsTOF test data where analog propagation stalled. Model checkpoint shipped as a private Kaggle dataset (`isaackjoshua/casmi26-fp-model`). **Score trajectory: 0.063 → 0.075 → 0.138 → 0.194 → 0.193 → 0.214.** - [x] Fingerprint model improvement round 1 — see `scripts/eval_pipeline.py` (dual validation: the massbank/mona hard set + a new 200-structure enveda-180 timsTOF holdout from the model's own structure split). The two sets disagree on which signal wins (propagation dominates on held-out drug-like timsTOF structures, the model on natural products); the real test's ordering matches the natural-product set, so alpha stays 0.3. Tried: heavier regularization (a wash), timsTOF- only fine-tuning (+11% timsTOF model-only, −3% natural products, and it *lowered* val Tanimoto@0.5 — that threshold proxy is misleading for a log-likelihood ranker), ensembles of similar models (no gain). **Winner: fine-tune v1 on timsTOF + natural-product libraries (`enveda-180`, `enveda-np-examples`, `gnps`, `riken`), 3 epochs at lr 2e-4, best-Tanimoto checkpoint (`fp_model_ft2_tan.pt`)** — a strict improvement over the deployed model on every metric: hard α=0.3 0.4932→0.4998, timsTOF α=0.3 0.5661→0.5838, timsTOF model-only 0.3966→0.4339. Uploaded as v2 of the model dataset; kernel rebuilt, awaiting the daily submission quota reset. - [x] Peak-level transformer (`src/peak_transformer.py`) — 4.6M params (13x smaller than the MLP), spectra as sets of up to 64 peaks with sinusoidal embeddings of exact m/z and neutral loss, so mass defect and fragment-pair structure are available to self-attention. 6 epochs, ~6 min each. Best val Tanimoto@0.5 0.3304 vs the MLP's 0.31 — but end-to-end it *splits*: better on the instrument-matched timsTOF set (0.5851/0.4466 vs 0.5838/0.4339 at α=0.3/0.0), worse on natural products (0.4797/0.4566 vs 0.4998/0.4905). Second time the Tanimoto@0.5 proxy has pointed the wrong way for a log-likelihood ranker. Ensembling the two (probability-level average, via `pipeline_v3.predict_bit_probs` dispatch) is a wash on the hard set (0.4959) and clearly better on timsTOF (0.5923/0.4650). - [x] Both candidates submitted. ft2_tan MLP: **0.229** (new best, up from 0.214). MLP+transformer ensemble: **0.230** — flat, within noise. The transformer earns nothing on the real test despite winning the timsTOF proxy, the third time the natural-product hard set predicted the real outcome correctly while the timsTOF set over-promised. **Chemistry match matters more than instrument match for this test set** — worth weighting the hard set accordingly from here on. - [x] **The bottleneck is candidate-pool coverage, not ranking.** We score ~0.50 on a validation set where pool coverage is 100%, and 0.23 on the real test — a ratio of ~46%. COCONUT covers classic natural products well (79% of RIKEN plant metabolites) but only 37% of the GNPS natural-product community library and 39% of our own hard-validation structures. The test set is explicitly "natural products, hypothesised natural products, natural product analogs, and synthetic molecules that might plausibly occur in nature" — the analogs and synthetics live in PubChem, not COCONUT. Built a PubChem pool from the CID-SMILES bulk dump (`scripts/build_pubchem_pool.py`): **94,043,986 structures, 28 GB**, deduplicated on InChIKey14 and sorted by exact mass so a query's mass window is a contiguous slice. It **contains 86 of the 91 hard-validation structures COCONUT misses (95%)**. Two passes (shard, then k-way merge) because one pass projected to ~84 GB of RAM; fingerprints live in their own `.npy` shards so a notebook can memory-map 24 GB instead of loading it (`src/large_pool.py`). - [x] **Merging that pool in is a net loss; tiering it in is a win.** Measured on the hard set split by whether COCONUT contains the answer — the 91 that it doesn't are exactly as unreachable as a real Class 2 molecule, which is the only honest way to simulate one (every training structure is in the pool by construction): | strategy | precision (59) | coverage (91) | weighted | |---|---|---|---| | curated only (deployed) | 0.3854 | 0.0147 | 0.1605 | | merged | 0.2052 | 0.1107 | 0.1479 | | **tiered** | **0.3854** | 0.0444 | **0.1785** | Merging loses because of how many mass-matched distractors it adds. Measured at the real test molecules' masses, a ±1 mDa window holds a median of **4,652** PubChem candidates (p90 14,727, max 30,103) against the curated pool's 49 — ~95x more competitors. (An earlier figure of 200–470 was measured at *round* masses like 350.000, which are unusually sparse: real molecules cluster by mass defect, so exact round masses are the emptiest regions of the mass axis. The conclusions below were all measured end-to-end and are unaffected, but the merged-pool coverage result is more impressive than it looked — 0.1107 means finding the answer at ~rank 9 of ~4,650.) But the two columns win in disjoint places and the cost is purely displacement, so `src/pipeline_v4.py` *consults* PubChem rather than merging it: curated candidates keep the ranks they earn and PubChem fills only the slots of the 25 they leave empty. That preserves precision exactly, so it cannot lose ground where the curated pool already works. Sweeping how many slots tier 1 may keep is flat inside noise (0.1783–0.1811 for caps 3–25), so it ships uncapped. - [x] **Two dead ends, both worth the cost of ruling out.** *A natural-product filter on PubChem* would cut distractors ~2x, but it cuts the answers PubChem uniquely recovers harder than it cuts COCONUT (stereocentre: 84% of COCONUT, 32% of those answers) — they are not natural-product-like, consistent with a test set of analogs and synthetics, so filtering toward nature removes exactly what coverage is for. *Gating between the pools per molecule* would be worth a lot (a perfect choice scores 0.2187 vs tiering's 0.1785), but there is no signal and it points the wrong way: the curated pool looks *more* convincing when it does **not** hold the answer (margin +0.13 vs −0.06), giving 42.8% accuracy against a 59.4% base rate. At Tanimoto ~0.37 the predicted fingerprint cannot make the true structure stand out from its isomers, which is the same weakness that collapses precision when the pool grows. **The ranker cannot self-diagnose, so remaining headroom is in the spectrum-to-structure model, not in pool logic.** - [x] Re-compared model configs in the Class 2 simulation (earlier comparisons used the full pool, which holds every answer by construction and so was blind to coverage). Ensembling does not help — the large transformer alone (0.1605/0.1785) beats every ensemble of it, including the deployed pair (0.1588/0.1768) — but by ~1%, inside noise at n=150. Kernel v5 therefore keeps the deployed ensemble and changes only the pool, one variable at a time. - [ ] Not yet retried after the session that launched it was killed: a larger transformer (d=384, 6 layers, 8 epochs, ~50 min). - [ ] Class 3 de novo exploration - [ ] Final ensemble + validation ## Design notes / known limitations - **Orthogonal fingerprints beat a better fingerprint.** Predicting the same 2048-bit Morgan r2 target more accurately is a dead end — eleven checkpoints spanning 62M to 4.6M params sit within 5% of each other, and across them +0.01 Tanimoto buys only +0.0031 MRR. But predicting *other* descriptor families alongside it does help: a transformer trained on Morgan r2 + Morgan r3 + MACCS (3239 bits) was worth +0.005 on the leaderboard (0.240 → 0.245). Scoring is a sum over bits, so independent fingerprints extend the sum; two predictions at Tanimoto 0.3 that make *different* mistakes constrain a candidate more than one at 0.35. How they are combined matters more than that they are. Scoring on the new model's 3239 bits alone **loses**, and so does naively summing the block log-likelihoods: MACCS is dense (28% of bits set) and by far the best-predicted block (Tanimoto 0.72 against 0.37 for Morgan r2), so its spread swamps the Morgan signal. Each block has to be standardized within the mass window first, with the orthogonal ones down-weighted to ~0.3 (`multi_fingerprint.block_weighted_loglik`); the result is flat over weights 0.2–0.45 and significant on two disjoint validation sets. - **Coverage was not the bottleneck; ranking is.** Phase 4 was built on the estimate that only ~46% of test molecules had their answer in the candidate pool, inferred from scoring ~0.50 locally at 100% coverage against 0.23 on the leaderboard. That ratio conflates two things, and the leaderboard says it is the other one. Two independent submissions each imply a reachable share near 1.0: | change | proxy deltas (reachable / unreachable) | observed | implied share | |---|---|---|---| | add the PubChem pool | +0.0000 / +0.0313 | +0.0020 | 0.94 | | cap tier 1 at 5 | −0.0023 / +0.0113 | −0.0030 | 1.05 | One parameter explains both results: essentially every real test molecule is already in the curated pool, so a 94M-structure pool adds almost nothing (+0.002) and taking slots away from tier 1 to give it room only costs precision (−0.003). The local/leaderboard gap is difficulty, not coverage — the real spectra are harder to rank, not absent from the pool. - **A validation proxy has to preserve production *density*, not just semantics.** The Class 2 simulation made answers unreachable by shrinking the curated pool to COCONUT-only (479,717 of 729,387). That is honest about coverage and wrong about density: tier 1 then filled a median of 10 of the 25 guess slots, where on the real test set it fills a median of **25**. Since tier 2 only takes slots tier 1 leaves, the simulation handed it early ranks it never gets in production — and predicted +15% for a change that delivered +0.002 (0.230 → 0.232). It also reported the tier-1 cap as flat, because at a fill of 10 the cap barely bound. The corrected design (`scripts/eval_realistic_tiering.py`) removes from the pool and library *only* the structures being scored, keeping everything else, and splits them so half are reachable and half are not. - **Read each array out of an `.npz` exactly once.** Indexing an `NpzFile` re-reads and re-allocates the entire array on every access, and any view kept from it pins that whole copy alive. So `{k: d["fp_table"][i] for i, k in enumerate(keys)}` allocates `len(keys) x sizeof(fp_table)` and gets the process OOM-killed — this cost two debugging cycles in `scripts/eval_reranker_full_window.py` and `scripts/mine_decoys.py`. Assign the array to a local first and store indices, not views. - **`np.isin` has no hash path for object dtype.** On arrays of Python strings (which is what the InChIKey14 columns are) it falls back to an O(n*m) loop: 34 minutes for 729k against 480k, where `pd.Index(a).isin(b)` returns the identical result in ~1 second. Worse, `np.isin(arr, some_set)` wraps the set in a 0-d object array and silently matches nothing. - Library dedup keeps one representative spectrum per (structure, adduct) — picking the one with the most peaks. (Phase 2's `pipeline_v2.merge_spectra` does merge a *query* molecule's multiple collision-energy spectra before anchor search; the *library* side is still one spectrum per structure/adduct.) - The frequency-based fallback (`baseline._global_fallback_candidates`) only fires when a spectrum gets zero similarity hits; `predict_test_set` prints a warning with the count when it happens, worth checking each run. - `PEAK_TOP_K`, `BIN_WIDTH`, `COARSE_TOP_K`, `ANALOG`-related bonus constants in `src/baseline.py` are initial guesses, not tuned — a natural next step once a real local CV split (not just the np-examples holdout) is built.