# Cross-Lingual Concept Distance

This repository implements the pipeline for measuring calibrated cross-lingual concept distance in Wikipedia lead sections.

Core metric:

```text
excess_distance(concept, lang_i, lang_j)
  = raw_cosine_distance(concept, lang_i, lang_j)
  - mean_raw_cosine_distance(calibration_domain, lang_i, lang_j)
```

Raw LaBSE distance alone must not be interpreted as cultural divergence.

## Setup

```bash
pip install -r requirements.txt
```

Scripts read from `data/` and write to `outputs/` (both relative to the repo root; `outputs/` is generated and gitignored). Most scripts accept `--phase0` to run against the small pilot sample instead of the full Phase 1 sample.

## Core pipeline

Data acquisition and validation (network calls; safe on a login node):

```bash
python scripts/resolve_qids.py
python scripts/fetch_sitelinks.py
python scripts/fetch_extracts.py
python scripts/validate_alignment.py
```

Embeddings and distances (GPU; submit `slurm_phase0_embeddings.sh` / `slurm_phase1_embeddings.sh`, do not run directly on a login node):

```bash
python scripts/compute_embeddings.py
python scripts/compute_distance_matrices.py
python scripts/calibration_baseline.py
```

Results and figures:

```bash
python scripts/visualize.py
python scripts/significance_tests.py
python scripts/bootstrap_uncertainty.py
python scripts/make_paper_tables.py
```

## Robustness checks

Each of these is an independent validation of the main-pipeline result and can be run after the core pipeline completes:

```bash
python scripts/bot_check.py                  # bot/ContentTranslation article audit
python scripts/calibration_sensitivity.py     # calibration-subdomain leave-one-out
python scripts/encoder_replication.py         # re-run with a second/third multilingual encoder
python scripts/flores_parallel_floor.py       # FLORES-200 parallel-text floor vs. Wikipedia floor
```

`encoder_replication.py` and `flores_parallel_floor.py` need a GPU; submit `slurm_encoder_replication.sh` / `slurm_flores_floor.sh`. `slurm_robustness.sh` runs the CPU-only checks together.

## Diagnostics and experiments (not part of the reported pipeline)

- `diag_token_lengths.py`: tokenization/truncation diagnostic for the embedding step.

## Slurm

Heavy embedding, encoder-replication, or FLORES-floor runs must be submitted through Slurm, not run directly on a shared login node. See `scripts/slurm_*.sh` for the job scripts used on this project's cluster (adjust `--account` and the `cd`/`HF_HOME` paths for your own environment).

## Outputs

- `outputs/raw_extracts/`
- `outputs/embeddings/`
- `outputs/distance_matrices/`
- `outputs/figures/`
- `outputs/reports/`
- `outputs/replication/` (encoder-replication results)

`make_paper_tables.py` writes generated tables to `../overleaf/tables/`; do not hand-edit those files.