# McBopomofo Data Source

## Dictionary Data Structure

```
├── BPMFBase.txt          Single character Bopomofo mappings
├── BPMFMappings.txt      Multi-character phrases (2-6 chars)
│                         Originally simplified from tsi.src of libtabe
│                         (BSD Licensed) with modifications
├── BPMFPunctuations.txt  Punctuation marks
├── heterophony1.list     Heterophony reading priorities (1st order)
├── heterophony2.list     Heterophony reading priorities (2nd order)
├── heterophony3.list     Heterophony reading priorities (3rd order)
├── phrase.occ            Phrase frequency/occurrence data
├── exclusion.txt         Phrase frequency exclusions
├── Symbols.txt           Special symbols (era names, etc.)
├── Macros.txt            Text macros (date/time)
├── Makefile              Build automation
├── pyproject.toml        Python package configuration
├── textpool.rc           Corpus counting configuration
├── bpmfvs-pua.txt        Precompiled database bpmfvs PUA code points
├── bpmfvs-variants.txt   Precompiled database for bpmfvs variant selectors
│
├── curation/             Python package for data processing
│   │                     (exports PROJECT_ROOT, CONFIG_FILE for path resolution)
│   ├── builders/         frequency_builder, phrase_deriver
│   ├── compilers/        main_compiler, plain_bpmf_compiler, compiler_utils
│   ├── utils/            text_filter
│   ├── validators/       score_validator
│   ├── notebooks/        Jupyter analysis notebooks
│   └── requirements.txt  Package dependencies
│
├── scripts/              Standalone CLI tools (with side effects)
│   │                     (imports paths from curation package)
│   ├── count_occurrences.py  Count phrase occurrences in corpus
│   ├── analyze_data.py       Analyze dictionary data quality
│   └── map_bpmf.py           Primitive BPMF mapping helper
│
├── tests/                Unit tests for curation package
│
└── bin_legacy/           DEPRECATED (Oct 2024): Historical tools (2012-2025)
    ├── DEPRECATED.md     Complete migration guide & tool history
    ├── audit_encoding.swift  Still usable encoding validator
    ├── C_Version/        Fast C implementation (phased out 2013)
    ├── Sample_Prep/      Historical corpus preparation methods
    └── disabled/         Legacy Perl/Ruby/Bash implementations
```

## Using the Build System

Use Makefile targets (automatically uses new curation/ structure):

```
  make all      # Build all data files
  make sort     # Sort all data files
  make check    # Validate data integrity
  make clean    # Remove generated files
```

For detailed build pipeline, see `AGENTS.md` and `../algorithm.md`.

## Updating the bpmfvs Databases

The bpmfvs files are precompiled since we work under the assumption that they
are fairly stable. If you need to update them, assuming `$REPO_ROOT` is the
base directory in which you have checked out [bpmfvs](https:
//github.com/ButTaiwan/bpmfvs). Then:

```
python3 -m curation.compilers.update_bpmfvs_pua_db \
  --input $REPO_ROOT/bpmfvs/phonetic/phonic_types.txt
python3 -m curation.compilers.update_bpmfvs_variant_db \
  --input $REPO_ROOT/bpmfvs/phonetic/phonic_table_Z.txt
  --bpmf_base BPMFBase.txt
  --bpmf_mappings BPMFMappings.txt
```

This will update `bpmfvs-pua.txt` and `bpmfvs-variants.txt` respectively. After
that, make a pull request so that the maintainers can review the changes.

## Editorial Rule
* when in doubt, use google/yahoo search to confirm the rarity of the phrases
  for example search: "一瞻丰采" site:.tw
  - if the amount of the results is under 1000, it's likely okay to remove this
    phrase
  - if tsi.src variants showed up in the first page of the results, you should
    remove this phrase. Tsi.src variants include SEO sites, this phenomenom is
    quite amazing itself.
  - for idioms, it's possible the first hit is the idiom database, it's up to
    the editor to decide.
  - if this candidate is apparent part of a long phrase and the segmentation is
    apparently incorrect, find a way to remove the phrase candidate and replace
    it with the complete one, but forget about it if the length is longer than
    6.
* We need to constantly remind ourself, IME is not an idiom database.