# document-extraction

Structured extraction of data from bankruptcy (PACER) PDF documents. First real
service under `bankruptcy-data/`. This is the **Python port of the Laravel Intel
layer** (`web/Laravel13/app/Models/TRBK/Intel`); it preserves that system's
behavior so extractions are interchangeable:

- **Default provider:** Grok 4.3 (xAI Responses API). Anthropic / OpenAI are
  alternate arms (`INTEL_AI_PROVIDER`).
- **Storage:** MongoDB `BankruptcyIntel.<collection>` (one collection per doc
  type) with full provenance and `is_canonical` promotion per case.
- **Field format:** every extracted field is `{ "value": string, "source_page": int }`.
- **Wasabi-only rule:** a docket is extractable only when `vw_docket.amazonId` is set.

See the strategy doc: `web/Laravel13/docs/bankruptcy/gen-ai-mcp-expansion.md`
("The PDF Document Corpus"), the field spec `extraction-data-dictionary.md`, and
the port reference `intel-document-extraction-status.md`. Read
`../docs/database-access-patterns.md` before touching the data layer.

## Conda environment

This service has its own conda env, **`bko-document-extraction`** (Python 3.11),
per the `bko-<service>` precedent.

```bash
conda env create -f environment.yml
conda activate bko-document-extraction
pip install -e .            # core deps
# pip install -e '.[parsing]'   # adds Docling (heavy ML) for tabular Schedules
# pip install -e '.[dev]'       # pytest + ruff
```

## Layout

```
document-extraction/
├── environment.yml                 # conda env (python + pip only)
├── pyproject.toml                  # deps + console script (doc-extraction)
├── .env.example                    # DATABASE_URL, ANTHROPIC_API_KEY, WASABI_*
├── src/doc_extraction/
│   ├── config.py                   # env-driven settings + SCHEMA_VERSION
│   ├── db.py                       # session/engine (copied from shared/db.py)
│   ├── models.py                   # SQLAlchemy ORM (Vw views + owned tables)
│   ├── schemas.py                  # Pydantic contracts (DocumentType, ...)
│   ├── cli.py                      # entrypoint: `doc-extraction extract|backfill`
│   ├── documents/                  # fetch from Wasabi + classify by doc type
│   ├── parsing/                    # Docling pre-parse (tabular) + vision fallback
│   ├── extraction/                 # per-doc-type Claude structured extraction
│   └── persistence/                # write system-of-record + projection tables
└── tests/
```

## Architecture notes

- **Schema ownership:** Laravel owns all DDL/migrations for the shared database.
  The owned tables in `models.py` are this service's *contract* against tables
  Laravel creates — they are marked DRAFT until finalized with those migrations.
  This service never runs migrations.
- **Storage:** each extraction stores provenance + raw + normalized JSON
  (system-of-record), from which queryable per-doc-type projection tables are
  derived. Models served to MCP/REST come from Laravel reading those tables.
- **Models:** Sonnet for judgment docs, Haiku for predictable forms (proof of
  claim), Opus reserved for measured failures. Batch API for proactive bulk,
  streaming for on-demand.
- **Parsing:** v1 uses native Claude PDF. Docling (the `parsing` extra) is added
  for tabular Schedules (A/B, D/E/F), with a native-vision fallback for scans.

## Running

The **spine is implemented end-to-end for Disclosure Statements**:

```bash
# resolve docketId -> amazonId (Wasabi) -> fetch PDF -> Grok extraction -> Mongo
doc-extraction extract <docketId>                 # auto-detects disclosure_statement / combined
doc-extraction extract <docketId> --provider grok
doc-extraction extract <docketId> --doc-type disclosure_statement
```

Pipeline stages (mirroring the Laravel Intel concerns):

| Stage | Module | Ports from (Laravel) |
|---|---|---|
| resolve docket + Wasabi-only rule | `documents/resolver.py` | `ResolvesAmazonIdFromDocket` |
| fetch PDF bytes | `documents/wasabi.py` | `Wasabi::get_Object` (`trbk-docs`, `w-` key fallbacks) |
| provider dispatch + Grok call | `extraction/providers/` | `IntelPdfExtraction`, `IntelGrokResponses` |
| per-doc prompts + JSON schema | `extraction/documents/` | `Documents/*.php` (DisclosureStatement done) |
| Mongo persist + canonical promotion | `persistence/repository.py` | `IntelPersistsMongoExtraction` |

**Not yet ported:** the other 13 document classes, the Anthropic page-slicing /
context-retry loop (`IntelPdfExtraction::extractWithAnthropic`), Docling parsing
for tabular Schedules, and the `backfill` command. Like other backend jobs, this
will be driven by the existing Laravel job-table + cron runner.