---
name: onboard-city
description: Stands up a new Project Sidewalk city end to end — neighborhood data hunting, the street build and its QGIS QA loop, an imagery preflight across providers, `make onboard-city`, translations, config review, and the server handoff. The scripts do the deterministic work; this skill supplies the judgment calls around them.
---

# Onboard a city

Two commands and this skill. Read `docs/onboarding-a-city.md` once; it is the runbook this skill drives.

```
make build-city-data id=<city-id> args="..."   # tools/city/onboard_city.py → db/onboarding/<city-id>/ (QA gpkg, SQL, report, endpoints csv)
make check-imagery   id=<city-id> args="--sample --<provider>"   # preflight, one row per provider in preflight_report.md
make onboard-city    id=<city-id>              # tools/city/setup_new_city.py → configs, GA, schema, fill, scan, gradient, dump + handoff
```

Everything under `db/onboarding/` is git-ignored. The db container sees it at `/opt/onboarding/` **only from the main
checkout** (`db/` is the bind mount), so run these from the main checkout, not a worktree.

## 1. Intake — ask before building

- **City id.** Lowercase kebab-case ending in the state for US cities (`laurens-ia`), the country elsewhere
  (`bayonne-fr`). It becomes `SIDEWALK_CITY_ID`, the schema (`sidewalk_laurens_ia`), and the output dir.
- **Server name.** The id without its suffix (`sidewalk-laurens`), unless a clearly larger city shares the name
  (`sidewalk-newport-ky`).
- **Imagery provider.** `gsv` unless the partner says otherwise; `mapillary` / `panoramax` / `infra3d` need the
  matching credentials in the web container (Panoramax needs none). If unsure, the preflight in step 3 decides.
- **Neighborhood boundaries**, in this order of preference, and record where they came from (`--regions-source`, a
  URL or the collaborator's email — it lands in `region.data_source`):
  1. A partner- or city-supplied dataset (any OGR format/CRS; note its name column for `--region-name-col`).
  2. The city's open-data portal (ArcGIS Hub / Socrata / CKAN / data.gouv.fr …): search "neighborhoods",
     "quartiers", "barrios", "council districts". Prefer official, non-overlapping polygons that tile the city.
  3. Let the tool fall back: OSM neighbourhood polygons (used only above 75% coverage), then US census tracts
     (TIGERweb), then the whole city as one region (small towns; split later in QGIS if it grows).
- **A one-region city** takes the city's name (from `--place`), not the source's — otherwise a small town is the
  neighbourhood "Census Tract 7801" in every mission message and API response. Use `--single-region-name` with a
  `--boundary-file`, or to override. Names you chose (`--regions-file`, `--rename-regions`, a `--merge-regions`
  target) are left alone.
- **City boundary.** `--place "<City, State, Country>"` geocodes the OSM admin boundary; check the report's
  "Boundary:" line names the right place. A partner file goes in with `--boundary-file`.
- **Scope.** Whole city, or a phased launch opening some regions first (`onboard-city` asks; the imagery scan
  covers everything either way).

## 2. Build and QA the streets

```
make build-city-data id=laurens-ia args="--place 'Laurens, Iowa, USA'"
make build-city-data id=bayonne-fr args="--boundary-file /path/city.geojson --regions-file /path/quartiers.geojson \
    --region-name-col nom --regions-source 'https://…'"
```

Read `db/onboarding/<city-id>/report.md` before anything else and put the numbers in front of the maintainer:

- **Tiny segments (#4717):** the sub-20 m share should be at or below the production average (18%). Bayonne came in
  at 16%. Higher means the tier-1 merge is being defeated — check `--merge-tiny-m` and whether the source data is
  split oddly.
- **Regions:** `OVERSIZED` (> 60 km of streets) means split in QGIS; `SPARSE`/`EMPTY` means fold into a neighbour
  with `--merge-regions "A:B"` (names, not ids) and rerun. Seattle's regions carry 20–36 km each.
- **Coverage:** regions should cover ≥ 95% of the boundary; streets outside every region are trimmed.

The human QA gate: open `<city-id>_qa.gpkg` in QGIS (`qgis_road`, `qgis_region`, `city_boundary`,
`dropped_segments`, `rider_merges`) over a basemap. Fixes come back two ways: parameter changes rerun the build;
hand edits (delete a street, move a boundary) are re-exported with
`make build-city-data id=<city-id> args="--from-gpkg"`, which validates the layers and rewrites the SQL. Never
load a stale SQL over hand edits.

### Region names

Nothing else checks them: the build keeps the source's names and the fill stores them exactly as staged, and they
show up in the region picker, mission messages, and the API. Review them for every city once the region set is
settled (merges and boundary edits done), so the list you review is the one that ships.

1. **List them** with `repr`, which shows stray, doubled and non-breaking spaces a plain listing hides:
   ```
   docker exec projectsidewalk-web sh -c "cd /home && python3.13 -c \"import geopandas as gpd; \
   gpkg = 'db/onboarding/<city-id>/<city-id>_qa.gpkg'; \
   [print(r.region_id, repr(r.name)) for r in gpd.read_file(gpkg, layer='qgis_region').itertuples()]\""
   ```
2. **Look for** shape problems (ALL CAPS, all lowercase, leading/trailing/double spaces, non-breaking spaces or
   control characters, empty names); repeats, which the build numbers `"X (2)"`: find what actually tells them
   apart (#5252 found 1437 shared names across 12 prod cities); and damage from the source — CDMX's file had
   accents stripped or letters deleted (`SECCIN` for *Sección*, `CAADA` for *Cañada*) and plain typos.
3. **Propose fixes by the city's own conventions, not a rule.** What #4619 learned fixing 1806 prod names:
   - Spanish: `de`/`del` lowercase inside a name but capitalized opening one (`Del Valle`); `en`, `para`, `y`
     lowercase; articles lowercase only after `de`/`en` (`Santa Cruz de las Salinas`, but `Barrio Los Reyes`);
     ordinals lowercase (`1a Sección`, `2do Reacomodo`), but a block letter keeps its capital (`Picos Iztacalco 1B`).
   - Codes and acronyms stay as written: Rancagua's `UV 2` (Unidad Vecinal), `PSE&G`, `P.I.C.O.`, `LA-32`.
     Pronounceable ones follow local usage (`Infonavit`, `Pemex`).
   - Check spellings against an official list (the city's catalogue, the postal registry). Don't guess an accent
     or spelling you can't source: leave it and say so.
4. **Show the maintainer** a table of only the names that change (`region_id | current | proposed | why`), with the
   uncertain ones marked, and apply only what they approve.
5. **Apply** by writing the approved rows to `db/onboarding/<city-id>/region_renames.csv` (`current_name,new_name`;
   write it with Python's `csv` module so commas, quotes and padding survive — `current_name` must match exactly,
   spaces included). Then rerun the last build command with `--rename-regions` added, so the GeoPackage, the SQL
   and the full report all carry the new names; list them again to confirm. If the GeoPackage holds hand edits a
   fetch rerun would lose, use `make build-city-data id=<city-id> args="--from-gpkg --rename-regions"` instead:
   only the SQL and report change (the GeoPackage keeps the old names, and the report loses its fetch-only lines),
   so confirm from the `Renamed N region(s)` log line.

   From then on pass `--rename-regions` on **every** build of this city: rows already applied are skipped, and
   renames run before `--merge-regions`, so a merge added later uses the new names.

## 3. Imagery preflight — before any database work

```
make check-imagery id=<city-id> args="--sample --gsv"
make check-imagery id=<city-id> args="--sample --mapillary"     # run every plausible provider
```

Each run adds a row to `preflight_report.md`: coverage of a 150-street sample, failures, and how fresh the newest
captures are. Recommend the provider from that table; a `failed` count that is not zero means a key or quota
problem, not a coverage figure. Under ~70% coverage, say so plainly before continuing — the full scan will hide
that share of the city.

## 4. Run the setup

```
make onboard-city id=<city-id>
```

It shows the report and preflight, asks for the display name, country/state, provider, status (default private),
launch date (the Friday of next week), and URLs; registers the city in `conf/cityparams.conf`, `conf/messages`, and
the docs City IDs table; creates GA properties when `ga-service-account.json` is present; asks to add both hostnames to the **live production
Maps key** (skipped with a pointer when gcloud can't edit it); clones a donor schema
(the dev container's city by default — refused, with the schemas that disagree named, if its top evolution is another
branch's under the same number, i.e. its hash is neither the file's nor the other schemas'; pass `--donor` then);
boots the app once to apply any missing evolutions — and, right after a clone, to let Play verify every applied
hash; loads and fills; runs the full imagery scan for the chosen provider; samples the street gradient (step 8,
from the build's `street_structures.csv`, so it needs no nightly `osm_way` cache; a country with no registered
elevation model gets the `--dem-dir` recipe printed and the run goes on — `docs/street-gradient.md`); checks that
nothing but onboarding has written to the schema; dumps it to `db/<schema>-dump`; prints the handoff. Rerunning
skips finished steps. The script's flags go through `args=` (`make onboard-city id=<city-id> args="--skip-scan"`):

- `--dry-run` previews the file edits and drives no container (the one mode allowed from a checkout the
  containers do not mount — the script hashes the file each step uses against the container's copy, so a
  worktree or a second clone is refused).
- `--yes` takes every default without asking — the only way to run unattended; without it, a run with nothing
  on stdin stops at the first question that is a choice. Pair it with `--donor`, `--country`, `--pano-type`,
  `--tutorial-region` and `--regions` for the answers that have no default worth taking, and `--recreate` to
  drop an existing schema without being asked.
- **The boot gate (step 4).** The boot listens on its own port (`:9100`), so `npm start` and `make qa-worktree`
  stay up; what it needs is the main checkout's `target/`. If a build holds that, the step stops and names its
  pid. A build in a worktree is not in the way (the caches are shared by design). `--allow-running-apps` boots
  past an idle build in the main checkout; a boot an earlier run left behind is never overridable. The nightly
  actors are off for the boot, so it writes no job rows into the new schema.
- **The dump (step 9).** The dump leaves out the data of every table the clone, fill, scan and gradient do not write
  (`region_completion` too, which the app recomputes), so a local QA pass or a job run as the city (from
  `/clustering`, or an app left pointed at the schema) stays local and out of the dump; the step prints what it
  left out. The one value it changes is `street_edge_priority`, which a QA walk moves: it resets them to 1 on
  `y` (the default; `--yes` takes it). `--dump-only` reruns only this step, which is how a city QA'd after its
  first dump gets a clean one without passing "drop and recreate?"; it refuses a schema that is still unfilled.

Watch the fill's closing summary (streets, km, sub-20 m share, per-region km, center/zoom) against the report.

## 5. What the scripts leave to you

- **Translations.** The orchestrator prints the exact keys and the files that lack each. Every `messages.<lang>`
  gets a line: `zh-TW` transliterated; `es`/`nl`/`de`/`pt-BR`/`fr` with the exonym where one exists (`Nueva York`,
  `États-Unis`) and the English spelling otherwise — the line goes in even then, so a missing line always means
  "not looked at yet". English city/state/country names go in the base `messages` (proper nouns are
  language-neutral); the US state abbreviation goes in `messages.en`. `make lint-locales` must stay green.
- **`config` row review.** The clone carries the donor's `excluded_tags` (a European city may want a different tag
  set), `update_offset_hours` (Mikey's load-spreading spreadsheet assigns these), and `make_crops`; the fill prints
  all three and clears the donor's `mapathon_event_link` and official contact (#5462; a city that wants one sets it on
  `/admin/partners`). Ask; don't guess.
- **Optional flags** left unset on purpose: `private-profiles-by-default`, `global-leaderboard-excluded`,
  `ai-label-submission-enabled` (all false by default).
- **GA.** If step 2 was skipped, `python3 tools/city/create_ga_properties.py <city-id>` fills both id maps later; new
  properties go inside the existing "Project Sidewalk - Prod/Test" accounts, never new accounts.
- **Docs.** The City IDs row is added for you; `docs/dev-environment.md` is a convenience copy of `cityparams.conf`.

## 6. Hand it off

Follow the checklist the orchestrator prints: dump to the server (`scp` to `<netid>@makelab1.cs.washington.edu`,
renamed to `<schema>-empty-dump` at the destination), the IT tooling's `setup-new.pl`, Maps-key referrers (step 2
asks before adding them to the live production key; if gcloud can't edit it, `python3 tools/city/maps_key_referrers.py
<city-id>`), DNS, the pano scraper's manifest row (`<city-id>,<prod fqdn>` in `/etc/sidewalk/cities.csv` on the
scraper host), then the PR (configs + messages + docs). The checklist's step 7 says whether the street grades are in
the dump; when they are not (no registered elevation model, or `--skip-gradient`), the maintainer either samples
before the dump (for a hand-downloaded model, a rerun with `args="--dem-dir … --dem-name … --dem-resolution-m …"`)
or backfills the live city later. Point the maintainer at the QA items only a person can do:
open the landing page as the new city (map centered, neighborhood names right), walk one street in Explore on the
chosen imagery, check the Explore tag lists against `excluded_tags`. That walk leaves an `audit_task`, thousands of
interaction rows and a moved `audited_distance` in the schema, so **after local QA, dump again**: `make onboard-city
id=<city-id> args="--dump-only"` lists what the walk left, clears it on `y`, and writes the dump the server should get.