# StockInfo developer guide

How to run StockInfo from a checkout, change it, test it and publish it.
Installing and running the finished container is described in the
[Docker guide](../docker/README.md); what StockInfo does is in the
[README](../README.md).

## Contents

- [Requirements](#requirements)
- [Set up and start](#set-up-and-start)
- [Configuration (.env)](#configuration-env)
- [Dashboard development](#dashboard-development)
- [REST API](#rest-api)
- [Docker image](#docker-image)
- [Tests](#tests)
- [Releases and changelog](#releases-and-changelog)
- [Project layout](#project-layout)

## Requirements

- **Python 3.11+** (`make setup` creates the project `.venv`; override the
  interpreter with `PYTHON_BOOTSTRAP=python3.12 make setup`)
- **make** (drives setup and start/stop of the services)
- **overmind** and **tmux** for `make dev-up` (backend and dashboard together)
- optional **Docker** (to run the container)
- **Node.js 24+ with npm** (`make setup` installs the dashboard packages;
  the backend and prebuilt Docker image do not use Node.js on the host)
- **Deno 2** for `make docs-check`, only when ProjectTools is linked under
  `.libs/`; `make setup` does not install it. Without ProjectTools the check
  is skipped, so `make test`, `make check` and the pre-commit hook do not
  need Deno

**You do not need anything under `.libs/` to develop.** `make setup`, the
`dev-*` targets, `make test`, `make check` and `make up`/`down` work on a plain
clone. Targets marked **[Maintainer]** in `make help` (`build`, `push`,
`status`, `tag-*`, `version`) need the author's private BashLib and stop with a
note without it. To build the image yourself, run this instead of
`make build`:

```bash
docker build -f docker/Dockerfile -t mangolila/stockinfo:latest .
```

For the maintainer, `make setup` also links the shared libraries under
`.libs/`. The Docker image needs none of this.

[↑ Contents](#contents)

## Set up and start

```bash
# 1. Create the configuration
cp .env.example .env

# 2. Install dependencies (Python .venv, dashboard packages, Git hook)
make setup

# 3. Start backend :8000 and dashboard :5173 with auto-reload
make dev-up         # stop with make dev-down · logs with make dev-logs
```

`make setup` can be run again at any time. It keeps an existing `.venv` and
only brings packages up to date. A package pinned to a Git commit in
`requirements.txt` is reinstalled when the installed commit differs from the
pin, even if the package version stayed the same.

`make dev-up` runs both in the background via
[overmind](https://github.com/DarthSim/overmind) (needs overmind and tmux).

`make dev-down` also cleans up after a crash: it ends leftover overmind, tmux
and app processes of this project, removes a stale `.overmind.sock` and frees
ports 5173 and 8000. It only ends your own processes inside this project and
can be run as often as you like.

The server then runs at `http://localhost:8000`. Interactive API docs (Swagger UI,
dark theme) live at `http://localhost:8000/docs` (use `http://` in the browser, not
`https://`). The dashboard runs at `http://localhost:5173`.

First query:

```bash
curl http://localhost:8000/quote/IE00B3RBWM25
```

`make help` lists all available commands, `make hints` shows useful URLs.

Plugin data migrations run automatically before normal operation when an
active source increases its `data_version`. StockInfo backs up first and commits
each migration together with its new version. Failed or missing migrations
block business requests and refreshes; the server log identifies the plugin,
versions and cause. See [plugin data migrations](plugin-authors.md#plugin-data-migrations).

[↑ Contents](#contents)

## Configuration (.env)

All values can be overridden in `.env`; step 1 of the setup above creates it
from `.env.example`:

| Variable | Meaning | Default |
|---|---|---|
| `HOST` / `PORT` | server address | `0.0.0.0` / `8000` |
| `DATABASE_PATH` | path of the SQLite file | `data/stockinfo.db` |
| `CACHE_TTL_HOURS` | age at which a quote is re-fetched on request | `6` |
| `REFRESH_INTERVAL_HOURS` | interval of the background refresh | `6` |
| `METADATA_TTL_DAYS` | refresh cadence for ETF metadata | `7` |
| `DEFAULT_EXCHANGE` | preferred exchange for ISIN queries (MIC) | `XETR` (Xetra) |
| `STRICT_EXCHANGE` | require the preferred exchange when resolving an ISIN | `false` |
| `FX_TTL_HOURS` | cache lifetime for exchange rates | `1` |
| `OPENFIGI_API_KEY` | optional key for a higher OpenFIGI rate limit | empty |
| `EXTRAETF_ETF_URL` / `EXTRAETF_STOCK_URL` | profile link templates (placeholder `{isin}`) | extraetf.com/… |
| `YAHOO_URL` | Yahoo link template (placeholder `{symbol}`) | de.finance.yahoo.com/… |
| `CORS_ORIGINS` | web pages on another address that may call the API, as a JSON list | `["http://localhost:5173"]` |

The default bind address `0.0.0.0` makes a natively started StockInfo
reachable from your network. To use it only on your own computer, set
`HOST=127.0.0.1`. The [security model](../docker/README.md#security) explains
what anyone who can reach the port can do.

The settings above are read by `app/config.py`. `make docs-check` does not
see them, so a new setting has to be added here and to the
[Docker guide](../docker/README.md#configuration) by hand.

[↑ Contents](#contents)

## Dashboard development

The web frontend lives in [`dashboard/`](../dashboard/) (Vue 3 + Vite +
TypeScript + SCSS). `make dev-up` starts it together with the backend. To run
it on its own:

```bash
cd dashboard
npm install
npm run dev                # http://localhost:5173
```

The Vite dev proxy forwards API requests to the backend (`http://localhost:8000`)
automatically — `VITE_API_BASE_URL` is not needed in dev (optionally overridable,
see `dashboard/vite.config.ts`). The backend must run in parallel.

[↑ Contents](#contents)

## REST API

![Swagger UI (dark)](../unraid/screenshots/swagger.png)

| Method & path | Purpose |
|---|---|
| `GET /health` | liveness — answers as long as the process is alive |
| `GET /ready` | readiness — is normal operation released? `503` with `status` = `degraded` (database unreachable **or** startup failed, told apart by `database`), `migration_pending` or `starting` |
| `GET /operational` | can the process do its current job? The Docker healthcheck hangs on this one. `200` while a migration is pending or the service is starting, `503`/`degraded` only when something is broken |
| `GET /migration` | what a confirmed migration would do — without changing anything |
| `POST /migration/confirm` | run it, once, on explicit confirmation |
| `GET /migration/report` | what actually happened, still available long after |
| `GET /quote/{isin}` | quote by ISIN (prefers Xetra/EUR) |
| `GET /quote?symbol=VGWL.DE` | quote by full Yahoo symbol (suffix = exchange) |
| `GET /quote/{isin}/history` | intraday history (collected ticks) |
| `GET /quote/{isin}/daily?period=1w\|1m\|3m\|1y\|max` | real end-of-day closes (EOD, cached) |
| `GET /fx?base=EUR&quote=USD` | exchange rate: 1 `base` = rate `quote`; `quote_time` is the time the source states, or the fetch time if it states none |
| `GET /instruments` | all cached instruments with their latest quote |
| `POST /instruments/intake` | resolve an identifier and add the instrument; optionally confirm a different exchange before saving |
| `GET /fields` | core and plugin field descriptions, detail schema and schema versions |
| `GET /instrument-types` | declared asset types from the running plugin configuration, including completeness and source status |
| `GET /sources` | configured source chains and local source diagnostics |
| `GET /env` | current configuration (secrets masked) |
| `POST /refresh` · `POST /refresh/{isin}` | refresh all / a single instrument |
| `PUT /instruments/by-symbol/{symbol}/isin` | add an ISIN after the fact |
| `PATCH /instruments/by-id/{listing_id}/details` | set or remove manual detail values |
| `DELETE /instruments/{isin}` | delete an instrument including its history |
| `GET /backups` · `POST /backups` · `POST /backups/{name}/restore` | list backups / create one / schedule a restore (`202`; it takes effect on the next restart) |

For instruments **without an ISIN**, use `GET /quote?symbol=…` for a quote.
History, daily, refresh and delete have `…/by-symbol/{symbol}` variants.
An ambiguous symbol returns `409` with the candidate listings.

For dynamic type filters, read `GET /instrument-types` instead of collecting
types from stored instruments. New plugin types appear without a fixed client
enumeration. Known types remain listed during source outages; `complete: false`
indicates missing or invalid declarations. See the
[type catalog contract](rest-core-contract.md#asset-typen-aus-der-plugin-konfiguration).

**Example response excerpt** (`GET /quote/IE00B3RBWM25`; optional fields omitted):

```json
{
  "identity": {
    "kind": "listed",
    "ticker": "VGWL",
    "mic": "XETR",
    "isin": "IE00B3RBWM25"
  },
  "symbol": "VGWL.DE",
  "exchange": "Xetra",
  "name": "Vanguard FTSE All-World UCITS ETF",
  "type": "etf",
  "currency": "EUR",
  "price": 160.98,
  "quote_time": "2026-07-10T15:35:46+00:00",
  "volume": 14403,
  "ter": 0.19,
  "provider": "Vanguard",
  "replication": "Physical(Optimized sampling)",
  "volatility": 9.95,
  "accumulating": false,
  "source": "yfinance+justetf",
  "cached": false,
  "stale": false,
  "fetched_at": "2026-07-12T18:16:28+00:00"
}
```

Read `identity.kind` before accessing identity fields: listed instruments carry
`ticker` and `mic`, pairs carry `base` and `quote_currency`, and ISIN-only
instruments carry `isin`. Quote and instrument responses also contain a
`details` map with values, units, currencies and provenance. Instruments have
a persistent `listing_id`. See the [REST reference](rest-core-contract.md)
and [detail fields](plugin-authors.md#open-detail-fields).

`volatility` is the 1-year volatility in percent — from justETF for ETFs, otherwise
computed (annualized) from the cached EOD closes. `accumulating` states whether an
ETF accumulates (`true`) or distributes (`false`).

Fields that cannot be determined are `null` (e.g. `ter` for individual stocks). If a
live fetch fails but an old value exists in the cache, that value is returned with
`"stale": true` instead of an error. Unknown ISIN → `404`.

**ETFs outside Europe.** justETF is a database of European UCITS funds, so a
US or Canadian ETF finds nothing there. For those, Yahoo supplies the fund
provider and deliberately nothing else: it reports the expense ratio in two
fields with two different units (`0.03` vs `0.0003` for the same fund) and the
fund size as an absolute amount, while StockInfo keeps fund sizes in millions.
A wrong number would be worse than none, because a value from a source hides
one entered by hand. TER, fund size and domicile therefore stay empty for
those funds until someone enters them.

The exchange suffix selects the exchange; the currency comes from the quote.
The current Yahoo plugin rejects pence codes such as `GBp` instead of
converting them to `GBP`. Yahoo daily prices are adjusted for splits and
dividends. Both behaviors differ from the REST artifact; see the
[known contract differences](rest-core-contract.md#die-begriffe-die-sich-sonst-niemand-erschließt).

[↑ Contents](#contents)

## Docker image

Backend **and** dashboard run in a single image on one port. Built with
`docker/build.sh` (ecosystem convention, versioned via `gitDockerTag`):

```bash
make build     # build the image (docker/build.sh --build)
make up        # start the container → http://localhost:8000/
make down      # stop & remove
make docker-logs   # follow logs
```

On ARM Macs, the dashboard build runs natively; the final image still uses the
platform selected by `PLATFORM` (`linux/amd64` by default for Unraid;
`PLATFORM=arm` selects `linux/arm64`). The checked build publishes one platform
at a time.

FastAPI serves the dashboard itself (relative API calls) — no separate web server
required. `make up` keeps the data in the `stockinfo-data` volume (`/data` inside
the container); the [Docker guide](../docker/README.md#storage-and-permissions)
explains storage and permissions.

To select the file-only source profile **before the first start**, build the
image and write the profile to the same named volume that `make up` mounts:

```bash
make build
./scripts/sources-profile.sh --yaml --target docker
make up
```

Use `--online --target docker` to restore the online chains with a YAML
fallback. The script copies the matching example asset file only if it is
missing, backs up an existing `sources.yaml`, and leaves the application
container untouched. If the container is already running, restart it after
switching; the source chains are loaded at startup. The named-volume route
needs Docker access and a locally available `mangolila/stockinfo:latest`
image. For a different `make up DATA_VOLUME=other-volume`, pass
`--volume other-volume` to the script as well. No data is moved between
named volumes and host directories.

If `make up` uses a different `IMAGE_NAME`, set `STOCKINFO_IMAGE` to that
image with the `:latest` tag when running the script.

**Push to a registry:**

```bash
make build                   # Docker Hub image, checked locally
make push                    # push that exact build to Docker Hub
TARGET=ghcr make build       # alternatively build for GitHub Container Registry
TARGET=ghcr make push        # push to the same target used for the build
```

`make build` copies the [EUPL license](../LICENSE), its [German text](../LICENSE.de.txt),
[licensing and provider declaration](../LICENSING.md), the
[third-party notices](../THIRD_PARTY_NOTICES.md), the
[`plugin_api` MIT license](../plugin_api/LICENSE), and the example plugin's MIT
license into the image. It checks the files, their SHA-256 hashes, the target
architecture, and the OCI license and source labels before recording a build as
ready to push. No manual license-copy step is needed. A failed build clears the
previous push marker; `make push` also rejects an image from a different source
commit or registry target.

The versioned image tag contains the source commit hash. Publish that commit
before publishing the image so recipients can obtain the corresponding source.

**Docker Hub description.** After a successful Docker Hub image push,
`make push` uploads [`docker/README.md`](../docker/README.md), the
container-specific description, with absolute GitHub links for documents and
images. Other registries skip this step. Install Python 3.11+ and
[Pandoc](https://pandoc.org/installing.html). The Bash entry point installs its
dependencies in its own user-cache environment, leaving the project `.venv`
untouched. Links use the published `master` branch.

Set `DOCKER_PW_FILE` to a Docker Hub token file outside the repository
(default: `${DOCKER_CONFIG:-$HOME/.docker}/dockerhub.sec`). The token needs
Read, Write, Delete permissions. The script checks README and token-file
readability first; only Docker Hub receives the credentials.

Preview without Docker Hub credentials (initial setup may download packages):

```bash
./.libs/ProjectTools/src/bash/dockerhub-readme.sh --preview
# Output: docker/preview/README.md
```

The preview is versioned and belongs in the same commit as the change to
`docker/README.md`; otherwise `make docs-check` reports a problem.

If the README upload fails, `make push` fails too, but the image is already
published. Retry only the description with:

```bash
./.libs/ProjectTools/src/bash/dockerhub-readme.sh --publish
```

Defaults: repository from project settings, branch `master`. Override with
`--repository`, `--ref`, or `--output`. Use `--username` for an organization
login and `--description` to change the short summary. The script verifies
the saved overview and enforces the 25,000-byte limit without truncation. Run
without arguments for help.

[↑ Contents](#contents)

## Tests

```bash
make test                       # documentation, backend, plugin API, example plugin and dashboard
make test-backend               # backend only (pytest)
make test-dashboard             # dashboard only (ESLint and Vitest)
make docs-check                 # documentation only: links, commands, variables, dependent files
make check                      # everything that works offline, plus Ruff and type checks
```

`make docs-check` checks the documentation against the project; its rules
are in `.docs-check.conf.sh`. It needs ProjectTools under `.libs/` and Deno 2;
without ProjectTools, the target prints a note and the other tests still run.

`make setup` also installs a Git pre-commit hook that runs the check on the
staged files and stops a commit when it reports problems.

`README.md` and `README.de.md` must keep the same structure (`TRANSLATIONS` in
`.docs-check.conf.sh`): the same headings and, per section, the same links,
images, code, alerts and number of paragraphs. Only the text and Mermaid
labels are translated. Any difference is a problem that stops the pre-commit
hook and cannot be acknowledged; change both versions in the same commit.

The normal backend run includes tests marked `integration` that call real
external APIs or the package index. `make check` leaves them out and adds
`ruff` and the dashboard type check (`vue-tsc`). To exclude only the online
tests:

```bash
.venv/bin/pytest -m "not integration"
```

Backend tests use temporary databases. The test setup blocks connections to
the working database; additional test databases belong under `tmp_path`.

[↑ Contents](#contents)

## Releases and changelog

`make changelog` regenerates [CHANGELOG.md](../CHANGELOG.md) locally from release
tags and Conventional Commits. It does not commit or push. The generated file
is replaced on the next run; its content comes from commit and release tag
messages. To select another Python interpreter (Python 3.9+):

```bash
make changelog PYTHON=/path/to/python3
```

`make tag-major`, `make tag-minor` and `make tag-patch` bump the version,
commit, tag and push it, then generate, commit and push the changelog. Pass
`MSG="Release summary"` for the release tag message. A release needs a clean
working tree and the shared BashLib and ProjectTools libraries.

If the changelog step fails after the tag has been pushed, fix the cause and
retry without bumping the version again:

```bash
python3 .libs/ProjectTools/src/python/changelog.py --publish
```

[↑ Contents](#contents)

## Project layout

```
app/                    # FastAPI backend
  main.py               #   app setup, routes, scheduler startup
  config.py             #   configuration (.env)
  persistence/          #   SQLite via SQLModel: schema, migrations, data access (only here)
  resolver.py           #   ISIN → symbol/exchange (OpenFIGI + Yahoo)
  providers/            #   data sources: yfinance, justETF, OpenFIGI
  plugins/              #   bundled sources implementing the plugin API
  services/             #   quote fetching + cache/TTL logic
  scheduler.py          #   periodic background refresh
  routers/              #   HTTP endpoints (quotes, dashboard)
  docs.py               #   dark-themed Swagger UI (/docs)
dashboard/              # Vue dashboard (standalone app)
plugin_api/             # independent plugin contract, examples and role tests
contract/               # REST contract and OpenAPI snapshot
examples/               # source profiles and manually maintained asset examples
tests/                  # backend tests (pytest)
docker/                 # Dockerfile, build.sh (single-image build)
unraid/                 # Unraid guide and screenshots (README + CA template)
Makefile                # service start/stop (make help)
CHANGELOG.md            # generated history of tagged releases
```

References: [REST API](rest-core-contract.md),
[source configuration](plugins.md), [plugin development](plugin-authors.md)
and [contract checks](../contract/README.md). Historical designs and plans live
under [`superpowers/`](superpowers/).

[↑ Contents](#contents)