# Local usage analytics triage (PROD-11059)

For the architecture and trust boundaries, see [architecture](architecture.md).

Managed Analytics is enabled by default with validated commercial licensing and
configured usage storage. It creates a `DEFAULT` project with a backend-owned `analytics` provisioning marker
and an invisible DuckDB `analytics` connection.
No MotherDuck account, dbt project, or configurable connector UI is required.

## Configure and provision

Start an isolated development instance using the worktree runbook, with a valid
commercial license configured through `LIGHTDASH_LICENSE_KEY`. Configure the
usage-events writer's S3-compatible storage settings: `USAGE_EVENTS_S3_ENDPOINT`,
`USAGE_EVENTS_S3_BUCKET`, `USAGE_EVENTS_S3_ACCESS_KEY`,
`USAGE_EVENTS_S3_SECRET_KEY`, and `USAGE_EVENTS_S3_REGION`. The reader reuses this
server-owned configuration to issue signed GET URLs; it requires no CLI login,
OAuth exchange, or new cloud infrastructure. The usage-events endpoint defaults
to `S3_ENDPOINT`; override it when analytics uses GCS and local storage uses MinIO.

Legacy `analytics-project`
feature-flag overrides no longer control availability. Source data must belong
to the persisted project's org, even locally; legacy org/source-org overrides
are ignored. For production testing, follow [live testing](live-testing.md).

As a signed-in organization administrator, open **Organization settings →
Lightdash analytics** at `/generalSettings/lightdashAnalytics`. Click
**Create** to provision and open the project, or **Open** if it already exists.
The page shows the project's creation time, not its model-sync version or data
freshness. Deployment availability and org-admin backend restrictions still apply.

Alternatively, call the same-origin endpoint from the browser console on your
assigned local frontend:

```javascript
const response = await fetch('/api/v1/org/analytics-project', {
  method: 'POST',
});
const body = await response.json();
if (!response.ok) throw new Error(JSON.stringify(body));
window.location.assign(body.results.url);
```

The endpoint accepts no configuration: organization identity comes from the
authenticated session. After authorization, it verifies signed Parquet access
before creating any project, then compiles the backend-owned system explores in
memory and returns `{ projectUuid, url, created }`. Storage failures leave no new
project. The name produces `lightdash-analytics` unless that slug is taken;
normal collision handling chooses a unique suffix. An existing analytics project
is found by its marker, never by a name or slug.

Concurrent requests are serialized per organization using a database advisory
lock. Re-running refreshes the models and repairs a failed model-save attempt
without replacing the project or deleting saved charts. Managed Analytics
projects do not expire automatically. Project discovery and reads require
managed Analytics access plus normal project permissions; organization admins
retain access. See [access](access.md) for explicit non-admin grants.

### Status and reset

`GET /api/v1/org/analytics-project` returns `{ project: null }` when absent,
otherwise project UUID, name, slug, URL, and creation timestamp. This read does
not access object storage, compile models, or create anything.

The settings page's **Delete** action requires confirmation and calls
`DELETE /api/v1/org/analytics-project/{projectUuid}`. Only the current org's
analytics-marked project with that exact UUID can be deleted. Deletion uses the
same per-org lock as creation. A stale UUID cannot delete a replacement project.
This permanently removes the project and its saved content, but leaves the
collected usage events in object storage intact. Afterwards, **Create** provisions
a new project; it does not restore deleted charts or dashboards.

All three endpoints live in `AnalyticsProjectController`, backed by
`AnalyticsProjectService`, and require a session-authenticated org administrator
and enabled managed Analytics. Read/query access uses the permissions in
[managed analytics access](access.md).

## Read path and safeguards

The backend lists `events/compacted/org_id=<uuid>/stream=<name>/dt=<date>/*.parquet`
for `query_events`, `ai_usage`, `data_app_events` and `export_events` across all
retained dates, plus the exact current charts, dashboards and users snapshot keys
under the same org prefix. Filter dates in
Explore; legacy `LIGHTDASH_LOCAL_ANALYTICS_START_DATE` and
`LIGHTDASH_LOCAL_ANALYTICS_END_DATE` settings are no longer read.
Files and authentication are refreshed per session. DuckDB views bind exact
HTTPS URLs; Explore SQL uses table names. Reads use HTTP ranges, with no import
into a persistent database and no writes to the source bucket.

Each query gets an isolated in-memory instance, signed GET URLs, a 256 MB
memory limit and 32 threads for remote reads, unless explicitly overridden.
These reader settings differ from the single-threaded background exporter.
External access is restricted to the exact file manifest; disk spilling is
disabled and file/metadata caches are confined to the query's private instance. Existing
SELECT-only and file-function SQL validation remain enabled. Catalog queries that
could expose signed view SQL are blocked, and native errors are sanitized. Broad
writer keys never enter DuckDB; signed URLs are not persisted in project
credentials or returned through the API.

History, paginated results, streaming results and legacy result reads check the
deployment availability and managed analytics access. CSV and Excel downloads, including scheduled data exports, are supported when
export file storage is configured. Protected downloads recheck access; deliberately
shared signed delivery links can outlive permission changes until they expire.
See [downloads and scheduled deliveries](access.md#downloads-and-scheduled-deliveries).

## Verify dimension refreshes and joins

Use the [dimension smoke test](dimensions.md#local-verification) to exercise real
Postgres → Parquet → local MinIO → Explore-generated joins. It uses a disposable
schema and bucket, supports a million-row fixture, and checks overwrite behavior,
renames, missing lookups, tenant isolation and preservation of the previous file
after an interrupted export.

For manual testing, click **Sync content** or re-run
`POST /api/v1/org/analytics-project` to compile the lookup fields, then select
**Users → User name** and **Total queries** in Query Events. Sync updates models
and managed sample dashboards but does not refresh snapshots. Names update after
the next successful daily refresh.

To verify deployment before the first users snapshot exists, use an isolated
instance and a dedicated bucket:

1. Back up the organization's `dim=users/users.parquet` object, including its
   metadata, then remove that exact object while keeping event files available.
2. Click **Sync content** and re-run the provisioning endpoint for an existing
   project. Both should succeed. Run the joined query in Explore and verify
   `Unknown user` with unchanged event counts.
3. For an organization without an analytics project, keep a supported event file
   available and its users snapshot absent. Provisioning should create a project
   successfully. With no supported event files, it should instead return
   `No analytics data is available yet`.
4. Restore the original snapshot and verify a fresh query resolves the user name
   without recreating the project. Remove only the temporary project and fixtures
   created for the test. Do not delete shared MinIO prefixes.

## Before customer rollout

- Review effective IAM and shared-instance isolation. Dedicated read-only
  credentials are deferred hardening (PROD-11103). Signed URLs keep broad keys
  out of DuckDB but still trust the signer. Folder separation is not an IAM boundary.
- Review permissions and cached/downloaded-results surfaces against the current
  [managed analytics access rules](access.md), including signed delivery links.
- Production provisioning and final project lifecycle/role design.
- Additional entity-name lookups, domain metrics, schema evolution, empty
  streams, retention and query performance.
- Result-cache freshness, date-aware discovery and manifest/view caching at scale
  (PROD-11111). All retained dates are exposed, subject to fail-closed listing/file
  caps; large-history and concurrent query performance still need benchmarking.
- Validate background refresh database load and temporary disk capacity. The
  daily job now also overwrites current dimension snapshots; see
  [resource and retry limits](dimensions.md#export-and-failure-behavior).

Actual GCS reads and isolated MinIO tests verify the signed-URL path; AWS S3 and
production multi-tenant rollout remain unverified.
Production execution uses the same licensed default;
see [live testing](live-testing.md).

## Original provisioning stack

1. Internal Parquet connector and organization-bound source resolver.
2. Signed-URL bucket access using the existing writer's backend credentials.
3. Admin create-or-get endpoint, preview provisioning, signed-URL provider, and both
   system models (Query Events and AI Usage). Replaces the provisioning script.
4. Internal architecture and operational documentation (no runtime changes).

Dedicated read-only credentials remain a separate deferred follow-up (PROD-11103).

See [storage credential verification](credentials.md) for the
opt-in read-only cloud and isolated local security tests.