{ "cells": [ { "cell_type": "markdown", "id": "obs-intro", "metadata": {}, "source": [ "# Observability\n", "\n", "Pixeltable can export its internal telemetry as OpenTelemetry (OTel) spans, so you can watch\n", "operations in any OTel backend. This notebook wires the OTel bridge to **Grafana Cloud** and traces a\n", "single `insert()` that exercises the full span set: source preparation, lock acquisition, planning,\n", "external media downloads, UDF evaluation, media persistence, the store write path, and view propagation.\n", "\n", "The insert emits one nested span tree:\n", "\n", "```\n", "pixeltable.insert (operation span, root)\n", "├─ pixeltable.data_source.prepare (source resolution + schema validation)\n", "├─ pixeltable.catalog.begin_xact (connection + lock acquisition, one per attempt)\n", "├─ pixeltable.plan.create (insert plan construction, DEBUG level)\n", "├─ pixeltable.media.fetch (one per external file downloaded, DEBUG level)\n", "├─ pixeltable.row (one per inserted row, DEBUG level)\n", "│ └─ pixeltable.udf. (each UDF call, nested under its row)\n", "├─ pixeltable.media.save (one per generated media file persisted, DEBUG level)\n", "├─ pixeltable.store.build_rows (row -> store-row conversion, DEBUG level)\n", "├─ pixeltable.sa.insert_rows (the SQL INSERT)\n", "└─ pixeltable.view_load (one per mutable view, containing its own row/store spans)\n", "```\n", "\n", "Traces land in Grafana Cloud Traces (Tempo)." ] }, { "cell_type": "code", "execution_count": null, "id": "obs-pip", "metadata": {}, "outputs": [], "source": [ "%pip install -qU 'pixeltable[otel]'" ] }, { "cell_type": "markdown", "id": "obs-init-md", "metadata": {}, "source": [ "## Turn on the OpenTelemetry bridge\n", "\n", "Telemetry is opt-in: call `pxt_otel.init()` once per process, before the first table operation. It\n", "configures itself from the database's `[[pixeltable.database]]` entry in `pixeltable.toml`, from environment\n", "variables, or from the `[otel]` section of `~/.pixeltable/config.toml`, in that order of precedence.\n", "\n", "**Option 1: the database entry in `pixeltable.toml`.** Put the endpoint on the `[[pixeltable.database]]` entry of the database (the entry with no `name` is the local database; see [Configuration](/platform/configuration#per-database-settings)). The headers are a secret and stay out of the file: keep `OTEL_EXPORTER_OTLP_HEADERS` in the environment, and on a hosted database set it with `pxt secret set pxt://org:db OTEL_EXPORTER_OTLP_HEADERS=...`, then `pxt db restart` so the running database sees it.\n", "\n", "```toml\n", "[[pixeltable.database]]\n", "db_exporter_otlp_endpoint = 'https://otlp-gateway-.grafana.net/otlp'\n", "```\n", "\n", "**Option 2: environment variables.** Set these **before starting the Jupyter kernel** (the kernel\n", "inherits them):\n", "\n", "| Environment variable | Value |\n", "| --- | --- |\n", "| `OTEL_EXPORTER_OTLP_ENDPOINT` | Your OTLP endpoint, e.g. `https://otlp-gateway-.grafana.net/otlp` for Grafana Cloud |\n", "| `OTEL_EXPORTER_OTLP_HEADERS` | `Authorization=Basic `, where `` is the base64 encoding of `:` |\n", "\n", "For Grafana Cloud, the OpenTelemetry connection page (Connections -> Add new connection ->\n", "OpenTelemetry) shows your ``, ``, and lets you create a ``.\n", "Then run these in the terminal, substituting your values, and start Jupyter from that same terminal:\n", "\n", "```bash\n", "export OTEL_EXPORTER_OTLP_ENDPOINT='https://otlp-gateway-.grafana.net/otlp'\n", "export OTEL_EXPORTER_OTLP_HEADERS=\"Authorization=Basic $(echo -n ':' | base64 | tr -d '\\n')\"\n", "```\n", "\n", "**Option 3: `~/.pixeltable/config.toml`**, shared by every database on the installation:\n", "\n", "```toml\n", "[otel]\n", "exporter_otlp_endpoint = 'https://otlp-gateway-.grafana.net/otlp'\n", "```\n", "\n", "`init()` runs **once per process**. If it ran without an endpoint earlier in this kernel it can't be\n", "reconfigured, so fix the configuration, restart the kernel, and run this cell first." ] }, { "cell_type": "code", "execution_count": null, "id": "obs-init", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "OTel bridge active\n" ] } ], "source": [ "import opentelemetry.instrumentation.pixeltable as pxt_otel\n", "from opentelemetry import trace\n", "from opentelemetry.sdk.trace import TracerProvider\n", "\n", "pxt_otel.init(\n", " span_level='debug'\n", ") # 'debug' emits per-row + per-UDF spans (default 'info' = operation span only)\n", "\n", "\n", "# fail loud instead of silently dropping every span: a no-op ProxyTracerProvider means the SDK never came up\n", "assert isinstance(trace.get_tracer_provider(), TracerProvider), (\n", " 'OpenTelemetry is INERT: no OTLP endpoint was configured. '\n", " 'Set the env vars (or config.toml) above, restart the kernel, and run this cell first.'\n", ")\n", "print('OTel bridge active')" ] }, { "cell_type": "markdown", "id": "obs-table-md", "metadata": {}, "source": [ "## A pipeline that exercises every span type\n", "\n", "One table with an image column and two computed columns, plus a view:\n", "\n", "- inserting external image URLs triggers `pixeltable.media.fetch` (one download span per file, on worker threads)\n", "- the `add_one` UDF produces a `pixeltable.udf.add_one` span under each `pixeltable.row`\n", "- the stored `thumb` column generates thumbnails whose persistence produces `pixeltable.media.save` spans\n", "- the view is recomputed as part of the same insert, producing the nested `pixeltable.view_load` subtree\n", "\n", "`if_exists='replace_force'` also drops the dependent view on re-runs." ] }, { "cell_type": "code", "execution_count": null, "id": "obs-table", "metadata": {}, "outputs": [], "source": [ "import pixeltable as pxt\n", "\n", "t = pxt.create_table(\n", " 'otel_demo1',\n", " {'c': pxt.Int, 'img': pxt.Image},\n", " if_exists='replace_force',\n", ")\n", "\n", "# Simple expression (span not reported since no UDF is invoked)\n", "t.add_computed_column(inc=t.c + 1)\n", "# Complex expression (span reported since UDF is invoked)\n", "t.add_computed_column(thumb=t.img.resize((64, 64)))\n", "\n", "v = pxt.create_view(\n", " 'otel_demo_view',\n", " t,\n", " additional_columns={'inc2': t.inc + 2},\n", " if_exists='replace',\n", ")" ] }, { "cell_type": "markdown", "id": "obs-insert-md", "metadata": {}, "source": [ "## Insert\n", "\n", "This is the traced operation: 4 rows referencing external images (well under the 100-span-per-operation\n", "cap, so every row gets a span). The whole pipeline, downloads, UDFs, thumbnails, store writes, and the\n", "view reload, lands under one `pixeltable.insert` trace.\n", "\n", "Note: `pixeltable.media.fetch` spans only appear when the files are not yet in the local file cache; on a re-run of\n", "this cell the downloads are skipped." ] }, { "cell_type": "code", "execution_count": 5, "id": "obs-insert", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Inserted 8 rows with 0 errors in 0.04 s (224.62 rows/s)\n" ] }, { "data": { "text/plain": [ "8 rows inserted." ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "base = 'https://raw.githubusercontent.com/pixeltable/pixeltable/main/docs/resources/images'\n", "images = [\n", " '000000000001.jpg',\n", " '000000000016.jpg',\n", " '000000000019.jpg',\n", " '000000000025.jpg',\n", "]\n", "status = t.insert(\n", " [{'c': i, 'img': f'{base}/{name}'} for i, name in enumerate(images)]\n", ")\n", "status" ] }, { "cell_type": "markdown", "id": "obs-flush", "metadata": {}, "source": [ "## Wait a few seconds for export\n", "\n", "`init()` sets up a `BatchSpanProcessor`, which ships queued spans automatically about every 5 seconds while the kernel stays alive, so no manual flush is needed. Give it a few seconds after the insert, then check Grafana.\n", "\n", "(In a short-lived script that exits immediately you would call `trace.get_tracer_provider().force_flush()` before exit, but in a live notebook kernel the timer handles it.)" ] }, { "cell_type": "markdown", "id": "obs-view", "metadata": {}, "source": [ "## View it in Grafana\n", "\n", "- **Traces**: Explore -> your Traces (Tempo) data source -> search Service Name `pixeltable`, or span name\n", " `pixeltable.insert`. Expand the trace to see the full hierarchy: `pixeltable.media.fetch` downloads running in\n", " parallel, `pixeltable.row` -> `pixeltable.udf.add_one`/`pixeltable.udf.resize` nesting, `pixeltable.media.save`\n", " persistence, the `pixeltable.store.build_rows`/`pixeltable.sa.insert_rows` write path, and the\n", " `pixeltable.view_load` subtree with its own rows and store writes.\n", "\n", "For more detail use `span_level='trace'`; to reduce volume on a large insert, use the default `'info'`\n", "(operation, `data_source.prepare`, `begin_xact`, `sa.insert_rows`, and `view_load` spans only)." ] } ], "metadata": { "kernelspec": { "display_name": "pxt", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.10.20" } }, "nbformat": 4, "nbformat_minor": 5 }