# nlqueries-core [](https://github.com/nlqueries/nlqueries/actions/workflows/ci.yml) [](https://pypi.org/project/nlqueries-core/) [](https://pypi.org/project/nlqueries-core/) [](LICENSE) [](https://github.com/nlqueries/nlqueries) **NLQueries Core** is an open-source natural language to SQL engine — see [nlqueries.com](https://nlqueries.com). It turns plain-English questions into validated SQL, builds a YAML knowledge base from your schema and query history (rebuilt whenever you re-run `export-kb`), and exposes everything as an MCP server your AI assistant can call directly. It also answers questions from your documents (PDF, Word, Excel, Markdown, plain text, Notion, Confluence), with citations. ## Links - **Website** — [nlqueries.com](https://nlqueries.com) - **Documentation** — [nlqueries.com/docs/](https://nlqueries.com/docs/) - **Getting started** — [nlqueries.com/docs/getting-started](https://nlqueries.com/docs/getting-started) - **Supported databases** — [nlqueries.com/databases/](https://nlqueries.com/databases/) - **PyPI** — [nlqueries-core](https://pypi.org/project/nlqueries-core/) - **Docker Hub** — [nlqueries/core](https://hub.docker.com/r/nlqueries/core) --- ## Features | Capability | Description | |---|---| | **Database connectors** | PostgreSQL, Snowflake, BigQuery, Redshift, SQL Server / Azure SQL, DuckDB, SQLite — plus a generic SQLAlchemy connector, driven by a connection URL, for MySQL (`connect mysql`) and any other SQLAlchemy-reachable database | | **Document connectors** | PDF, Word, Excel, Markdown, plain text, Notion, Confluence — ask questions over ingested documents with citations | | **Query pipeline** | Filter, cluster, and parameterize query history into reusable `QueryCapsule` templates | | **Knowledge base** | Auto-generated YAML schema + capsule file, with coverage reporting via `kb-stats` | | **Document Q&A** | `doc-ingest` (or `doc-sync-notion` / `doc-sync-confluence`) chunks, embeds and stores a source in Qdrant; `doc-ask` answers from it with citations | | **Conversational follow-ups** | Carries context across questions so a follow-up like "and by region?" resolves against the previous query — on by default in `nlqueries query`, reset with `--new-session` | | **Semantic cache** | Recognises a question `query` or the MCP `query` tool has answered before, by meaning, and skips the LLM: it replays the stored answer and runs the stored SQL again so the rows are current (not with `--no-execute`). `ask` does not use it. Needs Qdrant | | **Embedding daemon** | Keeps the embedding model resident in memory — ~10 ms per call instead of ~9 s | | **Feedback loop** | Rate an answer with `feedback`, review with `feedback-stats`, and `promote-feedback` the thumbs-up ones — their SQL is re-validated against the current schema and blended into later prompts as verified examples | | **Regression checks** | `eval` re-asks the agent's own mined capsules — and a golden question set too, with `--golden` — then checks the generated SQL parses and references only tables the agent knows | | **dbt import** | `import-dbt` merges dbt model and column descriptions into the knowledge base | | **LLM client** | Anthropic, OpenAI, Amazon Bedrock, or any LiteLLM-supported provider | | **MCP server** | Query execution and schema/knowledge lookup exposed as MCP tools for Claude, Cursor, etc. | | **CLI** | `nlqueries` (or the shorter `nlq` alias) — connect, build, query, and inspect from your terminal | See [docs/architecture.md](docs/architecture.md) (or [read online](https://nlqueries.com/docs/architecture)) for how these pieces fit together. --- ## Quickstart **Prerequisite:** Python 3.11–3.14. ### Option A — Docker (recommended) Pulls the published [`nlqueries/core`](https://hub.docker.com/r/nlqueries/core) image from Docker Hub — no clone required, just the compose file: ```bash curl -O https://raw.githubusercontent.com/nlqueries/nlqueries/main/docker-compose.yml ``` Create a `.env` file next to it with one LLM key, a Qdrant key and an MCP token. All three are required — compose refuses to start without `QDRANT_API_KEY` or `NLQ_MCP_STATIC_TOKEN`: ```bash ANTHROPIC_API_KEY=sk-ant-... # or OPENAI_API_KEY=sk-... QDRANT_API_KEY=... # any random secret: openssl rand -hex 32 NLQ_MCP_STATIC_TOKEN=... # another one: openssl rand -hex 32 ``` The MCP server is reachable over the network, so it requires that token: an MCP client connects to `http://localhost:8080/sse` and sends `Authorization: Bearer `. The token has full access. To give different people different access, see [docs/mcp-authentication.md](docs/mcp-authentication.md). Then start the stack: ```bash docker compose up ``` This pulls `nlqueries/core:0.3.0` and starts it alongside Qdrant (`:6333`), with the MCP server on `:8080`. Run CLI commands against the running stack from a second terminal: ```bash docker exec -it nlqueries-core nlqueries health ``` The image includes the PostgreSQL, SQLite, Snowflake, BigQuery and DuckDB drivers. It does not include the MySQL, Redshift or SQL Server drivers, or the `docs` and `wiki` extras (PDF / Word / Excel ingestion, Notion / Confluence sync); for those, install with pip (Option B) and add the extra. ### Option B — pip install ```bash pip install nlqueries-core export ANTHROPIC_API_KEY=sk-ant-... # or OPENAI_API_KEY nlqueries health ``` Optional extras for specific connectors: ```bash pip install "nlqueries-core[mysql]" # MySQL (driver for the generic connector) pip install "nlqueries-core[snowflake]" # Snowflake pip install "nlqueries-core[bigquery]" # Google BigQuery pip install "nlqueries-core[redshift]" # Amazon Redshift pip install "nlqueries-core[mssql]" # SQL Server / Azure SQL pip install "nlqueries-core[duckdb]" # DuckDB pip install "nlqueries-core[docs]" # PDF / Word / Excel ingestion pip install "nlqueries-core[wiki]" # Notion / Confluence sync ``` PostgreSQL and SQLite need no extra. The generic `sqlalchemy` connector needs whichever driver its URL names. ### Option C — Clone and install from source No Docker required — for contributing, or to run against unreleased changes: ```bash git clone https://github.com/nlqueries/nlqueries.git cd nlqueries python -m venv .venv && source .venv/bin/activate # Windows: .venv\Scripts\Activate.ps1 pip install -e ".[dev]" export ANTHROPIC_API_KEY=sk-ant-... # or OPENAI_API_KEY nlqueries health ``` See [CONTRIBUTING.md](CONTRIBUTING.md#development-setup) for linting and test commands. ### First query ```bash nlqueries connect postgres --host localhost --database mydb --user alice --password secret --alias dev nlqueries process-history dev --days 30 --annotate nlqueries export-kb dev nlqueries query dev "How many orders shipped last month?" nlqueries query dev "and how many were returned?" # follow-up — keeps prior context ``` Follow-up context is on by default; pass `--new-session` to start fresh or `--no-session` to disable it for a one-off question. Full walkthrough: [docs/getting-started.md](docs/getting-started.md) (or [read online](https://nlqueries.com/docs/getting-started)). --- ## Documentation | Doc | Covers | |---|---| | [docs/getting-started.md](docs/getting-started.md) (or [read online](https://nlqueries.com/docs/getting-started)) | Step-by-step setup and your first query | | [docs/cli-reference.md](docs/cli-reference.md) (or [read online](https://nlqueries.com/docs/cli-reference)) | Every command and flag | | [docs/connectors.md](docs/connectors.md) (or [read online](https://nlqueries.com/docs/connectors)) | Database and document connector setup, per-connector notes and caveats | | [docs/configuration.md](docs/configuration.md) (or [read online](https://nlqueries.com/docs/configuration)) | Environment variables | | [docs/troubleshooting.md](docs/troubleshooting.md) (or [read online](https://nlqueries.com/docs/troubleshooting)) | Common warnings and errors explained | | [docs/qdrant-setup.md](docs/qdrant-setup.md) (or [read online](https://nlqueries.com/docs/qdrant-setup)) | Setting up Qdrant (required for embeddings, semantic cache, document search) | | [docs/mcp-authentication.md](docs/mcp-authentication.md) (or [read online](https://nlqueries.com/docs/mcp-authentication)) | Authenticating the MCP server — required to serve it over a network | | [docs/architecture.md](docs/architecture.md) (or [read online](https://nlqueries.com/docs/architecture)) | Module layout and request flow | | [docs/database-hardening.md](docs/database-hardening.md) (or [read online](https://nlqueries.com/docs/database-hardening)) | The read-only role to grant per engine, and exactly what each connector does and does not enforce | | [docs/dependency-locking.md](docs/dependency-locking.md) (or [read online](https://nlqueries.com/docs/dependency-locking)) | The hashed lock file the image builds from, the digest-pinned base image, and how to regenerate both | | [docs/contributing.md](docs/contributing.md) (or [read online](https://nlqueries.com/docs/contributing)) | What a new connector has to implement, and what to open an issue about first | --- ## Contributing See [CONTRIBUTING.md](CONTRIBUTING.md). All contributors must sign the CLA before a PR can be merged — see [CONTRIBUTOR_LICENSE_AGREEMENT.md](CONTRIBUTOR_LICENSE_AGREEMENT.md). --- ## License [Business Source License 1.1](LICENSE) — converting to Apache 2.0 no later than **June 4, 2030**. The licence sets that date and also converts each version on the fourth anniversary of its own first publication, whichever comes first, so a version first published before June 4, 2026 converts on its own anniversary, ahead of the fixed date. Until then the licence grants production use, including commercially and self-hosted, with one carve-out: you may not offer NLQueries Core to third parties as a hosted or managed service. Individual and non-commercial self-hosted deployments are unrestricted. [LICENSE](LICENSE) is the authority; this paragraph is a summary.