# Isaac Sim MCP Server A [Model Context Protocol (MCP)](https://modelcontextprotocol.io/) server that gives AI coding assistants deep knowledge of NVIDIA Isaac Sim — extensions, code examples, settings, and developer instructions — powered by semantic search and NVIDIA AI reranking. Built on the [NVIDIA AIQ Toolkit](https://github.com/NVIDIA/GenerativeAIExamples) (formerly NeMo Agent Toolkit / NAT 1.3+). --- ## What's new (Apr 2026) - **MCP protocol bumped to `2025-11-25`** (G-8, G-9): NAT 1.4+ accepts the new spec date end-to-end. - **Hybrid retrieval is now the default** (`OVAI_RETRIEVAL_MODE=hybrid`): RRF fusion of BM25 lexical + dense semantic with optional cross-encoder reranking. - **Query expansion active on every search** — 215+ term glossary + auto-generated USD v25.02/v25.11 glossary expand abbreviations (SSS, PBR, LIVRPS, Gf, Sdf, ...) before retrieval (REQ-QE-1..6). - **BM25 sidecar migrated from `bm25.pkl` to `bm25.json`** (G-7, pickle-safety hardening) — new `bm25_safe` schema is JSON-only. - **Result guardrails wired into every search tool** (G-1, REQ-RG-1/2/3): `sanitize_query` on input, deterministic "No relevant documentation found for this query." sentinel for empty results, and `search_isaac_sim_settings` caps output at 12000 chars with a `prefix_filter`/`type_filter` hint. - **Tool descriptions refreshed (Phase 14, 34 tools)** — every tool now has PRIMARY / WHEN-TO-USE / ARGUMENTS / RETURNS / USAGE EXAMPLES / WHEN-TO-USE-A-DIFFERENT-TOOL-INSTEAD / abbreviation-tip sections. ## 5-Minute Quickstart Get from zero to working Isaac Sim tools in your IDE. Follow every step in order. ### Prerequisites - [Docker](https://docs.docker.com/get-docker/) installed and running - [Poetry](https://python-poetry.org/docs/#installation) — `build-docker.sh` builds the AIQ + MCP wheels with poetry before `docker build` - Python **3.11 – 3.13** on the host. `pyproject.toml` requires `>=3.11,<3.14`; `poetry build` aborts on Python 3.10 with *"Current Python version (3.10.x) is not allowed by the project"*. If your default `python3` is 3.10, point poetry at a newer interpreter in **both** project dirs: ```bash cd source/aiq/isaacsim_fns && poetry env use python3.12 cd source/mcp/isaacsim_mcp && poetry env use python3.12 ``` - An NVIDIA API key (see Step 1 below) - [Git LFS](https://git-lfs.com/) installed — the Isaac Sim extension metadata and FAISS indices under `source/aiq/isaacsim_fns/src/isaacsim_fns/data/` are LFS-tracked. `build-wheels.sh` auto-runs `git lfs install --local && git lfs pull` for you on the first build, so you only need the binary on PATH (`sudo apt-get install git-lfs` or `brew install git-lfs`). Without LFS, the wheel ends up ~13× smaller and the container silently fails at first tool call with `Extension data is not available` — auto-recovery in `build-wheels.sh` catches this. - The repo cloned: `git clone https://github.com/NVIDIA-Omniverse/kit-usd-agents.git` — `build-docker.sh` expects `source/aiq/isaacsim_fns/` to exist next to `source/mcp/isaacsim_mcp/`, so a sparse-checkout of just the MCP subtree won't build. ### Step 1: Get Your API Key | Key | What It's For | Where to Get It | |-----|---------------|-----------------| | `NVIDIA_API_KEY` | Authenticates calls to NVIDIA's cloud endpoints for embeddings, reranking, and LLM inference | [build.nvidia.com/settings/api-keys](https://build.nvidia.com/settings/api-keys) — sign in, click **Generate API Key**, paste it in place of `REPLACE_WITH_NVIDIA_API_KEY` | > **Note:** A second key (`NGC_API_KEY` from [org.ngc.nvidia.com/setup/api-key](https://org.ngc.nvidia.com/setup/api-key)) is only required for local NIM deployment — see [Deployment Options](#deployment-options). ### Step 2: Configure Your `.env` ```bash cd kit-usd-agents/source/mcp cp .env.example .env ``` Open `.env` and set: ```env NVIDIA_API_KEY=REPLACE_WITH_NVIDIA_API_KEY ``` ### Step 3: Build and Run the Docker Container ```bash cd isaacsim_mcp # Build the image (~10–15 min cold, produces a ~1.35 GB image) ./build-docker.sh # Linux/macOS # build-docker.bat # Windows # Run the server (foreground, auto-removed on Ctrl-C) docker run --rm -p 9904:9904 --env-file ../.env isaacsim-mcp:latest # Or run as a managed service so you can stop/restart later: # docker run -d --name isaacsim-mcp -p 9904:9904 --env-file ../.env isaacsim-mcp:latest # docker logs -f isaacsim-mcp # docker stop isaacsim-mcp && docker rm isaacsim-mcp ``` > **`.env` location matters.** `--env-file ../.env` resolves relative to your current directory. Run `docker run` from `source/mcp/isaacsim_mcp/`, or use the absolute path: `--env-file "$(git rev-parse --show-toplevel)/source/mcp/.env"`. ### Step 4: Verify the Server is Running The server speaks Streamable HTTP at `/mcp` (the canonical endpoint in NAT 1.25). There is **no separate `/health` GET endpoint**; the canonical liveness probe is an MCP `initialize` POST. > **Trailing slash:** older NAT 1.3 builds required `/mcp/`. NAT 1.25 returns `307 Temporary Redirect` from `/mcp/` to `/mcp`, so both work — but using `/mcp` directly avoids a redirect on every call. If you keep the trailing slash, pass `-L` to curl. ```bash # Easiest: use the included Python health check. # Needs aiohttp on the host (`pip install aiohttp`); or run it inside the container: # docker exec isaacsim-mcp python /app/check_mcp_health.py python check_mcp_health.py # Or curl the MCP endpoint directly. The Accept header is required — # the server replies with text/event-stream: curl -s -X POST http://localhost:9904/mcp \ -H 'Content-Type: application/json' \ -H 'Accept: application/json, text/event-stream' \ -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-11-25","capabilities":{},"clientInfo":{"name":"health","version":"1.0"}}}' ``` A healthy server returns a JSON-RPC `result` payload listing the server's name and capabilities. ### Step 5: Connect Your IDE All IDE configs point at the same URL: `http://localhost:9904/mcp`. > **Name collision:** if you previously had an `isaac-sim-mcp` registered against a hosted endpoint (e.g. `http://isaac-sim-mcp.nvidia.com/mcp`), `claude mcp add isaac-sim-mcp` will refuse / clash. List with `claude mcp list | grep isaac-sim-mcp`, then remove the prior entry: `claude mcp remove isaac-sim-mcp -s `. > **MCP scoping note (Claude Code):** `claude mcp add ... -t http ` writes to `.claude.json` in your **current working directory**. The MCP is then only active when you launch Claude CLI from that directory. To register globally, pass `--scope user`.
Cursor ```json { "mcpServers": { "isaac-sim-mcp": { "url": "http://localhost:9904/mcp" } } } ``` Reload Cursor: `Cmd/Ctrl+Shift+P` → **Developer: Reload Window**
Claude Code ```bash # Project scope claude mcp add isaac-sim-mcp -t http http://localhost:9904/mcp # User (global) scope claude mcp add isaac-sim-mcp --scope user -t http http://localhost:9904/mcp ``` Or add to `~/.claude.json`: ```json { "mcpServers": { "isaac-sim-mcp": { "type": "http", "url": "http://localhost:9904/mcp" } } } ```
Windsurf ```json { "mcpServers": { "isaac-sim-mcp": { "url": "http://localhost:9904/mcp" } } } ```
VS Code (Copilot) ```json { "servers": { "isaac-sim-mcp": { "type": "http", "url": "http://localhost:9904/mcp" } } } ```
### Step 6: Verify Tools Appear In your IDE's AI chat, you should see **5 Isaac Sim tools**: | Tool | Description | |------|-------------| | `get_isaac_sim_instructions` | Top-level guidance on Isaac Sim development | | `search_isaac_sim_extensions` | Semantic search across the indexed Isaac Sim extension catalog | | `get_isaac_sim_extension_details` | Full details for one or more extensions | | `search_isaac_sim_code_examples` | Find Isaac Sim code patterns by description | | `search_isaac_sim_settings` | Find an Isaac Sim setting by name or purpose | Try asking: *"Find me an Isaac Sim extension for cameras"* — if you get the `isaacsim.sensors.camera` family back, the index is intact. --- ## Architecture Overview ``` ┌──────────────────────┐ │ Your IDE │ │ (Cursor / Claude │ │ Code / Windsurf / │ │ VS Code Copilot) │ └────────┬─────────────┘ │ MCP (Streamable HTTP, POST /mcp) ▼ ┌──────────────────────┐ │ Isaac Sim MCP Server │ ← This package │ (port 9904) │ └────────┬─────────────┘ │ AIQ Workflow ▼ ┌──────────────────────┐ │ RAG Pipeline │ │ ┌────────────────┐ │ │ │ Embedder │ │ ← NVIDIA cloud or local NIM │ │ Reranker │ │ ← NVIDIA cloud or local NIM │ └────────────────┘ │ └────────┬─────────────┘ │ ▼ ┌──────────────────────┐ │ Isaac Sim Atlas DB │ ← Extensions, code examples, │ │ settings, instructions └──────────────────────┘ ``` --- ## Deployment Options ### Option A: Cloud Endpoints (Recommended) ```env NVIDIA_API_KEY=REPLACE_WITH_NVIDIA_API_KEY ``` ### Option B: Local NIM Containers (Advanced, GPU required) ```env NVIDIA_API_KEY=REPLACE_WITH_NVIDIA_API_KEY NGC_API_KEY=REPLACE_WITH_NGC_API_KEY KIT_EMBEDDER_BACKEND=local KIT_LOCAL_EMBEDDER_URL=http://localhost:8080 KIT_RERANKER_BACKEND=local KIT_LOCAL_RERANKER_URL=http://localhost:8081 ``` > NGC API key: [org.ngc.nvidia.com/setup/api-key](https://org.ngc.nvidia.com/setup/api-key). Full local-NIM setup is in [`source/mcp/LOCAL_DEPLOYMENT.md`](../LOCAL_DEPLOYMENT.md). --- ## Troubleshooting | Problem | Likely Cause | Fix | |---------|-------------|-----| | `connection refused` on port 9904 | Docker container not running | `docker ps` to check; restart the container | | `404` on `GET /health` | No `/health` GET endpoint exists | Use `python check_mcp_health.py` or POST an MCP `initialize` to `/mcp` | | `307 Temporary Redirect` on `/mcp/` | NAT 1.25 canonicalizes to `/mcp` (no trailing slash); NAT 1.3 was the opposite | Use `/mcp` directly in IDE config and curl probes, or pass `-L` to curl to follow the redirect | | `406 Not Acceptable` from curl probe | Server replies with `text/event-stream`; default curl `Accept: */*` is rejected | Add `-H 'Accept: application/json, text/event-stream'` | | `401 Unauthorized` from cloud calls | Invalid `NVIDIA_API_KEY` | Regenerate at [build.nvidia.com/settings/api-keys](https://build.nvidia.com/settings/api-keys); update `.env` | | `[410] Gone — This endpoint has reached its end of life on 2026-05-18T00:00:00Z` during hosted rerank | An older config or image still pins retired model `nvidia/llama-3.2-nv-rerankqa-1b-v2` | Set `OVAI_RERANK_MODEL=nvidia/llama-nemotron-rerank-vl-1b-v2` (or another current model from [build.nvidia.com/explore/retrieval](https://build.nvidia.com/explore/retrieval)), or fall back to `OVAI_RERANK=false`. Full details + on-prem NIM workaround in [LOCAL_DEPLOYMENT.md](../LOCAL_DEPLOYMENT.md#hybrid-retrieval-ovgenai-retrieval-tuning) | | `ModuleNotFoundError: aiohttp` from `check_mcp_health.py` | The host doesn't have `aiohttp` | `pip install aiohttp`, or run inside the container: `docker exec isaacsim-mcp python /app/check_mcp_health.py` | | `--env-file: file not found` | Wrong cwd when invoking `docker run` | Run from `source/mcp/isaacsim_mcp/`, or use absolute path | | Port 9904 already in use | Another process on that port | `lsof -i :9904` or `netstat -aon \| findstr 9904`; stop or remap (e.g. `-p 9914:9904`) | | `Current Python version (3.10.x) is not allowed by the project` during `build-docker.sh` | poetry venv is bound to system Python 3.10 | `poetry env remove --all && poetry env use python3.12` in **both** `source/aiq/isaacsim_fns/` and `source/mcp/isaacsim_mcp/` | | `command not found: poetry` during `build-docker.sh` | Poetry isn't installed | Install per [python-poetry.org](https://python-poetry.org/docs/#installation); see Prerequisites | | `(1312, 'CredRead', 'A specified logon session does not exist. ...')` during `setup-dev` or `poetry install` / `poetry lock` | Poetry queries the Windows Credential Manager for every package URL, and this session has no logon-session credential store | `setup-dev.bat` / `setup-dev.sh` disable Poetry's keyring lookup automatically (this MCP only uses public PyPI). When running Poetry yourself, or if you set `POETRY_KEYRING_ENABLED` before running `setup-dev`: `set POETRY_KEYRING_ENABLED=false` (Windows) / `export POETRY_KEYRING_ENABLED=false` (Linux) | | `pyproject.toml changed significantly since poetry.lock was last generated` | `poetry.lock` is out of date with `pyproject.toml` | `setup-dev` detects this with `poetry check --lock` and runs `poetry lock` for you; manually: `poetry lock`, then `poetry install` | | `TypeError: Descriptors cannot be created directly` while loading `nat_opentelemetry` / `nat_langchain` | Environment installed from an older lock that paired `opentelemetry-proto` 1.11.1 with protobuf 7 | Re-run `setup-dev` to install the corrected lock. Do not set `PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python` | | `pip ResolutionImpossible` mentioning `nvidia-nat` during `docker build` | NAT modular packages were resolved from different releases | Keep the full NAT package set on `1.8.0a20260514`; RAGAS is not part of the MCP runtime | | Tools not appearing in IDE | MCP config not loaded or wrong URL | Verify with `check_mcp_health.py`; ensure URL is `http://localhost:9904/mcp`; reload IDE | | `isaac-sim-mcp` already registered / `claude mcp add` rejects | Earlier registration (e.g. against the hosted `isaac-sim-mcp.nvidia.com`) is still in your config | Remove first: `claude mcp remove isaac-sim-mcp -s `, then re-add against the local server | | `isaac-sim-mcp` missing from `claude mcp list` | `-t http` registered the MCP at project scope | Re-add with `--scope user`, or always launch Claude CLI from where you registered | --- **Stale or corrupt hybrid BM25 sidecar:** - As of Apr 2026 the sidecar is `bm25.json` (the old `bm25.pkl` pickle format was removed for pickle-safety). If hybrid search returns empty or mis-ranked results, delete `bm25.json` next to the FAISS index and restart — the server will rebuild it on first query using the `bm25_safe` JSON schema. ## Dependencies The Docker images install `ovgenai-retrieval` — the shared hybrid-retrieval library bundled under `source/aiq/ovgenai_retrieval/` and built into a wheel by `source/mcp/build-wheels.sh` — backed by `rank_bm25`, so this MCP and the sibling `ovgenai-agent-search` tool share the exact same RRF-fusion + query-expansion + JSON-BM25 code path. ## Related Projects - **`ovgenai-agent-search`** — the filesystem-search equivalent of this MCP. Same `ovgenai-retrieval` library, same guardrails, same query expansion, but exposed as a shell-first CLI / skill rather than an HTTP MCP. Use `ovgenai-agent-search` from agents that prefer shell access (e.g., Claude Code skills); use this MCP for agents that prefer HTTP tool calls. ## Development ### Running Locally (Without Docker) ```bash ./setup-dev.sh # Linux/macOS # setup-dev.bat # Windows ./run.sh # Linux/macOS # run.bat # Windows ``` --- ## License See the [LICENSE](../../../LICENSE) file in the root of this repository.