name: toolmenu description: Test your MCP server the way agents and clients meet it, and catch breaking menu changes on every PR. author: niksa90 branding: icon: list color: yellow inputs: command: description: How to start the server over stdio, e.g. "node dist/server.js". Use this or url. required: false url: description: A Streamable HTTP endpoint to check instead of a stdio command. required: false headers: description: >- HTTP headers for a url server, one "Name: value" per line. Pass keys in from the workflow's secrets (see docs/github-action.md), not as plain text. required: false env: description: >- Environment variables for a stdio server, one KEY=value per line (it only gets a minimal environment otherwise). required: false baseline: description: The committed menu file to diff against. required: false default: menu.json sarif: description: >- A folder to also write the findings to as SARIF (snapshot.sarif, diff.sarif, session.sarif), for GitHub code scanning: upload it with github/codeql-action/upload-sarif. Results point at the baseline file. Empty (the default): none. required: false default: '' scenario: description: >- Optional scenario.yml for a session run (tools not marked readOnlyHint are refused unless it allows writes), or "auto" to build the steps from the menu: every read-only tool whose required arguments the schema can fill. required: false baseline-from: description: >- Which menu to diff against the baseline: "snapshot" (default), or "session": every tool the scenario saw, for servers whose tools appear after an unlock. Needs a scenario. required: false default: snapshot catalog: description: >- Also read the operations behind a search tool (search and execute) and diff them, like snapshot --catalog. Queries come from toolmenu.config.json or the menu. required: false default: 'false' fail-on: description: Fail the job on findings at or above this level (error, warn or info). required: false default: error gate: description: >- What decides the job for the comparison with the baseline. "findings" (default): diff's findings at or above fail-on. "review": the review verdict, so breaking changes the release bump allows pass, and findings at or above fail-on the baseline didn't have fail. Snapshot and session findings count either way. The verdict is at the top of the report whichever you pick. required: false default: findings comment: description: Post (and update) one PR comment with the report. required: false default: 'true' release: description: >- Release versions for the version-bump check, ".." (e.g. "1.4.0..1.5.0"). "auto" (default): on a PR, the version in package.json, pyproject.toml or Cargo.toml on the base branch and head; on a tag push, the previous version tag and this one. "off" skips the check. required: false default: auto version: description: >- toolmenu version from npm. Default: the version of this Action (the tag you pinned), so the CLI and the Action match. Set "latest" to always run the newest. required: false cli: description: Override how toolmenu is run (for testing a local build), e.g. "node dist/cli.js". required: false token: description: Token for the PR comment. Needs pull-requests write. required: false default: ${{ github.token }} outputs: sarif: description: The folder the SARIF files were written to, when the sarif input was set. value: ${{ steps.run.outputs.sarif }} report: description: Path to the markdown report. value: ${{ steps.run.outputs.report }} exit-code: description: 0 clean, 1 findings at or above fail-on, 2 couldn't run. value: ${{ steps.run.outputs.exit-code }} verdict: description: >- Should this menu replace the baseline (toolmenu review): unchanged, safe, breaking-released, breaking or review. Empty when there was no baseline to compare with. value: ${{ steps.run.outputs.verdict }} accept: description: "'true' when the verdict says the new menu can replace the baseline as it is." value: ${{ steps.run.outputs.accept }} skipped: description: >- 'true' when the check was skipped because the PR runs without the repository's secrets. The job still passes then: gate on this output if a skip shouldn't. value: ${{ steps.run.outputs.skipped }} runs: using: composite steps: - id: run shell: bash run: '"$GITHUB_ACTION_PATH/action/run.sh"' env: TOOLMENU_COMMAND: ${{ inputs.command }} TOOLMENU_URL: ${{ inputs.url }} TOOLMENU_HEADERS: ${{ inputs.headers }} TOOLMENU_ENV: ${{ inputs.env }} TOOLMENU_BASELINE: ${{ inputs.baseline }} TOOLMENU_SCENARIO: ${{ inputs.scenario }} TOOLMENU_SARIF: ${{ inputs.sarif }} TOOLMENU_BASELINE_FROM: ${{ inputs.baseline-from }} TOOLMENU_CATALOG: ${{ inputs.catalog }} TOOLMENU_FAIL_ON: ${{ inputs.fail-on }} TOOLMENU_GATE: ${{ inputs.gate }} TOOLMENU_COMMENT: ${{ inputs.comment }} TOOLMENU_VERSION: ${{ inputs.version }} TOOLMENU_RELEASE: ${{ inputs.release }} TOOLMENU_BASE_REF: ${{ github.base_ref }} TOOLMENU_CLI: ${{ inputs.cli }} TOOLMENU_PR: ${{ github.event.pull_request.number }} # Runs without the repository's secrets: pull request events (pull_request # and its review events; pull_request_target does get secrets) from a fork, # where a deleted fork has no head repo at all, and anything Dependabot # triggers, which only gets Dependabot secrets. TOOLMENU_NO_SECRETS: ${{ (github.event.pull_request.number > 0 && github.event_name != 'pull_request_target' && github.event.pull_request.head.repo.full_name != github.repository) || github.actor == 'dependabot[bot]' }} GH_TOKEN: ${{ inputs.token }} # To tell the job's own token (comments by github-actions[bot]) from one passed in. TOOLMENU_JOB_TOKEN: ${{ github.token }}