aid: prompt-foo name: Promptfoo description: >- Promptfoo is an open-source CLI and TypeScript/Node.js library for evaluating, red-teaming, and security-testing LLM applications, agents, and RAG pipelines. It runs deterministic prompt evals with model-graded and rule-based assertions, generates dynamic adversarial attack probes across 50+ vulnerability categories (prompt injection, jailbreaks, RAG poisoning, PII leakage, harmful content, business rule violations), and integrates with CI/CD via a GitHub Action and a `code-scans` command for pull-request review. Promptfoo supports dozens of LLM providers — OpenAI, Anthropic Claude, Google Gemini, AWS Bedrock/SageMaker, Azure OpenAI, Mistral, Cohere, Groq, DeepSeek, Together, Fireworks, OpenRouter, LiteLLM, Vercel and Cloudflare AI gateways — plus local runtimes (Ollama, LocalAI, llama.cpp, vLLM, llamafile, Docker Model Runner) and custom HTTP/WebSocket/Python/JavaScript/Go/Ruby/Shell providers. It also ships an MCP server (`promptfoo mcp`), a Model Audit scanner (`scan-model`) for malicious ML artifacts, and a hosted Enterprise tier with team sharing, continuous monitoring, SSO, and a centralized compliance dashboard aligned to OWASP LLM Top 10, NIST AI RMF, MITRE ATLAS, and the EU AI Act. The project is MIT-licensed, has 21k+ GitHub stars, and is now part of OpenAI while remaining open source. type: Index position: Provider access: 3rd-Party image: https://kinlane-images.s3.amazonaws.com/shared/apis-json/apis-json-logo.jpg tags: - LLM Evaluation - LLM Red Teaming - LLM Security - AI Security - Prompt Engineering - Vulnerability Scanning - Adversarial Testing - Jailbreak Testing - Prompt Injection - RAG - Agents - MCP - Model Audit - OWASP LLM Top 10 - NIST AI RMF - MITRE ATLAS - EU AI Act - CI/CD - Open Source - TypeScript url: https://raw.githubusercontent.com/api-evangelist/prompt-foo/refs/heads/main/apis.yml created: '2026-05-25' modified: '2026-05-25' specificationVersion: '0.20' apis: [] common: - type: TrustCenter url: security/prompt-foo-trust-center.yml - type: DomainSecurity url: security/prompt-foo-domain-security.yml - type: Website url: https://www.promptfoo.dev - type: Portal url: https://www.promptfoo.dev - type: Documentation url: https://www.promptfoo.dev/docs/intro/ - type: GettingStarted url: https://www.promptfoo.dev/docs/getting-started/ - type: Documentation name: CLI Reference url: https://www.promptfoo.dev/docs/usage/command-line/ - type: Documentation name: Red Team Quickstart url: https://www.promptfoo.dev/docs/red-team/quickstart/ - type: Documentation name: Providers url: https://www.promptfoo.dev/docs/providers/ - type: Documentation name: Assertions & Metrics url: https://www.promptfoo.dev/docs/configuration/expected-outputs/ - type: Documentation name: API Reference url: https://www.promptfoo.dev/docs/api-reference/ - type: Pricing url: https://www.promptfoo.dev/pricing/ - type: Blog url: https://www.promptfoo.dev/blog/ - type: StatusPage url: https://status.promptfoo.app - type: TrustCenter url: https://trust.promptfoo.dev - type: GitHubOrganization url: https://github.com/promptfoo - type: SourceCode name: promptfoo (CLI + library) url: https://github.com/promptfoo/promptfoo - type: SDKs name: Python wrapper for the promptfoo CLI url: https://github.com/promptfoo/promptfoo-python - type: Tools name: Promptfoo GitHub Action url: https://github.com/promptfoo/promptfoo-action - type: Tools name: ModelAudit — ML model security scanner url: https://github.com/promptfoo/modelaudit - type: Tools name: MCP Agent Provider (custom provider for testing MCP servers) url: https://github.com/promptfoo/mcp-agent-provider - type: Tools name: evil-mcp-server (red-team MCP target) url: https://github.com/promptfoo/evil-mcp-server - type: CodeExamples name: example-app (industry/model/security demo app) url: https://github.com/promptfoo/example-app - type: CodeExamples name: demo-app url: https://github.com/promptfoo/demo-app - type: Library name: js-rouge (ROUGE summarization metric) url: https://github.com/promptfoo/js-rouge - type: Package name: npm — promptfoo url: https://www.npmjs.com/package/promptfoo - type: Package name: PyPI — promptfoo url: https://pypi.org/project/promptfoo/ - type: Package name: Homebrew — promptfoo url: https://formulae.brew.sh/formula/promptfoo - type: HelmChart name: Helm chart for self-hosted Promptfoo url: https://github.com/promptfoo/promptfoo/tree/main/helm/chart/promptfoo - type: License name: MIT License url: https://github.com/promptfoo/promptfoo/blob/main/LICENSE - type: Forums name: Discord community url: https://discord.gg/promptfoo - type: Twitter url: https://twitter.com/promptfoo - type: LinkedIn url: https://www.linkedin.com/company/promptfoo - type: ChangeLog url: https://github.com/promptfoo/promptfoo/releases - type: Features data: - Open-source CLI and Node.js/TypeScript library for LLM evaluation and red teaming - >- `promptfoo eval` — run deterministic prompt/model evals with caching, concurrency, and live reload - >- `promptfoo redteam` — generate and run dynamic adversarial probes across 50+ vulnerability categories - >- `promptfoo view` — local browser UI for browsing eval and red-team results - >- `promptfoo share` — publish a shareable URL for an eval or model audit - >- `promptfoo generate` — synthesize datasets, red-team tests, and assertions - "`promptfoo optimize` — improve prompts against a target provider" - "`promptfoo scan-model` — security-scan ML model files (ModelAudit)" - >- `promptfoo code-scans` — scan code changes for LLM security vulnerabilities in IDEs and CI/CD - >- `promptfoo mcp` — expose promptfoo tools as a Model Context Protocol server - "`promptfoo retry`, `list`, `export`, `import`, `validate`, `debug`, `cache`, `auth`" - >- Assertions: rule-based (equals, contains, regex, javascript, python, cost, latency) and model-graded (llm-rubric, classifier, factuality, answer-relevance, similarity) - >- Red-team plugins covering prompt injection, jailbreaks, PII leakage, harmful content, bias, business-rule violations, RAG poisoning, and agent/tool abuse - >- Attack strategies including multi-turn (Crescendo), GOAT (Meta), and iterative jailbreak techniques - >- Framework alignment: OWASP LLM Top 10, NIST AI RMF, MITRE ATLAS, EU AI Act - >- 50+ providers: OpenAI, Anthropic, Google Gemini, AWS Bedrock, SageMaker, Azure OpenAI, Mistral, Cohere, Groq, DeepSeek, Together, Fireworks, Perplexity, OpenRouter, LiteLLM, Vercel AI Gateway, Cloudflare AI Gateway - >- Local runtimes: Ollama, LocalAI, llama.cpp, vLLM, Docker Model Runner, llamafile - >- Custom providers via HTTP, WebSocket, Python, JavaScript, Go, Ruby, and Shell - GitHub Action (`promptfoo/promptfoo-action`) for PR-level eval gating - Self-hostable via Helm chart and on-premise enterprise deployment - >- Hosted Enterprise tier: team sharing, continuous monitoring, centralized compliance dashboard, SSO, granular permissions, managed cloud, SLA-backed support - SOC 2 and ISO 27001 certified; trust center at trust.promptfoo.dev - >- Distributed on npm (`promptfoo`), PyPI (`promptfoo`), and Homebrew (`brew install promptfoo`) - MIT-licensed; 21k+ GitHub stars; now part of OpenAI sources: - https://www.promptfoo.dev - https://github.com/promptfoo/promptfoo - https://www.promptfoo.dev/docs/usage/command-line/ - https://www.promptfoo.dev/docs/red-team/quickstart/ - https://www.promptfoo.dev/docs/providers/ - https://www.promptfoo.dev/pricing/ updated: '2026-05-25' maintainers: - FN: Kin Lane email: kin@apievangelist.com