Promptfoo
The best open-source AI red-teaming tool there is — MIT, free forever, 10,000 probes a month — and now owned by OpenAI, which is exactly the vendor most of its users want to test independently.
PRICING
| Community | Free forever | MIT-licensed; all evaluation features, all providers, red teaming capped at 10k probes/month, local or self-hosted |
| Enterprise | Custom | advanced vulnerability detection, remediation guidance, guardrail dashboard, SSO, RBAC, API and webhooks |
| Enterprise On-Premise | Custom | same feature set inside your own infrastructure |
Promptfoo's own pricing page lists only Community and Enterprise as of 04/09/2026. Third-party reviews still quote a $50/month Team plan — treat that as stale. You also pay your own provider's token costs for graders and attack generation.
checked 2026-09-04 · vendor pricing page
Element scores
Strengths
24,777 GitHub stars, MIT licence, and a free tier that is genuinely the whole tool: every assertion type, every provider, CI/CD, ModelAudit static scanning, and 10,000 red-team probes a month. The red-team engine generates attacks against your specific application rather than replaying a fixed list, covering 50+ vulnerability types mapped to the OWASP LLM Top 10. Tracing is a real differentiator — a built-in OTLP receiver, GenAI semantic conventions, trajectory assertions like tool-used and tool-sequence, and trace summaries fed back into iterative attacks so the grader can tell 'the agent said it would not do that' from 'the agent actually called the forbidden tool'. It speaks MCP in both directions. And OpenAI's own deprecation docs point developers here as the exit route from the retiring hosted Evals platform.
Honest dings
OpenAI announced the acquisition on 09/03/2026 and the technology is being folded into OpenAI Frontier. The licence has not changed and the team says the open-source suite stays multi-provider, but the roadmap now sits with the vendor whose models most people are testing — if evaluator independence is a procurement requirement, that is a real conflict to write down. The free self-hosted server is explicitly not recommended for production: local SQLite, no horizontal scaling, no auth or SSO, no RBAC, and multiple replicas break with 'Job not found'. Enterprise pricing is not published, so budgeting a production deployment needs a sales call. Red-team probes above 10,000/month are an Enterprise purchase, and dynamic attack generation and grading burn your own provider tokens — a single deep red team can run from cents to hundreds of dollars.
Sources (13) — every claim traceable
Every audit lists the research it rests on — transparency and traceability are the product. Tools evolve: each audit is a snapshot of its audit date, and re-audits supersede older versions (kept below for reference).
- promptfoo.dev/blog/promptfoo-joining-openai — Official company post: acquisition by OpenAI, commitment to stay open source and multi-provider, 350k developers / 130k monthly active / 25%+ of the Fortune 500, 23-person team (accessed 2026-09-04)
- openai.com/index/openai-to-acquire-promptfoo — OpenAI's 09/03/2026 announcement: Promptfoo technology integrates into OpenAI Frontier; open-source project continues alongside enterprise capabilities (accessed 2026-09-04)
- github.com/promptfoo/promptfoo — Repository state on 04/09/2026: 24,777 stars, 2,257 forks, MIT licence, Node >=22.22 requirement, README notice that Promptfoo is now part of OpenAI and remains MIT (accessed 2026-09-04)
- promptfoo.dev/pricing — Official pricing: Community free forever with 10k red-team probes/month, Enterprise and Enterprise On-Premise custom-priced; feature matrix showing SSO, RBAC, guardrail dashboard and API are Enterprise-only (accessed 2026-09-04)
- promptfoo.dev/docs/usage/self-hosting — Official warning that self-hosting is not recommended for production: local SQLite, no horizontal scaling, no auth or SSO, no multi-team support, replicaCount must stay at 1 (accessed 2026-09-04)
- promptfoo.dev/docs/tracing — OpenTelemetry tracing: built-in OTLP receiver, trace viewer, traceparent propagation, GenAI semantic conventions, trace-aware assertions, forwarding to Jaeger or Tempo (accessed 2026-09-04)
- promptfoo.dev/docs/red-team/agents — Agent red-teaming: trajectory assertions (tool-used, tool-args-match, tool-sequence, step-count, goal-success), excessive-agency / rbac / tool-discovery plugins, trace feedback into jailbreak strategies (accessed 2026-09-04)
- promptfoo.dev/docs/guides/llm-redteaming — Red-team plugin catalogue mapped to OWASP LLM01/LLM08/LLM09, default grader gpt-5, HTTP endpoint targets, vulnerability report view (accessed 2026-09-04)
- github.com/promptfoo/promptfoo/blob/ff310fef/site/d… — CLI reference: `promptfoo mcp` server exposing list_evaluations / run_evaluation / redteam_run / redteam_generate to agents; redteam discover / poison / report subcommands; self-hosted and remote-generation env flags (accessed 2026-09-04)
- promptfoo.dev/docs/configuration/expected-outputs/s… — Embedding-based `similar` assertion: cosine, dot and euclidean metrics, text-embedding-3-large default, provider override per test or suite (accessed 2026-09-04)
- promptfoo.dev/docs/red-team/foundation-models — ModelAudit static scanner (`promptfoo scan-model`) for pickled-model code execution, unsafe Keras Lambda layers and encoded payloads; live foundation-model scans (accessed 2026-09-04)
- developers.openai.com/api/docs/deprecations — OpenAI's own Evals deprecation entry links 'Moving from OpenAI Evals to Promptfoo' as the migration path (accessed 2026-09-04)
- giskard.ai/knowledge/promptfoo-alternatives-ai-red-… — Independent competitor view: strong CI/CD and MCP plugin coverage, and the vendor-neutrality question engineering orgs now ask post-acquisition (accessed 2026-09-04)