IA-QA Eval & RAG — 38 tools is an MCP server published by io.github.JcJamet, hosted at www.ia-qa.com. When last checked on 2026-11-22 it was working: it answers and lists its tools. It declares 39 tools, including get_testing_guidelines, find_tool, hallucination_check, run_eval_contract, run_semantic_tests.
LLM & RAG evals: hallucination, eval contracts, semantic tests, tokens & model info. Free, no key.
13 connections through 5 shared values. Tap any node.
Squares are this server's values; dots are other servers sharing them. A shared address or publisher is a lead, not proof of a common operator.
Inferred from the host name and headers: what a server runs on, not who runs it.
Query the IA-QA methodology knowledge base. Returns structured testing guidelines, assertion strategies, thresholds, best practices, and relevant MCP tools for a given topic. Call without a topic to list all available topics. Topics: llm-unit-testing, rag-pipeline, prompt-stability, prompt-ab-testing, embedding-quality, eval-framework, semantic-testing, auto-testing, security, api-testing, ci-cd, multimodal, llm-data-security, agent-observability, pro-tips, learning-paths, golden-dataset, selector-drift, qa-recipes, playbooks. Not sure where to start testing an LLM, RAG pipeline or agent? Call without a topic (or with "start-here"): it maps what you are testing to the tools to call and the output field to gate CI on. A plain question such as "how do I test my RAG" also resolves to the right topic.
| Parameter | Type | Description |
|---|---|---|
| topic | string | The testing topic to retrieve guidelines for. Omit to get the start-here map and the full list of available topics. |
{
"name": "get_testing_guidelines",
"description": "Query the IA-QA methodology knowledge base. Returns structured testing guidelines, assertion strategies, thresholds, best practices, and relevant MCP tools for a given topic. Call without a topic to list all available topics. Topics: llm-unit-testing, rag-pipeline, prompt-stability, prompt-ab-testing, embedding-quality, eval-framework, semantic-testing, auto-testing, security, api-testing, ci-cd, multimodal, llm-data-security, agent-observability, pro-tips, learning-paths, golden-dataset, selector-drift, qa-recipes, playbooks. Not sure where to start testing an LLM, RAG pipeline or agent? Call without a topic (or with \"start-here\"): it maps what you are testing to the tools to call and the output field to gate CI on. A plain question such as \"how do I test my RAG\" also resolves to the right topic.",
"inputSchema": {
"type": "object",
"examples": [
{
"topic": "rag-pipeline"
}
],
"properties": {
"topic": {
"enum": [
"start-here",
"llm-unit-testing",
"rag-pipeline",
"prompt-stability",
"prompt-ab-testing",
"embedding-quality",
"eval-framework",
"semantic-testing",
"auto-testing",
"security",
"api-testing",
"ci-cd",
"multimodal",
"llm-data-security",
"agent-observability",
"pro-tips",
"learning-paths",
"golden-dataset",
"selector-drift",
"qa-recipes",
"playbooks"
],
"type": "string",
"description": "The testing topic to retrieve guidelines for. Omit to get the start-here map and the full list of available topics."
}
}
},
"outputSchema": {
"type": "object",
"properties": {
"tip": {
"type": "string"
},
"topic": {},
"usage": {
"type": "string"
},
"keywords": {
"type": "array"
},
"start_here": {
"type": "object"
},
"available_topics": {
"type": "array"
}
},
"additionalProperties": true
},
"annotations": {
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}
}Search available MCP tools by keyword or category before calling them. Returns names, descriptions and, on request, inputSchemas. Queries are logged 90 days, screened for credentials and personal data. Categories: data, dev, encoding, eval, integrations, llm, meta, qa, rag, security, text, web.
| Parameter | Type | Description |
|---|---|---|
| query * | string | Keywords, e.g. "cors", "token", "vector", "json" |
| category | string | Optional filter, one of the categories above. Any other value is refused. |
| max_results | number | Maximum tools to return (default 10, max 50). Best match first. |
| with_schema | boolean | Set true to include inputSchema in results (default: false) |
{
"name": "find_tool",
"description": "Search available MCP tools by keyword or category before calling them. Returns names, descriptions and, on request, inputSchemas. Queries are logged 90 days, screened for credentials and personal data. Categories: data, dev, encoding, eval, integrations, llm, meta, qa, rag, security, text, web.",
"inputSchema": {
"type": "object",
"examples": [
{
"query": "leaked api key"
}
],
"required": [
"query"
],
"properties": {
"query": {
"type": "string",
"description": "Keywords, e.g. \"cors\", \"token\", \"vector\", \"json\""
},
"category": {
"type": "string",
"description": "Optional filter, one of the categories above. Any other value is refused."
},
"max_results": {
"type": "number",
"description": "Maximum tools to return (default 10, max 50). Best match first."
},
"with_schema": {
"type": "boolean",
"description": "Set true to include inputSchema in results (default: false)"
}
}
},
"outputSchema": {
"type": "object",
"properties": {
"hint": {},
"tool": {},
"count": {
"type": "number"
},
"query": {},
"score": {},
"tools": {
"type": "array"
},
"category": {},
"truncated": {
"type": "boolean"
},
"total_matches": {
"type": "number"
}
},
"additionalProperties": true
},
"annotations": {
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}
}Lexical hallucination check: an LLM answer's words, numbers and polarity against the source. Deterministic, no API key. Each answer sentence is aligned to its best source sentence, so a number is support only on the SAME statement ("founded in 1998" is not grounded by "sold 1998 units"), and a negation or antonym flip against it returns verdict "contradicted". Lexical only: no paraphrase, synonym or multi-sentence reasoning, so "well_grounded" means nothing lexical was found, never "verified true". Entailment: run_semantic_tests (NLI/embedding) or a calibrated judge.
| Parameter | Type | Description |
|---|---|---|
| cases | array | Batch instead of answer + context: one result per case, and the worst gate overall. |
| answer | string | The LLM answer to verify (with context). |
| strict | boolean | Every sentence must be supported (default false). |
| context | string | The source text that should ground it. |
{
"name": "hallucination_check",
"description": "Lexical hallucination check: an LLM answer's words, numbers and polarity against the source. Deterministic, no API key. Each answer sentence is aligned to its best source sentence, so a number is support only on the SAME statement (\"founded in 1998\" is not grounded by \"sold 1998 units\"), and a negation or antonym flip against it returns verdict \"contradicted\". Lexical only: no paraphrase, synonym or multi-sentence reasoning, so \"well_grounded\" means nothing lexical was found, never \"verified true\". Entailment: run_semantic_tests (NLI/embedding) or a calibrated judge.",
"inputSchema": {
"type": "object",
"examples": [
{
"answer": "Le remboursement prend plus de 7 jours ouvrés.",
"context": "Le remboursement prend plus de 7 jours ouvrés."
}
],
"properties": {
"cases": {
"type": "array",
"items": {
"type": "object",
"required": [
"answer",
"context"
],
"properties": {
"id": {
"type": "string"
},
"answer": {
"type": "string"
},
"context": {
"type": "string"
}
}
},
"maxItems": 50,
"description": "Batch instead of answer + context: one result per case, and the worst gate overall."
},
"answer": {
"type": "string",
"description": "The LLM answer to verify (with context)."
},
"strict": {
"type": "boolean",
"description": "Every sentence must be supported (default false)."
},
"context": {
"type": "string",
"description": "The source text that should ground it."
}
}
},
"outputSchema": {
"type": "object",
"properties": {
"detail": {
"type": "string"
},
"message": {
"type": "string"
},
"overlap": {
"type": "number"
},
"verdict": {
"type": "string"
},
"grounded": {
"type": "boolean"
},
"total_words": {
"type": "number"
},
"matched_words": {
"type": "number"
},
"grounded_count": {
"type": "number"
},
"total_sentences": {
"type": "number"
},
"ungrounded_count": {
"type": "number"
}
},
"additionalProperties": true
},
"annotations": {
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}
}Run an IA-EVAL test suite (.ia-eval.yaml contract): call the model per scenario; return passed (true, false, null if undecided) and a verdict (PASS, FAIL, PARTIAL). Expectations are assertions that fail a scenario whatever its score; evaluators give the score. Hosted: pass inline_contract. validate_only calls no model.
| Parameter | Type | Description |
|---|---|---|
| api_keys | object | Your keys per provider. Only Groq falls back to the server key; any other provider needs yours. |
| overrides | object | Override contract defaults |
| contract_path | string | A .ia-eval.yaml under the server's working directory (local use). |
| validate_only | boolean | Check the contract against the IA-EVAL schema and stop: no model call, no key, deterministic. Returns valid + issues instead of throwing. Gate a PR that edits a contract on valid. |
| inline_contract | object | The contract as an object: "metadata" ({name, version, model?, provider?}), "expectations", "evaluators" and "scenarios" ([{id, input, expectations?}]). generate_eval_yaml scaffolds one. |
{
"name": "run_eval_contract",
"description": "Run an IA-EVAL test suite (.ia-eval.yaml contract): call the model per scenario; return passed (true, false, null if undecided) and a verdict (PASS, FAIL, PARTIAL). Expectations are assertions that fail a scenario whatever its score; evaluators give the score. Hosted: pass inline_contract. validate_only calls no model.",
"inputSchema": {
"type": "object",
"examples": [
{
"inline_contract": {
"metadata": {
"name": "smoke",
"version": "1.0"
},
"scenarios": [
{
"id": "capital",
"input": "Capital of France? One word.",
"ground_truth": "Paris"
}
],
"expectations": {
"contains": [
"Paris"
],
"min_score": 70
}
}
}
],
"properties": {
"api_keys": {
"type": "object",
"properties": {
"hf": {
"type": "string"
},
"xai": {
"type": "string"
},
"groq": {
"type": "string"
},
"google": {
"type": "string"
},
"openai": {
"type": "string"
},
"mistral": {
"type": "string"
},
"deepseek": {
"type": "string"
},
"anthropic": {
"type": "string"
}
},
"description": "Your keys per provider. Only Groq falls back to the server key; any other provider needs yours."
},
"overrides": {
"type": "object",
"properties": {
"model": {
"type": "string"
},
"provider": {
"type": "string"
},
"temperature": {
"type": "number"
},
"system_prompt": {
"type": "string"
}
},
"description": "Override contract defaults"
},
"contract_path": {
"type": "string",
"description": "A .ia-eval.yaml under the server's working directory (local use)."
},
"validate_only": {
"type": "boolean",
"description": "Check the contract against the IA-EVAL schema and stop: no model call, no key, deterministic. Returns valid + issues instead of throwing. Gate a PR that edits a contract on valid."
},
"inline_contract": {
"type": "object",
"description": "The contract as an object: \"metadata\" ({name, version, model?, provider?}), \"expectations\", \"evaluators\" and \"scenarios\" ([{id, input, expectations?}]). generate_eval_yaml scaffolds one.",
"additionalProperties": true
}
}
},
"outputSchema": {
"type": "object",
"properties": {
"valid": {
"type": "boolean"
},
"issues": {},
"summary": {
"type": "object"
},
"metadata": {
"type": "object"
},
"warnings": {},
"contract_path": {},
"scenario_results": {}
},
"additionalProperties": true
},
"annotations": {
"readOnlyHint": true,
"openWorldHint": true,
"idempotentHint": false,
"destructiveHint": false
}
}Semantic assertion primitive: compare actual vs expected text pairs using cosine similarity + ROUGE-L. Two modes: tfidf (default, free, no API key) or embeddings (OpenAI text-embedding-3-small, BYOK, true semantic similarity). Returns per-case PASS/FAIL verdicts and an overall verdict. CI-ready: pipe the JSON verdict field to gate a build.
| Parameter | Type | Description |
|---|---|---|
| mode | string | tfidf (default): fast, free, lexical. embeddings: OpenAI text-embedding-3-small, true semantic similarity, requires api_key. |
| cases * | array | Array of (actual, expected) pairs to evaluate. |
| api_key | string | OpenAI API key — required only when mode is embeddings. |
| thresholds | object | Pass/fail thresholds (defaults: cosine 0.75, rouge_l 0.5). |
| require_all | boolean | If true (default), all cases must pass for overall PASS. If false, at least one case passing returns PASS. |
{
"name": "run_semantic_tests",
"description": "Semantic assertion primitive: compare actual vs expected text pairs using cosine similarity + ROUGE-L. Two modes: tfidf (default, free, no API key) or embeddings (OpenAI text-embedding-3-small, BYOK, true semantic similarity). Returns per-case PASS/FAIL verdicts and an overall verdict. CI-ready: pipe the JSON verdict field to gate a build.",
"inputSchema": {
"type": "object",
"examples": [
{
"cases": [
{
"actual": "Paris is the capital of France",
"expected": "The capital of France is Paris"
}
]
}
],
"required": [
"cases"
],
"properties": {
"mode": {
"enum": [
"tfidf",
"embeddings"
],
"type": "string",
"description": "tfidf (default): fast, free, lexical. embeddings: OpenAI text-embedding-3-small, true semantic similarity, requires api_key."
},
"cases": {
"type": "array",
"items": {
"type": "object",
"required": [
"actual",
"expected"
],
"properties": {
"id": {
"type": "string",
"description": "Optional identifier for this case."
},
"actual": {
"type": "string",
"description": "The text produced by your LLM/system."
},
"expected": {
"type": "string",
"description": "The reference/ground-truth text."
}
}
},
"maxItems": 50,
"description": "Array of (actual, expected) pairs to evaluate."
},
"api_key": {
"type": "string",
"description": "OpenAI API key — required only when mode is embeddings."
},
"thresholds": {
"type": "object",
"properties": {
"cosine": {
"type": "number",
"maximum": 1,
"minimum": 0,
"description": "Minimum cosine similarity to pass (default: 0.75)."
},
"rouge_l": {
"type": "number",
"maximum": 1,
"minimum": 0,
"description": "Minimum ROUGE-L F1 to pass (default: 0.5)."
}
},
"description": "Pass/fail thresholds (defaults: cosine 0.75, rouge_l 0.5)."
},
"require_all": {
"type": "boolean",
"description": "If true (default), all cases must pass for overall PASS. If false, at least one case passing returns PASS."
}
}
},
"outputSchema": {
"type": "object",
"properties": {
"mode": {},
"cases": {},
"summary": {
"properties": {
"total": {},
"failed": {},
"passed": {},
"avg_cosine": {},
"avg_rouge_l": {}
}
},
"verdict": {},
"thresholds": {}
},
"additionalProperties": true
},
"annotations": {
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}
}Generate a ready-to-commit GitHub Actions workflow that gates a build on IA-QA: "eval_contract" runs a .ia-eval.yaml through ia-qa-com/eval-action@v1 (provider key as a repo secret), "cli_checks" runs deterministic tools via npx @ia-qa/cli and fails on their verdict, "selector_drift" gates E2E selectors with @ia-qa/self-healing. Deterministic template, no LLM; returns the YAML, the secrets and the next steps. Pair with generate_eval_yaml.
| Parameter | Type | Description |
|---|---|---|
| cron | string | For schedule (default '0 6 * * 1'). |
| gate | string | Default eval_contract. both = CLI checks + eval; all = CLI checks, drift, eval. |
| provider | string | Decides which repo secret is wired (default groq). |
| triggers | array | Default push + pull_request. |
| cli_tools | array | Tools to gate on, e.g. ["secret_scan","prompt_injection_scan"]. |
| min_score | number | Override the contract min_score (0-100). |
| app_base_url | string | [selector_drift] Default http://127.0.0.1:3000; must match .ia-qa/config.json. |
| fail_on_fail | boolean | Default true; false reports without gating. |
| node_version | string | Default "20". |
Generate a "needle in a haystack" test: embeds a target fact into a large block of filler text at a specified position. Use this to test LLM context window retrieval accuracy. Returns the full haystack, the question to ask, and metadata. No API key needed.
| Parameter | Type | Description |
|---|---|---|
| needle * | string | The fact to hide (e.g. "The secret code is ALPHA-42") |
| tokens | integer | Target haystack size in tokens (default: 5000, max: 100000) |
| position | string | Where to insert the needle: "start", "middle", "end", "random" (default: "middle") |
| question * | string | The question to ask the LLM (e.g. "What is the secret code?") |
{
"name": "needle_haystack_generate",
"description": "Generate a \"needle in a haystack\" test: embeds a target fact into a large block of filler text at a specified position. Use this to test LLM context window retrieval accuracy. Returns the full haystack, the question to ask, and metadata. No API key needed.",
"inputSchema": {
"type": "object",
"examples": [
{
"needle": "The code is 4242.",
"tokens": 2000,
"position": "middle",
"question": "What is the code?"
}
],
"required": [
"needle",
"question"
],
"properties": {
"needle": {
"type": "string",
"description": "The fact to hide (e.g. \"The secret code is ALPHA-42\")"
},
"tokens": {
"type": "integer",
"default": 5000,
"description": "Target haystack size in tokens (default: 5000, max: 100000)"
},
"position": {
"enum": [
"start",
"middle",
"end",
"random"
],
"type": "string",
"default": "middle",
"description": "Where to insert the needle: \"start\", \"middle\", \"end\", \"random\" (default: \"middle\")"
},
"question": {
"type": "string",
"description": "The question to ask the LLM (e.g. \"What is the secret code?\")"
}
}
},
"outputSchema": {
"type": "object",
"properties": {
"needle": {},
"haystack": {},
"position": {},
"question": {},
"insert_block": {},
"total_blocks": {
"type": "number"
},
"estimated_tokens": {}
},
"additionalProperties": true
},
"annotations": {
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": false,
"destructiveHint": false
}
}Generate a complete .ia-eval.yaml evaluation contract from a plain-language description of what your LLM should do. Uses Groq openai/gpt-oss-20b (server-side, no API key needed). Returns ready-to-run YAML for the LLM Test Runner (run_eval_contract). Picks appropriate evaluators (cosine_similarity, contains_check, hallucination_check, etc.) based on the task type.
| Parameter | Type | Description |
|---|---|---|
| task_type | string | Optional task type hint to guide evaluator selection. |
| description * | string | Plain-language description of what the LLM under test should do. Be specific: describe inputs, expected behaviour, and constraints. |
| system_prompt | string | Optional system prompt of the LLM under test. Helps generate more accurate test cases. |
| scenario_count | number | Number of scenarios to generate (default: 5). Covers happy path + edge cases + adversarial. |
{
"name": "generate_eval_yaml",
"description": "Generate a complete .ia-eval.yaml evaluation contract from a plain-language description of what your LLM should do. Uses Groq openai/gpt-oss-20b (server-side, no API key needed). Returns ready-to-run YAML for the LLM Test Runner (run_eval_contract). Picks appropriate evaluators (cosine_similarity, contains_check, hallucination_check, etc.) based on the task type.",
"inputSchema": {
"type": "object",
"examples": [
{
"description": "Summarize a customer support ticket in one sentence."
}
],
"required": [
"description"
],
"properties": {
"task_type": {
"enum": [
"rag",
"summarization",
"classification",
"safety",
"customer_support",
"code_gen"
],
"type": "string",
"description": "Optional task type hint to guide evaluator selection."
},
"description": {
"type": "string",
"description": "Plain-language description of what the LLM under test should do. Be specific: describe inputs, expected behaviour, and constraints."
},
"system_prompt": {
"type": "string",
"description": "Optional system prompt of the LLM under test. Helps generate more accurate test cases."
},
"scenario_count": {
"enum": [
3,
5,
8
],
"type": "number",
"description": "Number of scenarios to generate (default: 5). Covers happy path + edge cases + adversarial."
}
}
},
"outputSchema": {
"type": "object",
"properties": {
"yaml": {
"type": "string"
},
"task_type": {
"type": "string"
},
"model_used": {
"type": "string"
},
"scenario_count": {
"type": "number"
}
},
"additionalProperties": true
},
"annotations": {
"readOnlyHint": true,
"openWorldHint": true,
"idempotentHint": false,
"destructiveHint": false
}
}Reference-free stability check: does an assistant's answer stay the same when the QUESTION is transformed (case, typo, paraphrase, reorder, translation)? You bring the outputs; no model is called (tfidf mode is deterministic and free). Default thresholds: case .95, typo .90, paraphrase .80, reorder .80, translation .75 (embeddings only), specialization .60 (advisory, never gated). Returns PASS / FAIL / INVALID — INVALID when the BASE answer is a refusal or too short, so a model that refuses everything never scores 1.0. Pair with run_semantic_tests for correctness.
| Parameter | Type | Description |
|---|---|---|
| base * | object | The original run. |
| mode | string | tfidf (default, lexical: gate on case/typo, read paraphrase as a trend) or embeddings (OpenAI, needs api_key; required for translation). |
| api_key | string | OpenAI key, embeddings mode only. |
| variants * | array | Answers to transformed versions of the question. |
| thresholds | object | Per-relation overrides, 0-1. Calibrate on your own data before gating. |
| require_all | boolean | Default true: every gated variant must pass. Keep it for a gate — false lets one easy typo row hold the run at PASS while a paraphrase fails. |
| baseline_guard | object | Floor on the BASE answer; failing it returns INVALID. |
Test a SKILL.md. No model, every call: lint (gate on lint.valid) and review (description lacks what/when, words shared with other_skills, leftovers, no done-check, missing files; why + fix each). Unless lint_only, a model picks a skill per request from names+descriptions: trigger_accuracy PASS ≥80, FIX ≥50, else BLOCK; your scenarios = gate, generated = signal; unanswered = INCONCLUSIVE. Server key 10 runs/day.
| Parameter | Type | Description |
|---|---|---|
| files | array | Paths in its folder. |
| model | string | Default openai/gpt-oss-20b. |
| api_key | string | Used once, not stored. |
| dir_name | string | Its folder. |
| skill_md * | string | Full SKILL.md, frontmatter included. |
| lint_only | boolean | |
| scenarios | array | Max 20, near misses included. |
| other_skills | array | Installed next to it. |
| scenario_count | number | one of [4,6,8,10] |
Help a person write a SKILL.md they can test. No answers: the questions (why + example), written for the person; invented answers make the result meaningless. With answers: remarks per answer (no model), the SKILL.md, scenarios for test_skill, a starter .ia-eval.yaml.
| Parameter | Type | Description |
|---|---|---|
| name | string | |
| answers | object | Keyed by question id; each step can be sent alone. |
{
"name": "skill_interview",
"description": "Help a person write a SKILL.md they can test. No answers: the questions (why + example), written for the person; invented answers make the result meaningless. With answers: remarks per answer (no model), the SKILL.md, scenarios for test_skill, a starter .ia-eval.yaml.",
"inputSchema": {
"type": "object",
"examples": [
{}
],
"properties": {
"name": {
"type": "string"
},
"answers": {
"type": "object",
"properties": {
"bad": {
"type": "string"
},
"job": {
"type": "string"
},
"good": {
"type": "string"
},
"input": {
"type": "string"
},
"never": {
"type": "string"
},
"steps": {
"type": "string"
},
"format": {
"type": "string"
},
"output": {
"type": "string"
},
"context": {
"type": "string"
},
"unclear": {
"type": "string"
},
"boundary": {
"type": "string"
},
"requests": {
"type": "string"
}
},
"description": "Keyed by question id; each step can be sent alone."
}
}
},
"outputSchema": {
"type": "object",
"additionalProperties": true
},
"annotations": {
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}
}Validate an LLM response against QA criteria: format checks (JSON, code, markdown), content rules (must-include, must-not-include), length constraints, language detection, and safety patterns. Essential for QA testing LLM-powered features.
| Parameter | Type | Description |
|---|---|---|
| output * | string | The LLM output text to validate |
| max_length | number | Maximum character length for the output |
| min_length | number | Minimum character length for the output |
| check_safety | boolean | Check for PII patterns (emails, phones, SSN), profanity signals, and prompt leakage |
| must_include | string | Comma-separated strings that MUST appear in the output |
| expected_format | string | Expected output format |
| must_not_include | string | Comma-separated strings that must NOT appear (e.g. "TODO, FIXME, undefined, NaN") |
| check_json_schema | string | If expected_format is JSON, provide required keys as comma-separated list to validate the structure |
| expected_language | string | Expected language of the output (en, fr, es, de…). Checks for common words. |
Compare two ALREADY-PRODUCED outputs (e.g. model A vs model B on the same task) side by side. Returns deterministic metrics (token cosine, ROUGE-L, Jaccard, length/structure deltas, JSON diff) and a verdict. If a `reference` (ground truth) is given, scores each output against it and picks the closer one. If `model` + `api_key` are given, an LLM judge also picks a qualitative winner for the task. No re-execution — you bring the outputs.
| Parameter | Type | Description |
|---|---|---|
| task | string | The task/prompt both outputs were answering — used by the LLM judge for context |
| model | string | Optional judge model id (BYOK). When set with api_key, an LLM judge picks a qualitative winner. |
| api_key | string | Optional API key for the judge model (BYOK). Used only for the judge call; never stored. |
| label_a | string | Label for output A (e.g. "GPT-4o", "v1.0") |
| label_b | string | Label for output B (e.g. "GPT-5-nano", "v1.1") |
| reference | string | Optional ground-truth / expected answer. If set, each output is scored against it and the closer one wins (deterministic). |
| check_json | boolean | Try to parse as JSON and compare structurally (keys, types, values) |
| response_a * | string | First output (e.g. model A's answer) |
Semantically analyze N already-produced model outputs for the SAME task (the MCP counterpart to the LLM Sandbox). Without a reference: computes consensus — pairwise cosine agreement, the most-representative output, and the outlier. With a `reference` (ground truth): also ranks every output by closeness (token cosine + ROUGE-L composite) and names the closest. No LLM, no key, runs locally. The reference mode is gate-able in CI; the consensus/outlier mode is a signal (no reference to fail against). You bring the outputs (2+). A 2-way head-to-head with structural JSON diff is compare_responses.
| Parameter | Type | Description |
|---|---|---|
| reference | string | Optional ground-truth answer. If set, each output is also ranked by closeness to it and the closest one is named. |
| responses * | array | Outputs to analyze (same task, N models/prompts/versions): plain strings or { "label": "GPT-4o", "text": "..." }. At least 2. |
{
"name": "analyze_responses",
"description": "Semantically analyze N already-produced model outputs for the SAME task (the MCP counterpart to the LLM Sandbox). Without a reference: computes consensus — pairwise cosine agreement, the most-representative output, and the outlier. With a `reference` (ground truth): also ranks every output by closeness (token cosine + ROUGE-L composite) and names the closest. No LLM, no key, runs locally. The reference mode is gate-able in CI; the consensus/outlier mode is a signal (no reference to fail against). You bring the outputs (2+). A 2-way head-to-head with structural JSON diff is compare_responses.",
"inputSchema": {
"type": "object",
"examples": [
{
"reference": "Paris is the capital of France.",
"responses": [
{
"text": "Lyon is the capital.",
"label": "A"
},
{
"text": "Paris is the capital of France.",
"label": "B"
}
]
}
],
"required": [
"responses"
],
"properties": {
"reference": {
"type": "string",
"description": "Optional ground-truth answer. If set, each output is also ranked by closeness to it and the closest one is named."
},
"responses": {
"type": "array",
"items": {
"type": "object",
"required": [
"text"
],
"properties": {
"text": {
"type": "string",
"description": "The produced output"
},
"label": {
"type": "string",
"description": "Human name for this candidate (e.g. model id)"
}
}
},
"minItems": 2,
"description": "Outputs to analyze (same task, N models/prompts/versions): plain strings or { \"label\": \"GPT-4o\", \"text\": \"...\" }. At least 2."
}
}
},
"outputSchema": {
"type": "object",
"properties": {
"count": {},
"summary": {
"type": "string"
},
"consensus": {},
"reference_ranking": {}
},
"additionalProperties": true
},
"annotations": {
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}
}Define a test suite for a prompt: provide the system prompt, user prompt, and expected output criteria. Returns a test plan with scored rubric — use this as input for manual or automated LLM evaluation.
| Parameter | Type | Description |
|---|---|---|
| max_tokens | number | Max token budget for the test |
| temperature | number | Temperature to use |
| user_prompt * | string | The user prompt to send |
| check_safety | boolean | Include safety/PII checks in the rubric |
| must_include | string | Required content (comma-separated) |
| system_prompt * | string | The system prompt under test |
| expected_format | string | Expected output format |
| must_not_include | string | Forbidden content (comma-separated) |
| expected_behavior | string | Description of what the LLM should do (free text) |
| adversarial_prompts | boolean | Auto-generate adversarial test variants (jailbreak, injection, edge cases) |
Validate that an LLM output matches an expected format: JSON, Markdown, code block, bullet list, numbered list, table, YAML, XML, or custom regex. Essential for structured output testing.
| Parameter | Type | Description |
|---|---|---|
| output * | string | The LLM output to validate |
| regex_pattern | string | Custom regex pattern (only when expected_format is "regex") |
| expected_format * | string | Expected format |
{
"name": "llm_format_check",
"description": "Validate that an LLM output matches an expected format: JSON, Markdown, code block, bullet list, numbered list, table, YAML, XML, or custom regex. Essential for structured output testing.",
"inputSchema": {
"type": "object",
"examples": [
{
"output": "a: 1\nb: [1, 2]",
"expected_format": "yaml"
}
],
"required": [
"output",
"expected_format"
],
"properties": {
"output": {
"type": "string",
"description": "The LLM output to validate"
},
"regex_pattern": {
"type": "string",
"description": "Custom regex pattern (only when expected_format is \"regex\")"
},
"expected_format": {
"enum": [
"json",
"markdown_heading",
"code_block",
"bullet_list",
"numbered_list",
"table",
"yaml",
"xml",
"regex"
],
"type": "string",
"description": "Expected format"
}
}
},
"outputSchema": {
"type": "object",
"properties": {
"valid": {
"type": "boolean"
},
"checks": {},
"failed": {},
"passed": {},
"total_checks": {
"type": "number"
},
"expected_format": {}
},
"additionalProperties": true
},
"annotations": {
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}
}Compare multiple LLM responses to the same prompt and detect inconsistencies using Jaccard word-overlap similarity and fact drift (number comparison). Fast, deterministic, no API key needed. Limitations: relies on surface-level word matching — "Paris is the capital of France" vs "Paris is the French capital" may score low despite semantic equivalence. True semantic consistency: run_semantic_tests in embedding mode. Essential for determinism testing.
| Parameter | Type | Description |
|---|---|---|
| responses * | array | Array of 2+ LLM responses to compare (same prompt, different runs) |
| check_facts | boolean | Check for contradictory numbers/facts across responses (default: true) |
{
"name": "consistency_check",
"description": "Compare multiple LLM responses to the same prompt and detect inconsistencies using Jaccard word-overlap similarity and fact drift (number comparison). Fast, deterministic, no API key needed. Limitations: relies on surface-level word matching — \"Paris is the capital of France\" vs \"Paris is the French capital\" may score low despite semantic equivalence. True semantic consistency: run_semantic_tests in embedding mode. Essential for determinism testing.",
"inputSchema": {
"type": "object",
"examples": [
{
"responses": [
"The capital of France is Paris.",
"Paris is the capital of France."
]
}
],
"required": [
"responses"
],
"properties": {
"responses": {
"type": "array",
"items": {
"type": "string"
},
"description": "Array of 2+ LLM responses to compare (same prompt, different runs)"
},
"check_facts": {
"type": "boolean",
"description": "Check for contradictory numbers/facts across responses (default: true)"
}
}
},
"outputSchema": {
"type": "object",
"properties": {
"response_count": {
"type": "number"
}
},
"additionalProperties": true
},
"annotations": {
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}
}Validate that an LLM JSON output matches a JSON Schema definition. Tests required fields, types, enums, nested objects, and arrays. Critical for function-calling and structured output testing.
| Parameter | Type | Description |
|---|---|---|
| output * | string | The LLM JSON output (raw string, will be parsed) |
| schema * | object | JSON Schema (draft-07 subset) to validate against |
{
"name": "llm_json_schema_check",
"description": "Validate that an LLM JSON output matches a JSON Schema definition. Tests required fields, types, enums, nested objects, and arrays. Critical for function-calling and structured output testing.",
"inputSchema": {
"type": "object",
"examples": [
{
"output": "{\"name\":\"a\",\"age\":3}",
"schema": {
"type": "object",
"required": [
"name",
"age"
],
"properties": {
"age": {
"type": "integer",
"minimum": 0
},
"name": {
"type": "string"
},
"role": {
"enum": [
"admin",
"user"
]
}
}
}
}
],
"required": [
"output",
"schema"
],
"properties": {
"output": {
"type": "string",
"description": "The LLM JSON output (raw string, will be parsed)"
},
"schema": {
"type": "object",
"description": "JSON Schema (draft-07 subset) to validate against",
"additionalProperties": true
}
}
},
"outputSchema": {
"type": "object",
"properties": {
"valid": {
"type": "boolean"
},
"errors": {
"type": "array"
},
"error_count": {
"type": "number"
},
"parse_error": {},
"parsed_type": {}
},
"additionalProperties": true
},
"annotations": {
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}
}Score an LLM response against the criteria you pass: coverage of expected_keywords and compliance with max_length. Returns a 0-100 score over the criteria actually measured — or total_score null with the reason when none is given, because relevance and correctness cannot be read off word overlap (a correct one-word answer shares no words with its question). Meaning against a reference answer: run_semantic_tests. Also returns unscored signals: question-term overlap, average sentence length, markdown.
| Parameter | Type | Description |
|---|---|---|
| question * | string | The original question/prompt |
| response * | string | The LLM response to score |
| max_length | number | Maximum character length. Scored as a proportional penalty beyond it. |
| expected_keywords | array | Keywords a correct answer must contain, as whole words (case-insensitive; a plural in -s/-es counts). Scored as coverage. |
{
"name": "response_quality_score",
"description": "Score an LLM response against the criteria you pass: coverage of expected_keywords and compliance with max_length. Returns a 0-100 score over the criteria actually measured — or total_score null with the reason when none is given, because relevance and correctness cannot be read off word overlap (a correct one-word answer shares no words with its question). Meaning against a reference answer: run_semantic_tests. Also returns unscored signals: question-term overlap, average sentence length, markdown.",
"inputSchema": {
"type": "object",
"examples": [
{
"question": "q",
"response": "anything"
}
],
"required": [
"question",
"response"
],
"properties": {
"question": {
"type": "string",
"description": "The original question/prompt"
},
"response": {
"type": "string",
"description": "The LLM response to score"
},
"max_length": {
"type": "number",
"description": "Maximum character length. Scored as a proportional penalty beyond it."
},
"expected_keywords": {
"type": "array",
"items": {
"type": "string"
},
"description": "Keywords a correct answer must contain, as whole words (case-insensitive; a plural in -s/-es counts). Scored as coverage."
}
}
},
"outputSchema": {
"type": "object",
"properties": {
"grade": {},
"stats": {
"type": "object"
},
"reason": {
"type": "string"
},
"signals": {
"type": "object"
},
"breakdown": {
"type": "object"
},
"max_score": {
"type": "number"
},
"total_score": {},
"not_measured": {
"type": "array"
}
},
"additionalProperties": true
},
"annotations": {
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}
}Analyze a multi-turn conversation for context retention, topic drift, instruction following, and repetition. Accepts messages array [{role, content}]. Essential for chatbot QA.
| Parameter | Type | Description |
|---|---|---|
| messages * | array | Conversation messages in order |
{
"name": "conversation_analyze",
"description": "Analyze a multi-turn conversation for context retention, topic drift, instruction following, and repetition. Accepts messages array [{role, content}]. Essential for chatbot QA.",
"inputSchema": {
"type": "object",
"examples": [
{
"messages": [
{
"role": "user",
"content": "What is your refund policy?"
},
{
"role": "assistant",
"content": "Refunds are available within 30 days."
}
]
}
],
"required": [
"messages"
],
"properties": {
"messages": {
"type": "array",
"items": {
"type": "object",
"properties": {
"role": {
"enum": [
"system",
"user",
"assistant"
],
"type": "string"
},
"content": {
"type": "string"
}
}
},
"description": "Conversation messages in order"
}
}
},
"outputSchema": {
"type": "object",
"properties": {
"turn_count": {
"type": "number"
},
"repetitions": {},
"topic_drift": {},
"user_messages": {
"type": "number"
},
"context_retention": {},
"has_system_prompt": {
"type": "boolean"
},
"assistant_messages": {
"type": "number"
},
"avg_response_length": {
"type": "number"
},
"repetition_detected": {
"type": "boolean"
}
},
"additionalProperties": true
},
"annotations": {
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}
}Analyze a Chain-of-Thought (CoT) or reasoning trace from an LLM. Detects step count, logical flow, conclusion presence, backtracking, and estimates reasoning depth.
| Parameter | Type | Description |
|---|---|---|
| reasoning * | string | The CoT / reasoning trace text (e.g. from <think> tags or step-by-step output) |
| expected_conclusion | string | Expected final answer to check against (optional) |
{
"name": "cot_analyzer",
"description": "Analyze a Chain-of-Thought (CoT) or reasoning trace from an LLM. Detects step count, logical flow, conclusion presence, backtracking, and estimates reasoning depth.",
"inputSchema": {
"type": "object",
"examples": [
{
"reasoning": "Step 1: 12 apples minus 5 eaten.\nStep 2: 12 - 5 = 7.\nTherefore, the answer is 7.",
"expected_conclusion": "7"
}
],
"required": [
"reasoning"
],
"properties": {
"reasoning": {
"type": "string",
"description": "The CoT / reasoning trace text (e.g. from <think> tags or step-by-step output)"
},
"expected_conclusion": {
"type": "string",
"description": "Expected final answer to check against (optional)"
}
}
},
"outputSchema": {
"type": "object",
"properties": {
"step_count": {
"type": "number"
},
"total_chars": {
"type": "number"
},
"total_lines": {
"type": "number"
},
"has_conclusion": {
"type": "boolean"
}
},
"additionalProperties": true
},
"annotations": {
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}
}Generate an A/B test report comparing two prompts or model configurations. Accepts arrays of scores and returns mean, median, std deviation, improvement percentage, and a Welch t-test at 95 %: `winner` names a variant only when the difference is significant — otherwise "no_significant_difference", or "insufficient_samples" under 2 scores per variant.
| Parameter | Type | Description |
|---|---|---|
| variant_a * | object | First variant configuration with name and score array |
| variant_b * | object | Second variant configuration with name and score array |
{
"name": "ab_test_report",
"description": "Generate an A/B test report comparing two prompts or model configurations. Accepts arrays of scores and returns mean, median, std deviation, improvement percentage, and a Welch t-test at 95 %: `winner` names a variant only when the difference is significant — otherwise \"no_significant_difference\", or \"insufficient_samples\" under 2 scores per variant.",
"inputSchema": {
"type": "object",
"examples": [
{
"variant_a": {
"name": "A",
"scores": [
70,
71,
69
]
},
"variant_b": {
"name": "B",
"scores": [
70,
71,
69
]
}
}
],
"required": [
"variant_a",
"variant_b"
],
"properties": {
"variant_a": {
"type": "object",
"properties": {
"name": {
"type": "string",
"description": "Name/label for variant A"
},
"scores": {
"type": "array",
"items": {
"type": "number"
},
"description": "Array of scores (0-100)"
}
},
"description": "First variant configuration with name and score array"
},
"variant_b": {
"type": "object",
"properties": {
"name": {
"type": "string",
"description": "Name/label for variant B"
},
"scores": {
"type": "array",
"items": {
"type": "number"
},
"description": "Array of scores (0-100)"
}
},
"description": "Second variant configuration with name and score array"
}
}
},
"outputSchema": {
"type": "object",
"properties": {
"max": {},
"min": {},
"mean": {
"type": "string"
},
"count": {
"type": "number"
},
"median": {
"type": "string"
},
"winner": {},
"std_dev": {
"type": "string"
},
"variant_a": {
"type": "object"
},
"variant_b": {
"type": "object"
},
"recommendation": {
"type": "number"
},
"improvement_percent": {}
},
"additionalProperties": true
},
"annotations": {
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}
}Compute text similarity between reference and hypothesis using multiple metrics: Cosine (BoW, TF-IDF), Jaccard, ROUGE-1, ROUGE-2, ROUGE-L, and BLEU. No API key needed. Supports batch mode.
| Parameter | Type | Description |
|---|---|---|
| batch | array | Batch mode: array of {reference, hypothesis} pairs. |
| metrics | array | Metrics to compute (default: all). Options: "cosine_bow", "cosine_tfidf", "jaccard", "rouge1", "rouge2", "rougeL", "bleu" |
| reference | string | Reference / expected text (ground truth) |
| threshold | number | Optional pass/fail threshold (0-1). Applies to ROUGE-L F1 score. |
| hypothesis | string | Hypothesis / actual text (LLM output) |
{
"name": "similarity_score",
"description": "Compute text similarity between reference and hypothesis using multiple metrics: Cosine (BoW, TF-IDF), Jaccard, ROUGE-1, ROUGE-2, ROUGE-L, and BLEU. No API key needed. Supports batch mode.",
"inputSchema": {
"type": "object",
"examples": [
{
"reference": "the cat sat on the mat",
"threshold": 0.5,
"hypothesis": "the cat sat on the mat"
}
],
"properties": {
"batch": {
"type": "array",
"items": {
"type": "object",
"required": [
"reference",
"hypothesis"
],
"properties": {
"reference": {
"type": "string"
},
"hypothesis": {
"type": "string"
}
}
},
"description": "Batch mode: array of {reference, hypothesis} pairs."
},
"metrics": {
"type": "array",
"items": {
"type": "string"
},
"description": "Metrics to compute (default: all). Options: \"cosine_bow\", \"cosine_tfidf\", \"jaccard\", \"rouge1\", \"rouge2\", \"rougeL\", \"bleu\""
},
"reference": {
"type": "string",
"description": "Reference / expected text (ground truth)"
},
"threshold": {
"type": "number",
"description": "Optional pass/fail threshold (0-1). Applies to ROUGE-L F1 score."
},
"hypothesis": {
"type": "string",
"description": "Hypothesis / actual text (LLM output)"
}
}
},
"outputSchema": {
"type": "object",
"properties": {
"f1": {
"type": "number"
},
"mode": {
"type": "string"
},
"count": {
"type": "number"
},
"recall": {
"type": "number"
},
"results": {},
"precision": {
"type": "number"
}
},
"additionalProperties": true
},
"annotations": {
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}
}Test a vision model (OpenAI, Anthropic or Gemini) on one image: each test case asks a question and asserts on the answer (contains, not_contains, json_format, min_length, max_length, semantic_contains = TF-IDF cosine ≥ 0.4). Returns per-case PASS/FAIL, the pass rate, `overall` PASS/WARNING/FAIL against threshold (N/A when no assertion could run), and latency. BYOK: your key for the model's provider; the image is fetched only from a public address.
| Parameter | Type | Description |
|---|---|---|
| model * | string | one of ["gpt-4o","gpt-4o-mini","claude-sonnet-4-6","claude-haiku-4-5-20251001","gemini-3.5-flash","gemini-2.5-flash"] |
| api_key * | string | Key for the model's provider. |
| image_url | string | Public image URL (or image_base64). |
| threshold | number | Pass rate for an overall PASS (default 80). |
| test_cases * | array | |
| image_base64 | string | |
| system_prompt | string | |
| image_mime_type | string | For image_base64 (default image/jpeg). |
Compute text similarity using local algorithms (Bag of Words, TF-IDF, Character N-grams). No API key needed — runs entirely in-process. NOT real embeddings: true vector embeddings are run_semantic_tests with mode="embeddings" and an OpenAI key. Supports single pair or batch mode with pipe-separated pairs.
| Parameter | Type | Description |
|---|---|---|
| batch | array | Batch mode: array of { text_a, text_b } pairs. Overrides text_a/text_b if provided. |
| text_a | string | First text to compare (single-pair mode) |
| text_b | string | Second text to compare (single-pair mode) |
| methods | array | Algorithms to use (default: all three). Options: "bow", "tfidf", "ngram" |
{
"name": "embedding_similarity",
"description": "Compute text similarity using local algorithms (Bag of Words, TF-IDF, Character N-grams). No API key needed — runs entirely in-process. NOT real embeddings: true vector embeddings are run_semantic_tests with mode=\"embeddings\" and an OpenAI key. Supports single pair or batch mode with pipe-separated pairs.",
"inputSchema": {
"type": "object",
"examples": [
{
"text_a": "the cat sat",
"text_b": "quantum field theory"
}
],
"properties": {
"batch": {
"type": "array",
"items": {
"type": "object",
"required": [
"text_a",
"text_b"
],
"properties": {
"text_a": {
"type": "string"
},
"text_b": {
"type": "string"
}
}
},
"description": "Batch mode: array of { text_a, text_b } pairs. Overrides text_a/text_b if provided."
},
"text_a": {
"type": "string",
"description": "First text to compare (single-pair mode)"
},
"text_b": {
"type": "string",
"description": "Second text to compare (single-pair mode)"
},
"methods": {
"type": "array",
"items": {
"enum": [
"bow",
"tfidf",
"ngram"
],
"type": "string"
},
"description": "Algorithms to use (default: all three). Options: \"bow\", \"tfidf\", \"ngram\""
}
}
},
"outputSchema": {
"type": "object",
"properties": {
"mode": {
"type": "string"
},
"count": {
"type": "number"
},
"scores": {},
"text_a": {},
"text_b": {},
"results": {}
},
"additionalProperties": true
},
"annotations": {
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}
}Rank an array of text chunks by relevance to a query using TF-IDF scoring. Simulates retrieval ranking for RAG testing without needing embeddings or an API.
| Parameter | Type | Description |
|---|---|---|
| query * | string | The user query |
| top_k | number | Return top K results (default: all) |
| chunks * | array | Array of text chunks to rank |
{
"name": "rag_relevance_rank",
"description": "Rank an array of text chunks by relevance to a query using TF-IDF scoring. Simulates retrieval ranking for RAG testing without needing embeddings or an API.",
"inputSchema": {
"type": "object",
"examples": [
{
"query": "quantum",
"chunks": [
"a b c",
"d e f"
]
}
],
"required": [
"query",
"chunks"
],
"properties": {
"query": {
"type": "string",
"description": "The user query"
},
"top_k": {
"type": "number",
"description": "Return top K results (default: all)"
},
"chunks": {
"type": "array",
"items": {
"type": "string"
},
"description": "Array of text chunks to rank"
}
}
},
"outputSchema": {
"type": "object",
"properties": {
"rank": {
"type": "number"
},
"index": {},
"query": {},
"score": {
"type": "string"
},
"results": {},
"returned": {
"type": "number"
},
"total_chunks": {
"type": "number"
},
"chunk_preview": {},
"keyword_overlap": {}
},
"additionalProperties": true
},
"annotations": {
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}
}Compute BM25 relevance score between a query and one or more documents. BM25 is the industry-standard keyword-based ranking algorithm used in Elasticsearch, OpenSearch, and Weaviate hybrid search. Returns ranked results with normalized scores.
| Parameter | Type | Description |
|---|---|---|
| b | number | Length normalization factor (default: 0.75) |
| k1 | number | Term frequency saturation (default: 1.5) |
| query * | string | The search query |
| top_k | number | Return top K results (default: all) |
| documents * | array | Array of documents to rank |
{
"name": "bm25_score",
"description": "Compute BM25 relevance score between a query and one or more documents. BM25 is the industry-standard keyword-based ranking algorithm used in Elasticsearch, OpenSearch, and Weaviate hybrid search. Returns ranked results with normalized scores.",
"inputSchema": {
"type": "object",
"examples": [
{
"query": "refund damaged",
"documents": [
"shipping times",
"refund for damaged items",
"office hours"
]
}
],
"required": [
"query",
"documents"
],
"properties": {
"b": {
"type": "number",
"description": "Length normalization factor (default: 0.75)"
},
"k1": {
"type": "number",
"description": "Term frequency saturation (default: 1.5)"
},
"query": {
"type": "string",
"description": "The search query"
},
"top_k": {
"type": "number",
"description": "Return top K results (default: all)"
},
"documents": {
"type": "array",
"items": {
"type": "string"
},
"description": "Array of documents to rank"
}
}
},
"outputSchema": {
"type": "object",
"properties": {
"b": {},
"k1": {},
"index": {},
"query": {},
"results": {},
"bm25_score": {
"type": "number"
},
"doc_length": {
"type": "number"
},
"doc_preview": {},
"avg_doc_length": {
"type": "number"
},
"documents_count": {
"type": "number"
}
},
"additionalProperties": true
},
"annotations": {
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}
}Assemble a complete RAG (Retrieval-Augmented Generation) prompt from retrieved context chunks and a user query. Handles token budgeting, citation numbering, system instruction injection, and source attribution.
| Parameter | Type | Description |
|---|---|---|
| query * | string | The user question to answer |
| chunks * | array | Retrieved context chunks with .text (required), .source (optional), .score (optional) |
| language | string | Response language instruction (e.g. "French", "Spanish") |
| cite_sources | boolean | Add [1], [2] citation numbers (default: true) |
| max_context_tokens | number | Max tokens for context section (default: 2000) |
| system_instruction | string | Custom system instruction (default: standard RAG grounding instruction) |
{
"name": "build_rag_prompt",
"description": "Assemble a complete RAG (Retrieval-Augmented Generation) prompt from retrieved context chunks and a user query. Handles token budgeting, citation numbering, system instruction injection, and source attribution.",
"inputSchema": {
"type": "object",
"examples": [
{
"query": "q?",
"chunks": [
{
"text": "c1 text",
"source": "a.md"
}
]
}
],
"required": [
"query",
"chunks"
],
"properties": {
"query": {
"type": "string",
"description": "The user question to answer"
},
"chunks": {
"type": "array",
"items": {
"type": "object",
"properties": {
"text": {
"type": "string"
},
"score": {
"type": "number"
},
"source": {
"type": "string"
}
}
},
"description": "Retrieved context chunks with .text (required), .source (optional), .score (optional)"
},
"language": {
"type": "string",
"description": "Response language instruction (e.g. \"French\", \"Spanish\")"
},
"cite_sources": {
"type": "boolean",
"description": "Add [1], [2] citation numbers (default: true)"
},
"max_context_tokens": {
"type": "number",
"description": "Max tokens for context section (default: 2000)"
},
"system_instruction": {
"type": "string",
"description": "Custom system instruction (default: standard RAG grounding instruction)"
}
}
},
"outputSchema": {
"type": "object",
"properties": {
"prompt": {},
"system_prompt": {},
"chunks_included": {
"type": "number"
},
"included_chunks": {},
"chunks_truncated": {
"type": "number"
},
"total_tokens_estimate": {
"type": "number"
},
"context_tokens_estimate": {
"type": "number"
}
},
"additionalProperties": true
},
"annotations": {
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}
}Evaluate RAG retrieval: rank passages against a query, compute Precision@k / Recall@k and a PASS/FAIL verdict from your relevance labels (INCONCLUSIVE without them). Modes: (1) give every passage the `score` YOUR reranker produced — deterministic, offline, the mode to gate CI on; (2) no scores, no key — a local BM25 keyword baseline, never a judgement of a neural reranker; (3) api_key for an NVIDIA account that still has reranking (NVIDIA retired the hosted endpoints on 2026-05-18).
| Parameter | Type | Description |
|---|---|---|
| query * | string | |
| top_k | integer | Default 3. |
| api_key | string | NVIDIA key, used only when no passage has a score; never stored. |
| passages * | array | 2 to 20 passages. |
| threshold | number | Minimum Precision@k to PASS (default 0.5). |
{
"name": "rerank_evaluate",
"description": "Evaluate RAG retrieval: rank passages against a query, compute Precision@k / Recall@k and a PASS/FAIL verdict from your relevance labels (INCONCLUSIVE without them). Modes: (1) give every passage the `score` YOUR reranker produced — deterministic, offline, the mode to gate CI on; (2) no scores, no key — a local BM25 keyword baseline, never a judgement of a neural reranker; (3) api_key for an NVIDIA account that still has reranking (NVIDIA retired the hosted endpoints on 2026-05-18).",
"inputSchema": {
"type": "object",
"examples": [
{
"query": "capital of France",
"api_key": "<your-api-key>",
"passages": [
{
"text": "Paris is the capital of France.",
"relevant": true
},
{
"text": "Bananas are rich in potassium.",
"relevant": false
}
]
}
],
"required": [
"query",
"passages"
],
"properties": {
"query": {
"type": "string"
},
"top_k": {
"type": "integer",
"maximum": 10,
"minimum": 1,
"description": "Default 3."
},
"api_key": {
"type": "string",
"description": "NVIDIA key, used only when no passage has a score; never stored."
},
"passages": {
"type": "array",
"items": {
"type": "object",
"required": [
"text"
],
"properties": {
"id": {
"type": "string"
},
"text": {
"type": "string"
},
"score": {
"type": "number",
"description": "Your reranker's score (higher = more relevant). On all passages or none."
},
"relevant": {
"type": "boolean",
"description": "Ground truth."
}
}
},
"description": "2 to 20 passages."
},
"threshold": {
"type": "number",
"maximum": 1,
"minimum": 0,
"description": "Minimum Precision@k to PASS (default 0.5)."
}
}
},
"outputSchema": {
"type": "object",
"properties": {
"k": {
"type": "number"
},
"mode": {
"type": "string"
},
"model": {
"type": "string"
},
"ranked": {
"type": "array",
"items": {
"type": "object"
}
},
"verdict": {
"type": "string"
},
"threshold": {
"type": "number"
},
"latency_ms": {
"type": "number"
},
"recall_at_k": {
"type": "number"
},
"precision_at_k": {
"type": "number"
},
"total_passages": {
"type": "number"
},
"has_ground_truth": {
"type": "boolean"
}
},
"additionalProperties": true
},
"annotations": {
"readOnlyHint": true,
"openWorldHint": true,
"idempotentHint": false,
"destructiveHint": false
}
}Estimate the token count of a text string using the cl100k_base approximation (~4 chars/token). Shows whether a text fits the context window of a model and what it costs. Returns token estimate, character count, and word count.
| Parameter | Type | Description |
|---|---|---|
| input * | string | Text to count tokens for |
{
"name": "count_tokens",
"description": "Estimate the token count of a text string using the cl100k_base approximation (~4 chars/token). Shows whether a text fits the context window of a model and what it costs. Returns token estimate, character count, and word count.",
"inputSchema": {
"type": "object",
"examples": [
{
"input": "Hello world, this is a test."
}
],
"required": [
"input"
],
"properties": {
"input": {
"type": "string",
"description": "Text to count tokens for"
}
}
},
"outputSchema": {
"type": "object",
"properties": {
"chars": {},
"words": {},
"tokens_estimate": {}
},
"additionalProperties": true
},
"annotations": {
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}
}Extract the first valid JSON object or array embedded in chaotic LLM output (surrounded by markdown fences, prose, or explanatory text). Handles ```json blocks and inline JSON.
| Parameter | Type | Description |
|---|---|---|
| input * | string | Raw text (e.g., LLM output) that may contain a JSON object or array |
{
"name": "extract_json_from_text",
"description": "Extract the first valid JSON object or array embedded in chaotic LLM output (surrounded by markdown fences, prose, or explanatory text). Handles ```json blocks and inline JSON.",
"inputSchema": {
"type": "object",
"examples": [
{
"input": "Here:\n```json\n{\"a\":[1,2]}\n```\nthanks"
}
],
"required": [
"input"
],
"properties": {
"input": {
"type": "string",
"description": "Raw text (e.g., LLM output) that may contain a JSON object or array"
}
}
},
"outputSchema": {
"type": "object",
"properties": {
"json": {},
"source": {
"type": "string"
}
},
"additionalProperties": true
},
"annotations": {
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}
}Estimate the API cost in USD for a given model and token counts, at the price read daily from the provider's own pricing page (prices_as_of, price_status). Any model from list_llm_models.
| Parameter | Type | Description |
|---|---|---|
| model * | string | Model name, e.g. "gpt-4o", "claude-3.5-sonnet", "deepseek-v3" |
| input_tokens * | number | Number of input/prompt tokens |
| output_tokens | number | Number of output/completion tokens (default: 0) |
{
"name": "estimate_llm_cost",
"description": "Estimate the API cost in USD for a given model and token counts, at the price read daily from the provider's own pricing page (prices_as_of, price_status). Any model from list_llm_models.",
"inputSchema": {
"type": "object",
"examples": [
{
"model": "gpt-4o",
"input_tokens": 1000000,
"output_tokens": 1000000
}
],
"required": [
"model",
"input_tokens"
],
"properties": {
"model": {
"type": "string",
"description": "Model name, e.g. \"gpt-4o\", \"claude-3.5-sonnet\", \"deepseek-v3\""
},
"input_tokens": {
"type": "number",
"description": "Number of input/prompt tokens"
},
"output_tokens": {
"type": "number",
"description": "Number of output/completion tokens (default: 0)"
}
}
},
"outputSchema": {
"type": "object",
"properties": {
"model": {},
"rates": {
"type": "object"
},
"input_tokens": {},
"output_tokens": {},
"input_cost_usd": {
"type": "string"
},
"total_cost_usd": {
"type": "string"
},
"output_cost_usd": {
"type": "string"
}
},
"additionalProperties": true
},
"annotations": {
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}
}Plan token allocation across system prompt, user input, context/RAG chunks, and expected output. Warns if budget exceeds the model's context window (any model from list_llm_models).
| Parameter | Type | Description |
|---|---|---|
| model * | string | Model id from list_llm_models (e.g. gpt-6.1-sol, claude-sonnet-5-5, gemini-3.8-flash) |
| context | string | Actual context text (will estimate tokens) |
| user_input | string | Actual user input text (will estimate tokens) |
| system_prompt | string | Actual system prompt text (will estimate tokens) |
| context_tokens | number | Token count for RAG context / documents |
| user_input_tokens | number | Token count for user message |
| system_prompt_tokens | number | Token count for system prompt |
| expected_output_tokens | number | Expected max output tokens |
{
"name": "token_budget_calculator",
"description": "Plan token allocation across system prompt, user input, context/RAG chunks, and expected output. Warns if budget exceeds the model's context window (any model from list_llm_models).",
"inputSchema": {
"type": "object",
"examples": [
{
"model": "gpt-4o",
"context_tokens": 1000,
"expected_output_tokens": 500
}
],
"required": [
"model"
],
"properties": {
"model": {
"type": "string",
"description": "Model id from list_llm_models (e.g. gpt-6.1-sol, claude-sonnet-5-5, gemini-3.8-flash)"
},
"context": {
"type": "string",
"description": "Actual context text (will estimate tokens)"
},
"user_input": {
"type": "string",
"description": "Actual user input text (will estimate tokens)"
},
"system_prompt": {
"type": "string",
"description": "Actual system prompt text (will estimate tokens)"
},
"context_tokens": {
"type": "number",
"description": "Token count for RAG context / documents"
},
"user_input_tokens": {
"type": "number",
"description": "Token count for user message"
},
"system_prompt_tokens": {
"type": "number",
"description": "Token count for system prompt"
},
"expected_output_tokens": {
"type": "number",
"description": "Expected max output tokens"
}
}
},
"outputSchema": {
"type": "object",
"properties": {
"breakdown": {
"type": "object"
}
},
"additionalProperties": true
},
"annotations": {
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}
}Given an array of message objects [{role, content}], estimate total token usage and check if it fits in the target model's context window. Warns about truncation risk.
| Parameter | Type | Description |
|---|---|---|
| model * | string | Model id from list_llm_models (e.g. gpt-6.1-sol, claude-sonnet-5-5) |
| messages * | array | Array of messages (system/user/assistant) |
| max_output_tokens | number | Reserved tokens for output (default: 4096) |
{
"name": "context_window_check",
"description": "Given an array of message objects [{role, content}], estimate total token usage and check if it fits in the target model's context window. Warns about truncation risk.",
"inputSchema": {
"type": "object",
"examples": [
{
"model": "gpt-4o",
"messages": [
{
"role": "user",
"content": "hi"
}
]
}
],
"required": [
"messages",
"model"
],
"properties": {
"model": {
"type": "string",
"description": "Model id from list_llm_models (e.g. gpt-6.1-sol, claude-sonnet-5-5)"
},
"messages": {
"type": "array",
"items": {
"type": "object",
"properties": {
"role": {
"type": "string"
},
"content": {
"type": "string"
}
}
},
"description": "Array of messages (system/user/assistant)"
},
"max_output_tokens": {
"type": "number",
"description": "Reserved tokens for output (default: 4096)"
}
}
},
"outputSchema": {
"type": "object",
"properties": {
"chars": {
"type": "number"
},
"breakdown": {
"type": "object"
},
"message_count": {
"type": "number"
},
"utilization_percent": {
"type": "string"
}
},
"additionalProperties": true
},
"annotations": {
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}
}Get detailed specs for an AI model: context window, pricing per 1K tokens, knowledge cutoff, provider, multimodal support, reasoning capabilities, and feature list. Covers 30+ models from OpenAI, Anthropic, Google, DeepSeek, Meta, Mistral, Cohere, xAI.
| Parameter | Type | Description |
|---|---|---|
| model * | string | Model name (e.g. "gpt-4o", "claude-3.5-sonnet", "gemini-3.1-pro-preview") |
{
"name": "model_info",
"description": "Get detailed specs for an AI model: context window, pricing per 1K tokens, knowledge cutoff, provider, multimodal support, reasoning capabilities, and feature list. Covers 30+ models from OpenAI, Anthropic, Google, DeepSeek, Meta, Mistral, Cohere, xAI.",
"inputSchema": {
"type": "object",
"examples": [
{
"model": "gpt-4o"
}
],
"required": [
"model"
],
"properties": {
"model": {
"type": "string",
"description": "Model name (e.g. \"gpt-4o\", \"claude-3.5-sonnet\", \"gemini-3.1-pro-preview\")"
}
}
},
"outputSchema": {
"type": "object",
"properties": {
"model": {},
"pricing_per_1k": {
"type": "object"
}
},
"additionalProperties": true
},
"annotations": {
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}
}Put 2-5 AI models side by side: price per 1k tokens, context window, multimodal, reasoning, provider, pricing page. Facts only: names the cheapest and the largest context, never a "best model" — run your cases on each to choose.
| Parameter | Type | Description |
|---|---|---|
| models * | array | Array of 2-5 model names (e.g. ["gpt-5","claude-sonnet-4-6","gemini-3.5-flash"]) |
{
"name": "compare_models",
"description": "Put 2-5 AI models side by side: price per 1k tokens, context window, multimodal, reasoning, provider, pricing page. Facts only: names the cheapest and the largest context, never a \"best model\" — run your cases on each to choose.",
"inputSchema": {
"type": "object",
"examples": [
{
"models": [
"gpt-6.1-sol",
"claude-sonnet-5-5"
]
}
],
"required": [
"models"
],
"properties": {
"models": {
"type": "array",
"items": {
"type": "string"
},
"description": "Array of 2-5 model names (e.g. [\"gpt-5\",\"claude-sonnet-4-6\",\"gemini-3.5-flash\"])"
}
}
},
"outputSchema": {
"type": "object",
"properties": {
"rows": {},
"cheapest": {},
"prices_as_of": {},
"largest_context": {},
"models_compared": {
"type": "number"
}
},
"additionalProperties": true
},
"annotations": {
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}
}List all LLM models available on ia-qa.com with their provider, API endpoint, and capabilities. Filter by provider name (e.g. "Groq", "HuggingFace", "OpenAI") or return the full catalog.
| Parameter | Type | Description |
|---|---|---|
| provider | string | Provider name, case-insensitive (e.g. "Groq", "OpenAI", "Anthropic"). Omit for the full catalogue. |
{
"name": "list_llm_models",
"description": "List all LLM models available on ia-qa.com with their provider, API endpoint, and capabilities. Filter by provider name (e.g. \"Groq\", \"HuggingFace\", \"OpenAI\") or return the full catalog.",
"inputSchema": {
"type": "object",
"examples": [
{
"provider": "groq"
}
],
"properties": {
"provider": {
"type": "string",
"description": "Provider name, case-insensitive (e.g. \"Groq\", \"OpenAI\", \"Anthropic\"). Omit for the full catalogue."
}
}
},
"outputSchema": {
"type": "object",
"properties": {
"total": {
"type": "number"
},
"filter": {},
"models": {},
"providers": {}
},
"additionalProperties": true
},
"annotations": {
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}
}Generate text with an open-source model hosted on Groq, on the server's key — no API key needed. Models: openai/gpt-oss-20b, openai/gpt-oss-120b, qwen/qwen3.8-27b (default "openai/gpt-oss-20b"). HuggingFace models are not served on the server key. A reasoning model that spends max_tokens thinking is retried once at 4096. Rate-limited.
| Parameter | Type | Description |
|---|---|---|
| model | string | Model ID (default: "openai/gpt-oss-20b"). Server-keyed whitelist only — Groq: openai/gpt-oss-20b, openai/gpt-oss-120b, qwen/qwen3.8-27b. Other ids from list_llm_models, HuggingFace ones included, are BYOK-only and will be rejected. |
| prompt * | string | The user prompt / instruction to send to the model |
| system | string | Optional system prompt to set context or persona |
| max_tokens | number | Maximum tokens to generate (default: 2048, max: 4096) |
| temperature | number | Sampling temperature 0.0–1.5 (default: 0.7) |
{
"name": "llm_generate",
"description": "Generate text with an open-source model hosted on Groq, on the server's key — no API key needed. Models: openai/gpt-oss-20b, openai/gpt-oss-120b, qwen/qwen3.8-27b (default \"openai/gpt-oss-20b\"). HuggingFace models are not served on the server key. A reasoning model that spends max_tokens thinking is retried once at 4096. Rate-limited.",
"inputSchema": {
"type": "object",
"examples": [
{
"prompt": "Reply with the single word OK."
}
],
"required": [
"prompt"
],
"properties": {
"model": {
"type": "string",
"description": "Model ID (default: \"openai/gpt-oss-20b\"). Server-keyed whitelist only — Groq: openai/gpt-oss-20b, openai/gpt-oss-120b, qwen/qwen3.8-27b. Other ids from list_llm_models, HuggingFace ones included, are BYOK-only and will be rejected."
},
"prompt": {
"type": "string",
"description": "The user prompt / instruction to send to the model"
},
"system": {
"type": "string",
"description": "Optional system prompt to set context or persona"
},
"max_tokens": {
"type": "number",
"description": "Maximum tokens to generate (default: 2048, max: 4096)"
},
"temperature": {
"type": "number",
"description": "Sampling temperature 0.0–1.5 (default: 0.7)"
}
}
},
"outputSchema": {
"type": "object",
"properties": {
"model": {},
"usage": {},
"content": {},
"provider": {},
"latency_ms": {
"type": "number"
}
},
"additionalProperties": true
},
"annotations": {
"readOnlyHint": true,
"openWorldHint": true,
"idempotentHint": false,
"destructiveHint": false
}
}Give honest usage feedback on an IA-QA MCP tool. Give a score (1-5) and a comment. Rate low (1-2) if the tool was wrong, irrelevant, or a poor fit; rate high (4-5) only if it genuinely solved your need. Maintainers read ratings, one vote per agent and tool a month: comments and per-tool scores are not published, only the overall average and vote count. Comments are screened: one carrying a credential is not stored, personal data becomes placeholders — describe what happened, not what a model should do. Skip routine successes: signal, not praise.
| Parameter | Type | Description |
|---|---|---|
| score * | number | Rating from 1 (poor) to 5 (excellent) |
| comment | string | Strongly encouraged: what you were trying to do, whether the tool got you there, and what was missing, wrong or a poor fit. Up to 2000 chars are stored; beyond, the response says truncated: true — send the rest as a second call. |
| tool_name * | string | Name of the MCP tool to rate (e.g. "format_json", "shield_analyze") |
{
"name": "rate_tool",
"description": "Give honest usage feedback on an IA-QA MCP tool. Give a score (1-5) and a comment. Rate low (1-2) if the tool was wrong, irrelevant, or a poor fit; rate high (4-5) only if it genuinely solved your need. Maintainers read ratings, one vote per agent and tool a month: comments and per-tool scores are not published, only the overall average and vote count. Comments are screened: one carrying a credential is not stored, personal data becomes placeholders — describe what happened, not what a model should do. Skip routine successes: signal, not praise.",
"inputSchema": {
"type": "object",
"examples": [
{
"score": 4,
"comment": "Formatted a 2 MB payload correctly.",
"tool_name": "format_json"
}
],
"required": [
"tool_name",
"score"
],
"properties": {
"score": {
"type": "number",
"maximum": 5,
"minimum": 1,
"description": "Rating from 1 (poor) to 5 (excellent)"
},
"comment": {
"type": "string",
"description": "Strongly encouraged: what you were trying to do, whether the tool got you there, and what was missing, wrong or a poor fit. Up to 2000 chars are stored; beyond, the response says truncated: true — send the rest as a second call."
},
"tool_name": {
"type": "string",
"description": "Name of the MCP tool to rate (e.g. \"format_json\", \"shield_analyze\")"
}
}
},
"outputSchema": {
"type": "object",
"properties": {
"ok": {
"type": "boolean"
},
"score": {},
"comment": {},
"message": {
"type": "string"
},
"rated_at": {
"type": "string"
},
"tool_name": {},
"truncated": {
"type": "boolean"
},
"stored_chars": {
"type": "number"
}
},
"additionalProperties": true
},
"annotations": {
"readOnlyHint": false,
"openWorldHint": false,
"idempotentHint": false,
"destructiveHint": true
}
}This is the IA-QA "eval" pack: 39 of 156 tools are listed. find_tool searches all of them and says which pack lists each one; for a tool outside this pack, the user adds https://www.ia-qa.com/mcp or that pack. IA-QA serves two kinds of user, and every tool produces an actionable verdict (pass/fail, score, diff) — the testing tools also return gate: pass | review | fail, the one field to branch a CI step on; LLM tools are BYOK (the user supplies their own API key). USE AI TO TEST — a QA engineer who wants AI to help: test cases from a ticket (jira_to_test_suite, generate_test_cases), likely bugs and risk from a diff (analyze_diff_bugs, run_pr_gate_pipeline), UI selector drift between two page captures (diff_mappings; sandbox_scenario supplies captures whose verdict is known). Repairing their end-to-end tests needs a browser on their machine: that is the local app @ia-qa/pal (with its own stdio MCP server, ia-qa-pal-mcp), not this server — point them to https://www.ia-qa.com/devtools/pal/tutorial, and to the step-by-step method at https://www.ia-qa.com/method#augmented-qa-journey. TEST WHAT USES AI — someone shipping an LLM, a RAG pipeline or an agent: get_testing_guidelines with no topic maps each goal to the tools that fit and the output field to gate on. find_tool finds any tool by keyword or category without loading every schema. These tools are operated on behalf of a human: before running a tool, say in one line what it does and what its verdict means. The same tools are also available from the shell via @ia-qa/cli, and local selector self-healing via @ia-qa/self-healing. rate_tool records feedback the maintainers read: a low score on a wrong or ill-fitting output is as useful as a high one, and a rating with nothing to report is noise. Report a verdict as a careful tester would: gate review means it could not decide, so say that and never round it to a pass; relay what a result says it did not prove, and explain why it matters to the person you work for.
Names, descriptions and schemas were written by the server and are shown as data.
Pattern matches on what the server declares. They show what it says it can do, not what it does.
| contract_path | string | Default evals/smoke.ia-eval.yaml. |
| workflow_name | string |
| app_start_command | string | [selector_drift] Boots the app, e.g. "npm run start:ci". Never guessed: omitted, the step is a TODO. |
{
"name": "generate_ci_workflow",
"description": "Generate a ready-to-commit GitHub Actions workflow that gates a build on IA-QA: \"eval_contract\" runs a .ia-eval.yaml through ia-qa-com/eval-action@v1 (provider key as a repo secret), \"cli_checks\" runs deterministic tools via npx @ia-qa/cli and fails on their verdict, \"selector_drift\" gates E2E selectors with @ia-qa/self-healing. Deterministic template, no LLM; returns the YAML, the secrets and the next steps. Pair with generate_eval_yaml.",
"inputSchema": {
"type": "object",
"examples": [
{}
],
"properties": {
"cron": {
"type": "string",
"description": "For schedule (default '0 6 * * 1')."
},
"gate": {
"enum": [
"eval_contract",
"cli_checks",
"both",
"selector_drift",
"all"
],
"type": "string",
"description": "Default eval_contract. both = CLI checks + eval; all = CLI checks, drift, eval."
},
"provider": {
"enum": [
"groq",
"openai",
"anthropic",
"google"
],
"type": "string",
"description": "Decides which repo secret is wired (default groq)."
},
"triggers": {
"type": "array",
"items": {
"enum": [
"push",
"pull_request",
"workflow_dispatch",
"schedule"
],
"type": "string"
},
"description": "Default push + pull_request."
},
"cli_tools": {
"type": "array",
"items": {
"type": "string"
},
"description": "Tools to gate on, e.g. [\"secret_scan\",\"prompt_injection_scan\"]."
},
"min_score": {
"type": "number",
"description": "Override the contract min_score (0-100)."
},
"app_base_url": {
"type": "string",
"description": "[selector_drift] Default http://127.0.0.1:3000; must match .ia-qa/config.json."
},
"fail_on_fail": {
"type": "boolean",
"description": "Default true; false reports without gating."
},
"node_version": {
"type": "string",
"description": "Default \"20\"."
},
"contract_path": {
"type": "string",
"description": "Default evals/smoke.ia-eval.yaml."
},
"workflow_name": {
"type": "string"
},
"app_start_command": {
"type": "string",
"description": "[selector_drift] Boots the app, e.g. \"npm run start:ci\". Never guessed: omitted, the step is a TODO."
}
}
},
"outputSchema": {
"type": "object",
"properties": {
"gate": {
"type": "string"
},
"path": {
"type": "string"
},
"yaml": {
"type": "string"
},
"notes": {
"type": "array",
"items": {
"type": "string"
}
},
"next_steps": {
"type": "array",
"items": {
"type": "string"
}
},
"secrets_required": {
"type": "array",
"items": {
"type": "string"
}
}
},
"additionalProperties": true
},
"annotations": {
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}
}{
"name": "metamorphic_check",
"description": "Reference-free stability check: does an assistant's answer stay the same when the QUESTION is transformed (case, typo, paraphrase, reorder, translation)? You bring the outputs; no model is called (tfidf mode is deterministic and free). Default thresholds: case .95, typo .90, paraphrase .80, reorder .80, translation .75 (embeddings only), specialization .60 (advisory, never gated). Returns PASS / FAIL / INVALID — INVALID when the BASE answer is a refusal or too short, so a model that refuses everything never scores 1.0. Pair with run_semantic_tests for correctness.",
"inputSchema": {
"type": "object",
"examples": [
{
"base": {
"output": "Paris is the capital and largest city of France."
},
"variants": [
{
"output": "Paris is the capital and largest city of France.",
"relation": "case"
}
]
}
],
"required": [
"base",
"variants"
],
"properties": {
"base": {
"type": "object",
"properties": {
"output": {
"type": "string",
"description": "The answer to the original question."
},
"question": {
"type": "string",
"description": "Optional, echoed back."
}
},
"description": "The original run."
},
"mode": {
"enum": [
"tfidf",
"embeddings"
],
"type": "string",
"description": "tfidf (default, lexical: gate on case/typo, read paraphrase as a trend) or embeddings (OpenAI, needs api_key; required for translation)."
},
"api_key": {
"type": "string",
"description": "OpenAI key, embeddings mode only."
},
"variants": {
"type": "array",
"items": {
"type": "object",
"required": [
"relation",
"output"
],
"properties": {
"id": {
"type": "string"
},
"output": {
"type": "string",
"description": "The answer to the transformed question."
},
"question": {
"type": "string",
"description": "Optional, echoed back."
},
"relation": {
"enum": [
"case",
"typo",
"paraphrase",
"reorder",
"translation",
"specialization"
],
"type": "string",
"description": "Transformation applied to the question; sets the metric and threshold."
}
}
},
"maxItems": 20,
"description": "Answers to transformed versions of the question."
},
"thresholds": {
"type": "object",
"properties": {
"case": {
"type": "number",
"maximum": 1,
"minimum": 0
},
"typo": {
"type": "number",
"maximum": 1,
"minimum": 0
},
"reorder": {
"type": "number",
"maximum": 1,
"minimum": 0
},
"paraphrase": {
"type": "number",
"maximum": 1,
"minimum": 0
},
"translation": {
"type": "number",
"maximum": 1,
"minimum": 0
},
"specialization": {
"type": "number",
"maximum": 1,
"minimum": 0
}
},
"description": "Per-relation overrides, 0-1. Calibrate on your own data before gating."
},
"require_all": {
"type": "boolean",
"description": "Default true: every gated variant must pass. Keep it for a gate — false lets one easy typo row hold the run at PASS while a paraphrase fails."
},
"baseline_guard": {
"type": "object",
"properties": {
"min_length": {
"type": "number",
"description": "Default 40 chars."
},
"must_not_match": {
"type": "array",
"items": {
"type": "string"
},
"description": "Refusal patterns, checked in the first 200 chars. Replaces the default FR+EN list; [] disables."
}
},
"description": "Floor on the BASE answer; failing it returns INVALID."
}
}
},
"outputSchema": {
"type": "object",
"properties": {
"mode": {
"type": "string"
},
"summary": {
"type": "object"
},
"verdict": {
"type": "string"
},
"weakest": {
"type": "object"
},
"baseline": {
"type": "object"
},
"variants": {
"type": "array",
"items": {
"type": "object"
}
},
"warnings": {
"type": "array",
"items": {
"type": "string"
}
},
"thresholds": {
"type": "object"
}
},
"additionalProperties": true
},
"annotations": {
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}
}{
"name": "test_skill",
"description": "Test a SKILL.md. No model, every call: lint (gate on lint.valid) and review (description lacks what/when, words shared with other_skills, leftovers, no done-check, missing files; why + fix each). Unless lint_only, a model picks a skill per request from names+descriptions: trigger_accuracy PASS ≥80, FIX ≥50, else BLOCK; your scenarios = gate, generated = signal; unanswered = INCONCLUSIVE. Server key 10 runs/day.",
"inputSchema": {
"type": "object",
"examples": [
{
"skill_md": "---\nname: json-formatter\ndescription: Pretty-print a JSON string the user pastes.\n---\nFormat the JSON with 2-space indentation."
}
],
"required": [
"skill_md"
],
"properties": {
"files": {
"type": "array",
"items": {
"type": "string"
},
"description": "Paths in its folder."
},
"model": {
"type": "string",
"description": "Default openai/gpt-oss-20b."
},
"api_key": {
"type": "string",
"description": "Used once, not stored."
},
"dir_name": {
"type": "string",
"description": "Its folder."
},
"skill_md": {
"type": "string",
"description": "Full SKILL.md, frontmatter included."
},
"lint_only": {
"type": "boolean"
},
"scenarios": {
"type": "array",
"items": {
"type": "object",
"required": [
"prompt",
"should_trigger"
],
"properties": {
"prompt": {
"type": "string"
},
"should_trigger": {
"type": "boolean"
}
}
},
"description": "Max 20, near misses included."
},
"other_skills": {
"type": "array",
"items": {
"type": "object",
"required": [
"name",
"description"
],
"properties": {
"name": {
"type": "string"
},
"description": {
"type": "string"
}
}
},
"description": "Installed next to it."
},
"scenario_count": {
"enum": [
4,
6,
8,
10
],
"type": "number"
}
}
},
"outputSchema": {
"type": "object",
"properties": {
"lint": {
"type": "object"
},
"review": {
"type": "object"
},
"verdict": {
"type": "string"
}
},
"additionalProperties": true
},
"annotations": {
"readOnlyHint": true,
"openWorldHint": true,
"idempotentHint": false,
"destructiveHint": false
}
}{
"name": "llm_output_validator",
"description": "Validate an LLM response against QA criteria: format checks (JSON, code, markdown), content rules (must-include, must-not-include), length constraints, language detection, and safety patterns. Essential for QA testing LLM-powered features.",
"inputSchema": {
"type": "object",
"examples": [
{
"output": "{\"a\":1}",
"must_include": "a",
"expected_format": "json",
"must_not_include": "TODO"
}
],
"required": [
"output"
],
"properties": {
"output": {
"type": "string",
"description": "The LLM output text to validate"
},
"max_length": {
"type": "number",
"description": "Maximum character length for the output"
},
"min_length": {
"type": "number",
"description": "Minimum character length for the output"
},
"check_safety": {
"type": "boolean",
"description": "Check for PII patterns (emails, phones, SSN), profanity signals, and prompt leakage"
},
"must_include": {
"type": "string",
"description": "Comma-separated strings that MUST appear in the output"
},
"expected_format": {
"enum": [
"json",
"markdown",
"code",
"plain",
"any"
],
"type": "string",
"description": "Expected output format"
},
"must_not_include": {
"type": "string",
"description": "Comma-separated strings that must NOT appear (e.g. \"TODO, FIXME, undefined, NaN\")"
},
"check_json_schema": {
"type": "string",
"description": "If expected_format is JSON, provide required keys as comma-separated list to validate the structure"
},
"expected_language": {
"type": "string",
"description": "Expected language of the output (en, fr, es, de…). Checks for common words."
}
}
},
"outputSchema": {
"type": "object",
"properties": {
"total": {
"type": "number"
},
"checks": {},
"failed": {},
"passed": {},
"verdict": {}
},
"additionalProperties": true
},
"annotations": {
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}
}| response_b * | string | Second output (e.g. model B's answer) |
{
"name": "compare_responses",
"description": "Compare two ALREADY-PRODUCED outputs (e.g. model A vs model B on the same task) side by side. Returns deterministic metrics (token cosine, ROUGE-L, Jaccard, length/structure deltas, JSON diff) and a verdict. If a `reference` (ground truth) is given, scores each output against it and picks the closer one. If `model` + `api_key` are given, an LLM judge also picks a qualitative winner for the task. No re-execution — you bring the outputs.",
"inputSchema": {
"type": "object",
"examples": [
{
"response_a": "Paris is the capital.",
"response_b": "Paris is the capital."
}
],
"required": [
"response_a",
"response_b"
],
"properties": {
"task": {
"type": "string",
"description": "The task/prompt both outputs were answering — used by the LLM judge for context"
},
"model": {
"type": "string",
"description": "Optional judge model id (BYOK). When set with api_key, an LLM judge picks a qualitative winner."
},
"api_key": {
"type": "string",
"description": "Optional API key for the judge model (BYOK). Used only for the judge call; never stored."
},
"label_a": {
"type": "string",
"description": "Label for output A (e.g. \"GPT-4o\", \"v1.0\")"
},
"label_b": {
"type": "string",
"description": "Label for output B (e.g. \"GPT-5-nano\", \"v1.1\")"
},
"reference": {
"type": "string",
"description": "Optional ground-truth / expected answer. If set, each output is scored against it and the closer one wins (deterministic)."
},
"check_json": {
"type": "boolean",
"description": "Try to parse as JSON and compare structurally (keys, types, values)"
},
"response_a": {
"type": "string",
"description": "First output (e.g. model A's answer)"
},
"response_b": {
"type": "string",
"description": "Second output (e.g. model B's answer)"
}
}
},
"outputSchema": {
"type": "object",
"properties": {
"judge": {},
"labelA": {},
"labelB": {},
"metrics": {},
"summary": {
"type": "string"
},
"verdict": {}
},
"additionalProperties": true
},
"annotations": {
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}
}{
"name": "prompt_test_suite",
"description": "Define a test suite for a prompt: provide the system prompt, user prompt, and expected output criteria. Returns a test plan with scored rubric — use this as input for manual or automated LLM evaluation.",
"inputSchema": {
"type": "object",
"examples": [
{
"user_prompt": "Say hi",
"system_prompt": "You are a helpful assistant."
}
],
"required": [
"system_prompt",
"user_prompt"
],
"properties": {
"max_tokens": {
"type": "number",
"description": "Max token budget for the test"
},
"temperature": {
"type": "number",
"description": "Temperature to use"
},
"user_prompt": {
"type": "string",
"description": "The user prompt to send"
},
"check_safety": {
"type": "boolean",
"description": "Include safety/PII checks in the rubric"
},
"must_include": {
"type": "string",
"description": "Required content (comma-separated)"
},
"system_prompt": {
"type": "string",
"description": "The system prompt under test"
},
"expected_format": {
"enum": [
"json",
"markdown",
"code",
"plain",
"any"
],
"type": "string",
"description": "Expected output format"
},
"must_not_include": {
"type": "string",
"description": "Forbidden content (comma-separated)"
},
"expected_behavior": {
"type": "string",
"description": "Description of what the LLM should do (free text)"
},
"adversarial_prompts": {
"type": "boolean",
"description": "Auto-generate adversarial test variants (jailbreak, injection, edge cases)"
}
}
},
"outputSchema": {
"type": "object",
"properties": {
"rubric": {},
"categories": {
"type": "array"
},
"total_tests": {
"type": "number"
},
"instructions": {
"type": "string"
},
"test_suite_name": {
"type": "string"
}
},
"additionalProperties": true
},
"annotations": {
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}
}{
"name": "run_vlm_test_suite",
"description": "Test a vision model (OpenAI, Anthropic or Gemini) on one image: each test case asks a question and asserts on the answer (contains, not_contains, json_format, min_length, max_length, semantic_contains = TF-IDF cosine ≥ 0.4). Returns per-case PASS/FAIL, the pass rate, `overall` PASS/WARNING/FAIL against threshold (N/A when no assertion could run), and latency. BYOK: your key for the model's provider; the image is fetched only from a public address.",
"inputSchema": {
"type": "object",
"examples": [
{
"model": "gemini-2.5-flash",
"api_key": "<your-api-key>",
"image_url": "https://www.ia-qa.com/media/discover-report-poster.png",
"test_cases": [
{
"question": "Does the image contain text? Answer yes or no.",
"assertion_type": "contains",
"assertion_value": "yes"
}
]
}
],
"required": [
"test_cases",
"model",
"api_key"
],
"properties": {
"model": {
"enum": [
"gpt-4o",
"gpt-4o-mini",
"claude-sonnet-4-6",
"claude-haiku-4-5-20251001",
"gemini-3.5-flash",
"gemini-2.5-flash"
],
"type": "string"
},
"api_key": {
"type": "string",
"description": "Key for the model's provider."
},
"image_url": {
"type": "string",
"description": "Public image URL (or image_base64)."
},
"threshold": {
"type": "number",
"description": "Pass rate for an overall PASS (default 80)."
},
"test_cases": {
"type": "array",
"items": {
"type": "object",
"required": [
"question"
],
"properties": {
"id": {
"type": "string"
},
"question": {
"type": "string"
},
"assertion_type": {
"enum": [
"contains",
"not_contains",
"json_format",
"min_length",
"max_length",
"semantic_contains"
],
"type": "string"
},
"assertion_value": {
"type": "string",
"description": "Not needed for json_format."
}
}
},
"maxItems": 10
},
"image_base64": {
"type": "string"
},
"system_prompt": {
"type": "string"
},
"image_mime_type": {
"type": "string",
"description": "For image_base64 (default image/jpeg)."
}
}
},
"outputSchema": {
"type": "object",
"properties": {
"model": {},
"total": {},
"passed": {},
"overall": {
"enum": [
"PASS",
"WARNING",
"FAIL",
"N/A"
]
},
"results": {},
"image_url": {},
"pass_rate": {},
"threshold": {},
"latencyStats": {}
},
"additionalProperties": true
},
"annotations": {
"readOnlyHint": true,
"openWorldHint": true,
"idempotentHint": false,
"destructiveHint": false
}
}