OpenMark is an MCP server published by ai.openmark, hosted at openmark.ai. When last checked on 2026-11-22 it was working: it answers and lists its tools. It declares 10 tools, including get_task_format, validate_task, list_models, prepare_attachments, estimate_cost.
Benchmark your own prompt on 100+ LLMs with real API calls; pick the cheapest model that passes.
Nothing else in the archive shares these values.
Squares are this server's values; dots are other servers sharing them. A shared address or publisher is a lead, not proof of a common operator.
Inferred from the host name and headers: what a server runs on, not who runs it.
How to write an OpenMark task: the YAML structure, all 18 scoring modes, and tips. Call this before writing a task. Pass a topic for the full detail on one subject (the app's Task Guide): fields, scoring, composite, chaining, documents, examples, best_practices, or a scoring mode name.
| Parameter | Type | Description |
|---|---|---|
| topic | string | Optional: fields, scoring, composite, chaining, documents, examples, best_practices, or a scoring mode name such as regex_match |
{
"name": "get_task_format",
"title": "Task YAML format",
"description": "How to write an OpenMark task: the YAML structure, all 18 scoring modes, and tips. Call this before writing a task. Pass a topic for the full detail on one subject (the app's Task Guide): fields, scoring, composite, chaining, documents, examples, best_practices, or a scoring mode name.",
"inputSchema": {
"type": "object",
"properties": {
"topic": {
"type": "string",
"description": "Optional: fields, scoring, composite, chaining, documents, examples, best_practices, or a scoring mode name such as regex_match"
}
}
}
}None yet. Changes appear after a server's second daily check.
Check a task YAML the way OpenMark will before running it: structure, scoring settings, and the test limit of the user's plan. Free, saves nothing. Returns the problems to fix and warnings worth a look.
| Parameter | Type | Description |
|---|---|---|
| task_yaml * | string | The full task YAML (a JSON object with the same fields also works) |
{
"name": "validate_task",
"title": "Check a task",
"description": "Check a task YAML the way OpenMark will before running it: structure, scoring settings, and the test limit of the user's plan. Free, saves nothing. Returns the problems to fix and warnings worth a look.",
"inputSchema": {
"type": "object",
"required": [
"task_yaml"
],
"properties": {
"task_yaml": {
"type": "string",
"description": "The full task YAML (a JSON object with the same fields also works)"
}
}
}
}Current OpenMark model ids with list prices ($ per 1M tokens), as text and as a structured list. Use these ids in run_benchmark. Append @low, @medium or @high to a reasoning model id to set its reasoning effort.
| Parameter | Type | Description |
|---|---|---|
| search | string | Filter by id, name or provider, e.g. 'claude', 'gemini flash' |
| images_only | boolean | Only models that accept image attachments |
| max_output_price | number | Only models at or below this $ per 1M output tokens |
{
"name": "list_models",
"title": "List models",
"description": "Current OpenMark model ids with list prices ($ per 1M tokens), as text and as a structured list. Use these ids in run_benchmark. Append @low, @medium or @high to a reasoning model id to set its reasoning effort.",
"inputSchema": {
"type": "object",
"properties": {
"search": {
"type": "string",
"description": "Filter by id, name or provider, e.g. 'claude', 'gemini flash'"
},
"images_only": {
"type": "boolean",
"description": "Only models that accept image attachments"
},
"max_output_price": {
"type": "number",
"description": "Only models at or below this $ per 1M output tokens"
}
}
}
}Get an upload_uid and a one-hour upload link (no API key needed) for images, PDFs or text documents, with the exact curl command. Upload the files, reference them by file name in each test's attachments list, then pass the upload_uid to estimate_cost / run_benchmark.
| Parameter | Type | Description |
|---|---|---|
| file_names | array | Optional: the local file paths to upload, to get a ready-to-run command |
{
"name": "prepare_attachments",
"title": "Prepare file attachments",
"description": "Get an upload_uid and a one-hour upload link (no API key needed) for images, PDFs or text documents, with the exact curl command. Upload the files, reference them by file name in each test's attachments list, then pass the upload_uid to estimate_cost / run_benchmark.",
"inputSchema": {
"type": "object",
"properties": {
"file_names": {
"type": "array",
"items": {
"type": "string"
},
"maxItems": 20,
"description": "Optional: the local file paths to upload, to get a ready-to-run command"
}
}
}
}Credits a run_benchmark call would use, before spending anything: the expected total, the split per model, the balance needed to start, the worst case, and the user's balance. Same inputs and run settings as run_benchmark. Use it to drop an expensive model, or to compare settings (e.g. max_tokens), before running.
| Parameter | Type | Description |
|---|---|---|
| models * | array | |
| preset | string | Like the app's presets: standard = 2 passes + snappy timeouts (the default), thorough = 3 passes + patient timeouts. Explicit stability_runs / timeout_profile override it |
| fail_fast | boolean | Stop a model after repeated errors to save credits. Set false to let every model finish, e.g. to diagnose failures |
| task_yaml * | string | The full task YAML (a JSON object with the same fields also works) |
| max_tokens | integer | Output token limit per answer, 16 to 16000; 0 = no limit. Keep the default unless results show it was too low: a higher limit raises the credits needed to start. get_results flags models whose answers came back empty or near the limit; re-run those with a higher value |
| upload_uid | string | From prepare_attachments, so image and document tokens are counted |
| custom_timeout | integer | Seconds per answer, with timeout_profile custom |
| stability_runs | integer | Passes per model; 2+ shows which models are consistent |
| temp_preference | string | With temperature_discovery: which temperature wins a tie |
| timeout_profile | string | How long to wait per answer: snappy 45s, patient 90s, custom = custom_timeout |
| temperature_discovery | boolean | Find each model's best temperature for this task. Costs about 6x as much |
{
"name": "estimate_cost",
"title": "Estimate a run's cost",
"description": "Credits a run_benchmark call would use, before spending anything: the expected total, the split per model, the balance needed to start, the worst case, and the user's balance. Same inputs and run settings as run_benchmark. Use it to drop an expensive model, or to compare settings (e.g. max_tokens), before running.",
"inputSchema": {
"type": "object",
"required": [
"task_yaml",
"models"
],
"properties": {
"models": {
"type": "array",
"items": {
"type": "string"
},
"maxItems": 20,
"minItems": 1
},
"preset": {
"enum": [
"standard",
"thorough"
],
"type": "string",
"description": "Like the app's presets: standard = 2 passes + snappy timeouts (the default), thorough = 3 passes + patient timeouts. Explicit stability_runs / timeout_profile override it"
},
"fail_fast": {
"type": "boolean",
"default": true,
"description": "Stop a model after repeated errors to save credits. Set false to let every model finish, e.g. to diagnose failures"
},
"task_yaml": {
"type": "string",
"description": "The full task YAML (a JSON object with the same fields also works)"
},
"max_tokens": {
"type": "integer",
"default": 1000,
"maximum": 16000,
"minimum": 0,
"description": "Output token limit per answer, 16 to 16000; 0 = no limit. Keep the default unless results show it was too low: a higher limit raises the credits needed to start. get_results flags models whose answers came back empty or near the limit; re-run those with a higher value"
},
"upload_uid": {
"type": "string",
"description": "From prepare_attachments, so image and document tokens are counted"
},
"custom_timeout": {
"type": "integer",
"maximum": 600,
"minimum": 120,
"description": "Seconds per answer, with timeout_profile custom"
},
"stability_runs": {
"type": "integer",
"default": 2,
"maximum": 5,
"minimum": 1,
"description": "Passes per model; 2+ shows which models are consistent"
},
"temp_preference": {
"enum": [
"low",
"high"
],
"type": "string",
"default": "low",
"description": "With temperature_discovery: which temperature wins a tie"
},
"timeout_profile": {
"enum": [
"snappy",
"patient",
"custom"
],
"type": "string",
"default": "snappy",
"description": "How long to wait per answer: snappy 45s, patient 90s, custom = custom_timeout"
},
"temperature_discovery": {
"type": "boolean",
"default": false,
"description": "Find each model's best temperature for this task. Costs about 6x as much"
}
}
}
}Save a task (YAML, see get_task_format) to the user's OpenMark account and run it on the given models with real API calls. Returns a job_id and the expected cost; poll get_results. Spends the user's credits; pass max_credits to only start within a budget. Optional run settings match the app's Configure & Run panel (preset, stability_runs, max_tokens, temperature_discovery, temp_preference, timeout_profile, custom_timeout, fail_fast).
| Parameter | Type | Description |
|---|---|---|
| models * | array | Model ids from list_models |
| preset | string | Like the app's presets: standard = 2 passes + snappy timeouts (the default), thorough = 3 passes + patient timeouts. Explicit stability_runs / timeout_profile override it |
| fail_fast | boolean | Stop a model after repeated errors to save credits. Set false to let every model finish, e.g. to diagnose failures |
| task_yaml * | string | The full task YAML (a JSON object with the same fields also works) |
| max_tokens | integer | Output token limit per answer, 16 to 16000; 0 = no limit. Keep the default unless results show it was too low: a higher limit raises the credits needed to start. get_results flags models whose answers came back empty or near the limit; re-run those with a higher value |
| upload_uid | string | From prepare_attachments, when tests use attachments |
| max_credits | integer | Budget: if the expected cost is higher, nothing runs and you get the quote instead |
| custom_timeout | integer | Seconds per answer, with timeout_profile custom |
| stability_runs | integer | Passes per model; 2+ shows which models are consistent |
| temp_preference | string | With temperature_discovery: which temperature wins a tie |
| timeout_profile | string | How long to wait per answer: snappy 45s, patient 90s, custom = custom_timeout |
| temperature_discovery | boolean | Find each model's best temperature for this task. Costs about 6x as much |
{
"name": "run_benchmark",
"title": "Run a benchmark",
"description": "Save a task (YAML, see get_task_format) to the user's OpenMark account and run it on the given models with real API calls. Returns a job_id and the expected cost; poll get_results. Spends the user's credits; pass max_credits to only start within a budget. Optional run settings match the app's Configure & Run panel (preset, stability_runs, max_tokens, temperature_discovery, temp_preference, timeout_profile, custom_timeout, fail_fast).",
"inputSchema": {
"type": "object",
"required": [
"task_yaml",
"models"
],
"properties": {
"models": {
"type": "array",
"items": {
"type": "string"
},
"maxItems": 20,
"minItems": 1,
"description": "Model ids from list_models"
},
"preset": {
"enum": [
"standard",
"thorough"
],
"type": "string",
"description": "Like the app's presets: standard = 2 passes + snappy timeouts (the default), thorough = 3 passes + patient timeouts. Explicit stability_runs / timeout_profile override it"
},
"fail_fast": {
"type": "boolean",
"default": true,
"description": "Stop a model after repeated errors to save credits. Set false to let every model finish, e.g. to diagnose failures"
},
"task_yaml": {
"type": "string",
"description": "The full task YAML (a JSON object with the same fields also works)"
},
"max_tokens": {
"type": "integer",
"default": 1000,
"maximum": 16000,
"minimum": 0,
"description": "Output token limit per answer, 16 to 16000; 0 = no limit. Keep the default unless results show it was too low: a higher limit raises the credits needed to start. get_results flags models whose answers came back empty or near the limit; re-run those with a higher value"
},
"upload_uid": {
"type": "string",
"description": "From prepare_attachments, when tests use attachments"
},
"max_credits": {
"type": "integer",
"minimum": 1,
"description": "Budget: if the expected cost is higher, nothing runs and you get the quote instead"
},
"custom_timeout": {
"type": "integer",
"maximum": 600,
"minimum": 120,
"description": "Seconds per answer, with timeout_profile custom"
},
"stability_runs": {
"type": "integer",
"default": 2,
"maximum": 5,
"minimum": 1,
"description": "Passes per model; 2+ shows which models are consistent"
},
"temp_preference": {
"enum": [
"low",
"high"
],
"type": "string",
"default": "low",
"description": "With temperature_discovery: which temperature wins a tie"
},
"timeout_profile": {
"enum": [
"snappy",
"patient",
"custom"
],
"type": "string",
"default": "snappy",
"description": "How long to wait per answer: snappy 45s, patient 90s, custom = custom_timeout"
},
"temperature_discovery": {
"type": "boolean",
"default": false,
"description": "Find each model's best temperature for this task. Costs about 6x as much"
}
}
}
}Status of a benchmark job, and once completed the ranked results (score, stability, cost per run, latency), the tests each model missed with its answer, a recommendation for the chosen objective and quick picks for the others.
| Parameter | Type | Description |
|---|---|---|
| job_id * | string | |
| tolerance | number | Score points below the top score that still count as good enough |
| show_misses | boolean | List the tests each model got wrong, with the expected answer and the model's reply |
| optimize_for | string | accuracy (same order as the app's results table): best score, then cheapest, then fastest. cost / speed / stability: cheapest / fastest / most consistent model within `tolerance` points of the top score |
{
"name": "get_results",
"title": "Get benchmark results",
"description": "Status of a benchmark job, and once completed the ranked results (score, stability, cost per run, latency), the tests each model missed with its answer, a recommendation for the chosen objective and quick picks for the others.",
"inputSchema": {
"type": "object",
"required": [
"job_id"
],
"properties": {
"job_id": {
"type": "string"
},
"tolerance": {
"type": "number",
"default": 2,
"maximum": 50,
"minimum": 0,
"description": "Score points below the top score that still count as good enough"
},
"show_misses": {
"type": "boolean",
"default": true,
"description": "List the tests each model got wrong, with the expected answer and the model's reply"
},
"optimize_for": {
"enum": [
"accuracy",
"cost",
"speed",
"stability"
],
"type": "string",
"default": "accuracy",
"description": "accuracy (same order as the app's results table): best score, then cheapest, then fastest. cost / speed / stability: cheapest / fastest / most consistent model within `tolerance` points of the top score"
}
}
}
}A completed job's results as the CSV the OpenMark Router plugin for OpenClaw reads (the same file as the app's Export > OpenClaw). Save it in the plugin's benchmarks folder, by default ~/.openclaw/workspace/plugins/openmark-router/benchmarks/, and OpenClaw routes this kind of task to the benchmark winner. Free.
| Parameter | Type | Description |
|---|---|---|
| job_id * | string |
{
"name": "export_openclaw_csv",
"title": "Export for the OpenClaw router",
"description": "A completed job's results as the CSV the OpenMark Router plugin for OpenClaw reads (the same file as the app's Export > OpenClaw). Save it in the plugin's benchmarks folder, by default ~/.openclaw/workspace/plugins/openmark-router/benchmarks/, and OpenClaw routes this kind of task to the benchmark winner. Free.",
"inputSchema": {
"type": "object",
"required": [
"job_id"
],
"properties": {
"job_id": {
"type": "string"
}
}
}
}The user's plan, credit balance and limits, and what a credit pack, Pro or Expert would unlock. Use it before large runs, or when a run is refused for credits or limits.
No parameters.
{
"name": "get_account",
"title": "Account, credits and plans",
"description": "The user's plan, credit balance and limits, and what a credit pack, Pro or Expert would unlock. Use it before large runs, or when a run is refused for credits or limits.",
"inputSchema": {
"type": "object",
"properties": {}
}
}Watch the task of a completed job: when new or cheaper models ship, OpenMark re-tests up to 4 of them next to the user's pick and emails the user only when one beats it. Each check uses the user's credits at the normal rate.
| Parameter | Type | Description |
|---|---|---|
| model | string | The model to keep as the pick (default: the best-scoring model). Use the model you recommended. |
| job_id * | string |
{
"name": "watch_task",
"title": "Watch a task for better models",
"description": "Watch the task of a completed job: when new or cheaper models ship, OpenMark re-tests up to 4 of them next to the user's pick and emails the user only when one beats it. Each check uses the user's credits at the normal rate.",
"inputSchema": {
"type": "object",
"required": [
"job_id"
],
"properties": {
"model": {
"type": "string",
"description": "The model to keep as the pick (default: the best-scoring model). Use the model you recommended."
},
"job_id": {
"type": "string"
}
}
}
}OpenMark benchmarks LLMs on the user's own task with real API calls and deterministic scoring. Typical flow: call get_task_format, write a task YAML from the user's real prompt and a few realistic inputs with expected outputs, check it with validate_task (free; fix what it reports), pick 3-8 models with list_models, check the price with estimate_cost (or pass max_credits to run_benchmark; both take the app's run settings such as max_tokens, preset and temperature_discovery), call run_benchmark, then poll get_results with the job_id until it is completed (usually 1-5 minutes). get_results ranks by accuracy by default (highest score; ties go to the cheaper, then the faster model) and also lists the best pick for cost, speed and consistency. Ask the user what matters most and pass it as optimize_for; if a much cheaper model is within a couple of points of the winner, say so. Explain losses with the misses get_results lists. For images or documents, call prepare_attachments first. Offer watch_task so OpenMark re-tests the task when new models ship. If the user runs OpenClaw with the OpenMark Router plugin, export_openclaw_csv gives the routing file to save. Runs spend the user's OpenMark credits; get_account shows the balance and what each plan unlocks.
Names, descriptions and schemas were written by the server and are shown as data.