URLpipe is an MCP server published by dev.urlpipe, hosted at urlpipe.dev. When last checked on 2026-11-22 it was working: it answers and lists its tools. It declares 17 tools, including list_projects, get_usage, fetch_markdown, fetch_html, capture_screenshot.
Read any page with its JavaScript run: Markdown, screenshots, metadata, console errors, Lighthouse.
1 connections through 5 shared values. Tap any node.
Squares are this server's values; dots are other servers sharing them. A shared address or publisher is a lead, not proof of a common operator.
Inferred from the host name and headers: what a server runs on, not who runs it.
The projects this token can reach, each with how many requests it has made in the last 30 days and where its results are delivered. Start here: every other tool takes a project_id from this list. A project is the unit a request is billed and recorded against — credits themselves belong to the organization, so which project you pick does not change what a call costs.
No parameters.
{
"name": "list_projects",
"title": "List projects",
"description": "The projects this token can reach, each with how many requests it has\nmade in the last 30 days and where its results are delivered.\n\nStart here: every other tool takes a project_id from this list. A project\nis the unit a request is billed and recorded against — credits themselves\nbelong to the organization, so which project you pick does not change\nwhat a call costs.",
"inputSchema": {
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"required": [],
"properties": {}
},
"outputSchema": {
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"required": [
"organization",
"projects"
],
"properties": {
"projects": {
"type": "array",
"items": {
"type": "object",
"required": [
"id",
"name"
],
"properties": {
"id": {
"type": "integer"
},
"name": {
"type": "string"
},
"default_webhook_url": {
"type": "string"
},
"requests_last_30_days": {
"type": "integer"
}
}
}
},
"organization": {
"type": "string"
}
}
},
"annotations": {
"title": "List projects",
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}
}What the organization's plan allows, how much of it is left this period, and what each operation costs. Worth reading before a run of expensive calls: the operations differ by seventeen times in price, so the difference between fetch_markdown and summarize_page over a hundred pages is 100 credits against 1,700. It is also how to read a refusal — a call that comes back with quota_exceeded is telling you this number ran out.
No parameters.
{
"name": "get_usage",
"title": "Check usage and prices",
"description": "What the organization's plan allows, how much of it is left this period,\nand what each operation costs.\n\nWorth reading before a run of expensive calls: the operations differ by\nseventeen times in price, so the difference between fetch_markdown and\nsummarize_page over a hundred pages is 100 credits against 1,700. It is\nalso how to read a refusal — a call that comes back with quota_exceeded\nis telling you this number ran out.",
"inputSchema": {
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"required": [],
"properties": {}
},
"outputSchema": {
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"required": [
"plan",
"credits",
"cost_per_operation"
],
"properties": {
"plan": {
"type": "string"
},
"credits": {
"type": "object",
"properties": {
"used": {
"type": "integer"
},
"limit": {
"type": [
"integer",
"string"
],
"description": "A number, or \"unlimited\"."
},
"remaining": {
"type": [
"integer",
"string"
],
"description": "A number, or \"unlimited\"."
},
"resets_at": {
"type": "string"
}
}
},
"plan_name": {
"type": "string"
},
"concurrency_limit": {
"type": "integer"
},
"cost_per_operation": {
"type": "object",
"additionalProperties": {
"type": "integer"
}
},
"residential_surcharge_per_page_visit": {
"type": "integer"
}
}
},
"annotations": {
"title": "Check usage and prices",
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}
}Exposes POST /markdown. Renders the page in headless Chrome and converts its main content to clean Markdown — headings, lists, links and code kept, navigation, sidebars and cookie banners dropped. Start here when you want to READ a page. It is the cheapest operation we sell (1 credit) and by far the most compact thing to put in front of a model: fetch_html returns the whole DOM, which is usually many times larger and says nothing extra about what the page means.
| Parameter | Type | Description |
|---|---|---|
| url * | string | The page to fetch. Must be a public http(s) URL. |
| sync | boolean | Wait for the result and return it (default false). When false the call returns a token immediately and you collect the result with get_result. |
| labels | object | Your own keys to find this request by later, e.g. {"client": "acme"}: they come back with the result, in the webhook and in list_requests, which can filter by them. Up to 16 keys of up to 40 letters, digits, _ - or .; string values up to 256 characters. |
| max_age | string | How fresh a stored result must be to be reused, e.g. "2 hours" or "3 days". Default 7 days, maximum 30. A reused result is free. Pass "0" to force a fresh fetch. |
| report_to | string | Async only: a webhook URL to deliver the result to. Defaults to the project's configured endpoint, if it has one. |
| project_id * | integer | Which project this request belongs to. From list_projects. |
| residential |
Exposes POST /html. Returns the page's HTML after JavaScript has run and redirects have been followed — the DOM a real browser sees, not the empty shell curl returns. 1 credit. Use it when you need the markup itself: a specific attribute, a script tag, a structured-data block. If you want to read the page's content, fetch_markdown costs the same and returns a fraction of the text.
| Parameter | Type | Description |
|---|---|---|
| url * | string | The page to fetch. Must be a public http(s) URL. |
| sync | boolean | Wait for the result and return it (default false). When false the call returns a token immediately and you collect the result with get_result. |
| labels | object | Your own keys to find this request by later, e.g. {"client": "acme"}: they come back with the result, in the webhook and in list_requests, which can filter by them. Up to 16 keys of up to 40 letters, digits, _ - or .; string values up to 256 characters. |
| max_age | string | How fresh a stored result must be to be reused, e.g. "2 hours" or "3 days". Default 7 days, maximum 30. A reused result is free. Pass "0" to force a fresh fetch. |
| report_to | string | Async only: a webhook URL to deliver the result to. Defaults to the project's configured endpoint, if it has one. |
| project_id * | integer | Which project this request belongs to. From list_projects. |
| residential | boolean |
Exposes POST /screenshot. Captures the whole rendered page as a PNG — JavaScript executed, web fonts and images included, exactly as a browser would draw it. 1 credit, whichever options you use. Use it to see a page rather than read it: layout, visual regressions, link previews, or checking what an anti-bot page actually showed us. screenshot_options size it (viewport, scale, one element, the fold only), encode it (png, jpeg or webp) and restyle it (dark mode, hidden elements, your own CSS); page_options wait for the page and take ads and cookie banners out of it.
| Parameter | Type | Description |
|---|---|---|
| url * | string | The page to fetch. Must be a public http(s) URL. |
| sync | boolean | Wait for the result and return it (default false). When false the call returns a token immediately and you collect the result with get_result. |
| labels | object | Your own keys to find this request by later, e.g. {"client": "acme"}: they come back with the result, in the webhook and in list_requests, which can filter by them. Up to 16 keys of up to 40 letters, digits, _ - or .; string values up to 256 characters. |
| max_age | string | How fresh a stored result must be to be reused, e.g. "2 hours" or "3 days". Default 7 days, maximum 30. A reused result is free. Pass "0" to force a fresh fetch. |
| report_to | string | Async only: a webhook URL to deliver the result to. Defaults to the project's configured endpoint, if it has one. |
| project_id * | integer | Which project this request belongs to. From list_projects. |
Exposes POST /pdf. Prints the whole rendered page to a PDF — JavaScript executed, web fonts and lazy-loaded images included, its headings as bookmarks. 1 credit, whichever options you use. Use it to archive or hand on a page as a document: a receipt, an article, a report, terms as they stood today. pdf_options choose the paper, the orientation, the margins and the scale, whether backgrounds print, whether the page's print stylesheet or its on-screen look is used, and whether to cut it into sheets at all; page_options wait for the page and take ads and cookie banners out of it.
| Parameter | Type | Description |
|---|---|---|
| url * | string | The page to fetch. Must be a public http(s) URL. |
| sync | boolean | Wait for the result and return it (default false). When false the call returns a token immediately and you collect the result with get_result. |
| labels | object | Your own keys to find this request by later, e.g. {"client": "acme"}: they come back with the result, in the webhook and in list_requests, which can filter by them. Up to 16 keys of up to 40 letters, digits, _ - or .; string values up to 256 characters. |
| max_age | string | How fresh a stored result must be to be reused, e.g. "2 hours" or "3 days". Default 7 days, maximum 30. A reused result is free. Pass "0" to force a fresh fetch. |
| report_to | string | Async only: a webhook URL to deliver the result to. Defaults to the project's configured endpoint, if it has one. |
| project_id * | integer | Which project this request belongs to. From list_projects. |
Exposes POST /console. Loads the page in headless Chrome and returns what it reported through console.error and console.warn during load, plus uncaught exceptions and unhandled promise rejections (not console.log). 1 credit. What you would open DevTools for: broken third-party scripts and client-side errors you cannot reproduce locally.
| Parameter | Type | Description |
|---|---|---|
| url * | string | The page to fetch. Must be a public http(s) URL. |
| sync | boolean | Wait for the result and return it (default false). When false the call returns a token immediately and you collect the result with get_result. |
| labels | object | Your own keys to find this request by later, e.g. {"client": "acme"}: they come back with the result, in the webhook and in list_requests, which can filter by them. Up to 16 keys of up to 40 letters, digits, _ - or .; string values up to 256 characters. |
| max_age | string | How fresh a stored result must be to be reused, e.g. "2 hours" or "3 days". Default 7 days, maximum 30. A reused result is free. Pass "0" to force a fresh fetch. |
| report_to | string | Async only: a webhook URL to deliver the result to. Defaults to the project's configured endpoint, if it has one. |
| project_id * | integer | Which project this request belongs to. From list_projects. |
| residential | boolean |
Exposes POST /lighthouse. Runs a real Google Lighthouse audit against the live page: performance, accessibility, best-practices and SEO scores plus Core Web Vitals. 2 credits — dearer than a page fetch because the audit runs in its own limited lane, so it also takes longer than anything else here. It fetches the page itself rather than sharing a visit, so asking for it inside scrape_url costs two page visits, not one.
| Parameter | Type | Description |
|---|---|---|
| url * | string | The page to fetch. Must be a public http(s) URL. |
| sync | boolean | Wait for the result and return it (default false). When false the call returns a token immediately and you collect the result with get_result. |
| device | string | Which profile to audit under. Default mobile. |
| labels | object | Your own keys to find this request by later, e.g. {"client": "acme"}: they come back with the result, in the webhook and in list_requests, which can filter by them. Up to 16 keys of up to 40 letters, digits, _ - or .; string values up to 256 characters. |
| max_age | string | How fresh a stored result must be to be reused, e.g. "2 hours" or "3 days". Default 7 days, maximum 30. A reused result is free. Pass "0" to force a fresh fetch. |
| report_to | string | Async only: a webhook URL to deliver the result to. Defaults to the project's configured endpoint, if it has one. |
| project_id * | integer |
Exposes POST /meta. Reads everything a page declares about itself — Open Graph, Twitter cards, meta tags, JSON-LD, microdata and links — into one object with the same keys on every page: title, description, site name, kind of page, language, authors, publication and update dates, share image and video, favicon and icons, logo, feeds, keywords, canonical URL, robots directives, hreflang alternates, oEmbed endpoint and structured-data types, plus the raw og: and twitter: tags. A value the page does not declare is null or an empty list. Values injected by JavaScript are included, because the page is rendered first. 1 credit.
| Parameter | Type | Description |
|---|---|---|
| url * | string | The page to fetch. Must be a public http(s) URL. |
| sync | boolean | Wait for the result and return it (default false). When false the call returns a token immediately and you collect the result with get_result. |
| labels | object | Your own keys to find this request by later, e.g. {"client": "acme"}: they come back with the result, in the webhook and in list_requests, which can filter by them. Up to 16 keys of up to 40 letters, digits, _ - or .; string values up to 256 characters. |
| max_age | string | How fresh a stored result must be to be reused, e.g. "2 hours" or "3 days". Default 7 days, maximum 30. A reused result is free. Pass "0" to force a fresh fetch. |
| report_to | string | Async only: a webhook URL to deliver the result to. Defaults to the project's configured endpoint, if it has one. |
| project_id * | integer |
Exposes POST /technologies. Detects what a page is built with — CMS, e-commerce platform, JavaScript frameworks and libraries, analytics and tag managers, advertising and marketing tools, CDN, web server, programming language, payment providers, chat widgets, consent managers and more — from an open database of over 7,000 fingerprints. Each technology comes with its version when the page gives it away, a confidence from 0 to 100, its categories, its website and CPE, and which signals gave it away (html, meta tags, headers, cookies, JavaScript globals, DOM, script URLs and contents, stylesheets, DNS records, robots.txt). The page is rendered first, so technologies loaded by JavaScript are found too. 1 credit.
| Parameter | Type | Description |
|---|---|---|
| url * | string | The page to fetch. Must be a public http(s) URL. |
| sync | boolean | Wait for the result and return it (default false). When false the call returns a token immediately and you collect the result with get_result. |
| labels | object | Your own keys to find this request by later, e.g. {"client": "acme"}: they come back with the result, in the webhook and in list_requests, which can filter by them. Up to 16 keys of up to 40 letters, digits, _ - or .; string values up to 256 characters. |
| max_age | string | How fresh a stored result must be to be reused, e.g. "2 hours" or "3 days". Default 7 days, maximum 30. A reused result is free. Pass "0" to force a fresh fetch. |
| report_to | string | Async only: a webhook URL to deliver the result to. Defaults to the project's configured endpoint, if it has one. |
| project_id |
Exposes POST /keywords. Returns the 5–15 terms and phrases that best represent the page, ordered by relevance — ranked by a language model, not by raw frequency. 15 credits. As with summarize_page: if you are going to reason over the result yourself, fetch_markdown costs 1 credit and gives you everything.
| Parameter | Type | Description |
|---|---|---|
| url * | string | The page to fetch. Must be a public http(s) URL. |
| sync | boolean | Wait for the result and return it (default false). When false the call returns a token immediately and you collect the result with get_result. |
| labels | object | Your own keys to find this request by later, e.g. {"client": "acme"}: they come back with the result, in the webhook and in list_requests, which can filter by them. Up to 16 keys of up to 40 letters, digits, _ - or .; string values up to 256 characters. |
| max_age | string | How fresh a stored result must be to be reused, e.g. "2 hours" or "3 days". Default 7 days, maximum 30. A reused result is free. Pass "0" to force a fresh fetch. |
| report_to | string | Async only: a webhook URL to deliver the result to. Defaults to the project's configured endpoint, if it has one. |
| project_id * | integer | Which project this request belongs to. From list_projects. |
| residential | boolean | Fetch the page from a residential (home ISP) exit instead of a datacentre one. Costs a surcharge per page visit — see get_usage. Use it for sites that block datacentre traffic. |
Exposes POST /summarize. Returns a concise Markdown summary of the page's main content, with the navigation, ads and boilerplate left out. The most expensive operation we sell, at 17 credits, because it runs a language model over the page. If YOU are the model that will read it, fetch_markdown gives you the whole page for 1 credit and you can summarize it yourself — this tool is for when the summary is the artefact being produced, not a step on the way to one.
| Parameter | Type | Description |
|---|---|---|
| url * | string | The page to fetch. Must be a public http(s) URL. |
| sync | boolean | Wait for the result and return it (default false). When false the call returns a token immediately and you collect the result with get_result. |
| labels | object | Your own keys to find this request by later, e.g. {"client": "acme"}: they come back with the result, in the webhook and in list_requests, which can filter by them. Up to 16 keys of up to 40 letters, digits, _ - or .; string values up to 256 characters. |
| max_age | string | How fresh a stored result must be to be reused, e.g. "2 hours" or "3 days". Default 7 days, maximum 30. A reused result is free. Pass "0" to force a fresh fetch. |
| report_to | string | Async only: a webhook URL to deliver the result to. Defaults to the project's configured endpoint, if it has one. |
| project_id * | integer | Which project this request belongs to. From list_projects. |
| residential |
Exposes POST /extract. Reads the page and fills an object YOU define: pass `fields`, each with a name, a type and a description of what to take from the page, and get back exactly those keys in that order. Anything the page does not give comes back null — values are never guessed. Types: string, number, integer, boolean, date, datetime, url, email, object (with its own fields) and array (with items). Use it for product details, prices, contact data, job listings, event dates, lists of items with several attributes each. 24 credits: it reads the whole page with an AI model, so prefer extract_metadata when the standard metadata is all you need.
| Parameter | Type | Description |
|---|---|---|
| url * | string | The page to fetch. Must be a public http(s) URL. |
| sync | boolean | Wait for the result and return it (default false). When false the call returns a token immediately and you collect the result with get_result. |
| fields * | array | The fields to extract, at most 50 in all. |
| labels | object | Your own keys to find this request by later, e.g. {"client": "acme"}: they come back with the result, in the webhook and in list_requests, which can filter by them. Up to 16 keys of up to 40 letters, digits, _ - or .; string values up to 256 characters. |
| max_age | string | How fresh a stored result must be to be reused, e.g. "2 hours" or "3 days". Default 7 days, maximum 30. A reused result is free. Pass "0" to force a fresh fetch. |
| report_to | string | Async only: a webhook URL to deliver the result to. Defaults to the project's configured endpoint, if it has one. |
Exposes POST /scrape. Runs any subset of the other operations in one request, served from a single page visit where possible. Each operation is billed and stored exactly as the individual call would be, so this is not a discount — with one exception, and it is the reason to use it: a residential exit is charged per page VISIT, so several results off one visit pay the surcharge once. Answers with one object keyed by operation, each entry carrying its own success and result, so a partial failure still returns everything that worked. lighthouse does not share the visit — it runs its own audit on its own engine — so including it means two page fetches, and two surcharges when residential is on. So does a screenshot whose screenshot_options set viewport_width, viewport_height, device_scale_factor, dark_mode or block_ads: those change how the page loads, so the screenshot gets a visit of its own.
| Parameter | Type | Description |
|---|---|---|
| url * | string | The page to fetch. Must be a public http(s) URL. |
| sync | boolean | Wait for the result and return it (default false). When false the call returns a token immediately and you collect the result with get_result. |
| device | string | Lighthouse only, when it is among the operations. Default mobile. |
| fields | array | extract only, and required when it is among the operations: the fields to fill, exactly as extract_data takes them. |
| labels | object | Your own keys to find this request by later, e.g. {"client": "acme"}: they come back with the result, in the webhook and in list_requests, which can filter by them. Up to 16 keys of up to 40 letters, digits, _ - or .; string values up to 256 characters. |
Exposes GET /result/:token. Returns the result of a past request, in the same form the call that made it would have returned — Markdown as Markdown, a structured operation as its object, a screenshot as an image, a PDF as an embedded application/pdf resource, a scrape as its combined object. This is how an async call is collected: any tool called without sync: true answers with a token, and this turns that token into the result. It also re-reads a result somebody already paid for, which is free — results are kept for 30 days. A request that has not finished yet answers {"status": "processing"}; call again in a moment.
| Parameter | Type | Description |
|---|---|---|
| token * | string | The token the original call returned. |
| project_id * | integer | From list_projects. |
{
"name": "get_result",
"title": "Get a result",
"description": "Exposes GET /result/:token. Returns the result of a past request, in the\nsame form the call that made it would have returned — Markdown as\nMarkdown, a structured operation as its object, a screenshot as an image,\na PDF as an embedded application/pdf resource,\na scrape as its combined object.\n\nThis is how an async call is collected: any tool called without\nsync: true answers with a token, and this turns that token into the\nresult. It also re-reads a result somebody already paid for, which is\nfree — results are kept for 30 days.\n\nA request that has not finished yet answers {\"status\": \"processing\"};\ncall again in a moment.",
"inputSchema": {
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"required": [
"project_id",
"token"
],
"properties": {
"token": {
"type": "string",
"description": "The token the original call returned."
},
"project_id": {
"type": "integer",
"description": "From list_projects."
}
}
},
"outputSchema": {
"type": "object",
"anyOf": [
{
"type": "object",
"required": [
"markdown"
],
"properties": {
"markdown": {
"type": "string",
"description": "The page's main content as Markdown."
}
}
},
{
"type": "object",
"required": [
"html"
],
"properties": {
"html": {
"type": "string",
"description": "The rendered HTML."
}
}
},
{
"type": "object",
"required": [
"summary"
],
"properties": {
"summary": {
"type": "string",
"description": "A Markdown summary of the page."
}
}
},
{
"type": "object",
"required": [
"keywords"
],
"properties": {
"keywords": {
"type": "array",
"items": {
"type": "string"
}
}
}
},
{
"type": "object",
"required": [
"entries"
],
"properties": {
"entries": {
"type": "array",
"items": {
"type": "object",
"properties": {
"text": {
"type": "string"
},
"type": {
"enum": [
"error",
"warning",
"exception"
],
"type": "string"
}
}
}
}
}
},
{
"type": "object",
"required": [
"mime_type"
],
"properties": {
"mime_type": {
"type": "string",
"description": "The image's type. The image itself is the image content block."
},
"result_url": {
"type": [
"string",
"null"
],
"description": "A link to the image that needs no key."
}
}
},
{
"type": "object",
"required": [
"mime_type"
],
"properties": {
"mime_type": {
"type": "string",
"description": "Always application/pdf. The document itself is the resource content block."
},
"result_url": {
"type": [
"string",
"null"
],
"description": "A link to the PDF that needs no key."
}
}
},
{
"type": "object",
"required": [
"final_url",
"canonical_url",
"title",
"description",
"site_name",
"type",
"language",
"locale",
"authors",
"published_at",
"modified_at",
"image",
"video",
"favicon_url",
"icons",
"logo_url",
"feeds",
"keywords",
"theme_color",
"generator",
"manifest_url",
"robots",
"open_graph",
"twitter",
"alternates",
"oembed_url",
"structured_data_types"
],
"properties": {
"type": {
"enum": [
"article",
"website",
"profile",
"book",
"product",
"video",
"music",
"place",
"event",
"recipe",
null
],
"type": [
"string",
"null"
],
"description": "The kind of page, as the page declares it."
},
"feeds": {
"type": "array",
"items": {
"type": "object",
"required": [
"url",
"type",
"title"
],
"properties": {
"url": {
"type": "string"
},
"type": {
"type": [
"string",
"null"
]
},
"title": {
"type": [
"string",
"null"
]
}
},
"additionalProperties": false
}
},
"icons": {
"type": "array",
"items": {
"type": "object",
"required": [
"url",
"rel",
"sizes",
"type"
],
"properties": {
"rel": {
"type": "string"
},
"url": {
"type": "string"
},
"type": {
"type": [
"string",
"null"
]
},
"sizes": {
"type": [
"string",
"null"
]
}
},
"additionalProperties": false
}
},
"image": {
"type": [
"object",
"null"
],
"required": [
"url",
"width",
"height",
"alt",
"type"
],
"properties": {
"alt": {
"type": [
"string",
"null"
]
},
"url": {
"type": "string"
},
"type": {
"type": [
"string",
"null"
]
},
"width": {
"type": [
"integer",
"null"
]
},
"height": {
"type": [
"integer",
"null"
]
}
},
"additionalProperties": false
},
"title": {
"type": [
"string",
"null"
],
"description": "The page's own title, without the site's name around it."
},
"video": {
"type": [
"object",
"null"
],
"required": [
"url",
"type",
"width",
"height"
],
"properties": {
"url": {
"type": "string"
},
"type": {
"type": [
"string",
"null"
]
},
"width": {
"type": [
"integer",
"null"
]
},
"height": {
"type": [
"integer",
"null"
]
}
},
"additionalProperties": false
},
"locale": {
"type": [
"string",
"null"
],
"description": "The most specific language tag the page declares for that language, BCP 47 (en-US)."
},
"robots": {
"type": "object",
"required": [
"index",
"follow"
],
"properties": {
"index": {
"type": "boolean"
},
"follow": {
"type": "boolean"
}
},
"additionalProperties": false
},
"authors": {
"type": "array",
"items": {
"type": "object",
"required": [
"name",
"url",
"profiles"
],
"properties": {
"url": {
"type": [
"string",
"null"
]
},
"name": {
"type": "string"
},
"profiles": {
"type": "array",
"items": {
"type": "string"
}
}
},
"additionalProperties": false
}
},
"twitter": {
"type": "object",
"required": [
"card",
"site",
"creator",
"title",
"description",
"image"
],
"properties": {
"card": {
"type": [
"string",
"null"
]
},
"site": {
"type": [
"string",
"null"
]
},
"image": {
"type": [
"string",
"null"
]
},
"title": {
"type": [
"string",
"null"
]
},
"creator": {
"type": [
"string",
"null"
]
},
"description": {
"type": [
"string",
"null"
]
}
},
"description": "The page's twitter: tags as declared.",
"additionalProperties": false
},
"keywords": {
"type": "array",
"items": {
"type": "string"
}
},
"language": {
"type": [
"string",
"null"
],
"description": "ISO 639-1 code of the language the content is written in."
},
"logo_url": {
"type": [
"string",
"null"
]
},
"final_url": {
"type": [
"string",
"null"
],
"description": "The address the page ended on, after its redirects."
},
"generator": {
"type": [
"string",
"null"
]
},
"site_name": {
"type": [
"string",
"null"
]
},
"alternates": {
"type": "array",
"items": {
"type": "object",
"required": [
"hreflang",
"url"
],
"properties": {
"url": {
"type": "string"
},
"hreflang": {
"type": "string"
}
},
"additionalProperties": false
}
},
"oembed_url": {
"type": [
"string",
"null"
]
},
"open_graph": {
"type": "object",
"required": [
"title",
"description",
"image",
"url",
"type",
"site_name",
"locale"
],
"properties": {
"url": {
"type": [
"string",
"null"
]
},
"type": {
"type": [
"string",
"null"
]
},
"image": {
"type": [
"string",
"null"
]
},
"title": {
"type": [
"string",
"null"
]
},
"locale": {
"type": [
"string",
"null"
]
},
"site_name": {
"type": [
"string",
"null"
]
},
"description": {
"type": [
"string",
"null"
]
}
},
"description": "The page's og: tags as declared.",
"additionalProperties": false
},
"description": {
"type": [
"string",
"null"
]
},
"favicon_url": {
"type": [
"string",
"null"
]
},
"modified_at": {
"type": [
"string",
"null"
],
"description": "ISO 8601, as published_at."
},
"theme_color": {
"type": [
"string",
"null"
]
},
"manifest_url": {
"type": [
"string",
"null"
]
},
"published_at": {
"type": [
"string",
"null"
],
"description": "ISO 8601: a date (2024-05-01) or a moment with its offset."
},
"canonical_url": {
"type": [
"string",
"null"
],
"description": "The page's canonical URL, absolute."
},
"structured_data_types": {
"type": "array",
"items": {
"type": "string"
}
}
},
"additionalProperties": false
},
{
"type": "object",
"required": [
"final_url",
"technologies"
],
"properties": {
"final_url": {
"type": [
"string",
"null"
],
"description": "The address the page ended on, after its redirects."
},
"technologies": {
"type": "array",
"items": {
"type": "object",
"required": [
"name",
"slug",
"version",
"confidence",
"categories",
"website",
"description",
"cpe",
"detected_by",
"implied_by"
],
"properties": {
"cpe": {
"type": [
"string",
"null"
],
"description": "The CPE 2.3 name, for looking the technology up in vulnerability databases."
},
"name": {
"type": "string",
"description": "The technology's name, e.g. WordPress."
},
"slug": {
"type": "string",
"description": "A stable identifier for the technology, e.g. wordpress."
},
"version": {
"type": [
"string",
"null"
],
"description": "The version, when the page gives it away."
},
"website": {
"type": [
"string",
"null"
]
},
"categories": {
"type": "array",
"items": {
"type": "object",
"required": [
"slug",
"name"
],
"properties": {
"name": {
"type": "string"
},
"slug": {
"type": "string"
}
},
"additionalProperties": false
}
},
"confidence": {
"type": "integer",
"maximum": 100,
"minimum": 0,
"description": "How sure the detection is, 0–100. Most are 100."
},
"implied_by": {
"type": "array",
"items": {
"type": "string"
},
"description": "The detected technologies that imply this one (WordPress implies PHP)."
},
"description": {
"type": [
"string",
"null"
]
},
"detected_by": {
"type": "array",
"items": {
"type": "string"
},
"description": "Which signals gave it away: html, meta, headers, cookies, js, dom, script_src, scripts, css, text, url, xhr, cert_issuer, dns, robots. Empty for a technology only implied by another."
}
},
"additionalProperties": false
},
"description": "Every technology detected, most certain first."
}
},
"additionalProperties": false
},
{
"type": "object",
"description": "One key per requested field, in the order requested. A value the page does not give is null.",
"additionalProperties": true
},
{
"type": "object",
"properties": {
"url": {
"type": "string"
},
"audits": {
"type": "object"
},
"device": {
"type": "string"
},
"metrics": {
"type": "object"
},
"fetchTime": {
"type": "string"
},
"categories": {
"type": "object"
}
}
},
{
"type": "object",
"properties": {
"url": {
"type": "string"
},
"final_url": {
"type": [
"string",
"null"
],
"description": "The address the page ended on, after its redirects."
},
"operations": {
"type": "object",
"description": "One key per operation requested, in the order asked for.",
"additionalProperties": {
"type": "object",
"properties": {
"error": {
"type": "string"
},
"cached": {
"type": "boolean"
},
"result": {},
"success": {
"type": "boolean"
}
}
}
}
}
},
{
"type": "object",
"required": [
"token",
"status"
],
"properties": {
"token": {
"type": "string"
},
"labels": {
"type": "object"
},
"status": {
"type": "string",
"const": "processing"
}
}
}
],
"$schema": "https://json-schema.org/draft/2020-12/schema"
},
"annotations": {
"title": "Get a result",
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}
}A project's recent requests, newest first, with each one's token. Look here before fetching a page: if somebody already pulled it, passing that token to get_result returns the same result for nothing. A /scrape appears as one entry — its per-operation children are internal, and the scrape's own token returns all of them. Pass labels to see only the requests made with them — every key given must match its value exactly.
| Parameter | Type | Description |
|---|---|---|
| url | string | Only requests for this exact URL. |
| limit | integer | How many to return. Default 25, maximum 100. |
| labels | object | Only requests made with all of these labels, each value matched exactly. |
| offset | integer | Skip this many, for paging. |
| operation | string | Only requests running this operation. |
| project_id * | integer | From list_projects. |
{
"name": "list_requests",
"title": "List past requests",
"description": "A project's recent requests, newest first, with each one's token.\n\nLook here before fetching a page: if somebody already pulled it, passing\nthat token to get_result returns the same result for nothing. A /scrape\nappears as one entry — its per-operation children are internal, and the\nscrape's own token returns all of them.\n\nPass labels to see only the requests made with them — every key given\nmust match its value exactly.",
"inputSchema": {
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"required": [
"project_id"
],
"properties": {
"url": {
"type": "string",
"description": "Only requests for this exact URL."
},
"limit": {
"type": "integer",
"description": "How many to return. Default 25, maximum 100."
},
"labels": {
"type": "object",
"description": "Only requests made with all of these labels, each value matched exactly.",
"additionalProperties": {
"type": "string"
}
},
"offset": {
"type": "integer",
"description": "Skip this many, for paging."
},
"operation": {
"enum": [
"html",
"screenshot",
"markdown",
"meta",
"summarize",
"keywords",
"lighthouse",
"console",
"technologies",
"pdf",
"extract",
"scrape"
],
"type": "string",
"description": "Only requests running this operation."
},
"project_id": {
"type": "integer",
"description": "From list_projects."
}
}
},
"outputSchema": {
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"required": [
"project_id",
"total",
"offset",
"requests"
],
"properties": {
"total": {
"type": "integer"
},
"offset": {
"type": "integer"
},
"requests": {
"type": "array",
"items": {
"type": "object",
"required": [
"token",
"url",
"status"
],
"properties": {
"url": {
"type": "string"
},
"token": {
"type": "string"
},
"labels": {
"type": "object",
"additionalProperties": {
"type": "string"
}
},
"status": {
"type": "string"
},
"credits": {
"type": "integer"
},
"success": {
"type": [
"boolean",
"null"
]
},
"final_url": {
"type": "string",
"description": "Where the page ended up after its redirects."
},
"operation": {
"type": [
"string",
"null"
]
},
"created_at": {
"type": "string"
},
"duration_ms": {
"type": "integer"
},
"finished_at": {
"type": "string"
},
"served_from_cache": {
"type": [
"boolean",
"null"
]
}
}
}
},
"project_id": {
"type": "integer"
}
}
},
"annotations": {
"title": "List past requests",
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}
}Everything about one past request except its result: what was asked for, how long each stage took, whether it came from the store, and how the webhook delivery went. This is the metadata; get_result returns the page itself. Reach for this one when a call did not do what you expected — it says whether the result was reused, why an analysis failed, and whether the delivery to the project's endpoint succeeded.
| Parameter | Type | Description |
|---|---|---|
| token * | string | The token the original call returned. |
| project_id * | integer | From list_projects. |
{
"name": "get_request",
"title": "Inspect a past request",
"description": "Everything about one past request except its result: what was asked for,\nhow long each stage took, whether it came from the store, and how the\nwebhook delivery went.\n\nThis is the metadata; get_result returns the page itself. Reach for this\none when a call did not do what you expected — it says whether the result\nwas reused, why an analysis failed, and whether the delivery to the\nproject's endpoint succeeded.",
"inputSchema": {
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"required": [
"project_id",
"token"
],
"properties": {
"token": {
"type": "string",
"description": "The token the original call returned."
},
"project_id": {
"type": "integer",
"description": "From list_projects."
}
}
},
"outputSchema": {
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"required": [
"token",
"url",
"status"
],
"properties": {
"url": {
"type": "string"
},
"meta": {
"type": "object",
"description": "What the HTTP response headers would say: cache, cost, quota, result_url, final_url."
},
"async": {
"type": [
"boolean",
"null"
]
},
"token": {
"type": "string"
},
"labels": {
"type": "object",
"additionalProperties": {
"type": "string"
}
},
"status": {
"type": "string"
},
"credits": {
"type": "integer"
},
"max_age": {
"type": [
"integer",
"null"
]
},
"outcome": {
"type": [
"object",
"null"
]
},
"success": {
"type": [
"boolean",
"null"
]
},
"final_url": {
"type": "string",
"description": "Where the page ended up after its redirects."
},
"operation": {
"type": [
"string",
"null"
]
},
"report_to": {
"type": [
"string",
"null"
]
},
"created_at": {
"type": "string"
},
"operations": {
"type": "array",
"items": {
"type": "object",
"required": [
"token",
"url",
"status"
],
"properties": {
"url": {
"type": "string"
},
"token": {
"type": "string"
},
"labels": {
"type": "object",
"additionalProperties": {
"type": "string"
}
},
"status": {
"type": "string"
},
"credits": {
"type": "integer"
},
"success": {
"type": [
"boolean",
"null"
]
},
"final_url": {
"type": "string",
"description": "Where the page ended up after its redirects."
},
"operation": {
"type": [
"string",
"null"
]
},
"created_at": {
"type": "string"
},
"duration_ms": {
"type": "integer"
},
"finished_at": {
"type": "string"
},
"served_from_cache": {
"type": [
"boolean",
"null"
]
}
}
}
},
"duration_ms": {
"type": "integer"
},
"finished_at": {
"type": "string"
},
"webhook_error": {
"type": [
"string",
"null"
]
},
"webhook_status": {
"type": [
"string",
"null"
]
},
"request_options": {
"type": [
"object",
"null"
]
},
"served_from_cache": {
"type": [
"boolean",
"null"
]
},
"webhook_delivered_at": {
"type": [
"string",
"null"
]
}
}
},
"annotations": {
"title": "Inspect a past request",
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}
}URLpipe turns a URL into clean data — Markdown, rendered HTML, a screenshot, a PDF, page metadata, an AI summary or keywords, structured data in a schema you define, console output, a Lighthouse audit, or the technologies a site is built with. Pages are fetched with a real headless browser, so JavaScript-rendered sites work. These tools are the URLpipe HTTP API — same arguments, same responses. The one addition is project_id, because this credential reaches the whole organization rather than one project. Start with list_projects to get a project_id. ASYNC IS THE DEFAULT. A call returns a token straight away and the work continues in the background; pass sync: true when you want to wait and be handed the result. Either way, get_result turns a token into the result. An async call also delivers to the project's webhook, if it has one configured. COST. Every call spends credits from one monthly allowance, and the operations are not priced alike: fetch_markdown, fetch_html, extract_metadata, detect_technologies, capture_screenshot, capture_pdf and capture_console_errors cost 1, run_lighthouse_audit 2, extract_keywords 15, summarize_page 17, extract_data 24. get_usage has the live numbers. So: to READ a page, use fetch_markdown. It is the cheapest operation and the most compact input for a model — reach for summarize_page or extract_keywords only when the summary or the keyword list is the artefact you are producing, rather than a step towards reasoning you would do yourself. A repeated request is free. Results are stored for 30 days and reused when they are fresher than max_age (7 days by default), so re-fetching a page costs nothing unless you ask for a shorter window. A repeat of a request that is still running waits for it, also for free. list_requests shows what this project has already fetched. To retry a call safely, pass the same idempotency_key: within 24 hours it returns the first call's token and result rather than starting new work. scrape_url runs several operations off one page visit. That is not a discount — each operation is billed as usual — except when residential is on, where the exit surcharge is charged per visit rather than per operation. Every fetching tool takes labels: your own keys for the request, e.g. {"client": "acme"}. They come back with the result, and list_requests filters by them, so work done for one client or project can be found again. A page behind a login is read with auth_profile: the id (ap_…) of a signed-in browser state the user uploaded from the project's Auth profiles page in the dashboard, which lists their ids, or over the HTTP API (POST /auth_profiles). Every tool but run_lighthouse_audit takes it, and it costs nothing extra.
Names, descriptions and schemas were written by the server and are shown as data.
Pattern matches on what the server declares. They show what it says it can do, not what it does.
None yet. Changes appear after a server's second daily check.
| boolean |
| Fetch the page from a residential (home ISP) exit instead of a datacentre one. Costs a surcharge per page visit — see get_usage. Use it for sites that block datacentre traffic. |
| auth_profile | string | Load the page signed in, as this auth profile's visitor: its id, ap_…. A profile is a browser's signed-in state uploaded once, from the project's Auth profiles page in the dashboard or over the HTTP API (POST /auth_profiles); only URLs on its domains are accepted. Costs nothing extra. https://urlpip… |
| page_options | object | What to do to the page before anything is read off it. Applies to the result itself: removed ads and banners are gone from html, markdown and summaries too, and what scrolling loaded is in all of them. |
| idempotency_key | string | Makes the call safe to retry: sending it again with the same key within 24 hours returns the first call's token and result instead of starting (and charging for) new work, even with max_age "0". Up to 255 printable ASCII characters; a UUID is ideal. |
{
"name": "fetch_markdown",
"title": "Read a page as Markdown",
"description": "Exposes POST /markdown. Renders the page in headless Chrome and converts\nits main content to clean Markdown — headings, lists, links and code\nkept, navigation, sidebars and cookie banners dropped.\n\nStart here when you want to READ a page. It is the cheapest operation we\nsell (1 credit) and by far the most compact thing to put in front of a\nmodel: fetch_html returns the whole DOM, which is usually many times\nlarger and says nothing extra about what the page means.",
"inputSchema": {
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"required": [
"project_id",
"url"
],
"properties": {
"url": {
"type": "string",
"description": "The page to fetch. Must be a public http(s) URL."
},
"sync": {
"type": "boolean",
"description": "Wait for the result and return it (default false). When false the call returns a token immediately and you collect the result with get_result."
},
"labels": {
"type": "object",
"description": "Your own keys to find this request by later, e.g. {\"client\": \"acme\"}: they come back with the result, in the webhook and in list_requests, which can filter by them. Up to 16 keys of up to 40 letters, digits, _ - or .; string values up to 256 characters.",
"additionalProperties": {
"type": "string"
}
},
"max_age": {
"type": "string",
"description": "How fresh a stored result must be to be reused, e.g. \"2 hours\" or \"3 days\". Default 7 days, maximum 30. A reused result is free. Pass \"0\" to force a fresh fetch."
},
"report_to": {
"type": "string",
"description": "Async only: a webhook URL to deliver the result to. Defaults to the project's configured endpoint, if it has one."
},
"project_id": {
"type": "integer",
"description": "Which project this request belongs to. From list_projects."
},
"residential": {
"type": "boolean",
"description": "Fetch the page from a residential (home ISP) exit instead of a datacentre one. Costs a surcharge per page visit — see get_usage. Use it for sites that block datacentre traffic."
},
"auth_profile": {
"type": "string",
"description": "Load the page signed in, as this auth profile's visitor: its id, ap_…. A profile is a browser's signed-in state uploaded once, from the project's Auth profiles page in the dashboard or over the HTTP API (POST /auth_profiles); only URLs on its domains are accepted. Costs nothing extra. https://urlpipe.dev/docs/auth-profiles"
},
"page_options": {
"type": "object",
"properties": {
"delay": {
"type": "integer",
"description": "Milliseconds to wait after the page has settled, 0 to 10000."
},
"block_ads": {
"type": "boolean",
"description": "Block the major ad networks and remove their slots from the page."
},
"block_popups": {
"type": "boolean",
"description": "Remove newsletter, app-install and promotional popups laid over the page."
},
"remove_selectors": {
"type": "array",
"items": {
"type": "string"
},
"description": "CSS selectors of elements to remove from the page, up to 50."
},
"scroll_to_bottom": {
"type": "boolean",
"description": "Scroll down the page before reading it, for as long as scrolling loads more (up to 25s). For listings and feeds that load their content as they are scrolled."
},
"wait_for_selector": {
"type": "string",
"description": "Wait (up to 10s) for an element matching this CSS selector before reading the page."
},
"block_cookie_banners": {
"type": "boolean",
"description": "Remove cookie-consent banners from the page."
}
},
"description": "What to do to the page before anything is read off it. Applies to the result itself: removed ads and banners are gone from html, markdown and summaries too, and what scrolling loaded is in all of them."
},
"idempotency_key": {
"type": "string",
"description": "Makes the call safe to retry: sending it again with the same key within 24 hours returns the first call's token and result instead of starting (and charging for) new work, even with max_age \"0\". Up to 255 printable ASCII characters; a UUID is ideal."
}
}
},
"outputSchema": {
"type": "object",
"anyOf": [
{
"type": "object",
"required": [
"markdown"
],
"properties": {
"markdown": {
"type": "string",
"description": "The page's main content as Markdown."
}
}
},
{
"type": "object",
"required": [
"token",
"status"
],
"properties": {
"token": {
"type": "string"
},
"labels": {
"type": "object"
},
"status": {
"type": "string",
"const": "accepted"
}
}
}
],
"$schema": "https://json-schema.org/draft/2020-12/schema"
},
"annotations": {
"title": "Read a page as Markdown",
"readOnlyHint": false,
"openWorldHint": true,
"idempotentHint": true,
"destructiveHint": false
}
}| Fetch the page from a residential (home ISP) exit instead of a datacentre one. Costs a surcharge per page visit — see get_usage. Use it for sites that block datacentre traffic. |
| auth_profile | string | Load the page signed in, as this auth profile's visitor: its id, ap_…. A profile is a browser's signed-in state uploaded once, from the project's Auth profiles page in the dashboard or over the HTTP API (POST /auth_profiles); only URLs on its domains are accepted. Costs nothing extra. https://urlpip… |
| page_options | object | What to do to the page before anything is read off it. Applies to the result itself: removed ads and banners are gone from html, markdown and summaries too, and what scrolling loaded is in all of them. |
| idempotency_key | string | Makes the call safe to retry: sending it again with the same key within 24 hours returns the first call's token and result instead of starting (and charging for) new work, even with max_age "0". Up to 255 printable ASCII characters; a UUID is ideal. |
{
"name": "fetch_html",
"title": "Get the rendered HTML",
"description": "Exposes POST /html. Returns the page's HTML after JavaScript has run and\nredirects have been followed — the DOM a real browser sees, not the empty\nshell curl returns. 1 credit.\n\nUse it when you need the markup itself: a specific attribute, a script\ntag, a structured-data block. If you want to read the page's content,\nfetch_markdown costs the same and returns a fraction of the text.",
"inputSchema": {
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"required": [
"project_id",
"url"
],
"properties": {
"url": {
"type": "string",
"description": "The page to fetch. Must be a public http(s) URL."
},
"sync": {
"type": "boolean",
"description": "Wait for the result and return it (default false). When false the call returns a token immediately and you collect the result with get_result."
},
"labels": {
"type": "object",
"description": "Your own keys to find this request by later, e.g. {\"client\": \"acme\"}: they come back with the result, in the webhook and in list_requests, which can filter by them. Up to 16 keys of up to 40 letters, digits, _ - or .; string values up to 256 characters.",
"additionalProperties": {
"type": "string"
}
},
"max_age": {
"type": "string",
"description": "How fresh a stored result must be to be reused, e.g. \"2 hours\" or \"3 days\". Default 7 days, maximum 30. A reused result is free. Pass \"0\" to force a fresh fetch."
},
"report_to": {
"type": "string",
"description": "Async only: a webhook URL to deliver the result to. Defaults to the project's configured endpoint, if it has one."
},
"project_id": {
"type": "integer",
"description": "Which project this request belongs to. From list_projects."
},
"residential": {
"type": "boolean",
"description": "Fetch the page from a residential (home ISP) exit instead of a datacentre one. Costs a surcharge per page visit — see get_usage. Use it for sites that block datacentre traffic."
},
"auth_profile": {
"type": "string",
"description": "Load the page signed in, as this auth profile's visitor: its id, ap_…. A profile is a browser's signed-in state uploaded once, from the project's Auth profiles page in the dashboard or over the HTTP API (POST /auth_profiles); only URLs on its domains are accepted. Costs nothing extra. https://urlpipe.dev/docs/auth-profiles"
},
"page_options": {
"type": "object",
"properties": {
"delay": {
"type": "integer",
"description": "Milliseconds to wait after the page has settled, 0 to 10000."
},
"block_ads": {
"type": "boolean",
"description": "Block the major ad networks and remove their slots from the page."
},
"block_popups": {
"type": "boolean",
"description": "Remove newsletter, app-install and promotional popups laid over the page."
},
"remove_selectors": {
"type": "array",
"items": {
"type": "string"
},
"description": "CSS selectors of elements to remove from the page, up to 50."
},
"scroll_to_bottom": {
"type": "boolean",
"description": "Scroll down the page before reading it, for as long as scrolling loads more (up to 25s). For listings and feeds that load their content as they are scrolled."
},
"wait_for_selector": {
"type": "string",
"description": "Wait (up to 10s) for an element matching this CSS selector before reading the page."
},
"block_cookie_banners": {
"type": "boolean",
"description": "Remove cookie-consent banners from the page."
}
},
"description": "What to do to the page before anything is read off it. Applies to the result itself: removed ads and banners are gone from html, markdown and summaries too, and what scrolling loaded is in all of them."
},
"idempotency_key": {
"type": "string",
"description": "Makes the call safe to retry: sending it again with the same key within 24 hours returns the first call's token and result instead of starting (and charging for) new work, even with max_age \"0\". Up to 255 printable ASCII characters; a UUID is ideal."
}
}
},
"outputSchema": {
"type": "object",
"anyOf": [
{
"type": "object",
"required": [
"html"
],
"properties": {
"html": {
"type": "string",
"description": "The rendered HTML."
}
}
},
{
"type": "object",
"required": [
"token",
"status"
],
"properties": {
"token": {
"type": "string"
},
"labels": {
"type": "object"
},
"status": {
"type": "string",
"const": "accepted"
}
}
}
],
"$schema": "https://json-schema.org/draft/2020-12/schema"
},
"annotations": {
"title": "Get the rendered HTML",
"readOnlyHint": false,
"openWorldHint": true,
"idempotentHint": true,
"destructiveHint": false
}
}| residential | boolean | Fetch the page from a residential (home ISP) exit instead of a datacentre one. Costs a surcharge per page visit — see get_usage. Use it for sites that block datacentre traffic. |
| auth_profile | string | Load the page signed in, as this auth profile's visitor: its id, ap_…. A profile is a browser's signed-in state uploaded once, from the project's Auth profiles page in the dashboard or over the HTTP API (POST /auth_profiles); only URLs on its domains are accepted. Costs nothing extra. https://urlpip… |
| page_options | object | What to do to the page before anything is read off it. Applies to the result itself: removed ads and banners are gone from html, markdown and summaries too, and what scrolling loaded is in all of them. |
| idempotency_key | string | Makes the call safe to retry: sending it again with the same key within 24 hours returns the first call's token and result instead of starting (and charging for) new work, even with max_age "0". Up to 255 printable ASCII characters; a UUID is ideal. |
| screenshot_options | object | How to take the screenshot. Every key is optional; leave it out for a full-page PNG. |
{
"name": "capture_screenshot",
"title": "Take a screenshot",
"description": "Exposes POST /screenshot. Captures the whole rendered page as a PNG —\nJavaScript executed, web fonts and images included, exactly as a browser\nwould draw it. 1 credit, whichever options you use.\n\nUse it to see a page rather than read it: layout, visual regressions,\nlink previews, or checking what an anti-bot page actually showed us.\nscreenshot_options size it (viewport, scale, one element, the fold\nonly), encode it (png, jpeg or webp) and restyle it (dark mode, hidden\nelements, your own CSS); page_options wait for the page and take ads\nand cookie banners out of it.",
"inputSchema": {
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"required": [
"project_id",
"url"
],
"properties": {
"url": {
"type": "string",
"description": "The page to fetch. Must be a public http(s) URL."
},
"sync": {
"type": "boolean",
"description": "Wait for the result and return it (default false). When false the call returns a token immediately and you collect the result with get_result."
},
"labels": {
"type": "object",
"description": "Your own keys to find this request by later, e.g. {\"client\": \"acme\"}: they come back with the result, in the webhook and in list_requests, which can filter by them. Up to 16 keys of up to 40 letters, digits, _ - or .; string values up to 256 characters.",
"additionalProperties": {
"type": "string"
}
},
"max_age": {
"type": "string",
"description": "How fresh a stored result must be to be reused, e.g. \"2 hours\" or \"3 days\". Default 7 days, maximum 30. A reused result is free. Pass \"0\" to force a fresh fetch."
},
"report_to": {
"type": "string",
"description": "Async only: a webhook URL to deliver the result to. Defaults to the project's configured endpoint, if it has one."
},
"project_id": {
"type": "integer",
"description": "Which project this request belongs to. From list_projects."
},
"residential": {
"type": "boolean",
"description": "Fetch the page from a residential (home ISP) exit instead of a datacentre one. Costs a surcharge per page visit — see get_usage. Use it for sites that block datacentre traffic."
},
"auth_profile": {
"type": "string",
"description": "Load the page signed in, as this auth profile's visitor: its id, ap_…. A profile is a browser's signed-in state uploaded once, from the project's Auth profiles page in the dashboard or over the HTTP API (POST /auth_profiles); only URLs on its domains are accepted. Costs nothing extra. https://urlpipe.dev/docs/auth-profiles"
},
"page_options": {
"type": "object",
"properties": {
"delay": {
"type": "integer",
"description": "Milliseconds to wait after the page has settled, 0 to 10000."
},
"block_ads": {
"type": "boolean",
"description": "Block the major ad networks and remove their slots from the page."
},
"block_popups": {
"type": "boolean",
"description": "Remove newsletter, app-install and promotional popups laid over the page."
},
"remove_selectors": {
"type": "array",
"items": {
"type": "string"
},
"description": "CSS selectors of elements to remove from the page, up to 50."
},
"scroll_to_bottom": {
"type": "boolean",
"description": "Scroll down the page before reading it, for as long as scrolling loads more (up to 25s). For listings and feeds that load their content as they are scrolled."
},
"wait_for_selector": {
"type": "string",
"description": "Wait (up to 10s) for an element matching this CSS selector before reading the page."
},
"block_cookie_banners": {
"type": "boolean",
"description": "Remove cookie-consent banners from the page."
}
},
"description": "What to do to the page before anything is read off it. Applies to the result itself: removed ads and banners are gone from html, markdown and summaries too, and what scrolling loaded is in all of them."
},
"idempotency_key": {
"type": "string",
"description": "Makes the call safe to retry: sending it again with the same key within 24 hours returns the first call's token and result instead of starting (and charging for) new work, even with max_age \"0\". Up to 255 printable ASCII characters; a UUID is ideal."
},
"screenshot_options": {
"type": "object",
"properties": {
"format": {
"enum": [
"png",
"jpeg",
"webp"
],
"type": "string",
"description": "Image format. Default png; webp is usually the smallest."
},
"styles": {
"type": "string",
"description": "CSS to inject into the page before capturing, up to 20000 characters."
},
"quality": {
"type": "integer",
"description": "jpeg and webp only, 1 to 100. Default 80."
},
"selector": {
"type": "string",
"description": "CSS selector of one element to capture instead of the page."
},
"dark_mode": {
"type": "boolean",
"description": "Render with prefers-color-scheme: dark."
},
"full_page": {
"type": "boolean",
"description": "Capture the whole page top to bottom (default true). false captures the viewport only."
},
"hide_selectors": {
"type": "array",
"items": {
"type": "string"
},
"description": "CSS selectors of elements to hide before capturing, up to 50. They stay in the page's html; page_options.remove_selectors takes them out of it."
},
"viewport_width": {
"type": "integer",
"description": "Browser viewport width in CSS pixels, 320 to 1920. Default 1350."
},
"omit_background": {
"type": "boolean",
"description": "Transparent background where the page sets none. png and webp only."
},
"viewport_height": {
"type": "integer",
"description": "Browser viewport height in CSS pixels, 240 to 1080. Default 797."
},
"device_scale_factor": {
"type": "integer",
"description": "Pixel density, 1 to 3: 2 gives a retina image twice the size. Default 1."
},
"full_page_max_height": {
"type": "integer",
"description": "The tallest a full-page or element capture may be, in CSS pixels: 100 to 16384, the default."
}
},
"description": "How to take the screenshot. Every key is optional; leave it out for a full-page PNG."
}
}
},
"outputSchema": {
"type": "object",
"anyOf": [
{
"type": "object",
"required": [
"mime_type"
],
"properties": {
"mime_type": {
"type": "string",
"description": "The image's type. The image itself is the image content block."
},
"result_url": {
"type": [
"string",
"null"
],
"description": "A link to the image that needs no key."
}
}
},
{
"type": "object",
"required": [
"token",
"status"
],
"properties": {
"token": {
"type": "string"
},
"labels": {
"type": "object"
},
"status": {
"type": "string",
"const": "accepted"
}
}
}
],
"$schema": "https://json-schema.org/draft/2020-12/schema"
},
"annotations": {
"title": "Take a screenshot",
"readOnlyHint": false,
"openWorldHint": true,
"idempotentHint": true,
"destructiveHint": false
}
}| pdf_options | object | How to print the PDF. Every key is optional; leave it out for A4 portrait. |
| residential | boolean | Fetch the page from a residential (home ISP) exit instead of a datacentre one. Costs a surcharge per page visit — see get_usage. Use it for sites that block datacentre traffic. |
| auth_profile | string | Load the page signed in, as this auth profile's visitor: its id, ap_…. A profile is a browser's signed-in state uploaded once, from the project's Auth profiles page in the dashboard or over the HTTP API (POST /auth_profiles); only URLs on its domains are accepted. Costs nothing extra. https://urlpip… |
| page_options | object | What to do to the page before anything is read off it. Applies to the result itself: removed ads and banners are gone from html, markdown and summaries too, and what scrolling loaded is in all of them. |
| idempotency_key | string | Makes the call safe to retry: sending it again with the same key within 24 hours returns the first call's token and result instead of starting (and charging for) new work, even with max_age "0". Up to 255 printable ASCII characters; a UUID is ideal. |
{
"name": "capture_pdf",
"title": "Save a page as PDF",
"description": "Exposes POST /pdf. Prints the whole rendered page to a PDF — JavaScript\nexecuted, web fonts and lazy-loaded images included, its headings as\nbookmarks. 1 credit, whichever options you use.\n\nUse it to archive or hand on a page as a document: a receipt, an article,\na report, terms as they stood today. pdf_options choose the paper, the\norientation, the margins and the scale, whether backgrounds print,\nwhether the page's print stylesheet or its on-screen look is used, and\nwhether to cut it into sheets at all; page_options wait for the page and\ntake ads and cookie banners out of it.",
"inputSchema": {
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"required": [
"project_id",
"url"
],
"properties": {
"url": {
"type": "string",
"description": "The page to fetch. Must be a public http(s) URL."
},
"sync": {
"type": "boolean",
"description": "Wait for the result and return it (default false). When false the call returns a token immediately and you collect the result with get_result."
},
"labels": {
"type": "object",
"description": "Your own keys to find this request by later, e.g. {\"client\": \"acme\"}: they come back with the result, in the webhook and in list_requests, which can filter by them. Up to 16 keys of up to 40 letters, digits, _ - or .; string values up to 256 characters.",
"additionalProperties": {
"type": "string"
}
},
"max_age": {
"type": "string",
"description": "How fresh a stored result must be to be reused, e.g. \"2 hours\" or \"3 days\". Default 7 days, maximum 30. A reused result is free. Pass \"0\" to force a fresh fetch."
},
"report_to": {
"type": "string",
"description": "Async only: a webhook URL to deliver the result to. Defaults to the project's configured endpoint, if it has one."
},
"project_id": {
"type": "integer",
"description": "Which project this request belongs to. From list_projects."
},
"pdf_options": {
"type": "object",
"properties": {
"media": {
"enum": [
"print",
"screen"
],
"type": "string",
"description": "print (default) uses the page's print stylesheet; screen prints it as a browser shows it."
},
"scale": {
"type": "integer",
"description": "Scale of the page, 10 to 200 percent. Default 100."
},
"margin": {
"type": "integer",
"description": "Margin on every side, 0 to 50 mm. Default 10."
},
"landscape": {
"type": "boolean",
"description": "Landscape orientation. Default false."
},
"paper_size": {
"enum": [
"letter",
"legal",
"tabloid",
"a3",
"a4",
"a5"
],
"type": "string",
"description": "The paper. Default a4."
},
"single_page": {
"type": "boolean",
"description": "One page as wide as the browser window and as tall as the whole document, instead of paper-sized sheets. Default false."
},
"print_background": {
"type": "boolean",
"description": "Print background colours and images. Default true."
}
},
"description": "How to print the PDF. Every key is optional; leave it out for A4 portrait."
},
"residential": {
"type": "boolean",
"description": "Fetch the page from a residential (home ISP) exit instead of a datacentre one. Costs a surcharge per page visit — see get_usage. Use it for sites that block datacentre traffic."
},
"auth_profile": {
"type": "string",
"description": "Load the page signed in, as this auth profile's visitor: its id, ap_…. A profile is a browser's signed-in state uploaded once, from the project's Auth profiles page in the dashboard or over the HTTP API (POST /auth_profiles); only URLs on its domains are accepted. Costs nothing extra. https://urlpipe.dev/docs/auth-profiles"
},
"page_options": {
"type": "object",
"properties": {
"delay": {
"type": "integer",
"description": "Milliseconds to wait after the page has settled, 0 to 10000."
},
"block_ads": {
"type": "boolean",
"description": "Block the major ad networks and remove their slots from the page."
},
"block_popups": {
"type": "boolean",
"description": "Remove newsletter, app-install and promotional popups laid over the page."
},
"remove_selectors": {
"type": "array",
"items": {
"type": "string"
},
"description": "CSS selectors of elements to remove from the page, up to 50."
},
"scroll_to_bottom": {
"type": "boolean",
"description": "Scroll down the page before reading it, for as long as scrolling loads more (up to 25s). For listings and feeds that load their content as they are scrolled."
},
"wait_for_selector": {
"type": "string",
"description": "Wait (up to 10s) for an element matching this CSS selector before reading the page."
},
"block_cookie_banners": {
"type": "boolean",
"description": "Remove cookie-consent banners from the page."
}
},
"description": "What to do to the page before anything is read off it. Applies to the result itself: removed ads and banners are gone from html, markdown and summaries too, and what scrolling loaded is in all of them."
},
"idempotency_key": {
"type": "string",
"description": "Makes the call safe to retry: sending it again with the same key within 24 hours returns the first call's token and result instead of starting (and charging for) new work, even with max_age \"0\". Up to 255 printable ASCII characters; a UUID is ideal."
}
}
},
"outputSchema": {
"type": "object",
"anyOf": [
{
"type": "object",
"required": [
"mime_type"
],
"properties": {
"mime_type": {
"type": "string",
"description": "Always application/pdf. The document itself is the resource content block."
},
"result_url": {
"type": [
"string",
"null"
],
"description": "A link to the PDF that needs no key."
}
}
},
{
"type": "object",
"required": [
"token",
"status"
],
"properties": {
"token": {
"type": "string"
},
"labels": {
"type": "object"
},
"status": {
"type": "string",
"const": "accepted"
}
}
}
],
"$schema": "https://json-schema.org/draft/2020-12/schema"
},
"annotations": {
"title": "Save a page as PDF",
"readOnlyHint": false,
"openWorldHint": true,
"idempotentHint": true,
"destructiveHint": false
}
}| Fetch the page from a residential (home ISP) exit instead of a datacentre one. Costs a surcharge per page visit — see get_usage. Use it for sites that block datacentre traffic. |
| auth_profile | string | Load the page signed in, as this auth profile's visitor: its id, ap_…. A profile is a browser's signed-in state uploaded once, from the project's Auth profiles page in the dashboard or over the HTTP API (POST /auth_profiles); only URLs on its domains are accepted. Costs nothing extra. https://urlpip… |
| page_options | object | What to do to the page before anything is read off it. Applies to the result itself: removed ads and banners are gone from html, markdown and summaries too, and what scrolling loaded is in all of them. |
| idempotency_key | string | Makes the call safe to retry: sending it again with the same key within 24 hours returns the first call's token and result instead of starting (and charging for) new work, even with max_age "0". Up to 255 printable ASCII characters; a UUID is ideal. |
{
"name": "capture_console_errors",
"title": "Capture console errors",
"description": "Exposes POST /console. Loads the page in headless Chrome and returns\nwhat it reported through console.error and console.warn during load, plus\nuncaught exceptions and unhandled promise rejections (not console.log). 1 credit.\n\nWhat you would open DevTools for: broken third-party scripts and\nclient-side errors you cannot reproduce locally.",
"inputSchema": {
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"required": [
"project_id",
"url"
],
"properties": {
"url": {
"type": "string",
"description": "The page to fetch. Must be a public http(s) URL."
},
"sync": {
"type": "boolean",
"description": "Wait for the result and return it (default false). When false the call returns a token immediately and you collect the result with get_result."
},
"labels": {
"type": "object",
"description": "Your own keys to find this request by later, e.g. {\"client\": \"acme\"}: they come back with the result, in the webhook and in list_requests, which can filter by them. Up to 16 keys of up to 40 letters, digits, _ - or .; string values up to 256 characters.",
"additionalProperties": {
"type": "string"
}
},
"max_age": {
"type": "string",
"description": "How fresh a stored result must be to be reused, e.g. \"2 hours\" or \"3 days\". Default 7 days, maximum 30. A reused result is free. Pass \"0\" to force a fresh fetch."
},
"report_to": {
"type": "string",
"description": "Async only: a webhook URL to deliver the result to. Defaults to the project's configured endpoint, if it has one."
},
"project_id": {
"type": "integer",
"description": "Which project this request belongs to. From list_projects."
},
"residential": {
"type": "boolean",
"description": "Fetch the page from a residential (home ISP) exit instead of a datacentre one. Costs a surcharge per page visit — see get_usage. Use it for sites that block datacentre traffic."
},
"auth_profile": {
"type": "string",
"description": "Load the page signed in, as this auth profile's visitor: its id, ap_…. A profile is a browser's signed-in state uploaded once, from the project's Auth profiles page in the dashboard or over the HTTP API (POST /auth_profiles); only URLs on its domains are accepted. Costs nothing extra. https://urlpipe.dev/docs/auth-profiles"
},
"page_options": {
"type": "object",
"properties": {
"delay": {
"type": "integer",
"description": "Milliseconds to wait after the page has settled, 0 to 10000."
},
"block_ads": {
"type": "boolean",
"description": "Block the major ad networks and remove their slots from the page."
},
"block_popups": {
"type": "boolean",
"description": "Remove newsletter, app-install and promotional popups laid over the page."
},
"remove_selectors": {
"type": "array",
"items": {
"type": "string"
},
"description": "CSS selectors of elements to remove from the page, up to 50."
},
"scroll_to_bottom": {
"type": "boolean",
"description": "Scroll down the page before reading it, for as long as scrolling loads more (up to 25s). For listings and feeds that load their content as they are scrolled."
},
"wait_for_selector": {
"type": "string",
"description": "Wait (up to 10s) for an element matching this CSS selector before reading the page."
},
"block_cookie_banners": {
"type": "boolean",
"description": "Remove cookie-consent banners from the page."
}
},
"description": "What to do to the page before anything is read off it. Applies to the result itself: removed ads and banners are gone from html, markdown and summaries too, and what scrolling loaded is in all of them."
},
"idempotency_key": {
"type": "string",
"description": "Makes the call safe to retry: sending it again with the same key within 24 hours returns the first call's token and result instead of starting (and charging for) new work, even with max_age \"0\". Up to 255 printable ASCII characters; a UUID is ideal."
}
}
},
"outputSchema": {
"type": "object",
"anyOf": [
{
"type": "object",
"required": [
"entries"
],
"properties": {
"entries": {
"type": "array",
"items": {
"type": "object",
"properties": {
"text": {
"type": "string"
},
"type": {
"enum": [
"error",
"warning",
"exception"
],
"type": "string"
}
}
}
}
}
},
{
"type": "object",
"required": [
"token",
"status"
],
"properties": {
"token": {
"type": "string"
},
"labels": {
"type": "object"
},
"status": {
"type": "string",
"const": "accepted"
}
}
}
],
"$schema": "https://json-schema.org/draft/2020-12/schema"
},
"annotations": {
"title": "Capture console errors",
"readOnlyHint": false,
"openWorldHint": true,
"idempotentHint": true,
"destructiveHint": false
}
}| Which project this request belongs to. From list_projects. |
| residential | boolean | Fetch the page from a residential (home ISP) exit instead of a datacentre one. Costs a surcharge per page visit — see get_usage. Use it for sites that block datacentre traffic. |
| include_audits | boolean | Include the full per-audit detail, not just scores and metrics. Much larger; default false. |
| idempotency_key | string | Makes the call safe to retry: sending it again with the same key within 24 hours returns the first call's token and result instead of starting (and charging for) new work, even with max_age "0". Up to 255 printable ASCII characters; a UUID is ideal. |
{
"name": "run_lighthouse_audit",
"title": "Run a Lighthouse audit",
"description": "Exposes POST /lighthouse. Runs a real Google Lighthouse audit against the\nlive page: performance, accessibility, best-practices and SEO scores plus\nCore Web Vitals. 2 credits — dearer than a page fetch because the audit\nruns in its own limited lane, so it also takes longer than anything else\nhere.\n\nIt fetches the page itself rather than sharing a visit, so asking for it\ninside scrape_url costs two page visits, not one.",
"inputSchema": {
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"required": [
"project_id",
"url"
],
"properties": {
"url": {
"type": "string",
"description": "The page to fetch. Must be a public http(s) URL."
},
"sync": {
"type": "boolean",
"description": "Wait for the result and return it (default false). When false the call returns a token immediately and you collect the result with get_result."
},
"device": {
"enum": [
"mobile",
"desktop"
],
"type": "string",
"description": "Which profile to audit under. Default mobile."
},
"labels": {
"type": "object",
"description": "Your own keys to find this request by later, e.g. {\"client\": \"acme\"}: they come back with the result, in the webhook and in list_requests, which can filter by them. Up to 16 keys of up to 40 letters, digits, _ - or .; string values up to 256 characters.",
"additionalProperties": {
"type": "string"
}
},
"max_age": {
"type": "string",
"description": "How fresh a stored result must be to be reused, e.g. \"2 hours\" or \"3 days\". Default 7 days, maximum 30. A reused result is free. Pass \"0\" to force a fresh fetch."
},
"report_to": {
"type": "string",
"description": "Async only: a webhook URL to deliver the result to. Defaults to the project's configured endpoint, if it has one."
},
"project_id": {
"type": "integer",
"description": "Which project this request belongs to. From list_projects."
},
"residential": {
"type": "boolean",
"description": "Fetch the page from a residential (home ISP) exit instead of a datacentre one. Costs a surcharge per page visit — see get_usage. Use it for sites that block datacentre traffic."
},
"include_audits": {
"type": "boolean",
"description": "Include the full per-audit detail, not just scores and metrics. Much larger; default false."
},
"idempotency_key": {
"type": "string",
"description": "Makes the call safe to retry: sending it again with the same key within 24 hours returns the first call's token and result instead of starting (and charging for) new work, even with max_age \"0\". Up to 255 printable ASCII characters; a UUID is ideal."
}
}
},
"outputSchema": {
"type": "object",
"anyOf": [
{
"type": "object",
"properties": {
"url": {
"type": "string"
},
"audits": {
"type": "object"
},
"device": {
"type": "string"
},
"metrics": {
"type": "object"
},
"fetchTime": {
"type": "string"
},
"categories": {
"type": "object"
}
}
},
{
"type": "object",
"required": [
"token",
"status"
],
"properties": {
"token": {
"type": "string"
},
"labels": {
"type": "object"
},
"status": {
"type": "string",
"const": "accepted"
}
}
}
],
"$schema": "https://json-schema.org/draft/2020-12/schema"
},
"annotations": {
"title": "Run a Lighthouse audit",
"readOnlyHint": false,
"openWorldHint": true,
"idempotentHint": true,
"destructiveHint": false
}
}| Which project this request belongs to. From list_projects. |
| residential | boolean | Fetch the page from a residential (home ISP) exit instead of a datacentre one. Costs a surcharge per page visit — see get_usage. Use it for sites that block datacentre traffic. |
| auth_profile | string | Load the page signed in, as this auth profile's visitor: its id, ap_…. A profile is a browser's signed-in state uploaded once, from the project's Auth profiles page in the dashboard or over the HTTP API (POST /auth_profiles); only URLs on its domains are accepted. Costs nothing extra. https://urlpip… |
| page_options | object | What to do to the page before anything is read off it. Applies to the result itself: removed ads and banners are gone from html, markdown and summaries too, and what scrolling loaded is in all of them. |
| idempotency_key | string | Makes the call safe to retry: sending it again with the same key within 24 hours returns the first call's token and result instead of starting (and charging for) new work, even with max_age "0". Up to 255 printable ASCII characters; a UUID is ideal. |
{
"name": "extract_metadata",
"title": "Extract page metadata",
"description": "Exposes POST /meta. Reads everything a page declares about itself —\nOpen Graph, Twitter cards, meta tags, JSON-LD, microdata and links — into\none object with the same keys on every page: title, description, site\nname, kind of page, language, authors, publication and update dates, share\nimage and video, favicon and icons, logo, feeds, keywords, canonical URL,\nrobots directives, hreflang alternates, oEmbed endpoint and structured-data\ntypes, plus the raw og: and twitter: tags. A value the page does not\ndeclare is null or an empty list. Values injected by JavaScript are\nincluded, because the page is rendered first. 1 credit.",
"inputSchema": {
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"required": [
"project_id",
"url"
],
"properties": {
"url": {
"type": "string",
"description": "The page to fetch. Must be a public http(s) URL."
},
"sync": {
"type": "boolean",
"description": "Wait for the result and return it (default false). When false the call returns a token immediately and you collect the result with get_result."
},
"labels": {
"type": "object",
"description": "Your own keys to find this request by later, e.g. {\"client\": \"acme\"}: they come back with the result, in the webhook and in list_requests, which can filter by them. Up to 16 keys of up to 40 letters, digits, _ - or .; string values up to 256 characters.",
"additionalProperties": {
"type": "string"
}
},
"max_age": {
"type": "string",
"description": "How fresh a stored result must be to be reused, e.g. \"2 hours\" or \"3 days\". Default 7 days, maximum 30. A reused result is free. Pass \"0\" to force a fresh fetch."
},
"report_to": {
"type": "string",
"description": "Async only: a webhook URL to deliver the result to. Defaults to the project's configured endpoint, if it has one."
},
"project_id": {
"type": "integer",
"description": "Which project this request belongs to. From list_projects."
},
"residential": {
"type": "boolean",
"description": "Fetch the page from a residential (home ISP) exit instead of a datacentre one. Costs a surcharge per page visit — see get_usage. Use it for sites that block datacentre traffic."
},
"auth_profile": {
"type": "string",
"description": "Load the page signed in, as this auth profile's visitor: its id, ap_…. A profile is a browser's signed-in state uploaded once, from the project's Auth profiles page in the dashboard or over the HTTP API (POST /auth_profiles); only URLs on its domains are accepted. Costs nothing extra. https://urlpipe.dev/docs/auth-profiles"
},
"page_options": {
"type": "object",
"properties": {
"delay": {
"type": "integer",
"description": "Milliseconds to wait after the page has settled, 0 to 10000."
},
"block_ads": {
"type": "boolean",
"description": "Block the major ad networks and remove their slots from the page."
},
"block_popups": {
"type": "boolean",
"description": "Remove newsletter, app-install and promotional popups laid over the page."
},
"remove_selectors": {
"type": "array",
"items": {
"type": "string"
},
"description": "CSS selectors of elements to remove from the page, up to 50."
},
"scroll_to_bottom": {
"type": "boolean",
"description": "Scroll down the page before reading it, for as long as scrolling loads more (up to 25s). For listings and feeds that load their content as they are scrolled."
},
"wait_for_selector": {
"type": "string",
"description": "Wait (up to 10s) for an element matching this CSS selector before reading the page."
},
"block_cookie_banners": {
"type": "boolean",
"description": "Remove cookie-consent banners from the page."
}
},
"description": "What to do to the page before anything is read off it. Applies to the result itself: removed ads and banners are gone from html, markdown and summaries too, and what scrolling loaded is in all of them."
},
"idempotency_key": {
"type": "string",
"description": "Makes the call safe to retry: sending it again with the same key within 24 hours returns the first call's token and result instead of starting (and charging for) new work, even with max_age \"0\". Up to 255 printable ASCII characters; a UUID is ideal."
}
}
},
"outputSchema": {
"type": "object",
"anyOf": [
{
"type": "object",
"required": [
"final_url",
"canonical_url",
"title",
"description",
"site_name",
"type",
"language",
"locale",
"authors",
"published_at",
"modified_at",
"image",
"video",
"favicon_url",
"icons",
"logo_url",
"feeds",
"keywords",
"theme_color",
"generator",
"manifest_url",
"robots",
"open_graph",
"twitter",
"alternates",
"oembed_url",
"structured_data_types"
],
"properties": {
"type": {
"enum": [
"article",
"website",
"profile",
"book",
"product",
"video",
"music",
"place",
"event",
"recipe",
null
],
"type": [
"string",
"null"
],
"description": "The kind of page, as the page declares it."
},
"feeds": {
"type": "array",
"items": {
"type": "object",
"required": [
"url",
"type",
"title"
],
"properties": {
"url": {
"type": "string"
},
"type": {
"type": [
"string",
"null"
]
},
"title": {
"type": [
"string",
"null"
]
}
},
"additionalProperties": false
}
},
"icons": {
"type": "array",
"items": {
"type": "object",
"required": [
"url",
"rel",
"sizes",
"type"
],
"properties": {
"rel": {
"type": "string"
},
"url": {
"type": "string"
},
"type": {
"type": [
"string",
"null"
]
},
"sizes": {
"type": [
"string",
"null"
]
}
},
"additionalProperties": false
}
},
"image": {
"type": [
"object",
"null"
],
"required": [
"url",
"width",
"height",
"alt",
"type"
],
"properties": {
"alt": {
"type": [
"string",
"null"
]
},
"url": {
"type": "string"
},
"type": {
"type": [
"string",
"null"
]
},
"width": {
"type": [
"integer",
"null"
]
},
"height": {
"type": [
"integer",
"null"
]
}
},
"additionalProperties": false
},
"title": {
"type": [
"string",
"null"
],
"description": "The page's own title, without the site's name around it."
},
"video": {
"type": [
"object",
"null"
],
"required": [
"url",
"type",
"width",
"height"
],
"properties": {
"url": {
"type": "string"
},
"type": {
"type": [
"string",
"null"
]
},
"width": {
"type": [
"integer",
"null"
]
},
"height": {
"type": [
"integer",
"null"
]
}
},
"additionalProperties": false
},
"locale": {
"type": [
"string",
"null"
],
"description": "The most specific language tag the page declares for that language, BCP 47 (en-US)."
},
"robots": {
"type": "object",
"required": [
"index",
"follow"
],
"properties": {
"index": {
"type": "boolean"
},
"follow": {
"type": "boolean"
}
},
"additionalProperties": false
},
"authors": {
"type": "array",
"items": {
"type": "object",
"required": [
"name",
"url",
"profiles"
],
"properties": {
"url": {
"type": [
"string",
"null"
]
},
"name": {
"type": "string"
},
"profiles": {
"type": "array",
"items": {
"type": "string"
}
}
},
"additionalProperties": false
}
},
"twitter": {
"type": "object",
"required": [
"card",
"site",
"creator",
"title",
"description",
"image"
],
"properties": {
"card": {
"type": [
"string",
"null"
]
},
"site": {
"type": [
"string",
"null"
]
},
"image": {
"type": [
"string",
"null"
]
},
"title": {
"type": [
"string",
"null"
]
},
"creator": {
"type": [
"string",
"null"
]
},
"description": {
"type": [
"string",
"null"
]
}
},
"description": "The page's twitter: tags as declared.",
"additionalProperties": false
},
"keywords": {
"type": "array",
"items": {
"type": "string"
}
},
"language": {
"type": [
"string",
"null"
],
"description": "ISO 639-1 code of the language the content is written in."
},
"logo_url": {
"type": [
"string",
"null"
]
},
"final_url": {
"type": [
"string",
"null"
],
"description": "The address the page ended on, after its redirects."
},
"generator": {
"type": [
"string",
"null"
]
},
"site_name": {
"type": [
"string",
"null"
]
},
"alternates": {
"type": "array",
"items": {
"type": "object",
"required": [
"hreflang",
"url"
],
"properties": {
"url": {
"type": "string"
},
"hreflang": {
"type": "string"
}
},
"additionalProperties": false
}
},
"oembed_url": {
"type": [
"string",
"null"
]
},
"open_graph": {
"type": "object",
"required": [
"title",
"description",
"image",
"url",
"type",
"site_name",
"locale"
],
"properties": {
"url": {
"type": [
"string",
"null"
]
},
"type": {
"type": [
"string",
"null"
]
},
"image": {
"type": [
"string",
"null"
]
},
"title": {
"type": [
"string",
"null"
]
},
"locale": {
"type": [
"string",
"null"
]
},
"site_name": {
"type": [
"string",
"null"
]
},
"description": {
"type": [
"string",
"null"
]
}
},
"description": "The page's og: tags as declared.",
"additionalProperties": false
},
"description": {
"type": [
"string",
"null"
]
},
"favicon_url": {
"type": [
"string",
"null"
]
},
"modified_at": {
"type": [
"string",
"null"
],
"description": "ISO 8601, as published_at."
},
"theme_color": {
"type": [
"string",
"null"
]
},
"manifest_url": {
"type": [
"string",
"null"
]
},
"published_at": {
"type": [
"string",
"null"
],
"description": "ISO 8601: a date (2024-05-01) or a moment with its offset."
},
"canonical_url": {
"type": [
"string",
"null"
],
"description": "The page's canonical URL, absolute."
},
"structured_data_types": {
"type": "array",
"items": {
"type": "string"
}
}
},
"additionalProperties": false
},
{
"type": "object",
"required": [
"token",
"status"
],
"properties": {
"token": {
"type": "string"
},
"labels": {
"type": "object"
},
"status": {
"type": "string",
"const": "accepted"
}
}
}
],
"$schema": "https://json-schema.org/draft/2020-12/schema"
},
"annotations": {
"title": "Extract page metadata",
"readOnlyHint": false,
"openWorldHint": true,
"idempotentHint": true,
"destructiveHint": false
}
}| integer |
| Which project this request belongs to. From list_projects. |
| residential | boolean | Fetch the page from a residential (home ISP) exit instead of a datacentre one. Costs a surcharge per page visit — see get_usage. Use it for sites that block datacentre traffic. |
| auth_profile | string | Load the page signed in, as this auth profile's visitor: its id, ap_…. A profile is a browser's signed-in state uploaded once, from the project's Auth profiles page in the dashboard or over the HTTP API (POST /auth_profiles); only URLs on its domains are accepted. Costs nothing extra. https://urlpip… |
| page_options | object | What to do to the page before anything is read off it. Applies to the result itself: removed ads and banners are gone from html, markdown and summaries too, and what scrolling loaded is in all of them. |
| idempotency_key | string | Makes the call safe to retry: sending it again with the same key within 24 hours returns the first call's token and result instead of starting (and charging for) new work, even with max_age "0". Up to 255 printable ASCII characters; a UUID is ideal. |
{
"name": "detect_technologies",
"title": "Detect a page's technologies",
"description": "Exposes POST /technologies. Detects what a page is built with — CMS,\ne-commerce platform, JavaScript frameworks and libraries, analytics and\ntag managers, advertising and marketing tools, CDN, web server,\nprogramming language, payment providers, chat widgets, consent managers\nand more — from an open database of over 7,000 fingerprints. Each\ntechnology comes with its version when the page gives it away, a\nconfidence from 0 to 100, its categories, its website and CPE, and which\nsignals gave it away (html, meta tags, headers, cookies, JavaScript\nglobals, DOM, script URLs and contents, stylesheets, DNS records,\nrobots.txt). The page is rendered first, so technologies loaded by\nJavaScript are found too. 1 credit.",
"inputSchema": {
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"required": [
"project_id",
"url"
],
"properties": {
"url": {
"type": "string",
"description": "The page to fetch. Must be a public http(s) URL."
},
"sync": {
"type": "boolean",
"description": "Wait for the result and return it (default false). When false the call returns a token immediately and you collect the result with get_result."
},
"labels": {
"type": "object",
"description": "Your own keys to find this request by later, e.g. {\"client\": \"acme\"}: they come back with the result, in the webhook and in list_requests, which can filter by them. Up to 16 keys of up to 40 letters, digits, _ - or .; string values up to 256 characters.",
"additionalProperties": {
"type": "string"
}
},
"max_age": {
"type": "string",
"description": "How fresh a stored result must be to be reused, e.g. \"2 hours\" or \"3 days\". Default 7 days, maximum 30. A reused result is free. Pass \"0\" to force a fresh fetch."
},
"report_to": {
"type": "string",
"description": "Async only: a webhook URL to deliver the result to. Defaults to the project's configured endpoint, if it has one."
},
"project_id": {
"type": "integer",
"description": "Which project this request belongs to. From list_projects."
},
"residential": {
"type": "boolean",
"description": "Fetch the page from a residential (home ISP) exit instead of a datacentre one. Costs a surcharge per page visit — see get_usage. Use it for sites that block datacentre traffic."
},
"auth_profile": {
"type": "string",
"description": "Load the page signed in, as this auth profile's visitor: its id, ap_…. A profile is a browser's signed-in state uploaded once, from the project's Auth profiles page in the dashboard or over the HTTP API (POST /auth_profiles); only URLs on its domains are accepted. Costs nothing extra. https://urlpipe.dev/docs/auth-profiles"
},
"page_options": {
"type": "object",
"properties": {
"delay": {
"type": "integer",
"description": "Milliseconds to wait after the page has settled, 0 to 10000."
},
"block_ads": {
"type": "boolean",
"description": "Block the major ad networks and remove their slots from the page."
},
"block_popups": {
"type": "boolean",
"description": "Remove newsletter, app-install and promotional popups laid over the page."
},
"remove_selectors": {
"type": "array",
"items": {
"type": "string"
},
"description": "CSS selectors of elements to remove from the page, up to 50."
},
"scroll_to_bottom": {
"type": "boolean",
"description": "Scroll down the page before reading it, for as long as scrolling loads more (up to 25s). For listings and feeds that load their content as they are scrolled."
},
"wait_for_selector": {
"type": "string",
"description": "Wait (up to 10s) for an element matching this CSS selector before reading the page."
},
"block_cookie_banners": {
"type": "boolean",
"description": "Remove cookie-consent banners from the page."
}
},
"description": "What to do to the page before anything is read off it. Applies to the result itself: removed ads and banners are gone from html, markdown and summaries too, and what scrolling loaded is in all of them."
},
"idempotency_key": {
"type": "string",
"description": "Makes the call safe to retry: sending it again with the same key within 24 hours returns the first call's token and result instead of starting (and charging for) new work, even with max_age \"0\". Up to 255 printable ASCII characters; a UUID is ideal."
}
}
},
"outputSchema": {
"type": "object",
"anyOf": [
{
"type": "object",
"required": [
"final_url",
"technologies"
],
"properties": {
"final_url": {
"type": [
"string",
"null"
],
"description": "The address the page ended on, after its redirects."
},
"technologies": {
"type": "array",
"items": {
"type": "object",
"required": [
"name",
"slug",
"version",
"confidence",
"categories",
"website",
"description",
"cpe",
"detected_by",
"implied_by"
],
"properties": {
"cpe": {
"type": [
"string",
"null"
],
"description": "The CPE 2.3 name, for looking the technology up in vulnerability databases."
},
"name": {
"type": "string",
"description": "The technology's name, e.g. WordPress."
},
"slug": {
"type": "string",
"description": "A stable identifier for the technology, e.g. wordpress."
},
"version": {
"type": [
"string",
"null"
],
"description": "The version, when the page gives it away."
},
"website": {
"type": [
"string",
"null"
]
},
"categories": {
"type": "array",
"items": {
"type": "object",
"required": [
"slug",
"name"
],
"properties": {
"name": {
"type": "string"
},
"slug": {
"type": "string"
}
},
"additionalProperties": false
}
},
"confidence": {
"type": "integer",
"maximum": 100,
"minimum": 0,
"description": "How sure the detection is, 0–100. Most are 100."
},
"implied_by": {
"type": "array",
"items": {
"type": "string"
},
"description": "The detected technologies that imply this one (WordPress implies PHP)."
},
"description": {
"type": [
"string",
"null"
]
},
"detected_by": {
"type": "array",
"items": {
"type": "string"
},
"description": "Which signals gave it away: html, meta, headers, cookies, js, dom, script_src, scripts, css, text, url, xhr, cert_issuer, dns, robots. Empty for a technology only implied by another."
}
},
"additionalProperties": false
},
"description": "Every technology detected, most certain first."
}
},
"additionalProperties": false
},
{
"type": "object",
"required": [
"token",
"status"
],
"properties": {
"token": {
"type": "string"
},
"labels": {
"type": "object"
},
"status": {
"type": "string",
"const": "accepted"
}
}
}
],
"$schema": "https://json-schema.org/draft/2020-12/schema"
},
"annotations": {
"title": "Detect a page's technologies",
"readOnlyHint": false,
"openWorldHint": true,
"idempotentHint": true,
"destructiveHint": false
}
}| auth_profile | string | Load the page signed in, as this auth profile's visitor: its id, ap_…. A profile is a browser's signed-in state uploaded once, from the project's Auth profiles page in the dashboard or over the HTTP API (POST /auth_profiles); only URLs on its domains are accepted. Costs nothing extra. https://urlpip… |
| page_options | object | What to do to the page before anything is read off it. Applies to the result itself: removed ads and banners are gone from html, markdown and summaries too, and what scrolling loaded is in all of them. |
| idempotency_key | string | Makes the call safe to retry: sending it again with the same key within 24 hours returns the first call's token and result instead of starting (and charging for) new work, even with max_age "0". Up to 255 printable ASCII characters; a UUID is ideal. |
{
"name": "extract_keywords",
"title": "Extract keywords",
"description": "Exposes POST /keywords. Returns the 5–15 terms and phrases that best\nrepresent the page, ordered by relevance — ranked by a language model,\nnot by raw frequency. 15 credits.\n\nAs with summarize_page: if you are going to reason over the result\nyourself, fetch_markdown costs 1 credit and gives you everything.",
"inputSchema": {
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"required": [
"project_id",
"url"
],
"properties": {
"url": {
"type": "string",
"description": "The page to fetch. Must be a public http(s) URL."
},
"sync": {
"type": "boolean",
"description": "Wait for the result and return it (default false). When false the call returns a token immediately and you collect the result with get_result."
},
"labels": {
"type": "object",
"description": "Your own keys to find this request by later, e.g. {\"client\": \"acme\"}: they come back with the result, in the webhook and in list_requests, which can filter by them. Up to 16 keys of up to 40 letters, digits, _ - or .; string values up to 256 characters.",
"additionalProperties": {
"type": "string"
}
},
"max_age": {
"type": "string",
"description": "How fresh a stored result must be to be reused, e.g. \"2 hours\" or \"3 days\". Default 7 days, maximum 30. A reused result is free. Pass \"0\" to force a fresh fetch."
},
"report_to": {
"type": "string",
"description": "Async only: a webhook URL to deliver the result to. Defaults to the project's configured endpoint, if it has one."
},
"project_id": {
"type": "integer",
"description": "Which project this request belongs to. From list_projects."
},
"residential": {
"type": "boolean",
"description": "Fetch the page from a residential (home ISP) exit instead of a datacentre one. Costs a surcharge per page visit — see get_usage. Use it for sites that block datacentre traffic."
},
"auth_profile": {
"type": "string",
"description": "Load the page signed in, as this auth profile's visitor: its id, ap_…. A profile is a browser's signed-in state uploaded once, from the project's Auth profiles page in the dashboard or over the HTTP API (POST /auth_profiles); only URLs on its domains are accepted. Costs nothing extra. https://urlpipe.dev/docs/auth-profiles"
},
"page_options": {
"type": "object",
"properties": {
"delay": {
"type": "integer",
"description": "Milliseconds to wait after the page has settled, 0 to 10000."
},
"block_ads": {
"type": "boolean",
"description": "Block the major ad networks and remove their slots from the page."
},
"block_popups": {
"type": "boolean",
"description": "Remove newsletter, app-install and promotional popups laid over the page."
},
"remove_selectors": {
"type": "array",
"items": {
"type": "string"
},
"description": "CSS selectors of elements to remove from the page, up to 50."
},
"scroll_to_bottom": {
"type": "boolean",
"description": "Scroll down the page before reading it, for as long as scrolling loads more (up to 25s). For listings and feeds that load their content as they are scrolled."
},
"wait_for_selector": {
"type": "string",
"description": "Wait (up to 10s) for an element matching this CSS selector before reading the page."
},
"block_cookie_banners": {
"type": "boolean",
"description": "Remove cookie-consent banners from the page."
}
},
"description": "What to do to the page before anything is read off it. Applies to the result itself: removed ads and banners are gone from html, markdown and summaries too, and what scrolling loaded is in all of them."
},
"idempotency_key": {
"type": "string",
"description": "Makes the call safe to retry: sending it again with the same key within 24 hours returns the first call's token and result instead of starting (and charging for) new work, even with max_age \"0\". Up to 255 printable ASCII characters; a UUID is ideal."
}
}
},
"outputSchema": {
"type": "object",
"anyOf": [
{
"type": "object",
"required": [
"keywords"
],
"properties": {
"keywords": {
"type": "array",
"items": {
"type": "string"
}
}
}
},
{
"type": "object",
"required": [
"token",
"status"
],
"properties": {
"token": {
"type": "string"
},
"labels": {
"type": "object"
},
"status": {
"type": "string",
"const": "accepted"
}
}
}
],
"$schema": "https://json-schema.org/draft/2020-12/schema"
},
"annotations": {
"title": "Extract keywords",
"readOnlyHint": false,
"openWorldHint": true,
"idempotentHint": true,
"destructiveHint": false
}
}| boolean |
| Fetch the page from a residential (home ISP) exit instead of a datacentre one. Costs a surcharge per page visit — see get_usage. Use it for sites that block datacentre traffic. |
| auth_profile | string | Load the page signed in, as this auth profile's visitor: its id, ap_…. A profile is a browser's signed-in state uploaded once, from the project's Auth profiles page in the dashboard or over the HTTP API (POST /auth_profiles); only URLs on its domains are accepted. Costs nothing extra. https://urlpip… |
| page_options | object | What to do to the page before anything is read off it. Applies to the result itself: removed ads and banners are gone from html, markdown and summaries too, and what scrolling loaded is in all of them. |
| idempotency_key | string | Makes the call safe to retry: sending it again with the same key within 24 hours returns the first call's token and result instead of starting (and charging for) new work, even with max_age "0". Up to 255 printable ASCII characters; a UUID is ideal. |
{
"name": "summarize_page",
"title": "Summarize a page",
"description": "Exposes POST /summarize. Returns a concise Markdown summary of the page's\nmain content, with the navigation, ads and boilerplate left out.\n\nThe most expensive operation we sell, at 17 credits, because it runs a\nlanguage model over the page. If YOU are the model that will read it,\nfetch_markdown gives you the whole page for 1 credit and you can\nsummarize it yourself — this tool is for when the summary is the artefact\nbeing produced, not a step on the way to one.",
"inputSchema": {
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"required": [
"project_id",
"url"
],
"properties": {
"url": {
"type": "string",
"description": "The page to fetch. Must be a public http(s) URL."
},
"sync": {
"type": "boolean",
"description": "Wait for the result and return it (default false). When false the call returns a token immediately and you collect the result with get_result."
},
"labels": {
"type": "object",
"description": "Your own keys to find this request by later, e.g. {\"client\": \"acme\"}: they come back with the result, in the webhook and in list_requests, which can filter by them. Up to 16 keys of up to 40 letters, digits, _ - or .; string values up to 256 characters.",
"additionalProperties": {
"type": "string"
}
},
"max_age": {
"type": "string",
"description": "How fresh a stored result must be to be reused, e.g. \"2 hours\" or \"3 days\". Default 7 days, maximum 30. A reused result is free. Pass \"0\" to force a fresh fetch."
},
"report_to": {
"type": "string",
"description": "Async only: a webhook URL to deliver the result to. Defaults to the project's configured endpoint, if it has one."
},
"project_id": {
"type": "integer",
"description": "Which project this request belongs to. From list_projects."
},
"residential": {
"type": "boolean",
"description": "Fetch the page from a residential (home ISP) exit instead of a datacentre one. Costs a surcharge per page visit — see get_usage. Use it for sites that block datacentre traffic."
},
"auth_profile": {
"type": "string",
"description": "Load the page signed in, as this auth profile's visitor: its id, ap_…. A profile is a browser's signed-in state uploaded once, from the project's Auth profiles page in the dashboard or over the HTTP API (POST /auth_profiles); only URLs on its domains are accepted. Costs nothing extra. https://urlpipe.dev/docs/auth-profiles"
},
"page_options": {
"type": "object",
"properties": {
"delay": {
"type": "integer",
"description": "Milliseconds to wait after the page has settled, 0 to 10000."
},
"block_ads": {
"type": "boolean",
"description": "Block the major ad networks and remove their slots from the page."
},
"block_popups": {
"type": "boolean",
"description": "Remove newsletter, app-install and promotional popups laid over the page."
},
"remove_selectors": {
"type": "array",
"items": {
"type": "string"
},
"description": "CSS selectors of elements to remove from the page, up to 50."
},
"scroll_to_bottom": {
"type": "boolean",
"description": "Scroll down the page before reading it, for as long as scrolling loads more (up to 25s). For listings and feeds that load their content as they are scrolled."
},
"wait_for_selector": {
"type": "string",
"description": "Wait (up to 10s) for an element matching this CSS selector before reading the page."
},
"block_cookie_banners": {
"type": "boolean",
"description": "Remove cookie-consent banners from the page."
}
},
"description": "What to do to the page before anything is read off it. Applies to the result itself: removed ads and banners are gone from html, markdown and summaries too, and what scrolling loaded is in all of them."
},
"idempotency_key": {
"type": "string",
"description": "Makes the call safe to retry: sending it again with the same key within 24 hours returns the first call's token and result instead of starting (and charging for) new work, even with max_age \"0\". Up to 255 printable ASCII characters; a UUID is ideal."
}
}
},
"outputSchema": {
"type": "object",
"anyOf": [
{
"type": "object",
"required": [
"summary"
],
"properties": {
"summary": {
"type": "string",
"description": "A Markdown summary of the page."
}
}
},
{
"type": "object",
"required": [
"token",
"status"
],
"properties": {
"token": {
"type": "string"
},
"labels": {
"type": "object"
},
"status": {
"type": "string",
"const": "accepted"
}
}
}
],
"$schema": "https://json-schema.org/draft/2020-12/schema"
},
"annotations": {
"title": "Summarize a page",
"readOnlyHint": false,
"openWorldHint": true,
"idempotentHint": true,
"destructiveHint": false
}
}| project_id * | integer | Which project this request belongs to. From list_projects. |
| residential | boolean | Fetch the page from a residential (home ISP) exit instead of a datacentre one. Costs a surcharge per page visit — see get_usage. Use it for sites that block datacentre traffic. |
| auth_profile | string | Load the page signed in, as this auth profile's visitor: its id, ap_…. A profile is a browser's signed-in state uploaded once, from the project's Auth profiles page in the dashboard or over the HTTP API (POST /auth_profiles); only URLs on its domains are accepted. Costs nothing extra. https://urlpip… |
| instructions | string | Optional guidance that applies to every field, e.g. "prices in EUR". |
| page_options | object | What to do to the page before anything is read off it. Applies to the result itself: removed ads and banners are gone from html, markdown and summaries too, and what scrolling loaded is in all of them. |
| idempotency_key | string | Makes the call safe to retry: sending it again with the same key within 24 hours returns the first call's token and result instead of starting (and charging for) new work, even with max_age "0". Up to 255 printable ASCII characters; a UUID is ideal. |
{
"name": "extract_data",
"title": "Extract structured data with your own schema",
"description": "Exposes POST /extract. Reads the page and fills an object YOU define:\npass `fields`, each with a name, a type and a description of what to take\nfrom the page, and get back exactly those keys in that order. Anything the\npage does not give comes back null — values are never guessed.\nTypes: string, number, integer, boolean, date, datetime, url, email, object (with its own\nfields) and array (with items). Use it for product details, prices,\ncontact data, job listings, event dates, lists of items with several\nattributes each. 24 credits: it reads the\nwhole page with an AI model, so prefer extract_metadata when the standard\nmetadata is all you need.",
"inputSchema": {
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"required": [
"project_id",
"url",
"fields"
],
"properties": {
"url": {
"type": "string",
"description": "The page to fetch. Must be a public http(s) URL."
},
"sync": {
"type": "boolean",
"description": "Wait for the result and return it (default false). When false the call returns a token immediately and you collect the result with get_result."
},
"fields": {
"type": "array",
"items": {
"type": "object",
"required": [
"name",
"type"
],
"properties": {
"name": {
"type": "string",
"description": "The key in the result: letters, digits and underscores."
},
"type": {
"enum": [
"string",
"number",
"integer",
"boolean",
"date",
"datetime",
"url",
"email",
"object",
"array"
],
"type": "string"
},
"items": {
"anyOf": [
{
"type": "string"
},
{
"type": "object"
}
],
"description": "For an array: what it holds — a type such as \"string\", or {\"type\": \"object\", \"fields\": [...]}."
},
"fields": {
"type": "array",
"items": {
"type": "object"
},
"description": "For an object: its fields, shaped like this one."
},
"description": {
"type": "string",
"description": "What to take from the page. The more specific, the better."
}
}
},
"description": "The fields to extract, at most 50 in all."
},
"labels": {
"type": "object",
"description": "Your own keys to find this request by later, e.g. {\"client\": \"acme\"}: they come back with the result, in the webhook and in list_requests, which can filter by them. Up to 16 keys of up to 40 letters, digits, _ - or .; string values up to 256 characters.",
"additionalProperties": {
"type": "string"
}
},
"max_age": {
"type": "string",
"description": "How fresh a stored result must be to be reused, e.g. \"2 hours\" or \"3 days\". Default 7 days, maximum 30. A reused result is free. Pass \"0\" to force a fresh fetch."
},
"report_to": {
"type": "string",
"description": "Async only: a webhook URL to deliver the result to. Defaults to the project's configured endpoint, if it has one."
},
"project_id": {
"type": "integer",
"description": "Which project this request belongs to. From list_projects."
},
"residential": {
"type": "boolean",
"description": "Fetch the page from a residential (home ISP) exit instead of a datacentre one. Costs a surcharge per page visit — see get_usage. Use it for sites that block datacentre traffic."
},
"auth_profile": {
"type": "string",
"description": "Load the page signed in, as this auth profile's visitor: its id, ap_…. A profile is a browser's signed-in state uploaded once, from the project's Auth profiles page in the dashboard or over the HTTP API (POST /auth_profiles); only URLs on its domains are accepted. Costs nothing extra. https://urlpipe.dev/docs/auth-profiles"
},
"instructions": {
"type": "string",
"description": "Optional guidance that applies to every field, e.g. \"prices in EUR\"."
},
"page_options": {
"type": "object",
"properties": {
"delay": {
"type": "integer",
"description": "Milliseconds to wait after the page has settled, 0 to 10000."
},
"block_ads": {
"type": "boolean",
"description": "Block the major ad networks and remove their slots from the page."
},
"block_popups": {
"type": "boolean",
"description": "Remove newsletter, app-install and promotional popups laid over the page."
},
"remove_selectors": {
"type": "array",
"items": {
"type": "string"
},
"description": "CSS selectors of elements to remove from the page, up to 50."
},
"scroll_to_bottom": {
"type": "boolean",
"description": "Scroll down the page before reading it, for as long as scrolling loads more (up to 25s). For listings and feeds that load their content as they are scrolled."
},
"wait_for_selector": {
"type": "string",
"description": "Wait (up to 10s) for an element matching this CSS selector before reading the page."
},
"block_cookie_banners": {
"type": "boolean",
"description": "Remove cookie-consent banners from the page."
}
},
"description": "What to do to the page before anything is read off it. Applies to the result itself: removed ads and banners are gone from html, markdown and summaries too, and what scrolling loaded is in all of them."
},
"idempotency_key": {
"type": "string",
"description": "Makes the call safe to retry: sending it again with the same key within 24 hours returns the first call's token and result instead of starting (and charging for) new work, even with max_age \"0\". Up to 255 printable ASCII characters; a UUID is ideal."
}
}
},
"outputSchema": {
"type": "object",
"anyOf": [
{
"type": "object",
"description": "One key per requested field, in the order requested. A value the page does not give is null.",
"additionalProperties": true
},
{
"type": "object",
"required": [
"token",
"status"
],
"properties": {
"token": {
"type": "string"
},
"labels": {
"type": "object"
},
"status": {
"type": "string",
"const": "accepted"
}
}
}
],
"$schema": "https://json-schema.org/draft/2020-12/schema"
},
"annotations": {
"title": "Extract structured data with your own schema",
"readOnlyHint": false,
"openWorldHint": true,
"idempotentHint": true,
"destructiveHint": false
}
}| max_age | string | How fresh a stored result must be to be reused, e.g. "2 hours" or "3 days". Default 7 days, maximum 30. A reused result is free. Pass "0" to force a fresh fetch. |
| report_to | string | Async only: a webhook URL to deliver the result to. Defaults to the project's configured endpoint, if it has one. |
| operations * | array | Which operations to run off this page. |
| project_id * | integer | Which project this request belongs to. From list_projects. |
| pdf_options | object | PDF only, when it is among the operations: how to print it, exactly as capture_pdf takes it. |
| residential | boolean | Fetch the page from a residential (home ISP) exit instead of a datacentre one. Costs a surcharge per page visit — see get_usage. Use it for sites that block datacentre traffic. |
| auth_profile | string | Load the page signed in, as this auth profile's visitor: its id, ap_…. A profile is a browser's signed-in state uploaded once, from the project's Auth profiles page in the dashboard or over the HTTP API (POST /auth_profiles); only URLs on its domains are accepted. Costs nothing extra. https://urlpip… |
| instructions | string | Optional guidance that applies to every field, e.g. "prices in EUR". |
| page_options | object | What to do to the page before anything is read off it. Applies to the result itself: removed ads and banners are gone from html, markdown and summaries too, and what scrolling loaded is in all of them. |
| include_audits | boolean | Lighthouse only: include the full per-audit detail. Default false. |
| idempotency_key | string | Makes the call safe to retry: sending it again with the same key within 24 hours returns the first call's token and result instead of starting (and charging for) new work, even with max_age "0". Up to 255 printable ASCII characters; a UUID is ideal. |
| screenshot_options | object | Screenshot only, when it is among the operations: how to take it, exactly as capture_screenshot takes it. |
{
"name": "scrape_url",
"title": "Several results from one page visit",
"description": "Exposes POST /scrape. Runs any subset of the other operations in one\nrequest, served from a single page visit where possible. Each operation\nis billed and stored exactly as the individual call would be, so this is\nnot a discount — with one exception, and it is the reason to use it: a\nresidential exit is charged per page VISIT, so several results off one\nvisit pay the surcharge once.\n\nAnswers with one object keyed by operation, each entry carrying its own\nsuccess and result, so a partial failure still returns everything that\nworked.\n\nlighthouse does not share the visit — it runs its own audit on its own\nengine — so including it means two page fetches, and two surcharges when\nresidential is on. So does a screenshot whose screenshot_options set\nviewport_width, viewport_height, device_scale_factor, dark_mode or\nblock_ads: those change how the page loads, so the screenshot gets a\nvisit of its own.",
"inputSchema": {
"type": "object",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"required": [
"project_id",
"url",
"operations"
],
"properties": {
"url": {
"type": "string",
"description": "The page to fetch. Must be a public http(s) URL."
},
"sync": {
"type": "boolean",
"description": "Wait for the result and return it (default false). When false the call returns a token immediately and you collect the result with get_result."
},
"device": {
"enum": [
"mobile",
"desktop"
],
"type": "string",
"description": "Lighthouse only, when it is among the operations. Default mobile."
},
"fields": {
"type": "array",
"items": {
"type": "object",
"required": [
"name",
"type"
],
"properties": {
"name": {
"type": "string",
"description": "The key in the result: letters, digits and underscores."
},
"type": {
"enum": [
"string",
"number",
"integer",
"boolean",
"date",
"datetime",
"url",
"email",
"object",
"array"
],
"type": "string"
},
"items": {
"anyOf": [
{
"type": "string"
},
{
"type": "object"
}
],
"description": "For an array: what it holds — a type such as \"string\", or {\"type\": \"object\", \"fields\": [...]}."
},
"fields": {
"type": "array",
"items": {
"type": "object"
},
"description": "For an object: its fields, shaped like this one."
},
"description": {
"type": "string",
"description": "What to take from the page. The more specific, the better."
}
}
},
"description": "extract only, and required when it is among the operations: the fields to fill, exactly as extract_data takes them."
},
"labels": {
"type": "object",
"description": "Your own keys to find this request by later, e.g. {\"client\": \"acme\"}: they come back with the result, in the webhook and in list_requests, which can filter by them. Up to 16 keys of up to 40 letters, digits, _ - or .; string values up to 256 characters.",
"additionalProperties": {
"type": "string"
}
},
"max_age": {
"type": "string",
"description": "How fresh a stored result must be to be reused, e.g. \"2 hours\" or \"3 days\". Default 7 days, maximum 30. A reused result is free. Pass \"0\" to force a fresh fetch."
},
"report_to": {
"type": "string",
"description": "Async only: a webhook URL to deliver the result to. Defaults to the project's configured endpoint, if it has one."
},
"operations": {
"type": "array",
"items": {
"enum": [
"html",
"screenshot",
"markdown",
"meta",
"summarize",
"keywords",
"lighthouse",
"console",
"technologies",
"pdf",
"extract"
],
"type": "string"
},
"description": "Which operations to run off this page."
},
"project_id": {
"type": "integer",
"description": "Which project this request belongs to. From list_projects."
},
"pdf_options": {
"type": "object",
"properties": {
"media": {
"enum": [
"print",
"screen"
],
"type": "string",
"description": "print (default) uses the page's print stylesheet; screen prints it as a browser shows it."
},
"scale": {
"type": "integer",
"description": "Scale of the page, 10 to 200 percent. Default 100."
},
"margin": {
"type": "integer",
"description": "Margin on every side, 0 to 50 mm. Default 10."
},
"landscape": {
"type": "boolean",
"description": "Landscape orientation. Default false."
},
"paper_size": {
"enum": [
"letter",
"legal",
"tabloid",
"a3",
"a4",
"a5"
],
"type": "string",
"description": "The paper. Default a4."
},
"single_page": {
"type": "boolean",
"description": "One page as wide as the browser window and as tall as the whole document, instead of paper-sized sheets. Default false."
},
"print_background": {
"type": "boolean",
"description": "Print background colours and images. Default true."
}
},
"description": "PDF only, when it is among the operations: how to print it, exactly as capture_pdf takes it."
},
"residential": {
"type": "boolean",
"description": "Fetch the page from a residential (home ISP) exit instead of a datacentre one. Costs a surcharge per page visit — see get_usage. Use it for sites that block datacentre traffic."
},
"auth_profile": {
"type": "string",
"description": "Load the page signed in, as this auth profile's visitor: its id, ap_…. A profile is a browser's signed-in state uploaded once, from the project's Auth profiles page in the dashboard or over the HTTP API (POST /auth_profiles); only URLs on its domains are accepted. Costs nothing extra. https://urlpipe.dev/docs/auth-profiles"
},
"instructions": {
"type": "string",
"description": "Optional guidance that applies to every field, e.g. \"prices in EUR\"."
},
"page_options": {
"type": "object",
"properties": {
"delay": {
"type": "integer",
"description": "Milliseconds to wait after the page has settled, 0 to 10000."
},
"block_ads": {
"type": "boolean",
"description": "Block the major ad networks and remove their slots from the page."
},
"block_popups": {
"type": "boolean",
"description": "Remove newsletter, app-install and promotional popups laid over the page."
},
"remove_selectors": {
"type": "array",
"items": {
"type": "string"
},
"description": "CSS selectors of elements to remove from the page, up to 50."
},
"scroll_to_bottom": {
"type": "boolean",
"description": "Scroll down the page before reading it, for as long as scrolling loads more (up to 25s). For listings and feeds that load their content as they are scrolled."
},
"wait_for_selector": {
"type": "string",
"description": "Wait (up to 10s) for an element matching this CSS selector before reading the page."
},
"block_cookie_banners": {
"type": "boolean",
"description": "Remove cookie-consent banners from the page."
}
},
"description": "What to do to the page before anything is read off it. Applies to the result itself: removed ads and banners are gone from html, markdown and summaries too, and what scrolling loaded is in all of them."
},
"include_audits": {
"type": "boolean",
"description": "Lighthouse only: include the full per-audit detail. Default false."
},
"idempotency_key": {
"type": "string",
"description": "Makes the call safe to retry: sending it again with the same key within 24 hours returns the first call's token and result instead of starting (and charging for) new work, even with max_age \"0\". Up to 255 printable ASCII characters; a UUID is ideal."
},
"screenshot_options": {
"type": "object",
"properties": {
"format": {
"enum": [
"png",
"jpeg",
"webp"
],
"type": "string",
"description": "Image format. Default png; webp is usually the smallest."
},
"styles": {
"type": "string",
"description": "CSS to inject into the page before capturing, up to 20000 characters."
},
"quality": {
"type": "integer",
"description": "jpeg and webp only, 1 to 100. Default 80."
},
"selector": {
"type": "string",
"description": "CSS selector of one element to capture instead of the page."
},
"dark_mode": {
"type": "boolean",
"description": "Render with prefers-color-scheme: dark."
},
"full_page": {
"type": "boolean",
"description": "Capture the whole page top to bottom (default true). false captures the viewport only."
},
"hide_selectors": {
"type": "array",
"items": {
"type": "string"
},
"description": "CSS selectors of elements to hide before capturing, up to 50. They stay in the page's html; page_options.remove_selectors takes them out of it."
},
"viewport_width": {
"type": "integer",
"description": "Browser viewport width in CSS pixels, 320 to 1920. Default 1350."
},
"omit_background": {
"type": "boolean",
"description": "Transparent background where the page sets none. png and webp only."
},
"viewport_height": {
"type": "integer",
"description": "Browser viewport height in CSS pixels, 240 to 1080. Default 797."
},
"device_scale_factor": {
"type": "integer",
"description": "Pixel density, 1 to 3: 2 gives a retina image twice the size. Default 1."
},
"full_page_max_height": {
"type": "integer",
"description": "The tallest a full-page or element capture may be, in CSS pixels: 100 to 16384, the default."
}
},
"description": "Screenshot only, when it is among the operations: how to take it, exactly as capture_screenshot takes it."
}
}
},
"outputSchema": {
"type": "object",
"anyOf": [
{
"type": "object",
"properties": {
"url": {
"type": "string"
},
"final_url": {
"type": [
"string",
"null"
],
"description": "The address the page ended on, after its redirects."
},
"operations": {
"type": "object",
"description": "One key per operation requested, in the order asked for.",
"additionalProperties": {
"type": "object",
"properties": {
"error": {
"type": "string"
},
"cached": {
"type": "boolean"
},
"result": {},
"success": {
"type": "boolean"
}
}
}
}
}
},
{
"type": "object",
"required": [
"token",
"status"
],
"properties": {
"token": {
"type": "string"
},
"labels": {
"type": "object"
},
"status": {
"type": "string",
"const": "accepted"
}
}
}
],
"$schema": "https://json-schema.org/draft/2020-12/schema"
},
"annotations": {
"title": "Several results from one page visit",
"readOnlyHint": false,
"openWorldHint": true,
"idempotentHint": true,
"destructiveHint": false
}
}