Auth: required. Send an ivy_ API key (Authorization: Bearer ivy_… or X-Ivy-Api-Key) or a Clerk session JWT (Authorization: Bearer …). See Authentication.

Stateless benchmarking surface. Runs the full OrchestratorClient pipeline — intent detection, dynamic tool loading, tool execution, cost routing — without session persistence. Each request creates an ephemeral orchestrator instance discarded after the response. Use to measure: tool selection accuracy, intent detection quality, TTFT, and end-to-end pipeline latency.

Request

Headers

x-pleach-depthinteger

Delegation depth of this call. Omit it on a first call. An agent that calls this endpoint while running a turn for another agent call must send its own depth plus one, so a chain of agents calling agents stops after 4 hops. A value above 4 is refused with 400 DELEGATION_DEPTH_EXCEEDED; a value that is not a single non-negative integer is refused with 400 INVALID_DELEGATION_DEPTH. The header is a loop guard you set yourself, not a credential: it grants nothing.

Request body

promptstring

The user prompt to benchmark

modelstring

Model to use

providerenum ("anthropic" | "openai" | "openrouter")

Provider override

streamboolean

Enable SSE streaming (default: true)

enableIntentDetectionboolean

Enable intent detection (default: true)

enableDynamicToolLoadingboolean

Enable dynamic tool loading based on intent (default: true)

maxTokensinteger

Max tokens for generation

temperaturenumber

Sampling temperature

enableThinkingboolean

Enable extended thinking

thinkingBudgetinteger

Thinking budget in tokens

diagnosticsboolean

Opt-in headless diagnostics (IH-19/3b): return console breadcrumbs + a synthesized TurnReport inline in the response. Forces a non-streaming response. Default: false.

toolsstring[]

Allow-list. When present, only these tools may be offered to the model or run, plus a small exempt set the platform needs (review_tool_registry, get_data, sandbox_read). [] means no domain tools (exempt set only); omit for no restriction. Names are matched as canonical tool names after alias resolution (e.g. diffdock → diffdock_dock). Names must be tools this deployment offers: an unknown, deprecated or disabled name is a 400 UNKNOWN_TOOL (details.unknownTools echoes the first 20, details.unknownCount counts them all). At most 256 names of at most 128 characters each; more is a 400 INVALID_REQUEST. The list only narrows: it never admits a tool your tier or the guest tier refuses, and sub-agents inherit it.

excludeToolsstring[]

Deny-list. These tools are never offered to the model and a call to one is refused (tool error TOOL_EXCLUDED_BY_REQUEST) — including the exempt set if you name it. Same name matching and UNKNOWN_TOOL rule as tools. Wins over tools. Sub-agents inherit it. Withheld tools are reported in the response's withheldTools.

maxCreditsnumber

Most credits this benchmark turn's model calls may debit; the same ceiling as /api/v1/orchestrate maxCredits. Each model call reserves the size of the largest call so far (at least 1 credit) before it is sent, and is refused when the credits spent plus every reservation of a call still running plus its own would exceed the cap. Once one more tool round and the answer no longer fit, the tool loop stops and the answer is written from the results gathered. The bench response does not echo the cap: its usage stays token counts. Counted: the model calls this turn makes itself. Not counted or capped: GPU job credits and model calls a tool makes through a request to another API route. Must be a positive number, else 400 INVALID_REQUEST.

Responses

`200` OK

Benchmark response

Response shape:

contentstring

Assistant response text

toolCallsobject[]

toolResultsobject[]

usageobject

timingobject

Performance timing breakdown

withheldToolsobject[]

Tools the request's tools / excludeTools policy withheld this turn: offered-then-stripped intent or critical tools and refused calls, one entry per tool, at most 20 entries. Always present; an empty array means nothing was withheld.

gatesAutoApprovedobject[]

Tool calls this turn that an approval gate (cost or safety check) would have stopped and that ran without an approval step, because gates are not enabled. One entry per tool and gate, at most 64 entries. Names only. Always present; an empty array means no call would have been gated.

gatesAutoApprovedDroppedinteger

Present only when more distinct gated calls ran than gatesAutoApproved lists: how many were left off it.

diagnosticsobject

Present only when the request set diagnostics: true. Redacted console breadcrumbs + a synthesized TurnReport for the IH-21 predicates.

`400` Bad Request

INVALID_REQUEST: the body failed validation. UNKNOWN_TOOL: tools or excludeTools named a tool that does not exist. DELEGATION_DEPTH_EXCEEDED: x-pleach-depth is above 4. INVALID_DELEGATION_DEPTH: x-pleach-depth is not a single non-negative integer.

Response shape: object

`401` Unauthorized

Authentication required

Response shape: object

`429` Too Many Requests

RATE_LIMITED: the burst rate limit (30 requests per 3 seconds for /orchestrate, 15 for /bench). A call made with an API key is counted against that key and against the key owner's total, wherever it is called from; a signed-in session is counted per user, and a call with no credential per IP. API_KEY_FAILURE_RATE_LIMITED: too many calls with an invalid or refused API key from this network. API_KEY_CONCURRENCY_LIMITED: this API key already has 5 agent turns in flight across /orchestrate and /bench. Each carries a Retry-After header.

Response shape: object

curl -X POST 'https://ivybiosciences.com/api/v1/bench' \
  -H "Authorization: Bearer $IVY_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
  "prompt": "Predict the 3D structure of MKTVRQERLKSIVRILERSKEPVSGAQ"
}'

POST /bench