Robin Saige we check the tools AI agents call

DefaultBench

io.github.NathanYoshua/defaultbench

Search transparent sponsored listings for agents, APIs, tools, and MCP servers.

— the operator's own registry description reported

Publisher io.github.NathanYoshua · first seen 2026-09-03 · endpoint https://defaultbench.com/mcp

allow
verdict
mid-solo
cluster
ok_tools
answers
13
tools served

Dependency rating derived

allow
allow — alive & usable · retrieval-public ● · V1 shows its work ◐
allow — because answers ●, 13 tools legible (band B) ●, no confirmed drift in 1 probes ●; not yet checked: truth (0 runs)

Cluster mid-solo · distinctiveness 0.303 (typical). Every figure is a percentile or class within the 15,472-server censussee the whole spectrum.

Does it answer? Q1 · observed
answersalive_tools
latency116 ms
vs populationtypical
protocolcurrent
authopen
Can it be used? Q4 · observed
tools13
vs populationmid
capabilities2
harnessB
Did it hold when checked? Q3–Q4 · derived
identitysolo
servers on its host1
verifiabilityunassessed
duplicate inventoryno
drift events0
answers trueno primary

🔏 Signed receipt rr_5619502d95d11baff3247270 · ed25519 · key rs-rcpt-2026-08 — this rating is tamper-evident; an agent gets the full signature from check_server and can verify it against the published key.

Q1Does it answer?alive & usable — 1 probes ●evidence ↓
Q2What is it?retrieval-public ● evidence ↓
Q3Can it be checked?V1 shows its work ◐evidence ↓
Q4Did it hold?harness B · truth not yet run · no drift ●evidence ↓
Q5What company does it keep?mid-solo · 1 on its host (context, not verdict) ◐evidence ↓

● observed · ◐ derived · ○ reported — the verdict synthesizes Q1–Q4; Q5 is context. Methodology →

History observed

Every probe we made, oldest → newest (1 shown, 2026-09-06 → 2026-09-06). Green answered · amber answered-but-walled · red no useful answer. A gap in our cadence is a gap in coverage, not evidence about the server.

changelog (1 events)
datetypewhat changed
2026-09-06outcomefirst probe: ok_tools

What kind of tools these are derived

Verification only means something relative to a tool's NATURE — an email-sender has no "true answer", a generator has no answer key. Classification evidence: classified from tool names/descriptions/annotations WE OBSERVED. What checking applies to each kind (the full methodology →):

naturetoolsmeaning what checking applies
retrieval-public10serves public factstruth-checkable against a public source — the V-ladder applies in full
action1changes the worldno true answer exists; we check the CONTRACT (destructive/read-only/idempotent declarations, error legibility) and never fire real actions
computational1deterministic transformsself-checkable by re-computation and known-answer tests
unclassified1not classifiable from its textTier-1 only until its tools describe themselves

Tools last seen observed

compare_resources complete_seller_analytics_challenge get_claim_status get_discovery_sources get_evidence_history get_listing get_seller_analytics get_seller_analytics_challenge inspect_keyword inspect_organic_resource quote_rank search_agents validate_organic_resource

Harness readiness observed

Can an agent's harness pick this tool and call it correctly? Graded on the three things an agent reads — name, description, typed parameters. Band B — usable, with rough edges.

legibility checktools passingok
name is clear & specific13/13
has a description13/13
description is substantive13/13
parameters are typed13/13
parameters are described1/131/13

13 of 13 tools graded · 13 with a captured input schema. RS-008, Tier-1 — no domain knowledge, applies to any tool. “Alive” is not the same as “usable”.

Truth checks observed

Not in the Tier-2 trade lane, or not yet checked. Tier-2 re-derives a server's answers against published primary sources — see truth checks.

Protocol conformance observed

2025-06-18
protocol version
session model
1.1.0
server version defaultbench

Capabilities: resources tools

Drift observed

No confirmed drift on record. Baseline set 2026-09-06; changes appear here after human review.

raw probe records
probed (UTC) outcomehttpmsprotocol
2026-09-06T06:15:29Zok_tools2001162025-06-18

Watch or embed this verdict

verdict badge

Operators: put the live verdict in your README — it updates with every census, links back here, and is a dated observation, never a warranty:

[![Robin Saige verdict](https://robinsaige.com/badge/io.github.NathanYoshua/defaultbench.svg)](https://robinsaige.com/s/io.github.NathanYoshua/defaultbench)

Depend on it? Subscribe to its change feed — outcome changes and confirmed drift, no account needed. Building on it? The full dossier as JSON — verdict, rating, truth checks, history — stable enough to gate CI on.

← overview

Operator of this server? Everything we hold about it is on this page — free, no account, for anyone. Wrong attribution, stale probe, misclassification? Dispute this record →