How the Site Scan scores
Two tiers, all checks
deterministic, no LLM. Tier 1 (85 points) scores what agents measurably do today.
Tier 2 (15 points) is an early-adopter bonus for declared-intent standards — cheap to
add, little measured traffic yet. Weights are fixed in
scan_signals.json v1.0 (2026-07-31) and
reviewed quarterly against new crawl data. Grade bands: A ≥ 90 ·
B ≥ 75 · C ≥ 60 · D ≥ 45 · F below.
Rounding: int(x + 0.5), never banker's rounding.
Why our scores differ from other scanners. Most tools award
full points for llms.txt and MCP cards. The measured data says agents barely touch them today:
97% of llms.txt files receive zero crawler requests (Ahrefs, 137k domains, 06/2026); GPTBot
fetched robots.txt 3,990× vs llms.txt 7× (EZY, 07/2026); fewer than 15 of the top
200k sites have an MCP card (Cloudflare, 04/2026). We check them, weight them near zero, and
publish the evidence — when the data changes, the weights change.
| Weight | Signal & evidence | Tier |
|---|---|---|
| 20 | Bot-block probe Does your site answer AI crawlers at all, or does it serve them a challenge page? Evidence: Cloudflare blocks AI crawlers BY DEFAULT for new domains since 01/07/2025; 1M+ customers activated blocking; 416B AI-bot requests blocked Jul–Dec 2025 (digitalapplied.com, 2026). The single highest-value SMB finding: an unintentional, fixable, binary loss. | Tier 1 |
| 15 | robots.txt AI-crawler policy Do your robots.txt rules let retrieval and user-triggered AI bots in — deliberately? Evidence: robots.txt is where AI access is decided today: GPTBot fetched robots.txt 3,990× vs llms.txt 7× (EZY 83-site log study, 27/07/2026). 78% of top-200k domains have robots.txt written for search crawlers, not AI (Cloudflare, 17/04/2026). | Tier 1 |
| 15 | JS-render dependency Is your content in the raw HTML, or does it only exist after JavaScript runs? Evidence: GPTBot, ClaudeBot and PerplexityBot execute ZERO JavaScript (Vercel crawler study; searchoptimo.com, 28/06/2026). A client-side-rendered site is literally blank to AI crawlers. | Tier 1 |
| 10 | Structured data (JSON-LD) Can an agent extract who you are, what you sell and for how much — without guessing? Evidence: schema.org is the one machine-readability layer with genuinely widespread adoption and Google/Microsoft backing (Agent Protocol Atlas, 15/06/2026). | Tier 1 |
| 8 | Semantic HTML & auth walls Can a browser agent navigate your accessibility tree — and is content reachable without login or a consent wall? Evidence: Browser agents (Operator, Claude computer-use) navigate the accessibility tree; heading hierarchy, landmarks and labelled controls are their map (isready.ai, 15/06/2026). EU cookie walls that replace the body block crawlers and agents alike. | Tier 1 |
| 7 | Machine-readable pricing/products Can an agent collect your prices without executing JS? Evidence: Offer/Product schema is the deterministic check for price collection; agents extract prices from JSON-LD, not from rendered widgets (Agent Protocol Atlas, 15/06/2026). | Tier 1 |
| 5 | Sitemap Can an agent enumerate your pages? Evidence: /sitemap.xml is among the most common AI-agent requests (Readable CDN-edge data, ~2M agent requests/mo, 20/02/2026). | Tier 1 |
| 5 | Contact & action paths Can an agent find how to contact, book or buy from you in plain HTML? Evidence: Task-completing agents look for contact, booking and purchase endpoints in the HTML they can read (isready.ai Smart-agent dimensions, 15/06/2026). | Tier 1 |
| 4 | Markdown for agents Do you serve token-lean markdown when an agent asks for it? Evidence: 3.9% adoption in top-200k; measured benefit real — agents on markdown used 31% fewer tokens, answered 66% faster — but only ~3 of 7 tested agents send the header (Cloudflare, 17/04/2026). | Tier 2 |
| 3 | MCP server card Do you declare machine-usable capabilities at /.well-known/mcp.json? Evidence: Fewer than 15 sites in the top 200k had MCP Server Cards as of 04/2026 (Cloudflare) — an almost-empty bonus tier for websites, even though MCP is the de-facto agent-tool standard. | Tier 2 |
| 2 | Content Signals Does robots.txt state a deliberate ai-train / ai-input / search policy? Evidence: 4% adoption, momentum via Cloudflare (17/04/2026). A good 'deliberate policy vs accidental default' indicator. | Tier 2 |
| 2 | API discoverability Can an agent discover your API (api-catalog, OpenAPI, Link headers)? Evidence: RFC 9727 API Catalog + OpenAPI at common paths; near-zero SMB adoption today (Cloudflare, 17/04/2026) — bonus only. | Tier 2 |
| 2 | WebMCP Do you declare in-page tools for browser agents? Evidence: W3C draft, shipping behind Chrome 146 flags; claimed 67% fewer errors vs visual scraping (ByteIota, 05/2026). Transactional sites only; bonus. | Tier 2 |
| 2 | llms.txt Present and well-formed — scored honestly. Evidence: HONESTY NOTE, printed on every result: 97% of llms.txt files receive ZERO crawler requests (Ahrefs 137k-domain log study, 15/06/2026); GPTBot fetched robots.txt 3,990× vs llms.txt 7× (EZY, 27/07/2026); Google states in writing it is not needed for its systems. We check it, weight it near zero, and publish why. Main real consumers today: coding agents (Claude Code, Cursor). | Tier 2 |
Dispute a finding? Every result shows the fetched evidence. Mail scan@rxed.ai with your URL and the finding — probes can produce false positives on unusual WAF setups, and we correct them.
How this differs from the Agent Readiness Index: the ARI is an editorial, evidence-pack review of named services (published 0–10 passports); the Site Scan is an automated, deterministic check of any website (private A–F grade). Different objects, different depth — deliberately different scales.