How Ready Is Your Website for AI Agents?
Scan your website to see how ready it is for AI agents. We check multiple web standards and emerging protocols: from robots.txt, XML sitemaps, and Schema.org to Markdown content negotiation, llms.txt, and MCP.
Free live scan for any public domain
Evaluating...
Server Response Simulation
Deterministic extraction simulation across automated crawler user-agents
The server returns an empty root container (<div id="root"></div>). Automated clients and HTTP crawlers that do not execute JavaScript will extract zero document content from this response. Consider server rendering or static generation.
Assessment Disclaimer: These automated findings are technical assessments based on published web specifications and documented ecosystem conventions. Test all recommended configurations in a staging environment before production deployment. Web Specification Studio provides this tool as-is for technical guidance.
Why agent readiness matters for your website
A great website delivers a wonderful experience to people while also communicating smoothly with the automated tools that connect visitors to your work.
Every day, search crawlers, AI research assistants, API integrations, and developer agents visit your site to understand your content, catalog your services, and answer questions for users.
By providing clean HTTP responses, well-organized sitemaps, rich Schema.org structured data, and reliable endpoints, you make it easy for these systems to find your pages, understand your entities, and cite your content accurately.
Agent readiness gives your website a clear competitive advantage: faster discovery, higher citation accuracy in search, and dependable interfaces for automated tools and workflows.
The 3 Levels of Agent Readiness
How websites evolve from traditional human browsing to full AI agent automation:
Discovery: Can AI Crawlers Find You?
FoundationIf crawlers like GPTBot or ClaudeBot hit bot blocks, slow redirects, or missing sitemaps, your website remains invisible to AI answers.
Understanding: Can AI Models Read Your Facts?
AccuracyScrapers parsing cluttered HTML layouts and heavy JavaScript often hallucinate prices or misquote specifications when answering user queries.
Action: Can Autonomous Agents Use Your Tools?
AutomationBeyond reading articles, coding agents and assistants can query databases, run workflows, and execute tasks directly on your platform.
The 6 Layers of an Agent-Friendly Website
AI agents, search engines, and automated crawlers interact with your website across six practical areas. Each layer helps automated visitors find, read, and use your pages effectively:
Discovery
Can AI tools and search engines find your pages easily?
- robots.txt policy files
- XML sitemaps and sitemap indexes
- canonical URL declarations
- HTTP Link discovery headers
- standard well-known endpoints
Content Retrieval
Can automated visitors download clean, readable content quickly?
- clean HTTP status codes and headers
- server-rendered HTML pages
- Markdown content negotiation (Accept: text/markdown)
- proper caching headers (Vary: Accept)
- stable canonical URLs
Content Understanding
Can AI assistants understand what your content and data actually mean?
- Schema.org structured data
- JSON-LD knowledge graphs
- clear entity connections and links
- semantic HTML elements
- structured product, article, and organization data
Crawler Rules
Do you give search bots clear rules on what to visit and what to skip?
- robots.txt directives (RFC 9309)
- crawler-specific rules (ClaudeBot, GPTBot, PerplexityBot)
- Content Signals (search, ai-input, ai-train)
- login protection for private resources
- clear bot management guidelines
Tool Capabilities
Can AI agents interact with your tools, endpoints, or services?
- HTTP REST and OpenAPI definitions
- Model Context Protocol (MCP) server cards
- WebMCP browser tool invocation APIs
- machine-readable tool declarations
- OAuth 2.0 authorization metadata
Safe Actions & Transactions
Can automated clients complete actions and checkouts securely?
- Universal Commerce Protocol (UCP)
- Agentic Commerce Protocol (ACP)
- Machine Payment Protocol (MPP) and HTTP 402
- authenticated state transitions
- structured machine checkout interfaces
These layers are complementary. Implementing one layer does not automatically satisfy another, and not every website requires capabilities in all six layers.
Best Practices for an Agent-Friendly Site
Three simple guidelines help keep your website reliable for automated tools and completely secure for your users:
Clear Guidance & Strong Security
Publishing robots.txt guidelines helps search engines and automated crawlers discover your content responsibly. Pair these directives with standard authentication to keep private areas completely secure.
End-to-End Response Quality
Beyond confirming files exist, testing HTTP status codes, content-type headers, and schema syntax ensures automated tools parse your resources quickly and reliably.
Reliable Web Standards
Following established W3C and IETF standards ensures long-term compatibility across all search engines, AI assistants, and future web clients.
How the Audit Works
The scanner inspects your domain step by step across six clear phases:
Discover
Find Public Endpoints
Locate robots.txt, XML sitemaps, canonical tags, discovery headers, and well-known configuration paths.
Retrieve
Fetch Resources
Request resources with standard HTTP calls, checking response codes, content types, and caching headers.
Parse
Read Content Signals
Parse robots rules, XML sitemap URLs, Schema.org JSON-LD graphs, and document text structures.
Validate
Check Web Standards
Compare actual results against RFC web standards, Schema.org definitions, and official protocol specs.
Classify
Assign Clear Status
Label each finding as PASS, FAIL, WARNING, INFO, or N/A based on clear, reproducible evidence.
Explain
Provide Fixes
Give clear explanations, show the exact impact on automated tools, and provide ready-to-use fixes.
What We Check on Your Website
The scanner runs automated checks across seven practical areas on your production domain:
Web Discovery
We inspect the mechanisms that help automated clients locate your site's resources without brute-force crawling.
Objective: Verify whether an automated client can determine your site structure deterministically. DNS-AID TXT records are classified as an experimental discovery convention rather than an established web standard.
Content Delivery & Page Rendering
We examine how your website delivers content to automated clients, verifying real content delivery and distinguishing fast server responses from empty client-rendered shells.
We do not merely detect Markdown. We test whether the negotiated representation behaves correctly: preserving resource identity, matching canonical URLs, returning accurate media types, providing caching directives (Vary: Accept), and serving equivalent semantic content.
Crawler Rules & Bot Policies
We analyze the policies your site publishes for automated crawlers and automated systems.
Content Signals are an emerging machine-readable policy mechanism for communicating permitted uses of content. They complement crawler access directives but do not replace authentication or network-level enforcement.
Structured Data & Schema Markup
We inspect structured information exposed by your pages to assess explicit entity grounding.
Structured data gives machines explicit information about entities and relationships that would otherwise have to be inferred from page text.
AI Documentation & llms.txt
We look for curated indexes intended to help automated systems navigate a site with minimal token overhead.
llms.txt is an emerging convention for LLM-oriented site summaries. It is not an established web standard and is not required for search indexing or Google AI features.
Interactive Tools & MCP Protocols
We evaluate whether a site exposes machine-actionable tools and capability declarations.
WebMCP is an emerging W3C Community Group browser API allowing websites to expose in-page tools. MCP Server Cards are an ecosystem convention for advertising endpoints. Files like auth.md and skill directories are reported as informational machine documentation signals, not formal standards.
Automated Checkout & Payments
For transactional web applications, we inspect emerging machine payment and commerce signals.
Transactional capabilities are evaluated against specification metadata when applicable. Sites that do not conduct commerce are marked Not Applicable rather than penalized.
How Standards and Protocols Are Grouped
The web contains official standards, community protocols, and emerging AI conventions. We classify each check clearly so you know what is an established web standard and what is an emerging practice:
| Classification | Definition | Examples in Scanner |
|---|---|---|
| Web Standard | Formal specification maintained through a standards body or standards process | HTML living standard, RFC 9110 HTTP Semantics, RFC 9309 Robots Exclusion Protocol |
| Protocol | A defined machine-to-machine communication mechanism | HTTP, DNS, Model Context Protocol (MCP) |
| Vocabulary | A defined semantic vocabulary or data model | Schema.org, JSON-LD serialization |
| Practice | Widely adopted implementation practice across production websites | XML Sitemap protocol, Open Graph metadata, canonical link tags |
| Emerging | Developing specification, Community Group draft, or ecosystem convention | llms.txt, WebMCP, Content Signals, UCP, MPP |
| Vendor-Specific | Mechanism defined primarily by a particular provider or platform | Crawler-specific token rules, cloud bot headers |
| Informational | Observable signal without a normative compliance requirement | Representation efficiency, raw byte size, response timing |
The absence of an emerging convention or informational signal is never treated as a standards failure.
Agent Readiness Profile and Levels
Agent readiness is not a single competition where every website must reach the same destination. Different websites serve different operational roles. The Level is a summary headline, while the profile evaluates distinct capability dimensions independently:
| Capability Dimension | Assessment Focus | Target Outcome |
|---|---|---|
| Discovery | robots.txt, XML sitemaps, canonical links, Link headers | Deterministic discovery without brute-force crawling |
| Retrieval | Representation integrity, Content-Type, Vary headers, server HTML | Predictable payloads matching resource identity |
| Semantics | Schema.org vocabularies, JSON-LD graphs, entity grounding | Explicit entity disambiguation without text inference |
| Policy | RFC 9309 crawler directives, Content Signals permissions | Explicit machine-readable crawler and usage preferences |
| Capability | REST/OpenAPI catalog, MCP server convention, WebMCP tools | Discoverable and invocable machine interfaces |
| Transaction | Machine payments, HTTP 402, UCP/ACP commerce manifests | Structured state-changing commerce (or marked N/A) |
Unavailable
The machine-facing layer is substantially inaccessible or non-functional.
Typical conditions: HTTP connection errors • broken response headers • severe crawler restrictions • empty client-side DOM with zero server-rendered content
Crawlable
The website exposes basic, retrievable web documents.
Typical capabilities: valid 200 OK responses • retrievable HTML • valid robots.txt • fundamental web crawler access
Discoverable
The site provides explicit mechanisms for automated resource discovery.
Typical capabilities: XML sitemaps • canonical URL declarations • consistent internal linking • well-known discovery resources
Structured
The site exposes explicit machine semantics and entity grounding.
Typical capabilities: Schema.org types • valid JSON-LD graph • entity relationships • structured metadata
Interoperable
The site provides machine-oriented representations and clear policy signals beyond standard HTML.
Typical capabilities: HTTP content negotiation (Accept: text/markdown) • Vary: Accept headers • Content Signals • curated machine indexes
Agent-Interactive
The service exposes machine-invocable tools, interfaces, or transaction protocols.
Capabilities: Model Context Protocol (MCP) server endpoints • WebMCP browser tool declarations • API catalogs • authenticated tool execution
Describes an observable architectural capability, not certification and not a guarantee of third-party client behavior.
How We Report Audit Findings
Instead of giving you a mysterious single score, every check gives you transparent findings and clear explanations:
PASS • FAIL • WARN • INFO
Clear verification status accompanied by High, Medium, or Low certainty.
Tested Endpoints
The exact URL, HTTP method, and negotiation headers checked during the audit.
Direct Observations
Directly measured response status codes, header values, and parsed structures.
Why It Matters & Fixes
Clear analysis of tool impact plus exact, copy-paste code to fix issues.
Who This Tool Is Built For
Agent Ready Scanner provides technical teams with clear, reproducible signals to test machine-facing endpoints:
Web Developers
Validate HTTP behavior, content negotiation, Schema.org grounding, and machine interfaces before deploying to production.
Platform Developers
Inspect headers, discovery endpoints, crawler policy directives, and edge infrastructure behavior across environments.
Security Teams
Separate crawler policy from actual access control and identify unsafe assumptions regarding bot directives and endpoints.
API and Integration Teams
Evaluate machine-discoverable capabilities, Model Context Protocol (MCP) server endpoints, and automated client interfaces.
Technical SEO Teams
Understand the machine-readable infrastructure underlying automated retrieval, crawling efficiency, and entity grounding.
Frequently Asked Questions
Common questions regarding agent readiness, standards compliance, and how the scanner evaluates modern websites:
Is agent readiness an official web standard?
No single specification named 'Agent Readiness' exists. It is an operational profile evaluating how well your site implements official web standards (RFC 9110 HTTP Semantics, RFC 9309 Robots Exclusion Protocol, W3C Schema.org/JSON-LD, HTML living standard) alongside emerging community protocols (Model Context Protocol, WebMCP, Content Signals, llms.txt).
Does a website need llms.txt?
No. llms.txt is an emerging proposal for curated site summaries. Established search crawlers and AI assistants primarily discover and read standard HTML pages, XML sitemaps, robots.txt, and Schema.org markup. llms.txt is helpful context, not a prerequisite.
Does robots.txt control automated clients?
robots.txt is an advisory protocol (RFC 9309). Reputable crawlers respect it, but malicious scrapers and arbitrary clients can ignore it. It is a discovery and indexing policy, never a security boundary. Private resources must be protected by authentication.
Does this guarantee visibility in search engines or AI assistants?
No. The scanner evaluates technical infrastructure: whether your endpoints respond correctly, provide valid machine metadata, and support automated retrieval. Editorial relevance, domain authority, and content quality are assessed by the downstream platforms themselves.
What is WebMCP and how does WSS evaluate it?
WebMCP is an emerging W3C Community Group proposal for exposing machine-actionable tools inside web pages. The scanner checks whether your site declares tool capabilities via standard descriptors or in-page APIs. Because it is an emerging proposal, absence of WebMCP is reported as informational, not a failure.
How does the scanner audit MCP Server Cards?
The scanner inspects the ecosystem convention resource at /.well-known/mcp.json, testing if it returns 200 OK with valid JSON, contains expected metadata fields, and points to valid server interfaces.
What is the difference between a machine-readable website and an agent-ready website?
A machine-readable website focuses on retrieval and semantics: providing clean content, structured JSON-LD, and understandable text. An agent-ready website also provides machine-discoverable interfaces for invocation and state transitions: tool declarations, API endpoints, authorization flows, and transaction capabilities.
Related Specifications & Guides
Deepen your understanding of machine-readable protocols, technical SEO, and agent discovery with our documentation and guides:
Agent Readiness Specifications
Complete technical specifications covering llms.txt, Content Signals, Model Context Protocol, and machine content negotiation. Learn how automated clients discover and evaluate your site.
The Standards Checklist
Every spec item, flat and tickable. Use as a complete self-audit across web performance, technical SEO, accessibility, and machine-readable signals for your domain.
Sample Technical Audit
Review a real-world example of an in-depth audit report examining crawler access, rendering speed, and structured data.
Agent Web Architecture
How search crawlers, AI assistants, and autonomous agents discover and process content on modern sites.
Content Negotiation for AI
Implement standard HTTP Accept headers and Markdown responses to reduce token costs and improve retrieval.
Audit your site for the agent-powered web.
Agent Ready Scanner provides a live, point-in-time check of your site's machine-readable infrastructure. Find broken discovery links, invalid robots rules, missing structured data, and unexposed APIs so your site is ready for AI tools and automated agents.
This scanner checks publicly accessible HTTP signals at the time of the scan. Results reflect a point-in-time snapshot. Specifications and ecosystem conventions continue to develop. Review all configurations before deploying to production.
How to improve your score
Copy the instructions below and paste them into your coding agent (Cursor, Claude Code, Windsurf, Copilot, Antigravity, Kiro, etc.) to improve your site's agent readiness.
Generated Machine Configuration Files
Production-ready configuration files configured for your domain