Structuring for AI Crawlers: What Actually Works, and What llms.txt Doesn't Prove
A 300,000-domain study found llms.txt presence added noise, not signal, to predicting AI citation frequency - and no major AI vendor has confirmed it fetches the file. The techniques that do have evidence behind them are less exciting and already covered elsewhere on this site: raw HTML content, semantic tables, and structured data your crawler never has to render.
Start with the thing most AIO advice skips, because it changes how you should read the rest of this article and every other article on the topic.
The evidence problem with llms.txt
llms.txt - a plain-text or Markdown file at your domain root, styled after robots.txt, intended to give an AI system a curated summary of your site - is not a web standard. It is a community convention originated by Jeremy Howard of Answer.AI in September 2024, maintained through the llmstxt.org project, with no IETF or W3C standardization effort that has materialized as of this writing.
That alone wouldn’t disqualify it - plenty of useful web conventions never got a standards body’s blessing. The stronger problem is evidentiary.
No major AI vendor - OpenAI, Anthropic, Google, or Meta - has publicly confirmed that its crawlers consistently fetch or use /llms.txt. Google’s John Mueller has said directly that no AI crawler has claimed it extracts information via the file. And a study across roughly 300,000 domains, using an XGBoost model to test whether llms.txt presence predicted AI citation frequency, found something specific and worth sitting with: removing the llms.txt variable from the model improved its prediction accuracy. The file’s presence didn’t correlate weakly with citation - it added noise that made the model worse at predicting citation when included.
Adoption sits around 10% of domains studied, concentrated almost entirely in developer tools and technical documentation - Anthropic, Cloudflare, Vercel, Stripe, Mintlify among the visible adopters. That concentration is itself informative: the sites publishing it skew toward exactly the audience most likely to also have excellent semantic HTML, fast sites, and genuine technical authority for other reasons. A correlation between “has llms.txt” and “gets cited,” if one existed, would be extremely hard to disentangle from “is the kind of site that does everything else right too.” The 300,000-domain study suggests that once you control for that, there’s nothing left.
This doesn’t mean don’t publish one. It means don’t publish one instead of the things with actual evidence behind them, and don’t sell a client on llms.txt as if it were a lever with a measured effect. It has a real, narrow use case - see below - and it is not the AIO story.
What actually has evidence: the crawler doesn’t execute JavaScript
This is covered in depth in our article on SPA hydration and AI crawlers, and it is the single highest-confidence finding in this whole space, because it’s directly observed rather than inferred from a correlational study. Vercel and MERJ’s analysis of over 500 million GPTBot fetches found zero evidence of JavaScript execution. GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, and Meta-ExternalAgent all fetch raw HTML and do not run it. Google’s Gemini is the exception, via Googlebot’s rendering infrastructure.
The practical consequence, restated for this article: if your content isn’t in the initial HTTP response, it doesn’t exist for most AI answer engines, regardless of anything you do with llms.txt, schema, or crawler directives. Check with curl -A "GPTBot" your-url before optimizing anything else. This is the prerequisite every other technique in this article assumes is already true.
Semantic HTML, not visual layout
Also covered in depth in our article on table serialization, and worth restating as a general principle rather than a table-specific one: a crawler with no CSS and no JavaScript receives your markup as structure, or doesn’t receive structure at all. A CSS Grid built from <div class="cell"> elements serialises to an undifferentiated string once the stylesheet is gone - no row, no column, no relationship. A <table> with <thead>, <th scope>, and <caption> survives that stripping intact, because the relationships are encoded in the elements themselves, not in a stylesheet the crawler never requested.
This generalizes beyond tables:
<!-- Ambiguous once flattened: which paragraph is the answer to which question? -->
<div class="faq-item">
<div class="faq-q">How do I reset my password?</div>
<div class="faq-a">Go to Settings → Security → Reset.</div>
</div>
<!-- Unambiguous: the element names carry the relationship -->
<dl>
<dt>How do I reset my password?</dt>
<dd>Go to Settings → Security → Reset.</dd>
</dl><dl>/<dt>/<dd> for question-answer pairs, <table> for tabular data, <ol> for sequential steps, <figure>/<figcaption> for a labeled image or diagram - none of this is new HTML, and that’s the point. The elements that have carried semantic meaning since HTML4 are exactly the ones a text-flattening extraction pipeline can still parse correctly. Div-soup with visual CSS is the newer pattern, and it’s the one that fails.
Structured data: helps, doesn’t replace the markup
JSON-LD gives a crawler an explicit, typed summary alongside your content:
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "TechArticle",
"headline": "Structuring for AI Crawlers",
"datePublished": "2026-07-25",
"author": { "@type": "Person", "name": "[AUTHOR NAME]" },
"citation": ["https://vercel.com/blog/the-rise-of-the-ai-crawler"]
}
</script>Treat this as a supplement, not a substitute. Duplicating your content’s actual data into JSON-LD creates two sources of truth that drift apart on the first edit to either one - the same failure mode covered in the table-serialization article for Dataset schema. Mark up the content properly first; add JSON-LD to describe it, not to replace describing it.
Crawler directives: know which bot does what before writing a blanket rule
robots.txt rules for AI crawlers are not interchangeable, because the crawlers serve different purposes:
| Bot | Purpose | Blocking it costs you |
|---|---|---|
| GPTBot | OpenAI training data | Future model knowledge of your content - no immediate visibility effect |
| OAI-SearchBot | ChatGPT search retrieval | Appearing in ChatGPT’s live search results - immediate |
| ClaudeBot | Anthropic training data | Future model knowledge - no immediate visibility effect |
| Claude-SearchBot | Claude search retrieval | Appearing in Claude’s live search results - immediate |
| PerplexityBot | Perplexity retrieval | Appearing in Perplexity answers - immediate |
| Google-Extended | Gemini training data specifically (distinct from Googlebot) | Future Gemini training - does not affect Google Search ranking |
A blanket User-agent: * disallow rule blocks all of these indiscriminately, including the search-retrieval bots whose entire function is putting your content in front of someone asking a question right now. Write per-bot rules if you have an actual position on training-data use versus live retrieval - they are different decisions with different consequences, and most sites that block “AI crawlers” as a category haven’t distinguished them.
The narrow, legitimate case for llms.txt
Despite everything above, one use case has a real mechanism behind it rather than a correlational hope: developer-facing documentation with programmatic consumers. If your users are engineers pointing an LLM context window or an agent at your API docs, an /llms-full.txt - a single clean Markdown file - measurably reduces the token cost and noise of having that agent scrape and parse your HTML documentation pages instead. This isn’t about search-engine citation; it’s a direct efficiency win for a known, technical audience that will actually fetch the file deliberately, because you told them to, not because a search crawler discovered it.
# yoursite.com/llms.txt
> Web Specification Studio - technical documentation
> for performance, accessibility, and architecture standards.
## Docs
- [Specifications library](/specs/llms-full.txt): full technical reference
## Reference
- [API changelog](/changelog.md)Publish it for that reason, for that audience, with that framing. Don’t publish it as an AI-search visibility strategy, because the best evidence available says it isn’t one.
Frequently asked questions
Does llms.txt improve AI search visibility?
The best available evidence says no. A 300,000-domain study found its presence made a citation-frequency prediction model less accurate when included, and no major AI vendor has confirmed its crawlers fetch the file. It has a legitimate, narrower use for programmatic documentation consumers, which is a different claim than “helps you get cited.”
What actually predicts AI citation, then?
The evidenced factors, covered across articles on this site: content present in the raw HTTP response (not client-rendered), semantic HTML that survives being stripped of CSS and JavaScript, and - separately, unverified either way by the study above - general content quality and authority signals that would matter for classical search too.
Should I block AI crawlers entirely?
Understand the split first. Training crawlers (GPTBot, ClaudeBot, Google-Extended) affect whether your content trains future models, with no immediate visibility effect. Search-retrieval crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) affect whether you appear in AI answers today. A blanket block removes you from both, indiscriminately.
Is Google’s Gemini different from other AI crawlers?
Yes - Gemini shares Googlebot’s Web Rendering Service and does execute JavaScript, unlike GPTBot, ClaudeBot, and PerplexityBot, which fetch raw HTML only.
Does JSON-LD replace semantic HTML for AI readability?
No. It supplements it. If your primary content isn’t in properly structured HTML, JSON-LD describing that content is a second source of truth that can drift from the actual page - mark up the content first.
Sources
- OrganiKPI: llms.txt Adoption Data and When to Use It - the 300,000-domain XGBoost study
- llmtxt.info: What is llms.txt? - standardization status, vendor confirmation status
- LinkBuildingHQ: Should Websites Implement llms.txt in 2026? - John Mueller’s stated position
- Vercel: The Rise of the AI Crawler - the JavaScript-execution finding this article and others on this site rely on
- This site’s own articles on SPA hydration and AI crawlers and table serialization for LLM ingestion, which this article deliberately does not repeat