Robots.txt and AI Crawlers: Which Bots Follow It, Which Ignore It (12 Failures)

Three configuration failures account for most unwanted AI crawler access: a named user-agent group silently replacing the wildcard group, a 403 on /robots.txt granting full crawl access under RFC 9309, and edge CDN settings contradicting the file’s directives.

robots.txt is an advisory plaintext file at the root of a domain that instructs web crawlers which URL paths they are permitted to fetch. It carries zero cryptographic authentication and zero technical enforcement. A compliant crawler follows the rules. A misconfigured crawler or a headless scraper ignores them.

Most unwanted crawler access on production sites comes from these configuration errors, not from bots deliberately breaking rules.

This guide details a two-minute diagnostic check, 12 silent production failures, edge telemetry from 125,385 live requests, and a prioritized remediation checklist.

Two-minute diagnostic check

Run these five commands in your terminal against your live domain. Each unexpected output points directly to a numbered failure detailed below:

# Check 1: status code (expect 200)
curl -sI https://example.com/robots.txt | head -n 1

# Check 2: file size in bytes (expect well under 512,000)
curl -s https://example.com/robots.txt | wc -c

# Check 3: byte order mark (expect clean output, anything except efbbbf)
curl -s https://example.com/robots.txt | head -c 3 | xxd -p

# Check 4: number of User-agent groups (more than 1 requires checking group overrides)
curl -s https://example.com/robots.txt | grep -ic '^user-agent'

# Check 5: edge response per bot (compare against what your file intends to allow)
for ua in GPTBot OAI-SearchBot ClaudeBot Claude-SearchBot PerplexityBot Googlebot; do
  echo "$ua $(curl -s -o /dev/null -w '%{http_code}' -A "compatible; $ua/1.0" https://example.com/)"
done

Check	Bad result	What it means	See detailed section
Check 1	403 or 404	Crawlers assume full permission to crawl everything	Failure 2 of 12
Check 1	500, 503, or 429	Crawlers may stop fetching the entire domain	Failure 2 of 12
Check 2	Near or above 512,000 bytes	Rules near the bottom are silently discarded	Failure 3 of 12
Check 3	efbbbf	Invisible BOM skips the first rule block	Failure 4 of 12
Check 4	2 or more groups	Named bot groups replace wildcard rules entirely	Failure 1 of 12
Check 5	403 for a bot the file allows	CDN edge settings contradict the robots.txt	Failure 12 of 12

Check 5 inspects User-Agent string handling. Rules that verify IP ranges will not match requests dispatched from a local terminal.

Recent changes affecting bot policy

Verified against vendor updates through 14 October 2026:

Recent change	Why it matters	Official source
Cloudflare split AI controls	AI crawler controls reorganized into Search, Agent, and Training. Blocking Training can reach Googlebot, Bingbot, and Applebot. Rollout varies by account.	Cloudflare Radar and Dashboard
Anthropic crawler update	Anthropic updated crawler documentation to list ClaudeBot, Claude-SearchBot, and Claude-User. Older tokens like Claude-Web and anthropic-ai are legacy.	Anthropic Help Center
Google-Agent classification	Google lists Google-Agent among user-triggered agents. User-initiated fetchers skip robots.txt.	Google Search Central
OpenAI fetcher policy	OpenAI states robots.txt rules may not apply to ChatGPT-User queries. A GPTBot rule does not control live user fetches.	OpenAI Developer Documentation
IETF aipref working group	Active working group developing standardized AI usage preference vocabularies. Major crawlers do not follow these drafts yet.	IETF Datatracker

Verification sources and methodology

We do not rely on speculation or marketing summaries. Every claim in this guide is grounded in verifiable sources:

• Primary specifications: RFC 9309 (Robots Exclusion Protocol), RFC 9110 (HTTP Semantics), RFC 9421 (HTTP Message Signatures).
• Vendor documentation: Official crawler guides from OpenAI, Anthropic, Google, Perplexity, and Apple updated through October 2026.
• Production edge telemetry: 30 days of live Cloudflare edge analytics covering 125,385 requests and 1.66 GB of traffic.
• Diagnostic scanner tests: First-party automated audits from our live Agent Ready Scanner.

Production telemetry from our edge

Our site runs entirely on Cloudflare Pages with Cloudflare Workers, without an origin server behind it.

The data below represents 30 days of live edge telemetry collected between September 13 and October 12, 2026. These numbers count inbound requests at the global edge before any caching layer takes effect.

Traffic baseline

Metric	Value
Edge requests, 30 days	125,385
Daily average requests	4,179.5
Peak day	10 October, 8,314 requests
Unique visitors, sum of daily uniques	24,660
Daily average unique visitors	822
Peak unique visitors	8 October, 1,337
Data served	1,784,844,892 bytes (1.66 GiB)
Peak day bandwidth	10 October, 123.61 MB

Traffic distribution across the 30-day window:

• Requests per unique visitor: 5.08 overall, with a daily median of 5.22.
• Average response size: about 13.9 KB per request.
• Peak day versus average: October 10 carried 1.99 times the daily average volume.

Cloudflare calculates unique visitors by hashing client IP addresses. Recurring automated scanners and scraper networks are captured in the total.

Days worth inspecting

The ratio of requests per unique visitor is a dependable signal for separating ordinary human browsing from automated crawler sweeps:

Date	Requests	Uniques	Requests per unique
13 Sep	3,984	446	8.93
10 Oct	8,314	1,019	8.16
6 Oct	6,560	852	7.70
20 Sep	5,192	694	7.48

October 10 was a Saturday. It registered our highest request count of the month along with a request-to-unique ratio of 8.16, well above the 5.22 baseline median.

October 6 also stands out, serving 88.26 MB of bandwidth.

Because standard Cloudflare analytics do not log raw User-Agent headers, edge totals cannot prove whether an individual spike originated from AI scrapers, search indexing, or human sessions.

Figure 1: 30-day edge traffic telemetry

Click any day to inspect requests, unique visitors, bandwidth, and sweep signals.

All 30 days Spikes only

Selected Day 10 Oct (Monthly Peak)

Monthly Peak Volume (Ratio: 8.16)

Edge requests

8,314

Unique visitors

1,019

Reqs / unique

8.16

Data served

129.6 MB

Registered 1.99x our daily volume with an elevated request-to-unique ratio, characteristic of multi-worker crawler sweeps.

Requests to disallowed routes

Our production telemetry recorded repeated GET attempts against three specific routes explicitly forbidden in our robots.txt file:

Route	Robots.txt rule	Status returned
/api/chat	Disallow: /api/	404. The Pages Function implements POST only. GET requests fail immediately.
/.well-known/oauth-protected-resource	Explicit Disallow	410 Gone with X-Robots-Tag: noindex, nofollow
/.well-known/webauthn	Explicit Disallow	410 Gone with X-Robots-Tag: noindex, nofollow

A fully compliant crawler would not request these paths.

Software clients searching for authentication capabilities or security policies frequently hit .well-known endpoints during discovery routines. These clients are specialized tools, not web scrapers that parse robots.txt.

Without raw log headers, the requester behind each hit cannot be identified. Per-bot compliance was not measured from this dataset, because aggregate Cloudflare analytics do not include the user-agent field.

What the data shows: automated software requested paths that robots.txt forbids. Hard HTTP status codes at the edge stopped those requests from returning content.

Auditing our own robots.txt

Our live robots.txt uses a single User-agent: * group with Allow: / alongside specific disallow paths for /search/, /api/, /cdn-cgi/, /*$*, and seven .well-known endpoints.

Evaluating this configuration against RFC 9309 exposes two findings:

First, a single wildcard group governs every bot equally. Because we do not declare named bot groups, we avoid the named-group override trap. At the same time, this single-group configuration cannot treat training scrapers differently from search engines.

Second, the rule Disallow: /*$* is ineffective for wildcards. Under RFC 9309 §2.2.3, the dollar sign acts as an end-of-URL anchor only when placed at the very end of a pattern. In the middle of a string, standard parsers treat $ as a literal character.

Agent Ready Scanner self-audit

We ran our live diagnostic tool, the Agent Ready Scanner, against our domain. The audit returned an overall score of 53 out of 100, placing the site at Level 2 (Machine-Actionable Readiness):

Category	Score	Result
Discoverability	83%	Pass. robots.txt returns 200, sitemap-index.xml is active, and Link headers declare alternate Markdown representations.
Content Representation	83%	Pass. Server returns Markdown for Accept: text/markdown, /llms.txt passes checks, and the DOM is server-rendered.
Bot Access Governance	13%	Flagged

The governance score reflects the site’s open-access policy, not a technical bug.

Our site policy grants open access to all AI tokens, including GPTBot, ClaudeBot, and PerplexityBot. We do not separate model training scrapers from search indexing.

For a software business or content publisher, an unsegmented policy allows scrapers to harvest your data without giving you search visibility in return.

What aggregate edge data cannot show

Aggregate analytics have clear blind spots:

• They cannot show request totals broken down by individual crawler token.
• They cannot prove whether a specific bot hit a disallowed path.
• They cannot show whether user-triggered fetchers followed the stated directives.

Resolving these questions requires per-request access logs. We provide an automated Python audit script below to analyze those logs on your own server.

Which bot does what: training vs search vs user retrieval

AI operators deploy distinct bot fleets for different tasks. A rule written for one fleet does not apply to the others:

AI operator	Training fleet	Search indexer	User-triggered fetcher
OpenAI	GPTBot	OAI-SearchBot	ChatGPT-User
Anthropic	ClaudeBot	Claude-SearchBot	Claude-User
Google	Google-Extended (token)	Googlebot	Google-Agent, user fetchers
Perplexity	None documented	PerplexityBot	Perplexity-User
Apple	Applebot-Extended (token)	Applebot	Applebot
Meta	Meta-ExternalAgent	None	Meta-ExternalFetcher
ByteDance	Bytespider	None	None
Common Crawl	CCBot	None	None

Tokens like Google-Extended and Applebot-Extended are policy controls, not distinct HTTP user agents. When Google crawls your site for Gemini training, the request carries the standard Googlebot user-agent string. Google checks your robots.txt for the Google-Extended token to decide whether that content can be fed into model pre-training.

What each vendor documents about robots.txt

Vendor commitments vary widely across training, search, and user retrieval:

Bot token	Documented behavior	Source
GPTBot	Follows robots.txt. A Disallow opts out of training.	OpenAI official docs
OAI-SearchBot	Follows robots.txt. An opt-out removes content from ChatGPT search citations.	OpenAI official docs
ChatGPT-User	Robots.txt rules may not apply because a human user initiated the request.	OpenAI official docs
ClaudeBot	Follows robots.txt. A Disallow excludes content from future training cycles.	Anthropic official docs
Claude-SearchBot	Follows robots.txt. Disabling reduces search citations.	Anthropic official docs
Claude-User	Follows robots.txt. Disallow stops user query retrieval.	Anthropic official docs
Perplexity-User	Generally ignores robots.txt because an active user requested the URL.	Perplexity official docs
Google user fetchers	Ignore robots.txt rules entirely.	Google Search Central
Bytespider	Widely reported in developer forums and secondary reports to ignore robots.txt. No first-party documentation found.	Secondary reports

What blocking each category costs

Blocking crawlers creates different trade-offs depending on the bot type.

Cloudflare Radar tracks crawl-to-referral ratios, comparing pages scraped against human visitors sent back:

Operator and bot	Crawl-to-referral ratio	Measurement period	Source
Anthropic ClaudeBot	~38,000 requests per 1 referral	July 2025	Cloudflare Radar
Anthropic ClaudeBot	~286,000 requests per 1 referral	January 2025	Cloudflare Radar

While exact figures fluctuate across reporting cycles, the underlying economic reality is consistent:

Training crawlers take your content and server bandwidth while sending almost zero visitors back.

Search crawlers deliver real traffic and referral links.

For most web properties, a split policy is the most effective choice: opt out of training, stay eligible for search citations.

Twelve production failures

Failure 1 of 12: a named group replaces the wildcard group

A crawler selects the one group that best matches its product token. It does not merge that group with User-agent: *.

User-agent: *
Disallow: /admin/
Disallow: /api/

User-agent: PerplexityBot
Allow: /

PerplexityBot matches its own group, so the wildcard Disallow lines do not apply to it. Both /admin/ and /api/ are open to that crawler.

Repeat every shared rule inside each named group:

User-agent: PerplexityBot
Disallow: /admin/
Disallow: /api/
Allow: /

Token matching is case-insensitive. Several User-agent lines stacked above one rule set share that rule set.

Failure 2 of 12: status codes on /robots.txt change the policy

RFC 9309 §2.3.1 defines how a crawler treats each response to the robots.txt request. An error can produce the exact opposite of your intended policy:

Status on /robots.txt	RFC 9309 specification	Crawler action
200 OK	Parse rules	Enforced
3xx redirect	Follow at least 5 consecutive hops	Rules from final response apply
4xx (404, 410, 403)	File is unavailable	Crawlers assume full permission to crawl everything
5xx (500, 503)	Server is unreachable	Assume full disallow while error lasts

A deployment that routes /robots.txt to a 404 removes every Disallow rule at once.

A WAF or bot rule that returns 403 on /robots.txt grants full crawling access.

Google documents that it treats 429 on robots.txt like a server error. Exclude /robots.txt from challenges, rate limits, and authentication.

Failure 3 of 12: the file passes 500 KiB

RFC 9309 §2.5 lets a crawler cap parsing and requires the cap to be at least 500 kibibytes. Google documents the 500 KiB limit. Content after the limit is discarded.

Sites that generate thousands of Disallow lines for faceted navigation can easily pass it. If your AI rules sit at the bottom, the parser never reads them.

Move AI user-agent groups to the top or shrink the file.

Failure 4 of 12: a byte order mark skips the first rule

Windows tooling and some CMS exports prepend a Byte Order Mark (\xEF\xBB\xBF). A parser that does not strip it reads the first line as an unknown directive and skips the first record.

Several large parsers tolerate a BOM, but not every crawler does. Strip it immediately:

sed -i '1s/^\xEF\xBB\xBF//' public/robots.txt

Failure 5 of 12: path rules do not match what authors expect

Under RFC 9309 §2.2.2, the longest matching path wins. Rule order is irrelevant.

User-agent: GPTBot
Disallow: /
Allow: /docs/public/

For /docs/public/architecture.html, Disallow matches 1 character (/) and Allow matches 13 characters (/docs/public/). The longer match wins, so the request is allowed. On equal length, Allow is preferred.

Section 2.2.3 defines two pattern characters:

• An asterisk matches any sequence of characters.
• A dollar sign anchors the end of the URL, and only at the very end of a pattern.

Paths are case-sensitive, and rules match by prefix. Writing Disallow: /fish also blocks /fishing.

Our site’s own Disallow: /*$* rule illustrates this trap: in the middle of a pattern, a parser treats $ as a literal character.

Failure 6 of 12: Crawl-delay and noindex do not mean what they imply

Crawl-delay sits outside RFC 9309. Anthropic documents support for it. Google does not support it, and OpenAI does not document it.

Even where a vendor supports it, a crawler with many concurrent workers can exceed your intended rate. Limit rates at the edge.

Noindex inside robots.txt is dead. Google retired it in 2019, and RFC 9309 omits it. Use a response header instead:

X-Robots-Tag: noindex, noarchive

Google also documents that nosnippet, max-snippet, and data-nosnippet limit how page text appears in AI Overviews. Tags such as noai and noimageai belong to no standard.

Failure 7 of 12: Disallow hides the noindex header

A crawler blocked by Disallow: /internal/ never requests the page. It never sees the X-Robots-Tag header either.

If other sites link to that URL, search engines can still display the bare URL without a snippet.

To keep a URL out of search results, allow crawling and serve the noindex header.

Failure 8 of 12: user-triggered fetchers skip the file

When a user asks an assistant to read a URL, the platform sends a live request. Vendors disagree on whether robots.txt governs it.

OpenAI and Perplexity note rules may not apply. Google states its user-triggered fetchers ignore it. Anthropic states Claude-User follows it.

A robots.txt rule alone gives no guarantee across vendors. Some platforms route fetches through proxy networks, arriving with generic browser headers and unlisted IPs.

Failure 9 of 12: spoofed user agents pass as real bots

The User-Agent header is arbitrary plain text:

curl -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.3; +https://openai.com/gptbot" https://example.com/

Unauthorized scrapers frequently disguise themselves as ordinary desktop browsers. No robots.txt rule reaches a client that does not declare itself.

Verify bot identity using the method each operator documents:

Operator	Documented verification mechanism
Google	Reverse DNS to googlebot.com or google.com with forward DNS match, or public IP JSON
OpenAI	Published CIDR prefixes at openai.com/gptbot.json, searchbot.json, chatgpt-user.json
Anthropic	Published IP ranges at claude.com/crawling/bots.json
Perplexity	Published IP range JSON feed

curl -s https://openai.com/gptbot.json | jq -r '.prefixes[] | .ipv4Prefix // .ipv6Prefix'

A request carrying a bot token from an address outside the operator’s published list is spoofed.

Failure 10 of 12: IP blocking hides robots.txt

Anthropic documents that blocking its crawler IPs can stop its bots from reading robots.txt, so your opt-out is never seen.

Disallow in the file first. Reserve IP blocking rules for traffic that fails verification.

Failure 11 of 12: staging and preview environments are open

Preview deployments are public by default on many platforms and inherit no robots.txt policy. A public GitLab issue recorded Bytespider alone making 12,090 requests in 30 days against review app deployments.

For every non-production environment:

• Serve a restrictive robots.txt covering AI tokens.
• Add X-Robots-Tag: noindex for search engines.
• Put authentication in front of previews. Advisory files are not access barriers.

Failure 12 of 12: the CDN contradicts the file

A file that allows a bot does not mean the edge allows it. The crawler reads the allow rule, then receives a 403 on every page.

Cloudflare reorganized its AI crawler controls into Search, Agent, and Training categories. As of October 2026, our Cloudflare dashboard still shows a single toggle labelled AI Scrapers and Crawlers under Security > Bots.

Check which controls your account shows. Check whether a block also applies to Googlebot, Bingbot, and Applebot before enabling it, and run Check 5 from the top of this article after every change.

Production templates

Template 1: allow search citations, block training

Each named group repeats your private path rules:

User-agent: *
Disallow: /admin/
Disallow: /private/
Disallow: /api/

# Training crawlers and control tokens
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Bytespider
User-agent: CCBot
Disallow: /

# Search indexers and user-triggered fetchers
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
Disallow: /admin/
Disallow: /private/
Disallow: /api/
Allow: /

Template 2: block all AI bots

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: Google-Extended
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Meta-ExternalFetcher
User-agent: Bytespider
User-agent: CCBot
User-agent: Diffbot
Disallow: /

Blocking search indexers removes your site from AI answer citations. Some user-triggered fetchers may still reach the site.

Policy selection by site type

Platform type	Training	Search index	User fetch	Edge controls
Publisher with ad revenue	Block	Allow	Allow	Rate limit, verify IPs
Technical documentation	Allow or block by licensing intent	Allow	Allow	Rate limit
Ecommerce catalog	Block	Allow	Allow	Rate limit, bot score
Paywalled or member content	Block	Block	Block	Authentication, not robots.txt
Staging and previews	Block	Block	Block	Authentication gate

Usage preferences and rights reservation

Cloudflare’s Content Signals Policy defines search, ai-input, and ai-train signals in robots.txt.

The IETF aipref working group is drafting a shared vocabulary for AI usage preferences and headers. Major crawlers do not follow these drafts yet.

In the EU, text and data mining exceptions can depend on machine-readable opt-outs (robots.txt or W3C TDMRep). None of these enforce access; all rely on voluntary compliance.

Enforcement at the edge

Cloudflare WAF rules

Cloudflare rules enforce bot policy without Worker code. Confirm field names against current documentation before deploying:

Rule to block selected AI user agents on application paths:

(http.request.uri.path starts_with "/api/") and (http.user_agent contains "GPTBot" or http.user_agent contains "ClaudeBot" or http.user_agent contains "Bytespider" or http.user_agent contains "CCBot")

Set the rule action to Block.

Rule to return permanent errors for unpublished system paths:

(http.request.uri.path eq "/.well-known/oauth-protected-resource") or (http.request.uri.path eq "/.well-known/webauthn")

A Pages Function or Worker can return 410 Gone for these paths, signaling permanent absence.

Exclude /robots.txt from every rule by appending:

(http.request.uri.path ne "/robots.txt")

Cloudflare Worker implementation

This Worker rejects selected AI user agents on restricted paths and leaves /robots.txt untouched:

export default {
  async fetch(request: Request): Promise<Response> {
    const userAgent = request.headers.get("user-agent") || "";
    const url = new URL(request.url);
    const aiBotPattern = /(GPTBot|ClaudeBot|Bytespider|CCBot|Meta-ExternalAgent)/i;

    if (url.pathname === "/robots.txt") {
      return fetch(request);
    }

    if (aiBotPattern.test(userAgent) && url.pathname.startsWith("/api/")) {
      return new Response("Forbidden: AI crawler access restricted on this path.", {
        status: 403,
        headers: {
          "Content-Type": "text/plain",
          "X-Robots-Tag": "noindex, nofollow",
        },
      });
    }

    return fetch(request);
  },
};

A user agent match can be spoofed in both directions. Pair it with IP range verification on sensitive endpoints.

Static headers on Cloudflare Pages

A _headers file in the build output sets security headers for static assets without a Worker:

/internal/*
  X-Robots-Tag: noindex, noarchive

Rate limiting on Nginx

For sites outside Cloudflare, this pattern maps matching user agents to a rate-limited zone:

map $http_user_agent $ai_bot_key {
    default "";
    ~*(GPTBot|ClaudeBot|Claude-SearchBot|PerplexityBot|Bytespider|CCBot|Meta-ExternalAgent) $binary_remote_addr;
}

limit_req_zone $ai_bot_key zone=ai_crawler_limit:10m rate=2r/s;
limit_req_status 429;

server {
    location / {
        limit_req zone=ai_crawler_limit burst=5 nodelay;
    }
}

Return HTTP 429 with a Retry-After header, and never apply the limit to /robots.txt.

Cryptographic bot authentication

RFC 9421 defines HTTP Message Signatures. The IETF Web Bot Auth effort applies them so crawlers sign requests and servers verify them against a public key ledger. Adoption is in early stages.

Measure bot behavior in access logs

Counting what AI bots do on a specific server answers questions vendor documentation cannot. The script below reads combined access logs and reports:

• Total requests and /robots.txt fetches.
• Requests to paths that robots.txt disallows for that token.
• Requests originating from IP ranges the operator publishes.
• Status code distributions.

It parses robots.txt with Protego, implementing longest-match and wildcard rules. Python’s built-in urllib.robotparser applies rules in file order and does not match RFC 9309 behavior.

Per-request logs on Cloudflare Pages

Cloudflare analytics expose aggregates, while the audit script needs one line per request with a user agent:

1. Logpush streaming HTTP request logs to cloud storage.
2. Worker Logging inside edge functions.
3. GraphQL Analytics API queries grouped by user agent.

Convert exported logs to combined format, or adjust the LINE pattern at the top of the script.

Running the audit

pip install protego
curl -s https://openai.com/gptbot.json -o gptbot.json
curl -s https://openai.com/searchbot.json -o searchbot.json
curl -s https://openai.com/chatgpt-user.json -o chatgpt-user.json
curl -s https://claude.com/crawling/bots.json -o claude-bots.json
curl -s https://example.com/robots.txt -o robots.txt

python ai_bot_audit.py /var/log/nginx/access.log robots.txt \
  --ranges gptbot.json searchbot.json chatgpt-user.json

Sample output from a synthetic test log with placeholder addresses illustrates the format:

token                   requests   robots_fetches   hits_on_disallowed   disallowed_verified   verified_ip   unverified_ip
GPTBot                         4                1                    1                     1             3               0
ClaudeBot                      1                0                    1                     0             0               1

Evaluate the results in this order:

• Disallowed hits from verified IPs: proves an operator compliance failure or a stale robots.txt cache.
• Disallowed hits from unverified IPs: proves spoofing or proxy fetchers. Block these at the edge.
• Zero robots.txt fetches alongside high request counts: identifies crawlers that never inspect the file.
• High 4xx response share: shows crawlers hitting dead links or encountering active edge blocks.

The audit script

Save this as ai_bot_audit.py. It requires Python 3 and the protego package:

#!/usr/bin/env python3
"""Audit AI crawler behavior in an access log against a robots.txt file.

Usage:
  python ai_bot_audit.py access.log robots.txt [--ranges gptbot.json searchbot.json ...]

Input: combined log format (Nginx and Apache default). Range files use the
OpenAI/Perplexity style: {"prefixes": [{"ipv4Prefix": "..."}, {"ipv6Prefix": "..."}]}
or a plain text file with one CIDR per line.
"""
import argparse, collections, ipaddress, json, re, sys
from urllib.parse import urlsplit
from protego import Protego

# Google-Extended is a robots.txt control token only. Google documents that it
# has no HTTP user agent string, so it never identifies a real request. A log
# line that contains it is not from Google.
TOKENS = [
    "GPTBot", "OAI-SearchBot", "ChatGPT-User", "ClaudeBot", "Claude-SearchBot",
    "Claude-User", "PerplexityBot", "Perplexity-User", "Meta-ExternalAgent",
    "Meta-ExternalFetcher", "Bytespider", "CCBot", "Applebot", "Amazonbot",
    "Googlebot", "bingbot",
]
LINE = re.compile(
    r'^(?P<ip>\S+) \S+ \S+ \[[^\]]+\] "(?P<method>\S+) (?P<path>\S+) [^"]*" '
    r'(?P<status>\d{3}) (?P<bytes>\S+) "[^"]*" "(?P<ua>[^"]*)"'
)

def load_ranges(paths):
    nets = []
    for p in paths:
        text = open(p, encoding="utf-8").read().strip()
        try:
            data = json.loads(text)
            for item in data.get("prefixes", []):
                cidr = item.get("ipv4Prefix") or item.get("ipv6Prefix")
                if cidr:
                    nets.append(ipaddress.ip_network(cidr, strict=False))
        except json.JSONDecodeError:
            for ln in text.splitlines():
                ln = ln.strip()
                if ln and not ln.startswith("#"):
                    nets.append(ipaddress.ip_network(ln, strict=False))
    return nets

def in_ranges(ip, nets):
    try:
        addr = ipaddress.ip_address(ip)
    except ValueError:
        return False
    return any(addr in n for n in nets)

def token_for(ua):
    low = ua.lower()
    for t in TOKENS:
        if t.lower() in low:
            return t
    return None

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("log")
    ap.add_argument("robots")
    ap.add_argument("--ranges", nargs="*", default=[])
    a = ap.parse_args()

    rp = Protego.parse(open(a.robots, encoding="utf-8-sig").read())
    nets = load_ranges(a.ranges)
    stats = collections.defaultdict(lambda: collections.Counter())
    violations = collections.defaultdict(list)

    for line in open(a.log, encoding="utf-8", errors="replace"):
        m = LINE.match(line)
        if not m:
            continue
        tok = token_for(m["ua"])
        if not tok:
            continue
        s = stats[tok]
        s["requests"] += 1
        path = urlsplit(m["path"]).path or "/"
        if path == "/robots.txt":
            s["robots_fetches"] += 1
            continue
        if nets:
            s["verified_ip" if in_ranges(m["ip"], nets) else "unverified_ip"] += 1
        if not rp.can_fetch("https://example.com" + m["path"], tok):
            s["hits_on_disallowed"] += 1
            if nets and in_ranges(m["ip"], nets):
                s["disallowed_verified"] += 1
            if len(violations[tok]) < 5:
                violations[tok].append((m["ip"], m["path"], m["status"]))
        s[f"status_{m['status'][0]}xx"] += 1

    cols = [
        "requests", "robots_fetches", "hits_on_disallowed", "disallowed_verified",
        "verified_ip", "unverified_ip", "status_2xx", "status_4xx", "status_5xx",
    ]
    print(f"{'token':<22}" + "".join(f"{c:>21}" for c in cols))
    for tok, s in sorted(stats.items(), key=lambda kv: -kv[1]["requests"]):
        print(f"{tok:<22}" + "".join(f"{s[c]:>21}" for c in cols))

    if violations:
        print("\nSample requests to paths the file disallows for that token:")
        for tok, rows in violations.items():
            for ip, path, status in rows:
                print(f"  {tok:<20} {ip:<16} {status} {path}")

if __name__ == "__main__":
    main()

Adding a regression test to CI

A deployment can break robots.txt with no visible error. Assert your policy in continuous integration:

from protego import Protego
import urllib.request

body = urllib.request.urlopen("https://staging.example.com/robots.txt").read()
assert body[:3] != b"\xef\xbb\xbf", "BOM present"
assert len(body) < 500 * 1024, "robots.txt exceeds 500 KiB"

rp = Protego.parse(body.decode("utf-8"))
assert not rp.can_fetch("https://example.com/blog/post", "GPTBot")
assert rp.can_fetch("https://example.com/blog/post", "OAI-SearchBot")
assert not rp.can_fetch("https://example.com/admin/", "PerplexityBot")

The final assertion tests whether the named PerplexityBot group still carries the /admin/ Disallow. It does not test all private routes, only this specific rule.

Frequently asked questions

Does robots.txt stop AI training?

It stops compliant training crawlers like GPTBot and ClaudeBot from fetching new content. It does not remove content already collected, and it cannot bind scrapers that ignore the file.

Does blocking GPTBot remove a site from ChatGPT search?

No. Search visibility is controlled by OAI-SearchBot. Blocking GPTBot opts your content out of model training only.

Does Google-Extended affect Google Search ranking?

No. It is a control token for Gemini training and grounding. Googlebot crawling and Search rankings remain untouched.

Does Crawl-delay work for AI crawlers?

It depends on the vendor. Anthropic documents support for Crawl-delay on its bots. OpenAI and Google do not. Treat Crawl-delay as an unverified wish, and enforce hard rate limits at your edge.

Does Claude-User follow robots.txt?

Anthropic states that disabling Claude-User in robots.txt prevents on-demand retrieval for user queries. OpenAI and Perplexity state the exact opposite for ChatGPT-User and Perplexity-User.

Does blocking an AI crawler by IP address work?

It often backfires. Anthropic documents that blocking its IP ranges prevents its bots from fetching robots.txt in the first place, meaning your opt-out rules are never read.

How long does a robots.txt change take to apply?

RFC 9309 allows scrapers to cache the file for up to 24 hours. OpenAI documents roughly 24 hours before search crawlers refresh their cache.

Prioritized remediation checklist

Prioritize remediation by risk level:

1. Return HTTP 200 for /robots.txt and exclude it from every WAF rule, challenge, and rate limit.
2. Repeat shared Disallow lines inside every named user-agent group.
3. Keep the file under 500 KiB and strip any Byte Order Mark.
4. Put real authentication in front of staging, previews, and private endpoints.
5. Confirm that edge CDN rules and robots.txt directives agree by running Check 5 after every deploy.
6. Split training, search, and user-triggered fetch into three separate decisions.
7. Disallow in robots.txt before blocking any vendor by IP address.
8. Verify crawler identity with published CIDRs or reverse DNS lookups.
9. Run the Agent Ready Scanner for an automated baseline score.
10. Add automated CI assertions so commits cannot silently break policy.

Changelog

• 2026-10-14: Restructured around a two-minute diagnostic check and 12 numbered production failures. Added a dated section on recent policy changes and a prioritized remediation checklist.
• 2026-10-13: Initial publication. Added first-party edge analytics and the Agent Ready Scanner self-audit. Corrected statements about Crawl-delay support, Claude crawler names, IP verification, and 403 handling.
• Next review: 2026-11-13.

Written by

Rantideb Howlader

Platform Engineer and Technical Writer with 10+ years of full-stack development experience and 2+ years focused on DevOps and platform engineering.

About the authorLinkedIn

Related posts

• What Is llms.txt? How It Works, What It Does, and Whether You Need It
• Agent-Ready Architecture: How AI Systems Actually Read the Web
• Serving Markdown vs HTML to AI Agents: Content Negotiation
• Structuring for AI Crawlers: What Actually Works, and What llms.txt Doesn't Prove