robots.txt for AI crawlers
type: TechArticle
description: Major AI vendors publish named user-agents for their crawlers. Setting an explicit allow or disallow per agent is the clearest way to control how your content is used.
language: en
is part of: Web Specification Studio
author: Web Specification Studio
publisher: Web Specification Studio
license: https://creativecommons.org/licenses/by/4.0/
date modified: 2026-05-29T12:14:17.000Z
citation: OpenAI — GPTBot
  type: CreativeWork
  publisher: OpenAI
citation: Anthropic — Crawlers
  type: CreativeWork
  publisher: Anthropic
citation: Google — Google-Extended
  type: CreativeWork
  publisher: Google Search Central
citation: Apple — Applebot-Extended
  type: CreativeWork
  publisher: Apple

Web Specification Studio
type: WebSite

Web Specification Studio
type: Organization

breadcrumb
type: BreadcrumbList
item list element: Home
  type: ListItem
  position: 1
  item: https://webspecification.com/
item list element: Spec
  type: ListItem
  position: 2
  item: https://webspecification.com/spec/
item list element: Agent Readiness
  type: ListItem
  position: 3
  item: https://webspecification.com/spec/agent-readiness/
item list element: robots.txt for AI crawlers
  type: ListItem
  position: 4
  item: https://webspecification.com/spec/agent-readiness/robots-for-ai-crawlers/