Content Signals in robots.txt
type: TechArticle
description: Add Content-Signal directives to robots.txt to declare whether AI crawlers may search, ingest, or train on your content. An emerging IETF AI Preferences / IAB Tech Lab proposal that some validators already check.
language: en
is part of: Web Specification Studio
author: Web Specification Studio
publisher: Web Specification Studio
license: https://creativecommons.org/licenses/by/4.0/
date modified: 2026-05-29T12:14:17.000Z
citation: IETF AI Preferences WG (aipref) — drafts
  type: CreativeWork
  publisher: IETF
citation: IAB Tech Lab — Content Monetization Protocols (CoMP) for AI
  type: CreativeWork
  publisher: IAB Tech Lab
citation: Is It Agent Ready? — Content Signals check
  type: CreativeWork
  publisher: Is It Agent Ready?
citation: RFC 9309 — Robots Exclusion Protocol
  type: CreativeWork
  publisher: IETF

Web Specification Studio
type: WebSite

Web Specification Studio
type: Organization

breadcrumb
type: BreadcrumbList
item list element: Home
  type: ListItem
  position: 1
  item: https://webspecification.com/
item list element: Spec
  type: ListItem
  position: 2
  item: https://webspecification.com/spec/
item list element: Agent Readiness
  type: ListItem
  position: 3
  item: https://webspecification.com/spec/agent-readiness/
item list element: Content Signals in robots.txt
  type: ListItem
  position: 4
  item: https://webspecification.com/spec/agent-readiness/content-signals/