robots.txt
type: TechArticle
description: A plain-text file at the site root that tells crawlers which paths they may or may not fetch. Standardised in RFC 9309 and supported by every major search engine.
language: en
is part of: Web Specification Studio
author: Web Specification Studio
publisher: Web Specification Studio
license: https://creativecommons.org/licenses/by/4.0/
date modified: 2026-07-09T00:00:00.000Z
citation: RFC 9309 — Robots Exclusion Protocol
  type: CreativeWork
  publisher: IETF
citation: Introduction to robots.txt
  type: CreativeWork
  publisher: Google Search Central
citation: How Google interprets the robots.txt specification
  type: CreativeWork
  publisher: Google Search Central

Web Specification Studio
type: WebSite

Web Specification Studio
type: Organization

breadcrumb
type: BreadcrumbList
item list element: Home
  type: ListItem
  position: 1
  item: https://webspecification.com/
item list element: Spec
  type: ListItem
  position: 2
  item: https://webspecification.com/spec/
item list element: SEO
  type: ListItem
  position: 3
  item: https://webspecification.com/spec/seo/
item list element: robots.txt
  type: ListItem
  position: 4
  item: https://webspecification.com/spec/seo/robots-txt/