AI Crawlers: What robots.txt, llms.txt and Structured Data Actually Change

A professional website today receives traffic from crawlers that didn’t exist three years ago: GPTBot for OpenAI’s training data, ClaudeBot for Anthropic, Google-Extended for Google’s AI-generated overviews, PerplexityBot for Perplexity’s answers. Each one does — or doesn’t — follow the robots exclusion protocol, and each plays a different role depending on whether it’s used to train a model, index content for a conversational answer, or cite a source in real time. Ignoring that diversity means either mistakenly blocking acquisition channels that have become significant, or letting content get scraped that would have been better kept out of training sets.

#The families of AI crawlers and what they do

Three distinct uses hide behind the generic label “AI crawler,” and they don’t call for the same access policy.

The distinction between a training crawler and a real-time browsing agent is central. Blocking GPTBot doesn’t stop ChatGPT-User from visiting a page when a user explicitly asks for information about a given site: these are two separate agents, governed by separate rules in robots.txt.

#What robots.txt can actually do

The robots exclusion protocol remains a convention, not a technical barrier. A well-behaved crawler complies with it; a careless one simply ignores it, which explains the recurring controversies around Bytespider or other less identifiable agents. For the crawlers that do comply, the syntax stays the familiar one from the REP (Robots Exclusion Protocol) standard.

User-agent: GPTBot
Disallow: /account/
Disallow: /search/
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Bytespider
Disallow: /

This file illustrates a differentiated policy: allowing Google-Extended because visibility in Google’s AI overviews remains strategic, blocking ClaudeBot if the publisher doesn’t want its content used to train a third-party model, and closing off Bytespider as a precaution. Each site should weigh this against its own business goals rather than copy a generic template.

#llms.txt: an emerging convention, not a standard

Introduced in late 2024, the llms.txt file proposes a structured Markdown summary of a site’s content, meant to be read directly by a language model rather than to steer a crawler. It sits at the root, alongside robots.txt and sitemap.xml.

llms.txt isn’t officially adopted by any of the major model providers as a source of truth for indexing; its value remains the readability it offers an agent that chooses to consult it.

In practice, the file lists a site’s key sections with a short description and a link, in a format close to an annotated table of contents:

# Mail Studio

> Technical news magazine for web professions.

## Main sections
- [Front-end](https://www.mail-studio.com/en/frontend/): frameworks, performance, accessibility
- [AI](https://www.mail-studio.com/en/ai/): AI tooling for developers, MCP, agents
- [Security](https://www.mail-studio.com/en/security/): hardening, vulnerabilities, best practices

Adoption remains uneven: no major conversational engine has confirmed using it systematically to index or prioritize content. Publishing one is more of a reasonable, low-cost bet than a technical obligation — in the same spirit as exposing capabilities through an MCP-compliant server, which aims at a related goal: making a site or service natively readable by an agent rather than a human.

#The role of structured data in AI answers

Schema.org markup (Article, FAQPage, HowTo, Product) no longer serves only the rich snippets of classic search results. Conversational answer systems rely heavily on the same data to extract verifiable facts, cite a source with an author and a date, or understand the structure of a blog post. An article with no Article markup and no explicit author field starts at a disadvantage against an equivalent, properly marked-up competitor.

#Should training crawlers be blocked?

The question divides content publishers. Blocking GPTBot and ClaudeBot protects the value of original content against free reuse by third-party models, but it also removes the site from a form of presence in those same models’ future answers, as users increasingly turn to conversational assistants instead of classic search. There’s no universal answer: a law firm or a paid training provider has more to gain from blocking than a technical blog trying to build its reputation. The decision depends on the business model, not a generic best practice to apply without thought.

An increasingly common compromise is to allow indexing of public editorial content while blocking high commercial value areas — premium content, proprietary technical documentation, detailed pricing pages — through Disallow rules targeted at those directories rather than a blanket block on the whole domain.

#Checking what’s actually visiting a site

Beyond declarative configuration, the only way to know what’s really happening is to examine server logs. Filtering on the user agent, cross-checked against the source IP address, distinguishes a genuine GPTBot from an impersonator that merely copies the declared user agent: major publishers release official IP ranges or offer verification through reverse DNS lookup followed by a forward lookup, the same method historically used to authenticate Googlebot.

#Key takeaways

Managing AI crawlers plays out on three levels: robots.txt to set an access policy per agent, llms.txt as an optional complement for the models that choose to read it, and structured data so that authorized content is correctly interpreted once collected. None of these levers is an absolute guarantee, but combining them remains the best protection available for a site that wants to consciously choose its visibility in AI-generated answers.

Further reading: the AI for web developers guide and the GA4 and Search Console dashboard to track traffic from AI referrers.