A generated robots.txt with a wildcard rule plus one explicit rule per named AI agent, all sharing the same disallow list as the wildcard — the app, the platform panel, signed links, portals and API routes. Agents are grouped in code into ANSWER crawlers (fetched because a person just asked a question, and cited in the answer) and TRAINING crawlers (collecting text for model training).
Also called: robots.txt · should I block chatgpt · AI crawlers · GPTBot ClaudeBot
- 1AI_ANSWER_AGENTS lists eight fetch-on-demand agents that produce citations.
- 2AI_TRAINING_AGENTS lists twelve dataset crawlers, including Google-Extended, which gates Gemini training but not Google Search.
- 3robots() emits the wildcard rule plus one identical rule per named agent, and declares the sitemap and host.
- 4DISALLOWED_PATHS is shared, so no named agent ever gets more access than the wildcard.
The distinction is the whole point, and it is spelled out: "Two different jobs hide under one word, and conflating them is how sites accidentally make themselves invisible. ANSWER crawlers fetch a page because a person just asked a question, and cite it in the answer… Blocking these removes us from answers — it is the AI equivalent of blocking Googlebot." The business decision follows: "a builder asking an assistant 'how do I bid a post-frame job' and getting our answer with our name on it is the single cheapest distribution we have." And naming them when the wildcard already covers them is deliberate: "a named agent is an EXPLICIT decision that survives the next person who edits robots.txt."
- Answer crawlers and training crawlers were treated as one thing.
- A wildcard rule records no decision, so the next edit can reverse it silently.
- Private surfaces needed the same protection from named agents as from everyone else.
See it on your own jobs
Twenty minutes, your numbers, no slide deck. We’ll build one of your real buildings in front of you and send you the estimate link at the end — yours to keep either way.
or keep browsing features →