Skip to main content
Contact

robots.txt in 2026: Why Blanket Blocking or Allowing Is Both Wrong

6 min readPublished on May 19, 2026Updated on September 09, 2026

This article was written with AI assistance and editorially reviewed.

Contents

Most sites have not touched their robots.txt since it was first set up. For the AI crawlers that have appeared since 2023, that means they are not being handled by a deliberate decision, but by a default that predates ChatGPT. In practice, that leaves many sites either silently blocking every new bot or just as thoughtlessly leaving everything open - both are the wrong call, just for different reasons.

In short

Robots.txt decides which AI crawlers are even allowed to read your content. Block every bot and you disappear from AI answers. Allow every bot and you also feed scrapers that give you nothing back. The right configuration distinguishes bots by purpose, not by gut feeling.

Why robots.txt now decides AI visibility

Until a few years ago, robots.txt was a footnote: set up once to keep search engines out of admin areas, then never touched again. Generative AI changed that. Every major provider now runs its own bots that fetch content live, either to answer questions or to train models - and every one of them reads robots.txt before it accesses anything. One wrong or outdated line there directly decides whether your business can even be surfaced as a source in a ChatGPT or Perplexity answer.

The check takes a minute: open yourdomain.com/robots.txt in a browser. If it shows a blanket block for every bot, an empty shell left over from a hosting template, or nothing at all, that is not a neutral starting point - it is a silent decision against AI visibility that nobody actually made on purpose.

What robots.txt technically does, and does not do

Robots.txt is a plain text file in a domain's root directory that tells individual or all bots which paths they may crawl. The core syntax is two lines per block:

User-agent: GPTBot
Allow: /

User-agent names the bot, Allow or Disallow set which paths it may visit. A block only applies to a bot that matches its user-agent name exactly - the catch-all rule under User-agent: * only applies to bots that find no more specific block of their own. The limit worth understanding: robots.txt is a request, not access control. Reputable bots from OpenAI, Anthropic, Perplexity or Google honour it because doing so serves their own interest in a functioning bot ecosystem. An actor that does not want to comply simply will not - robots.txt does not protect against malicious access, it only steers the behaviour of cooperative systems.

The AI crawlers worth knowing

Not every bot serves the same purpose, and that is exactly what most robots.txt files ignore:

BotOperatorPurpose
GPTBot, OAI-SearchBot, ChatGPT-UserOpenAILive fetching for ChatGPT answers and search
ClaudeBot, Claude-SearchBot, Claude-UserAnthropicLive fetching for Claude answers and model training
PerplexityBot, Perplexity-UserPerplexityLive fetching for answers with source attribution
Google-ExtendedGoogleSeparate switch for AI training, independent of classic Googlebot
Applebot, Applebot-ExtendedAppleSpotlight search and Apple Intelligence training
CCBot, Bytespider, DiffbotCommon Crawl, ByteDance, DiffbotBulk scraping for third-party training sets, usually with no citation or traffic in return

The first four groups give something back that matters for a business: a chance at being cited with a link, or at least brand presence in the memory of a system potential customers use daily. The last group typically gives back neither - the data disappears into a training set with no user ever finding their way back through a link.

Blocking everything or allowing everything are both wrong

The most common reaction to AI crawlers is one of two overcorrections. Some block every bot with "AI" or "GPT" in the name out of caution - and cut themselves off from exactly the channels where potential customers now look for answers before they ever open a classic search engine. Others leave robots.txt wide open because that is the path of least resistance, and end up feeding unfiltered scrapers that provide no return whatsoever.

The considered position sits between the two and follows a single question: does your business get something back for allowing access - citation, traffic, brand presence in a system customers actually use? Bots that answer yes belong in robots.txt with an explicit Allow, not just under the silent *. Bots that answer no belong just as explicitly blocked. Both require actually naming the bots rather than relying on one catch-all line.

  • Give search and answer bots with citation potential (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) an explicit Allow
  • Explicitly Disallow pure bulk scrapers with no backlink value (CCBot, Bytespider, Diffbot)
  • Do not forget the sitemap line at the end of the file, so crawlers can even find the site's structure
  • Re-check robots.txt after every major structural change to the website

Concrete steps for the next 90 days

  • Days 1-30: open the current robots.txt in a browser, check every existing block against your own list of wanted and unwanted bots
  • Days 31-60: add missing Allow rules for answer bots and missing Disallow rules for pure scrapers, verify the sitemap line
  • Days 61-90: check server logs for actual traffic from the bots you allowed, and update robots.txt whenever new bot names appear

Conclusion

Robots.txt is the smallest technical decision with the largest effect on AI visibility, which is exactly why it is so often left to chance. A configuration that distinguishes answer bots from pure scrapers takes an hour to implement and keeps working for as long as the site exists. How Google and AI crawlers technically process the content behind that access is covered in our article on JavaScript SEO for SPAs. The sitemap line at the end of the file deserves just as much attention as the bot rules themselves, as we show in our article on XML Sitemaps 2026. What actually turns that access into a citable result is covered in our article on Schema.org for local businesses, and how solid HTTPS fundamentals fit in is covered in our article on HTTPS and HSTS. For a no-obligation review of your current robots.txt, feel free to get in touch.

Want to know which AI crawlers your website is currently blocking?
Get in touch

Frequently asked questions

What happens if you block GPTBot in robots.txt?

ChatGPT can no longer fetch and cite your content live. For a query your page would genuinely answer well, your business simply will not appear there - regardless of how good the content itself is.

Do AI crawlers hurt server performance?

Not noticeably with reputable providers. If you do see load spikes, you can set a crawl delay for a specific bot or restrict it to certain paths instead of blocking it outright.

Do you even need a robots.txt if you have nothing to hide?

Yes. Even without a single Disallow line, a maintained robots.txt with explicit allowances for the relevant bots signals that someone has actually thought about the site's visibility, rather than leaving it at whatever the hosting template shipped with.

Is robots.txt enough for AI visibility on its own?

No. Robots.txt only decides whether a crawler is allowed to access a page at all. Whether an AI then understands and cites the content also depends on server-rendered HTML and clean schema markup.

Where exactly does robots.txt need to live to work?

Only in the domain root, reachable at yourdomain.com/robots.txt. A file placed in a subfolder is invisible to every crawler and has no effect whatsoever.

Related service

You might also like

Get it for free

Enter your email address. You'll immediately receive a confirmation link - after clicking it the checklist is available right away.

Required field

Let's talk about your project

Message us on WhatsApp