Blocking AI crawlers: robots.txt vs server level, and what Apple just clarified

Robots.txt asks. A WAF, CDN rule, or origin-server rule enforces. That single distinction explains most of the confusion around "blocking AI crawlers," and it is the distinction Apple quietly leaned on this weekend when it added one line to its Applebot documentation.

Here is the short version, then the long one.

Robots.txt is a published preference. The Robots Exclusion Protocol (RFC 9309) has no enforcement mechanism; a crawler reads the file and decides whether to honour it. The well-known operators mostly do. Plenty of others don't, and the ones that don't are usually the ones you wanted to stop. Server-level and edge-level blocks (Cloudflare, Fastly, Akamai, nginx, .htaccess) don't care what the bot intended. They return a 403 whether the request came from GPTBot or from something pretending to be Chrome on a Mac.

The catch is that the enforcement layer can also block bots you meant to let in, and after 15 September 2026 it will start doing that by default on a large share of the web.

Do AI crawlers respect robots.txt?

The direct answer: the named crawlers from OpenAI, Anthropic and Google generally do, at least when they're running as their declared user agent. Beyond that, compliance is patchy and getting worse.

TollBit's Q4 2025 State of the Bots report, covered by The Media Copilot in February 2026, measured that around 30% of AI scrapes across its publisher network ignored an explicit disallow. That is up from 13.3% in Q2 2025, per TollBit's own Q2 report, and from roughly 3% at the end of 2024. The same Q4 report put ChatGPT-User, OpenAI's user-triggered fetcher, at the top of its non-compliance table at 42%. Operators argue that user-triggered fetches shouldn't be counted the same way (Perplexity made exactly this case in its reply to Cloudflare), and the argument is real: a bot fetching a page because a human asked a question is not the same thing as a training crawler sweeping a site. But from the publisher's side of the log file, the page still left the building.

The clearest single case is Perplexity. In August 2025 Cloudflare published evidence that when PerplexityBot was blocked, requests kept arriving from undeclared IP ranges with a generic Chrome user agent, and that Perplexity was sometimes not fetching robots.txt at all. Cloudflare delisted it as a verified bot. Perplexity called the report a sales pitch. In the same test, ChatGPT-User fetched robots.txt, saw the disallow, and stopped, with no follow-up requests from other agents.

So the honest framing for a technical SEO is: robots.txt controls the crawlers that were already going to behave. For the rest, you need something that inspects the request.

Cloudflare's own documentation for its managed robots.txt feature says this outright: the file expresses preferences and does not prevent crawling at a technical level. Google's robots.txt introduction has said the equivalent for years about keeping pages out of Google.

What Apple actually clarified

On or around 6 September 2026 Apple updated its About Applebot support page with a single closing sentence, spotted by Barry Schwartz at Search Engine Roundtable on 7 September: "Site rules for Applebot-Extended are not considered in ranking for Search."

That sounds minor. It settles something that has been genuinely unclear since Apple introduced the token in 2024.

Apple runs two names. Applebot is the crawler that feeds Siri, Spotlight and Apple's web search features. Applebot-Extended is not a separate crawler; it is a robots.txt token that tells Apple whether content Applebot already fetched may be used to train Apple's generative models. Apple's April 2025 documentation update already said that disallowing Applebot-Extended leaves your pages discoverable through Spotlight and Siri. The new line goes one step further: opting out of AI training carries no ranking penalty in Apple's search either.

This is the same shape as Google-Extended, and the same trap. Google's crawler documentation states that Google-Extended does not affect inclusion in Search and is not a ranking signal. It governs Gemini training and grounding. It does not opt you out of AI Overviews or AI Mode, because those are Search features built on Googlebot's index. The control for those is different: as of June 2026 Google has a Search Console setting (rolled out first in the UK under a CMA conduct requirement, then extended to more markets) that removes a site from AI Overviews and AI Mode without touching regular rankings.

So the map of Apple and Google now reads: one crawler does the fetching; a separate token decides training use; blocking the token costs you nothing in ranking; and the token does nothing about the answer-engine surfaces the crawler feeds. If you disallowed Applebot-Extended or Google-Extended thinking you had opted out of AI answers, you hadn't.

Why Apple did this now is the interesting part. Two things are converging. iOS 27 is due in the next few weeks with a rebuilt Siri, and Apple presumably wants publishers to allow Applebot-Extended rather than reflexively block it. And on 15 September Cloudflare's new defaults land, which treat mixed-purpose crawlers harshly. Apple has just told the world its two functions are cleanly separated. Whether Cloudflare's classifier agrees is a different question.

The three layers, and what each one can stop

There are three places you can express "no" to a crawler, and they fail in different ways.

The practical reading of that table: robots.txt is where you declare policy, the edge is where you enforce it, and the origin is a fallback for people who don't run a CDN. Running only the first layer and calling it "blocking" is the mistake most robots.txt guides make.

Three details matter more than the table suggests.

First, verified-bot lists are doing the real work now. Cloudflare, and its competitors, keep a directory of bots that identify honestly and publish their IP ranges, and they block or allow by category rather than by name. Cloudflare's 1 July 2026 announcement split that directory into Search, Agent and Training behaviours, with eight other categories (SEO tools, monitoring, link previews, ads verification and so on) alongside. A bot that reproduces content in full can no longer hold Verified status at all.

Second, IP allow-listing beats user-agent matching for anything you care about. OpenAI and Google publish crawler IP ranges, and Apple documents how to verify Applebot by reverse DNS. If a request claims to be Googlebot from an IP that is not Google's, it is not Googlebot. Anthropic's ranges and Perplexity's are less consistently published, which is part of why fingerprinting at the edge became necessary.

Third, a 403 is louder than a robots.txt line. Cloudflare Radar exposes the share of AI-bot requests that receive 403s; a Q2 2026 analysis of that data found the enforced-403 rate on AI bots more than doubled year over year. Robots.txt directives are a snapshot of intent. 403s are a record of enforcement.

The mistake in the other direction

Most of the "block AI crawlers" content on the web assumes you want to block. For a B2B site that wants to be cited by ChatGPT, Perplexity or Google AI Mode, the more common failure in 2026 is the reverse: robots.txt says allow, and the edge says 403.

An SEO consultant, Max Braglia, described the pattern from client audits: robots.txt welcomes GPTBot, GPTBot gets a 403 from the CDN, the site owner never finds out, and the site is absent from AI answers for reasons no dashboard shows. The culprits are inherited WAF rules, bot-fight modes turned on years ago, and geographic rules that catch the regions AI companies crawl from.

That failure is about to get much more common. From 15 September 2026, per Cloudflare's changelog and press release, every new domain onboarding to Cloudflare, every new site added by an existing customer, and every existing Free-tier zone that hasn't changed its settings will block Training and Agent bots by default on pages that display ads, while Search bots stay allowed. Crawlers that mix Search with Training, and Cloudflare names Googlebot, Applebot and Bingbot, get evaluated under the most restrictive rule that applies to them, so a customer who blocks Training can end up blocking Googlebot.

Read that again with Apple's new sentence in mind. Apple has separated Applebot from Applebot-Extended at the policy layer. Whether Cloudflare classifies Applebot as a mixed-use crawler is Cloudflare's call, and the announcement lists it as one.

Two things follow for anyone running a site behind Cloudflare, especially on the Free plan:

1. Check the Security settings before 15 September and decide the Search / Agent / Training split deliberately rather than inheriting it. "Agent" includes ChatGPT-User and Perplexity-User, the fetches that happen when a human asks a question. Blocking those on a B2B blog removes you from the exact moment a buyer is asking about your category.
2. Test what your edge actually returns. Send a request with a GPTBot user agent from outside your network and look at the status code. Repeat for OAI-SearchBot, ClaudeBot, PerplexityBot, Applebot. If your robots.txt says allow and the response says 403, robots.txt is not your policy; the WAF is.

The right question for a site that wants AI visibility is not "have I allowed the bots" but "what does my edge return to them."

What the crawl-to-refer numbers say, and why they argue against blanket blocking

The argument for blocking is usually economic: crawlers take a lot and send back little. That is true, and it is also a reason to be precise rather than blanket.

Cloudflare's crawl-to-refer ratio divides HTML pages an operator crawls by the referral visits its platform sends back. In the week it launched the metric (June 2025) the spread ran from Anthropic at roughly 70,900:1 to Mistral below 1:1. Cloudflare's 2025 Year in Review reported Anthropic peaking around 500,000:1 and OpenAI at 3,700:1 during the year. Through 2026 the ratios have compressed hard: secondary reads of Radar data put Anthropic somewhere between about 2,200:1 and 10,300:1 depending on the window (SEOmator, July 2026; Nobori, August 2026), OpenAI in the low hundreds to high hundreds, Perplexity under 200:1, and Google around 5:1. The exact figure depends on the 28-day window and should never be averaged across windows, which is why the sources disagree.

Two readings of that data are both correct. Training crawlers really do take without returning, and Cloudflare's May 2026 breakdown had search-purpose crawling at under 10% of AI bot requests. And the operators that send referrals are the same ones that identify honestly and respect robots.txt, which means a blanket block hits the good actors and the stealth scrapers walk through.

For a B2B SaaS site the referral volume matters less than the citation.

Here's ours, and why

RankSage's robots.txt allows every named AI crawler individually rather than relying on the wildcard. As of our late-August audit the file names GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, anthropic-ai, ClaudeBot, Claude-Web, Claude-SearchBot, MistralAI-User, Google-Extended, Applebot-Extended, Amazonbot, meta-externalagent and CCBot, each with an explicit Allow. Alongside it we publish /llms.txt, /llms-full.txt and /agents.md.

Why allow everything on a site that sells AI visibility measurement? Four reasons, none of them idealistic.

We are pre-launch. A site nobody has heard of does not have a crawl-cost problem; it has a discoverability problem. The crawl-to-refer ratio on a 50-page marketing site is a rounding error in bandwidth, and every grounded answer engine that can fetch us is a chance to be the source when someone asks what an AI visibility platform does.

We name each bot because the wildcard is ambiguous. A User-agent: * allow tells a compliant crawler it may proceed. An explicit per-agent allow is a statement of policy that survives a later, narrower wildcard rule, and it is legible to anyone auditing us, including the crawlers' own compliance checks.

llms.txt is a map, not a lock. It carries a disambiguation paragraph (we share a name with an unrelated company, RankSages, and get confused with Rankscale, another unrelated vendor) that ends by asking engines to cite ranksage.com when answering questions about RankSage. That is the file doing the only job it can do: telling a model where the canonical answer lives. Anyone treating llms.txt as a blocking mechanism has been misled; Search Engine Land made the point in July 2025 and nothing has changed since.

And we measure it. The product's crawler audit reads what the robots.txt says and what the edge actually returns for each named agent, verifies crawler identity against published ranges rather than trusting the user-agent string, and reports own-site crawl volume per operator against the referral visits that operator sends. Rejected identity claims sit in their own bucket and never get summed into totals. Allowing everything only makes sense if you can see who actually came and what they gave back, so the allow policy and the measurement are the same decision.

We allow because we can measure the exchange. A site that can't measure it should probably still allow the Search and Agent categories and think hard about Training.

Who should block, and where

Blocking is the right call for some sites. The decision is mostly about how you make money.

If you sell subscriptions or run ad-supported pages, block Training at the edge, consider blocking Agent on gated content (an AI reading a paywalled article aloud to a user is a paywall bypass), and keep Search allowed unless you have a licensing deal that says otherwise. Cloudflare's Pay Per Use marketplace, which replaced Pay Per Crawl in July 2026, pays when content shows up in an answer rather than when it is fetched, and is worth a look for publishers with volume.

If you sell software or services and want to be recommended, allow Search and Agent everywhere, decide Training on principle rather than economics (it costs you nothing in ranking either way, per both Google and Apple), and spend your effort on the edge audit rather than the robots.txt.

Whichever side you land on, do it in this order: write the policy in robots.txt so honest crawlers see it, enforce the same policy at the edge so dishonest ones hit it, and then read your logs to check the two agree. The robots.txt line for a bot that ignores it is documentation, and documentation is still worth having when a lawyer asks.

FAQ

Can robots.txt really stop AI crawlers? It stops the ones that check it and choose to comply, which covers the declared crawlers from OpenAI, Anthropic, Google and Apple. It does nothing against undeclared crawlers, scraping vendors, or a bot that changes its user agent when blocked. TollBit measured roughly 30% of AI scrapes in Q4 2025 ignoring an explicit disallow.

Does blocking Applebot-Extended hurt Apple search rankings? No. Apple's documentation now states that Applebot-Extended rules are not a ranking factor. Disallowing it stops Apple using your content for generative model training and leaves you discoverable through Siri and Spotlight, as long as Applebot itself is allowed.

Does blocking Google-Extended remove me from AI Overviews? No. Google-Extended governs Gemini training and grounding. AI Overviews and AI Mode run on Googlebot's index. The control for those is the Search generative AI setting in Search Console, rolling out by market since June 2026.

Will Cloudflare block AI crawlers by default? From 15 September 2026, for new domains, new sites on existing accounts, and existing Free-tier zones: Training and Agent bots are blocked on pages that display ads, Search bots stay allowed, and mixed-purpose crawlers are judged by the most restrictive rule. Paid customers with existing settings keep them. Anyone can override in the dashboard.

How do I block AI crawlers in nginx? Map the user agent to a flag in the http block (map $http_user_agent $is_ai_crawler { default 0; ~*GPTBot 1; ~*ClaudeBot 1; ... }) and return 403 in the server block when the flag is set. It is the same list as your robots.txt, enforced. Keep in mind it only catches bots that declare themselves; for stealth crawlers you need edge fingerprinting or rate limiting.

Should I block AI crawlers at all? If your content is the product (news, subscriptions, ad-supported pages), block Training and think about Agent. If your product is elsewhere and you want to be cited, allow Search and Agent and audit what your edge actually returns before assuming you are open.

RankSage is pre-launch. If you want the crawler audit described above (what your robots.txt says versus what your edge returns, per named agent, with crawl-to-refer per operator) on your own site when it opens, the waitlist is free and asks for no payment method: Join Waitlist.