A client who relies on Cloudflare for active management / WAF (web application firewall) had a longstanding rule in place in Cloudflare blocking AI training bots.
Cloudflare announced in July 2026 that beginning September 15, 2026, it would begin classifying Googlebot as an AI training bot and blocking it if AI training bots were blocked. They changed their mind on the date of the change, September 15, and instead announced they will block Google-Extended as an AI training bot and will leave Googlebot alone. Their early test of these rules – unannounced or accidentally deployed – reveals why.
In late August 2026 – a month earlier than announced – Cloudflare quietly began classifying Googlebot as an AI training bot for this site and reportedly many others.
How blocking Googlebot can invisibly wreak havoc on a site
When Googlebot attempts to crawl an actively managed Cloudflare website set to block it, it returns a 403 error. Google will then subsequently de-index the page and classify the URL as Blocked due to access forbidden (403).
A problem like this can go undetected until it is pervasive. The speed, extent, and impact of de-indexing is dependent on when and what Googlebot recrawls; and the Google Search Console Indexing report updates sporadically, resulting in a reporting lag that can sometimes take multiple weeks. Performance data is more up-to-date, but the issue won’t reveal itself until a noticeable drop happens.
How We Fixed It
The initial fix
On Monday morning we discovered the de-indexing issue and investigated the cause, since mass de-indexing can be the result of a number of problems (a classic example being someone accidentally checking a “Discourage search engines from indexing this site” box on a WordPress website or someone accidentally adding a Disallow: / to their robots.txt).
Checks using the URL inspection tool in Search Console and curl user-agent requests revealed Cloudflare was actively blocking Googlebot in all forms. Double checking their robots.txt confirmed the site was still actively blocking AI training bots through Cloudflare.
Within an hour of notifying the client, the Cloudflare rules were removed, and live URL tests using Search Console showed the pages were once again accessible. A quick manual submission confirmed the issue could be reversed immediately.
The recovery effort
Post-Cloudflare fix, we mass inspected the site using the URL Inspection Tool to determine the extent of the de-indexed pages and assembled a triage list by page type and importance. We submitted all sitemaps and began a daily campaign of mass status rechecks and manual URL submissions.
As a future just-in-case measure, daily user-agent checks for Googlebot accessibility were established for all clients.
Outcome
By Tuesday (the next day), all major product category pages that were affected were re-indexed.
By Friday, all product category and product pages were re-indexed, and of the remaining real URLs impacted on the site, only a small handful of blog posts were left to re-index.
By the following Monday, all outstanding affected blog posts were re-indexed as well.
