AI Answer Summary
Major publishers including USA Today, Reddit, Politico, Reuters, and The Economist are weighing whether to block Google's web crawler as AI Overviews and AI Mode reduce click-through traffic. Google search traffic to USA Today fell nearly 50% in the 12 months ending June 2026. Politico fell 20%, CNN and Business Insider 31% each. The core problem: Google uses the same crawler for traditional search indexing and AI features, so publishers cannot block AI use without also removing themselves from search results. When publishers disappear from the indexable web, the evidence base that AI engines cite fragments — and brands that have built AEO strategies on third-party editorial coverage face a less predictable citation environment.
The Story Behind the Story
The Wall Street Journal reported on 21–22 July 2026 that executives at USA Today, People Inc., Reuters, Reddit, and The Economist have discussed blocking Google's web crawler. USA Today CEO Mike Reed told the Journal: "It's time to take a stand and say enough is enough." Politico has discussed blocking Google and other bots from accessing its non-paywalled articles. Reddit is reconsidering its $60 million per year content deal with Google.
The numbers behind these decisions are stark. Google search traffic to USA Today fell nearly 50% in the 12 months ending June 2026, according to Semrush data cited by the Journal. Politico fell 20%. CNN and Business Insider each fell 31%. Reuters and the Washington Post saw steeper declines over the trailing year. People Inc. told the Journal that Google now accounts for roughly one-quarter of its traffic, compared with more than half two years ago.
The mechanism is well understood. AI Overviews and AI Mode answer queries directly on the results page. The reader gets the answer. The click never happens. The content that funded the answer receives nothing.
The Crawler Bind
The structural problem for publishers — and the reason this story matters beyond media industry economics — is that Google uses the same web crawler for both traditional search indexing and AI-generated features. A publisher cannot tell Google to stop using its content for AI responses while continuing to appear in regular search results.
The only clean exit is a full noindex instruction, which removes a site from Google search altogether. For most publishers, that trade is far steeper than the traffic they are already losing. The tool that is starving them of clicks is still the one delivering the clicks they have left.
This is why the debate has taken so long to reach this point. Publishers have been watching their traffic decline for two years. The reason they are only now discussing blocking is that the calculus is finally shifting: for some publishers, the traffic Google sends is no longer large enough to justify the cost of providing Google with the content it uses to answer queries without sending traffic.
Penske Media, the owner of Variety and Rolling Stone, sued Google in September 2025 on antitrust grounds, alleging that its AI summaries illegally use the publisher's reporting and depress online traffic. In the UK, Google agreed to provide publishers with an opt-out for AI Overviews following discussions with regulators. A similar option is not yet available for publishers in the US.
What This Means for AEO
Answer Engine Optimisation is built on a premise: that AI engines — Google AI Mode, ChatGPT, Perplexity, Gemini — synthesise answers from the indexable web, and that brands which appear in the sources those engines cite gain visibility, authority, and eventually pipeline.
That premise depends on the indexable web remaining a reasonably stable, high-quality evidence base. When major publishers start disappearing from it, three things happen.
The evidence base fragments. AI engines cite sources. The sources they cite are weighted toward authoritative, frequently updated, widely linked publications. When USA Today, Reuters, Politico, and The Economist reduce their crawlable footprint or disappear from it entirely, the pool of authoritative sources that AI engines can cite shrinks. The answers AI engines generate become less grounded in current reporting and more dependent on whatever remains crawlable — which may include lower-quality sources, older content, or content that was never designed to be cited as evidence.
Citation patterns shift unpredictably. Brands that have built AEO strategies around being cited alongside specific publications — or around appearing in the same answer as a Reuters or WSJ article — will find those citation patterns disrupted. The AI engine still needs to answer the query. It will find other sources. Whether those sources are as authoritative, as current, or as aligned with the brand's positioning is unpredictable.
The open web's role in AI answer generation becomes less reliable. The current model of AI answer generation assumes that the web is a reasonably open, crawlable, high-quality information environment. Publisher blocking accelerates a shift toward a more fragmented environment where the most authoritative sources are either paywalled, blocked, or available only through licensing deals that most brands cannot access.
The B2B Implication
For B2B marketers, the practical implication is not that AEO stops working. It is that the environment in which AEO operates is becoming less stable, and strategies that depend on a specific citation ecosystem need to account for that instability.
Three adjustments are worth making now.
Build primary source authority, not just third-party citation. The most durable AEO position is one where your own content — your research, your data, your structured answers — is directly cited by AI engines, not just your coverage in publications that may become less crawlable. Original data, proprietary research, and structured FAQ content that directly answers buyer questions are more resilient to publisher blocking than strategies that rely on earned media coverage.
Diversify across AI engines. Google AI Mode, ChatGPT, Perplexity, and Gemini use different source pools and different weighting systems. A brand that is visible in ChatGPT and Perplexity is less exposed to disruptions in Google's crawl ecosystem than one that has optimised only for Google AI Overviews.
Monitor citation sources, not just citation frequency. As the publisher landscape shifts, the sources that AI engines cite for your category will change. Tracking which sources are being cited alongside your brand — and which are disappearing — gives you early warning of shifts in the evidence base before they affect your visibility.
The Broader Pattern
The publisher blocking story is one symptom of a broader restructuring of the relationship between content creation, distribution, and monetisation on the web. The open web model — in which publishers created content, Google indexed it, and both parties benefited from the traffic exchange — is breaking down. AI-generated answers have disrupted the traffic side of that exchange without yet providing a replacement revenue model for publishers.
Cloudflare announced in July 2026 that starting 15 September 2026, its default settings will block "mixed-use" crawlers from any pages that host ads. This affects a significant portion of the web's content infrastructure. The direction of travel is toward a more fragmented, more gated, and more negotiated content environment.
For brands building long-term AEO strategies, the lesson is the same one that applied to SEO after the 2011 Panda update and the 2012 Penguin update: strategies built on the assumption that the current environment is stable are fragile. Strategies built on creating genuinely useful, original, structured content that directly serves buyer intent are more resilient to whatever the environment does next.
Frequently Asked Questions
Why are publishers considering blocking Google's web crawler? Google's AI Overviews and AI Mode increasingly answer search queries directly on the results page, reducing the click-throughs that sent readers to publisher sites. Google search traffic to USA Today fell nearly 50% in the 12 months ending June 2026. Politico fell 20%, CNN and Business Insider 31% each. Publishers are weighing whether the traffic Google still sends justifies providing Google with the content it uses to generate AI answers.
Can a publisher block Google's AI without losing search traffic? Not cleanly. Google uses the same web crawler for both traditional search indexing and AI-generated features. A publisher cannot tell Google to stop using its content for AI responses while continuing to appear in regular search results. A full noindex instruction — which removes a site from Google search entirely — is the only complete exit. For most publishers, that trade is far steeper than the traffic they are already losing.
What does publisher blocking mean for AEO strategies? When major publishers reduce their crawlable footprint, the evidence base that AI engines use to generate answers fragments. AI engines still need to answer queries, but the pool of authoritative, current sources they can cite shrinks. Citation patterns shift unpredictably. Brands that have built AEO strategies around being cited alongside specific publications will find those patterns disrupted.
What is the most resilient AEO strategy in this environment? Build primary source authority rather than relying solely on third-party citation. Original data, proprietary research, and structured FAQ content that directly answers buyer questions are more resilient to publisher blocking than strategies that depend on earned media coverage in publications that may become less crawlable. Diversifying across AI engines — Google AI Mode, ChatGPT, Perplexity, Gemini — also reduces exposure to disruptions in any single crawl ecosystem.
What is Cloudflare doing about AI crawlers? Cloudflare announced in July 2026 that starting 15 September 2026, its default settings will block "mixed-use" crawlers from any pages that host ads. This affects a significant portion of the web's content infrastructure and accelerates the shift toward a more fragmented, more gated content environment.
What happened with publishers and Google in the UK? Following discussions with UK regulators, Google agreed to provide publishers in the UK with an opt-out mechanism for AI Overviews — allowing publishers to exclude their content from AI-generated summaries while retaining normal search indexing. A similar option is not yet available for publishers in the US.
Modi Elnadi is the founder of Integrated.Social, a London-based AI growth marketing agency specialising in Answer Engine Optimisation, Agentic AI, and AI-native B2B demand generation.
