The handy “AI Overviews” that Google now places at the top of search results are robbing sites that users might otherwise reach from the search engine of traffic. When websites try to block Google’s AI, they are removed from search results and see their traffic plummet even further, since the AI answer generator and Google’s search bot are combined into a single system and cannot be controlled separately.
Google’s dominance in search, which a federal judge last week ruled an “unlawful monopoly,” gives the company a crucial advantage — publishers are forced to choose between making their content available to AI models or being removed from Google’s search engine, the primary source of their traffic. Many are unwilling to take that risk.
Google says that AI Overviews — the summaries that appear at the top of search results — are a result of its long-standing commitment to providing better information and opportunities for publishers.
“Every day, Google sends billions of users to sites across the web, and we’re committed to continuing this long-standing value exchange,” a Google spokesperson said. “AI reviews help people find Google Search more useful and keep them coming back to search for more, creating new opportunities for content discovery.”
The Dilemma for Publishers
Google uses separate crawlers for some products, like the Gemini chatbot. But Googlebot, the company’s primary crawler, serves both AI Overviews and Google Search because, according to a spokesperson, “they are tightly intertwined.” Google said publishers can block certain pages or parts of pages from appearing in AI Overviews, but that would also likely prevent them from appearing in all other Google search features.
iFixit CEO Kyle Wiens said iFixit’s relationship with Google is “much looser” than it is with other AI companies. “I can stop ClaudeBot from indexing us without hurting our business,” he said, referring to generative AI startup Anthropic. “But if I block Googlebot, we lose traffic and customers.”
“It’s like an existential crisis [for website owners],” says Joe Ragazzo, publisher of the news site Talking Points Memo. “There are two bad options. You leave and die immediately, or you partner with them and probably just slowly die because eventually they won’t need you either.”
The rise of generative AI has spawned a wave of startups offering AI-powered search products. The growing popularity of chatbots could pose a serious threat to Google’s search business. But competing with the search giant requires crawling and indexing the web as completely as possible, which is no easy task. This requires money, computing power, and large storage capacities.
Many publishers, fighting against unlicensed use of content for indexing, limit the crawling of their resources to third-party companies, giving the greatest freedom of action only to large search engines such as Google or Bing, which serve as sources of traffic for them.
The Future of Search and AI
Search startups can’t generate the kind of traffic that the big search players do, so they’re forced to pay publishers to license their content. Amid a wave of deals between media companies and AI startups, Google’s refusal to try to license content is especially noticeable, and publishers have little leverage over the company. Aside from a single $60 million deal with Reddit that sent traffic to the social media site soaring, Google has made it clear to publishers that it’s not interested in such negotiations.
Search startup Perplexity’s attempts to strike a similar deal with Reddit have failed because Google’s bar is too high, according to people familiar with the matter. Other search startups also have no way to access content from sites like Reddit. “It would take us 20 years of our current revenue just to pay Reddit,” said Vladimir Prelovac, founder of search startup Kagi. “I don’t even consider that as a possibility.”
It’s not just small startups that are struggling to index content. Major popular sites including Amazon, Goodreads, and Uniqlo have blocked OpenAI’s GPTBot crawler, potentially hampering the company’s ambitions in web search. OpenAI insists that sites can still appear in search results even if they block indexing. That’s because robots.txt files, which set crawling rules, have not been found to be legally binding, so public data can be indexed as long as no login or credentials are required, notes NIX Solutions.
In the wake of a landmark court ruling that found Google to have a monopoly on the online search market, the US Justice Department is considering a range of legal options, from allowing competitors to access Google’s search index to breaking up the company. The EU’s Digital Markets Act already requires Google to share some search query data.
Wiens believes that “separating Google search from their AI work would eliminate the conflicts.” Kamyl Bazbaz, vice president of public affairs at search engine DuckDuckGo, noted the importance of search indexes in the age of AI, saying that “the technological shifts happening in search make Google’s index even more problematic when it comes to antitrust issues.” Regardless of the outcome of Google’s antitrust case, the changes happening in the search landscape make it clear that publishers shouldn’t become overly dependent on any one technology platform, including Google. “We believe that you have to form real relationships with readers,” Ragazzo said, “and that’s how you build a publication that can withstand different eras.”
We’ll keep you updated on any developments in this evolving situation.