Originally Posted by
nm2582
There is also the issue that some sites (reddit being one of them) have started to limit the amount of data they expose to google, due to the increase in people using google to search reddit to find useful information (searches such as "best 747 model site:reddit.com") in a way that only adds additional load but doesn't actually benefit the underlying community. So, the available relevant content that google has available to show you is reduced (because less is in their index) AND then you add the AI and advertising layers on top of that....
Google has a commercial arrangement with Reddit that enables it to access and index Reddit’s content.
In early 2024, Google and Reddit struck a licensing agreement reportedly worth $60 million per year. This deal allows Google exclusive access to Reddit’s vast array of user-generated content through Reddit’s Data API, which Google uses to enhance its search results and train its AI models. As part of this arrangement, Reddit has updated its robots.txt file and technical settings to block other major search engines—such as Bing, DuckDuckGo, and Mojeek—from crawling and indexing new Reddit content, while still permitting Google to do so.
This exclusivity gives Google a unique advantage in surfacing Reddit content in its search results and in training AI, while competitors are now unable to access recent Reddit data. The arrangement has raised concerns about competition and the broader implications for AI development and internet search diversity.