Reddit has a warning for AI corporations and different scrapers: play by our guidelines or get blocked. The corporate mentioned in that it plans to replace its Robots Exclusion Protocol (robots.txt file), which permits it to dam automated scraping of its platform.
The corporate mentioned it’s going to additionally proceed to dam and rate-limit crawlers and different bots that don’t have a previous settlement with the corporate. The adjustments, it mentioned, shouldn’t have an effect on “good religion actors,” just like the Web Archive and researchers.
Reddit’s discover comes shortly after a number of experiences that Perplexity and different AI corporations recurrently web sites’ robots.txt protocol, which is utilized by publishers to inform net crawlers they don’t need their content material accessed. Perplexity’s CEO, in a latest with Quick Firm, mentioned that the protocol is “not a authorized framework.”
In a press release, a Reddit spokesperson instructed Engadget that it wasn’t focusing on a specific firm. “This replace isn’t meant to single anyone entity out; it’s meant to guard Reddit whereas retaining the web open,” the spokesperson mentioned. “Within the subsequent few weeks, we’ll be updating our robots.txt directions to be as clear as potential: if you’re utilizing an automatic agent to entry Reddit, no matter what sort of firm you might be, it’s essential to abide by our phrases and insurance policies, and it’s essential to discuss to us. We imagine within the open web, however we don’t imagine within the misuse of public content material.”
It’s not the primary time the corporate has taken a tough line with regards to knowledge entry. The corporate cited AI corporations’ use of its platform when it started charging for final 12 months. Since then, it has struck licensing offers with some AI corporations, together with and . The agreements permit AI companies to coach their fashions on Reddit’s archive and have been a major income for the newly-public Reddit. The “discuss to us” a part of that assertion is probably going a not-so-subtle reminder that the corporate is not within the enterprise of handing out its content material without spending a dime.
Trending Merchandise
