Cloudflare Reportedly Removes AI Bot Opt-Outs From 118 Sites' Robots.txt Files
· 3h ago · Originally reported by PPC Land
Research by HasData found that Cloudflare deleted AI crawler opt-outs from the robots.txt files of 118 websites. Despite 592 sites attempting to block GPTBot, many still served content to the bot, suggesting enforcement is inconsistent and may be shifting to CDN-level controls.
Why it matters
This highlights the challenges publishers face in controlling AI bots' access to their content, raising questions about data privacy and the effectiveness of current blocking methods.
Explain it like I’m new here
It's like putting up a 'No Trespassing' sign, but someone else keeps taking it down so robots can still come in.
Terms in this story
Publisher
A publisher is any owner of content or a digital property, such as a website, app, streaming service, podcast or screen network, that earns money by selling ad space to advertisers.
Bot traffic
Bot traffic is visits, ad impressions or clicks generated by automated software rather than people, ranging from honest search crawlers to fraudulent bots built to look human.
Go deeper
Related stories
Ad Tech Vendors Respond to iOS 27 Blocking Data and Identity Services
Apple's iOS 27 update, released in September, has started blocking major data and identity vendors from operating on its devices. Companies like LiveRamp, Permutive, and Audigent are now working to adapt their strategies in response to…
Apple Resolves The Trade Desk’s Safari Ad Issue, Hints at Wider Ad Tech Limits
Apple has addressed a technical problem that affected The Trade Desk’s ability to deliver ads on the Safari browser. However, there are indications that Apple may introduce broader restrictions impacting the digital advertising industry in…
Apple Restricts Additional Ad Tech Companies From Its Ecosystem
Apple has taken further action to block more advertising technology companies from operating within its platforms. This move is likely to impact how businesses access and use digital advertising tools on Apple devices.
Apple Invites The Trade Desk to Test iOS 27.2 Beta for Potential Ad Unblocking
Apple has requested The Trade Desk to evaluate a beta version of iOS 27.2, which could impact ad delivery in Safari. The move comes as Apple's ad-blocking measures have led publishers to consider IAB Tech Lab's Trusted Server, with several…
Apple’s Safari Ad Tech Blocklist Pushes Publishers Toward IAB Tech Lab’s Trusted Server
Apple has set a firm deadline for its Safari browser to block certain ad tech tools, prompting publishers to adopt the IAB Tech Lab’s Trusted Server solution. This move accelerates the industry’s shift toward privacy-focused ad…
Explainer: What first-party data is and why advertisers collect it
The article defines first-party data as information an organisation gathers directly from its own customers and audiences. It explains how this owned data is applied across targeting, measurement and personalisation efforts.