If you run an ecommerce store and got a wish from a Monkey’s Paw, I know exactly what you'd wish for: more shoppers.
But, as it is with the fable and, increasingly, the internet, your wishes can come true exactly as worded but never quite as intended.
So here you go... your wish is granted. Your sales are up and your traffic is up. Your analytics are looking fantastic. Except...
There’s just one small problem. Bots now make up the majority of your traffic and they are here to do worse than just price scraping. These mixed-use agents can find blogs, research, announcements that are hidden to the world and reveal it to everyone – without credit, without permission, possibly to your competitors.
That’s the kind of traffic Cloudflare decided to do something about.
On September 15, Cloudflare changed its default settings for pages that show ads. Training and Agent bots are now blocked by default, while Search bots can still get through. And this could be something big because Cloudlfare controls visitor access to +36% of all the ecommerce stores on the internet, so changing a setting that nobody reads can mean a chunk of the web also changes with it.
So, for you, would it mean a potential customer getting turned away at the door by a setting you never touched, or is it a much-needed layer of protection for your data?
Let's discuss why Cloudflare did this plus what does it actually cost a store your size?
A long time ago on a server far, far away, the web ran on a handshake. Search engines crawled your pages, and in exchange they sent people back to you. It just worked because both sides needed the other.
And then about 30 years later, AI answers broke the second half of that deal. The crawling kept happening but the sending back mostly stopped. The situation has worsened to the point that these “mixed-use” crawlers can crawl to the deepest crevices of your site and pull content that was never meant to see the light of a screen – as this redditor recounts:
"I've seen AI crawlers dig up internal articles on sites not meant for public and serving them as sources for chatbot. Since AI boom i've noticed many publications i had bookmarked for years go off site to avoid unwanted traffic or stolen."
And if you're assuming a firmly worded robots.txt settles it, Cloudflare ran that experiment. It created test domains that disallowed all automated access and added firewall rules blocking Perplexity's declared bots. It then asked Perplexity about the protected content and got detailed answers back anyway. The declared crawler had been swapped for a generic Chrome user agent, rotating through IP addresses and networks, running 3 to 6 million requests a day across tens of thousands of domains. Cloudflare delisted Perplexity as a verified bot over it. Perplexity called the whole thing a sales pitch and denied the bots were even theirs.
The models themselves wander too. OpenAI published six incidents from its own testing this September, including a model that searched public GitHub repositories for exposed API keys and used one to authenticate, then invented the data it was after when that still didn't work.
No wonder Cloudflare's CEO Matthew Prince has been making this point loudly all year, and here’s the reason: non-human traffic crossed the 50% mark roughly a year ago, and it’s only getting worse since then.
So Cloudflare sorted bots into categories and gave site owners a switch for the ones that matter most. Yes, you can change the settings yourself. But we all know how this works: a new setting appears, we click “Accept,” and six months later we’re staring at our analytics like they’ve personally betrayed us.
So, consider this your headache-prevention service. Better to spend five minutes checking these settings now than 6 months down the line when you are trying to figure why your traffic or sales data took a dip suddenly:
Cloudflare’s logic for that is pretty reasonable.
When you put an ad on a page, you presumably want a human eyeball to land there and see it. A bot reading the page doesn't generate an impression and doesn't click the ad. It doesn't buy anything as well. It’s basically a colleague who shows up, eats the snacks, uses the Wi-Fi, and contributes nothing to the meeting.
But when you apply that logic to an ecommerce store rather than a blog page, things become interesting.
Here a shopper can come asking ChatGPT: “Is this jacket actually waterproof" or "what happens if the zipper of this bag dies 4 months later?”
The Agent bot is the one that goes to your website, reads the product page, digs through your warranty policy and comes back with the answer. Blocking it is the digital version of locking your front door during a sale because you're tired of people browsing without buying.
So to answer the question is this change good for you or bad: it's kinda both.
It's real protection for your blog, your guides and your original research, which is the stuff genuinely being taken. And it's a self-inflicted wound on your product and policy pages, which is the stuff you desperately want to read because the right information can help you win the sale now and prevent a return, dispute, or chargeback later.
In November 2025, the ecommerce giant sued Perplexity over the same thing. The AI giant’s Comet browser agent was masquerading as an ordinary Chrome browser to browse and shop on Amazon without permission.
But for Amazon, no one's getting special treatment. Amazon's robots.txt names and locks out Claude, ChatGPT, Copilot, Perplexity, Manus, Grok and Google's AI crawlers, a blank Disallow against each one. You can try asking any of them to check a price on Amazon today and they'll tell you they can't get in.
Meanwhile, Alexa's "Buy for Me" feature shops other retailers and completes the purchase using your saved address and card. Amazon bolted every door it owns and sent its own agent out to walk through everybody else's.
Nice trick Amazon!
Now let's talk about how you can follow Amazon's footsteps.
Start by checking what you're already blocking, because you might probably be blocking the wrong half.
Someone parsed the robots.txt files of 109 well-known sites and found that 13% of ecommerce sites were blocking OpenAI’s training crawler. While just 6% were blocking the crawler that feeds ChatGPT Search and gives AI overviews.
So, you might think you've blocked training AI agent, but you've mistakenly blocked the agent responsible for showing you on SERPs and chats.
So go through this checklist:
There is one catch though: doing all this perfectly won't necessarily make your analytics look better.
A shopper might get a solid answer about your warranty in ChatGPT, come back three days later and buy from you, only for your analytics to call it direct traffic or, even worse, nothing at all. Welcome to the zero-click world and the slow death of attribution.
For now, remember one thing: “AI sent me no traffic” and “AI sent me no customers” are very different statements.
Your dashboard can measure the first. The second is getting increasingly difficult to see, and perhaps that’s intentional.
The first wish was more sellers, and the monkey's paw delivered it in bots.
Word the second one more carefully. You don't want every bot in the world crawling freely, and you don't want the drawbridge up either. You want the machines that arrive with a customer attached to get in, read the truth about your products, and make a purchase.
But your job doesn't end there. Your customer has purchased the product, but what happens if it breaks?
That's where SureBright helps you. We offer protection plans integrated right into your website so that your customer can enjoy lasting peace of mind while you can maximize your revenue.
Interested in seeing how that works? Schedule a demo with us today.
.jpg)
M Khizar is a writer enjoys making complicated things feel simple. He writes about warranties, ecommerce, and the small details people usually overlook, until they matter. His work focuses on clarity and helping readers make smarter decisions without overthinking it. Outside of work, he enjoys reading, writing personal blogs, and binge eating with friends.
.jpg)
