Cloudflare’s New AI Crawler Controls
Published · 5 min read
Reviewed by Andrew Liashok, Client Service Director, IN-STORE
A practical checklist for Cloudflare ai crawler controls: set Search, Training, and Agent options, then verify robots.txt before and after changes.
What To Set For Search, Training, And Agent, Then Verify Robots.txt Behavior Before And After
If your website brings in leads through content, local pages, or guides, AI crawler controls are now part of technical marketing operations. These settings can change how your content is accessed by bots while leaving the human experience unchanged, which makes them easy to miss until something breaks.
Cloudflare has published updated AI traffic controls, including an “Accountable” designation for major mixed use crawlers, plus clearer behavior for how “Block” and “Disallow AI Training” work together. This checklist is built to help small and mid sized site owners choose settings in Cloudflare’s Search, Training, and Agent categories, then confirm the resulting robots.txt behavior before and after changes.
What Cloudflare Updated, And Why It Matters Operationally
Cloudflare has published updated AI traffic controls that add an Accountable designation for major mixed use crawlers and clarifies how Block and Disallow AI Training work together. For service businesses and publishers, this matters because the same page can be valuable for search discovery, for lead education, and for training data collection. The control surface is becoming granular enough that you can express those intents separately, but only if you set it intentionally and verify what your site outputs.
Cloudflare also outlines recommended default settings for new domains beginning September 15, including a preset for ad monetized sites that allows Search while disallowing AI training and blocking certain agent behavior on pages with ads. Even if you are not launching a new domain, the existence of recommended defaults is a useful signal: crawler controls are shifting from a niche setting to a baseline config item.
SMB Checklist: Set Search, Training, And Agent Controls (Then Document Your Intent)
1) Decide What You Want Search To Do
Start by defining your desired outcome for “Search” in plain language, using your own content types. For most lead driven sites, the practical question is whether you want your pages accessible for discovery while controlling other uses of the same content. Cloudflare’s preset for ad monetized sites explicitly separates “allows Search” from “disallowing AI training”, which is a strong pattern to mirror when your content is both a marketing asset and a business asset.
2) Set Training Separately From Search
Cloudflare’s update clarifies how “Block” and “Disallow AI Training” work together. Treat “Training” as its own decision, separate from the visibility you want from search. This is the point where many teams make accidental trade offs, because they assume one toggle governs every bot behavior. Write down your intended stance on training before you change any control, then implement it, then verify it by checking the robots.txt behavior.
3) Review Agent Level Behavior, Especially For Mixed Use Crawlers
Cloudflare’s mention of an “Accountable” designation for major mixed use crawlers is your cue to think in terms of agent level behavior, not just a global allow or block. Mixed use crawlers can have more than one purpose, so your policy should be expressed at the agent level where possible. Operationally, this means you should keep a short internal note of which agents you expect to be treated as mixed use and what you want them to do on specific page types you care about (for example, lead gen pages versus education pages).
4) Use A Default, Then Add Exceptions Where Your Business Has Real Risk
Cloudflare’s recommended default settings for new domains beginning September 15 are useful as a baseline approach. The non obvious part is governance: if you have multiple people publishing content, updating landing pages, or managing ad placements, exceptions tend to proliferate and become hard to audit. Keep exceptions tied to a specific business reason and to a narrow set of page types, then track them in a simple change log.
Verification Workflow: Confirm Robots.txt Behavior Before And After
Run A Before Snapshot You Can Compare Against
Before you change any Cloudflare AI traffic options, capture your current robots.txt output as a baseline. Save it alongside a timestamped note of what you are about to change. This sounds basic, but it prevents the most common operational failure: losing the ability to prove which configuration produced which output when something downstream changes.
Make The Change, Then Re Check The Exact Output
After you update settings in Cloudflare, re check the robots.txt behavior and compare it to your baseline. Your goal is to confirm that the on page behavior matches your intended stance across Search, Training, and Agent controls.
Verify On The Page Types That Actually Drive Revenue
Do not validate only on the homepage. Check the specific page types that matter to a service business: service area pages, appointment or contact pages, quote request flows, and any content pages that reliably generate inquiries. The second order implication is simple: a change that leaves “what humans see” unchanged can still alter how bots access content that supports discovery and future leads. Verification needs to happen where the business value sits.
How To Make This Maintainable For A Lean Team
Treat Crawler Controls As Part Of Your Web Presence System
For SMBs without an in house marketing team, the practical risk is drift: settings get changed during a site refresh, a DNS update, a new domain launch, or a monetization experiment, and then nobody remembers the original intent. Put crawler control decisions into your regular website operations checklist, alongside content publishing, local pages, and lead capture forms. That keeps it in the same cadence as the work that produces demand.
Fold It Into Your Content And Local SEO Process
If content is a lead engine, crawler controls belong in the same workflow as editorial planning and local visibility. When IN-STORE sets up SMB growth systems, we treat technical settings, content production, and lead capture as connected operations so you can publish consistently without creating accidental access problems for bots that matter to discovery.
Cloudflare ai crawler controls are now a practical configuration topic, not a curiosity. Set your intent for Search, Training, and Agent behavior, then verify robots.txt behavior before and after every change so your site remains discoverable in the way you choose.
If you want a monthly marketing partner to manage your web presence, content operations, and technical checks as one system, explore IN-STORE’s Marketing services.
Looking for ongoing support? Explore IN-STORE's monthly marketing and website management at IN-STORE Marketing Services.
Source: Cloudflare Blog https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/
More articles from IN-STORE · This article as Markdown · RSS