Website owners increasingly have several decisions to make about automated visitors. A search engine indexing an article, a company collecting training data and an assistant fetching a page for a user may need different treatment.
Cloudflare's 15 September 2026 announcement introduces separate controls for search, AI training and AI agents. It also replaces Managed Robots.txt with Bot Preference Sync for applying preferences across supported crawlers.
Check the settings on each site
The announcement describes recommended settings for new websites based on their business model. Advertising-supported sites receive recommendations allowing search while disallowing training and blocking agents on pages with ads. Other sites receive recommendations allowing all three.
Those recommendations are not a reliable description of every existing domain. Open the dashboard for each site and check what is actually configured.
An agency may manage a publisher, a shop and a service business under the same account. Each has a different reason for wanting automated traffic, so one account-wide assumption can cause trouble.
Write down the business decision
Start with the site owner's aims. Do they want articles to appear in search? Do they permit their content to be used for training? Should an assistant be able to retrieve public product details for a shopper?
Record those choices in plain language before changing controls. That gives the team a way to evaluate a new crawler or revisit the decision later.
Consider which pages deserve separate handling. Public documentation and a paid resource library may have different access requirements. Customer accounts should already have proper authentication regardless of crawler preferences.
Understand what the controls promise
Cloudflare's Accountable designation covers operators meeting, or committing to meet, its stated criteria for independent choices and visibility. The company says mixed-use crawlers lacking appropriate controls can be blocked when site owners block training.
Check the current documentation and supported operator list when configuring a domain. A label, crawler name or announced commitment can change, and your site's settings need to reflect the behaviour you intend.
A crawler preference also does not provide a guarantee about future traffic or citations. Evaluate the effect using your own site's evidence.
Measure useful visits and unwanted load
Before changing settings, note the level of crawler traffic and the enquiries, sales or reader visits that matter to the business. Keep a record of the change and its date.
Afterwards, review crawl errors, server load and traffic from the channels you care about. If useful access disappears, investigate the relevant rule and crawler before relaxing every protection.
For a small publisher, the decision may be to allow discovery while restricting particular reuse. For another business, agent access may help customers find accurate product information. The separate controls make those choices easier to express.
Review your domains individually, agree the purpose of each setting and keep enough evidence to tell whether the choice is working.
Reporting checked on 13 October 2026.
