If you're using Cloudflare WAF to secure your website, you might have noticed that turning on strict security settings can sometimes lead to unintended problems. Search engines may have trouble crawling your site, Google Search Console might show access issues, and tools like Cloudflare Observatory could fail their security tests.
As website owners, we aim for strong security, but we don’t want it to harm our SEO, indexing, or advertising income.
After testing various Cloudflare WAF setups, I discovered three custom rule expressions that offer a good balance between security and performance. These rules let trusted services like Googlebot and Cloudflare Observatory access your site while presenting a Managed Challenge to regular visitors.
In this guide, I'll go through each rule, explain why it's important, and show you how to set it up properly.
Why These Cloudflare WAF Rules Matter
Many website owners set up strict security settings without thinking about how search engines and verification tools interact with their sites.
Incorrectly configured Cloudflare WAF rules can lead to:
- Googlebot being challenged or blocked
- Delays in indexing through Google Search Console
- Cloudflare Observatory failing security assessments
- Lower crawl efficiency
- A poor experience for trusted automated services
The three rules listed below help prevent these problems while still keeping your website safe from harmful traffic.
Rule 1: Allow Cloudflare Observatory Synthetic Checks
Why This Rule Is Important:
Cloudflare Observatory uses automated synthetic checks to track the availability and performance of websites. If your Cloudflare Web Application Firewall (WAF) blocks or challenges these requests, Observatory might show failed health checks, even if your website is actually accessible.
To avoid this issue, set up a rule that bypasses Cloudflare's security features specifically for requests coming from Observatory.
Rule Expression:
http.user_agent contains "CloudflareObservatory"
and
ip.geoip.asnum eq 396982
Action:
Skip
What This Expression Does
- The rule checks for two specific conditions:
- The request includes the CloudflareObservatory user agent.
- The request comes from Cloudflare's Autonomous System, which has the ASN number 396982.
Cloudflare will bypass extra security checks only if both of these conditions are true. This method is safer than depending only on the User-Agent, as User-Agent strings can be faked.
Recommended Skip Components
- Custom Rules
- Rate Limiting Rules
- Managed Rules
- Super Bot Fight Mode
refer below attched snapshot for the understanding, and make the options as it is and place this rule in 1st position
Rule 2: Allow Verified Search Bots
Why This Rule Is Important
Search engines play a key role in bringing organic traffic to your website. If search engine bots like Googlebot or Bingbot encounter Managed Challenges, JavaScript Challenges, or CAPTCHA pages, they might not be able to properly crawl your site.
This can lead to issues with:
- Google Search Console
- Google AdSense
- Bing Webmaster Tools
- Apple Search in Safari
- Search rankings
- Page indexing
Cloudflare keeps an up-to-date list of verified bots. Rather than having to manually track IP addresses or User-Agent strings, you can simply allow these verified bots.
Rule Expression
cf.client.bot
Action
Skip
Why This Is Better Than Whitelisting User Agents
Cloudflare automatically checks that requests are actually coming from trusted bots such as:
- Googlebot
- AdsBot-Google
- Bingbot
- Applebot
- ChatGPT
- Microsoft Copilot
- Other verified search and AI crawlers
This helps greatly reduce the chance of fake bots getting past your security measures.
Recommended Skip Components
- Custom Rules
- Rate Limiting Rules
- Managed Rules
- Super Bot Fight Mode
Please leave the options as it is, as shown in below attached snapshot. And place this rule order in 2nd position
Rule 3: Use Managed Challenge for Every Website Visitor
Why This Rule Matters
Once you've allowed trusted bots and monitoring services, you can safely use a Managed Challenge to protect the rest of your website. Unlike regular CAPTCHAs, Cloudflare's Managed Challenge automatically decides if a visitor should be let through without any challenge or if they need to complete a quick verification.
This helps protect your site from:
- Bad bots
- Attempts to guess login details
- Automated attacks
- Spam
- Website scrapers
At the same time, it keeps things easy and smooth for real users.
Rule Expression
(http.host in {"yourdomain.com" "www.yourdomain.com"})
Replace the hostnames with your own domain.
Action
Managed Challenge
Why Rule Order Matters in Cloudflare WAF
Cloudflare evaluates Custom WAF Rules sequentially, starting from the top of your rule list and moving downward. As soon as a request matches a rule, Cloudflare immediately performs the configured action, making the order of your rules an essential part of your website's security strategy.
To ensure that trusted services such as Cloudflare Observatory, Googlebot, Bingbot, and other verified search engine crawlers are not interrupted, configure your rules in the following order.
| Order | Cloudflare WAF Rule | Action | Purpose |
|---|---|---|---|
| 1 | Allow Cloudflare Observatory – Synthetic Checks | Skip | Allows Cloudflare Observatory to perform synthetic health checks without being blocked or challenged. |
| 2 | Allow Verified Search Bots | Skip | Ensures Googlebot, Bingbot, Applebot, ChatGPT, Copilot, and other verified bots can crawl your website without interruption. |
| 3 | Managed Challenge – Verify All Visitors | Managed Challenge | Applies Cloudflare's intelligent security challenge to regular website visitors while allowing trusted services to bypass verification. |
Best Practices
To get the best results, keep these recommendations in mind:
- Always place Skip rules above Managed Challenge rules.
- Use Cloudflare's Verified Bots feature instead of manually allowing bot IPs or User-Agents.
- Test your configuration after every rule change.
- Please regularly review Cloudflare Security events to ensure legitimate traffic is passing as expected.
- Keep Cloudflare Managed Rules enabled for ongoing protection against new threats.
A secure website shouldn't come at the expense of search engine visibility or advertising performance.
By combining these three Cloudflare WAF rule expressions, you can protect your website from unwanted traffic while allowing trusted services like Googlebot, Google Search Console, Google AdSense, and Cloudflare Observatory to function without interruption.
Frequently Asked Questions
1. Will these rules affect Google AdSense?
No. These rules help ensure that Google AdSense crawlers can access your website without unnecessary security challenges.
2. Is cf.client.bot safe to use?
Yes. Cloudflare verifies the identity of supported bots before setting this field, making it a safer option than relying on User-Agent strings alone.
3. Should I use Managed Challenge instead of CAPTCHA?
For most websites, yes. Managed Challenge adapts to the visitor's behavior and typically provides a smoother experience for legitimate users while still blocking suspicious traffic.
4. Can I use these rules on Blogger or WordPress?
Yes. These Cloudflare WAF expressions work regardless of your CMS, as long as your domain is protected by Cloudflare.
