Two different bots, two different jobs
Googlebot indexes public pages for Search. AdsBot reviews ad destinations so a paid click is not a 404, a geo block, or a login wall. Blocking AdsBot in robots.txt is a documented cause of ‘destination not accessible’ and ‘page not found’ in the landing-page test.
Google’s destination-not-accessible article is blunt: AdsBot often crawls from the United States. If your WAF treats that as an attack, the ad never gets approved in the locations you think you targeted.
Allow AdsBot, still deny /operator
robots.txt is not authentication. You can Allow GoogleAdsBot on public marketing paths and still Disallow /operator/, /api/, and app prefixes. Named user-agent groups replace the wildcard group — if you add a specific AdsBot group, restate the private disallows in that group.
Do not noindex a paid landing page you want AdsBot to review as a live destination. noindex is a Search hint. A 403 is an ads failure.
The checklist AdsBot actually fails
Most of these are infrastructure, not copy.
- robots.txt does not Disallow the landing path to GoogleAdsBot
- The URL returns 200, not 401/403/404, without a session cookie
- No ‘this site is not available in your region’ interstitial for US crawlers if you sell in those regions
- CDN bot fights do not fingerprint AdsBot as a scraper
- HTTP and HTTPS both resolve; mixed redirect chains are visible in the landing-page test
Staging, previews, and ‘works on my laptop’
A preview host that requires a password will fail AdsBot. Point the final URL at the public origin. If you use edge middleware to A/B the hero, make sure the crawler still receives real HTML, not an empty shell.
What to send if the destination keeps failing
Share the final URL, robots.txt, and whether a WAF sits in front. Related work at Kiwi: technical SEO and performance foundation. This is crawl hygiene, not a traffic promise.
Frequently asked questions
Is GoogleAdsBot the same as Googlebot?
No. Different user-agent, different job. Allowing one does not automatically allow the other. Read Google’s crawler overview and set robots.txt on purpose.
Should AdsBot be able to fetch /start?
If that is the final URL, yes — the public HTML. The Typeform embed can still wait on form consent. The page itself must load.
Can we geo-block the US and still run ads in Europe?
Risky. Google documents that AdsBot often crawls from the United States. If that request 403s, approval fails even when the human in Paris can load the page.
Does allowing AdsBot leak our operator panel?
Not if operator routes stay disallowed and authenticated. robots.txt is not a lock; authorization is. Use both.
Sources
- Google Ads policies — Destination not accessible
- Google Search Central — Google crawlers (user-agents)
- Google Search Central — robots.txt introduction
Figures cited above are drawn from the linked publications and are the responsibility of their sources; we date and scope them rather than presenting them as universal guarantees.