GeolyBot

What it does

• It crawls the websites that Geoly customers add as their own brand site, to check whether AI crawlers can read their pages (for example: pages blocked by robots.txt, content that only appears after JavaScript runs, missing page titles or structured data).
• It reads pages listed in your sitemap and linked from your homepage. It does not submit forms, log in, add to cart or check out.
• It does not train AI models on your content.

How it identifies itself

• User agent: Mozilla/5.0 (compatible; GeolyBot/1.0; +https://geoly.ai/bot)
• Requests we send ourselves are signed with Web Bot Auth. Our public keys are published at https://www.geoly.ai/.well-known/http-message-signatures-directory.
• When a site rate-limits us, and to render a small sample of pages with JavaScript, some requests are sent through our crawling provider DataForSEO instead. These requests use the user agent Mozilla/5.0 (compatible; GeolyBot/1.0; +https://geoly.ai/bot; via DataForSEO), come from DataForSEO's IP addresses, and are not signed. They follow the same robots.txt rules for GeolyBot.

How often and how fast

• A full crawl when a site is added, then regularly re-checked every week. Pages are also re-checked when a site owner marks an issue as fixed.
• At most about 2 requests per second per site. If the site asks us to slow down (HTTP 429 or 430, or 503 with a Retry-After header), it pauses and continues at about 1 request per second.
• It follows Crawl-delay in robots.txt.

How to limit or block it

GeolyBot follows robots.txt. To block it entirely:

User-agent: GeolyBot
Disallow: /

To slow it down:

User-agent: GeolyBot
Crawl-delay: 10

If you block GeolyBot, Geoly can't check your site and will report it as "blocked" to the brand that added it.

Contact

Questions or problems: [email protected]

Any website owner can write to [email protected] to ask us to stop crawling their site and to delete the content we have stored from it.