# ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF # RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT # AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET. # # search: building a search index and returning links and short excerpts. # ai-train: training or fine-tuning AI models. # use: how AI systems may consume the content. # /?q= (and //?q=) is the homepage with the search box prefilled: a duplicate of /, # canonical /. Profile pages used to link ten of them each (2026-11-18 crawl wave). User-agent: * Content-Signal: search=yes,ai-train=no,use=reference Disallow: /cms/ Disallow: /admin/ Disallow: /search/ Disallow: /profile-lookup/ Disallow: /hashtag-search/ Disallow: /img/ Disallow: /api/ Disallow: /?q= Disallow: /es/?q= Disallow: /pt/?q= Disallow: /id/?q= Disallow: /tr/?q= Disallow: /de/?q= Disallow: /fr/?q= Allow: / # --------------------------------------------------------------------------- # Crawl-delay throttling. # # Measured on twtdata.com 2026-08-11 and ranked by ORIGIN SECONDS, not request # count - request count mis-ranks badly: GPTBot was 66k requests at 0.152s each # while ClaudeBot was 21k requests at 0.905s each. # # AhrefsBot 5.6 h/day 73,752 req ClaudeBot 5.3 h/day 21,252 req # GPTBot 2.8 h/day 66,223 req Applebot 2.2 h/day 21,947 req # Amazonbot 1.2 h/day 13,113 req SemrushBot 0.4 h/day 8,024 req # # A cache does not absorb these. They are single-pass: AhrefsBot requested # 73,081 URLs of which 73,076 were distinct - a 0.08% hit rate. A cache is a bet # that a URL gets asked for twice, and that bet loses against a crawler that asks # exactly once. Googlebot is the counter-example at 1.11 req/URL and a 9.6% hit # rate, which is why it is NOT throttled. # # Throttled rather than blocked on purpose. We use Ahrefs ourselves, and blocking # it removes our own domain's data along with everyone else's. At 10s AhrefsBot # caps near 8,640 req/day against ~73,000 today - an ~88% cut - while keeping # full coverage over time. # # DELIBERATELY NOT THROTTLED: # Googlebot - ignores Crawl-delay entirely (Google's own spec). Use Search # Console crawl-rate settings if it ever needs limiting. # bingbot - honours Crawl-delay, but it is a real search engine sending # real traffic and costs only 0.5 h/day. Not worth it. # Claude-SearchBot, OAI-SearchBot, ChatGPT-User, PerplexityBot # - retrieval crawlers that can send referral traffic. Only the # training crawlers are throttled. # # IMPORTANT: every agent group below repeats the Disallow rules from # "User-agent: *" on purpose. A crawler obeys only the single most specific group # that matches it, so an agent-specific group without them would silently UNBLOCK # the admin and search URLs for that agent. # --------------------------------------------------------------------------- # Backlink crawlers. They follow links regardless of rel="nofollow" - # that is the product they sell - so nofollow cannot reach them, only Crawl-delay can. User-agent: AhrefsBot Crawl-delay: 10 Disallow: /cms/ Disallow: /admin/ Disallow: /search/ Disallow: /profile-lookup/ Disallow: /hashtag-search/ Disallow: /img/ Disallow: /api/ Disallow: /?q= Disallow: /es/?q= Disallow: /pt/?q= Disallow: /id/?q= Disallow: /tr/?q= Disallow: /de/?q= Disallow: /fr/?q= Allow: / User-agent: MJ12bot Crawl-delay: 10 Disallow: /cms/ Disallow: /admin/ Disallow: /search/ Disallow: /profile-lookup/ Disallow: /hashtag-search/ Disallow: /img/ Disallow: /api/ Disallow: /?q= Disallow: /es/?q= Disallow: /pt/?q= Disallow: /id/?q= Disallow: /tr/?q= Disallow: /de/?q= Disallow: /fr/?q= Allow: / # SEO platform crawlers. Semrush caps any value above 10s at 10s. User-agent: SemrushBot Crawl-delay: 10 Disallow: /cms/ Disallow: /admin/ Disallow: /search/ Disallow: /profile-lookup/ Disallow: /hashtag-search/ Disallow: /img/ Disallow: /api/ Disallow: /?q= Disallow: /es/?q= Disallow: /pt/?q= Disallow: /id/?q= Disallow: /tr/?q= Disallow: /de/?q= Disallow: /fr/?q= Allow: / User-agent: DotBot Crawl-delay: 10 Disallow: /cms/ Disallow: /admin/ Disallow: /search/ Disallow: /profile-lookup/ Disallow: /hashtag-search/ Disallow: /img/ Disallow: /api/ Disallow: /?q= Disallow: /es/?q= Disallow: /pt/?q= Disallow: /id/?q= Disallow: /tr/?q= Disallow: /de/?q= Disallow: /fr/?q= Allow: / # AI training crawlers. Their retrieval counterparts are left unthrottled # below on purpose - those can send referral traffic. User-agent: ClaudeBot Crawl-delay: 10 Disallow: /cms/ Disallow: /admin/ Disallow: /search/ Disallow: /profile-lookup/ Disallow: /hashtag-search/ Disallow: /img/ Disallow: /api/ Disallow: /?q= Disallow: /es/?q= Disallow: /pt/?q= Disallow: /id/?q= Disallow: /tr/?q= Disallow: /de/?q= Disallow: /fr/?q= Allow: / User-agent: GPTBot Crawl-delay: 10 Disallow: /cms/ Disallow: /admin/ Disallow: /search/ Disallow: /profile-lookup/ Disallow: /hashtag-search/ Disallow: /img/ Disallow: /api/ Disallow: /?q= Disallow: /es/?q= Disallow: /pt/?q= Disallow: /id/?q= Disallow: /tr/?q= Disallow: /de/?q= Disallow: /fr/?q= Allow: / User-agent: CCBot Crawl-delay: 10 Disallow: /cms/ Disallow: /admin/ Disallow: /search/ Disallow: /profile-lookup/ Disallow: /hashtag-search/ Disallow: /img/ Disallow: /api/ Disallow: /?q= Disallow: /es/?q= Disallow: /pt/?q= Disallow: /id/?q= Disallow: /tr/?q= Disallow: /de/?q= Disallow: /fr/?q= Allow: / # Amazon moved Amazonbot to standard robots.txt directives on 2026-06-15. User-agent: Amazonbot Crawl-delay: 10 Disallow: /cms/ Disallow: /admin/ Disallow: /search/ Disallow: /profile-lookup/ Disallow: /hashtag-search/ Disallow: /img/ Disallow: /api/ Disallow: /?q= Disallow: /es/?q= Disallow: /pt/?q= Disallow: /id/?q= Disallow: /tr/?q= Disallow: /de/?q= Disallow: /fr/?q= Allow: / # Applebot's Crawl-delay support is not documented either way. Harmless if # ignored. Applebot-Extended (the training opt-out) is a different agent. User-agent: Applebot Crawl-delay: 10 Disallow: /cms/ Disallow: /admin/ Disallow: /search/ Disallow: /profile-lookup/ Disallow: /hashtag-search/ Disallow: /img/ Disallow: /api/ Disallow: /?q= Disallow: /es/?q= Disallow: /pt/?q= Disallow: /id/?q= Disallow: /tr/?q= Disallow: /de/?q= Disallow: /fr/?q= Allow: / Sitemap: https://igdatahub.com/sitemap.xml