AhrefsSiteAudit

Ahrefs site auditing bot that checks technical SEO issues.

Overall rank

7 of 51 ▼ 6 m/m

Category rank

2 of 10 ▼ 1 m/m

What does AhrefsSiteAudit do?

AhrefsSiteAudit crawls and renders your website pages to identify technical and on-page SEO issues for the Ahrefs Site Audit tool. It checks for broken links, performance problems, indexability issues, and other SEO factors, then surfaces results in the Ahrefs dashboard for verified site owners to fix. It does not drive referral traffic or generate public citations back to your site.

Should I allow and optimize for AhrefsSiteAudit to drive organic growth?

AhrefsSiteAudit doesn't drive direct referral traffic to your site. However, Ahrefs data feeds into one of the most widely used SEO toolkits, and the crawled data also powers the Yep search engine. Allowing AhrefsSiteAudit ensures your site's technical SEO data is accurate in Ahrefs, which helps you (and competitors analyzing your site) see a complete picture. If you use Ahrefs yourself, blocking this bot undermines your own Site Audit reports. Even if you don't use Ahrefs, the data contributes to Ahrefs' broader index and Yep search results.

Here's how to optimize for AhrefsSiteAudit:

  • Allow AhrefsSiteAudit in your robots.txt if you use Ahrefs Site Audit
  • Verify your domain in Ahrefs to unlock full crawl configuration options
  • Add Ahrefs IP ranges to your CDN or firewall allowlist to prevent false blocks
  • Ensure your robots.txt loads quickly and returns a 200 status code
  • Use a sitemap directive in robots.txt to help the crawler discover all pages
  • Fix broken links and redirect chains flagged in Site Audit reports to improve crawl efficiency

Data Usage & Training

Crawled content populates Ahrefs' databases and powers Ahrefs products like Site Audit, backlink indexes, and the Yep search engine. Ahrefs' public documentation does not state whether crawled data is used for AI model training, so that use remains unclear.

How AhrefsSiteAudit Accesses Content

Here's how AhrefsSiteAudit accesses your site and understands your content:

  • Fetches HTML and renders JavaScript fully (headless browser rendering)
  • Identifies as Mozilla/5.0 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/site-audit)
  • Respects robots.txt Disallow, Allow, Crawl-delay, and Sitemap directives
  • Honors meta robots tags
  • Verified site owners can configure Site Audit to bypass robots.txt for their own sites
  • JavaScript rendering may request many assets in parallel, making Crawl-delay difficult to honor during rendering

On-demand or scheduled, depending on how the site owner configures their Site Audit project. Crawl rate is capped at up to 30 URLs per minute, and verified owners can adjust this in the Ahrefs tool settings.

How to Block or Control AhrefsSiteAudit

To block AhrefsSiteAudit via robots.txt:

User-agent: AhrefsSiteAudit
Disallow: /

You can also block by IP. Ahrefs publishes its IP ranges at https://help.ahrefs.com/en/articles/78658-what-is-the-list-of-your-ip-ranges. To verify a request is genuinely from Ahrefs, perform a reverse DNS lookup and confirm the hostname ends with ahrefs.com or ahrefs.net. The bot automatically backs off when it receives 4xx or 5xx responses. One caveat: verified site owners can configure Site Audit to ignore robots.txt for their own domains, so if you own the site in Ahrefs, your robots.txt rules won't necessarily apply to your own audits.

Common Issues & Troubleshooting

Watch out for these common problems when working with AhrefsSiteAudit:

  • Cloudflare, Incapsula, and ModSecurity often block the Ahrefs user-agent or IPs by default
  • A slow or failing robots.txt response can prevent crawling entirely
  • WordPress security plugins frequently block Ahrefs bots without warning
  • Crawl-delay directives may not be honored during JavaScript rendering, since rendering triggers parallel asset requests
  • Firewall rules blocking the user-agent string cause incomplete Site Audit reports

Quick Reference

Platform

Ahrefs

Agent Category

SEO

Growth Value

Medium

Official Documentation

ahrefs.com/robot

User Agent String

ahrefssiteaudit

robots.txt Entry

User-agent: ahrefssiteaudit Disallow: /

Frequently Asked Questions

Does AhrefsSiteAudit respect robots.txt?

Is AhrefsSiteAudit the same as AhrefsBot?

How do I verify that a request is really from AhrefsSiteAudit?

Why is AhrefsSiteAudit crawling my site if I don't use Ahrefs?

Does AhrefsSiteAudit render JavaScript?

Can I limit how fast AhrefsSiteAudit crawls my site?

Similar Agents & Bots

AhrefsBot

Ahrefs crawler used for backlink and SEO analysis.

Barkrowler

Qwant's SEO crawler for site analysis and indexing.

ClarityBot

Microsoft Clarity bot for SEO and verification checks.

DataForSeoBot

DataForSEO bot collecting ranking and SERP data.