# AhrefsSiteAudit Ahrefs site auditing bot that checks technical SEO issues. Overall rank ## 7 of 51 ▼ 6 m/m Category rank ## 2 of 10 ▼ 1 m/m ## What does AhrefsSiteAudit do? `AhrefsSiteAudit` crawls and renders your website pages to identify technical and on-page SEO issues for the [Ahrefs](https://ahrefs.com/) Site Audit tool. It checks for broken links, performance problems, indexability issues, and other SEO factors, then surfaces results in the [Ahrefs](https://ahrefs.com/) dashboard for verified site owners to fix. It does not drive referral traffic or generate public citations back to your site. ## Should I allow and optimize for AhrefsSiteAudit to drive organic growth? `AhrefsSiteAudit` doesn't drive direct referral traffic to your site. However, [Ahrefs](https://ahrefs.com/) data feeds into one of the most widely used SEO toolkits, and the crawled data also powers the Yep search engine. Allowing `AhrefsSiteAudit` ensures your site's technical SEO data is accurate in [Ahrefs](https://ahrefs.com/), which helps you (and competitors analyzing your site) see a complete picture. If you use [Ahrefs](https://ahrefs.com/) yourself, blocking this bot undermines your own Site Audit reports. Even if you don't use [Ahrefs](https://ahrefs.com/), the data contributes to [Ahrefs](https://ahrefs.com/)' broader index and Yep search results. ### Here's how to optimize for AhrefsSiteAudit: - Allow AhrefsSiteAudit in your robots.txt if you use Ahrefs Site Audit - Verify your domain in Ahrefs to unlock full crawl configuration options - Add Ahrefs IP ranges to your CDN or firewall allowlist to prevent false blocks - Ensure your robots.txt loads quickly and returns a 200 status code - Use a sitemap directive in robots.txt to help the crawler discover all pages - Fix broken links and redirect chains flagged in Site Audit reports to improve crawl efficiency ## Data Usage & Training Crawled content populates [Ahrefs](https://ahrefs.com/)' databases and powers [Ahrefs](https://ahrefs.com/) products like Site Audit, backlink indexes, and the Yep search engine. [Ahrefs](https://ahrefs.com/)' public documentation does not state whether crawled data is used for AI model training, so that use remains unclear. ## How AhrefsSiteAudit Accesses Content ### Here's how AhrefsSiteAudit accesses your site and understands your content: - Fetches HTML and renders JavaScript fully (headless browser rendering) - Identifies as Mozilla/5.0 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/site-audit) - Respects robots.txt Disallow, Allow, Crawl-delay, and Sitemap directives - Honors meta robots tags - Verified site owners can configure Site Audit to bypass robots.txt for their own sites - JavaScript rendering may request many assets in parallel, making Crawl-delay difficult to honor during rendering On-demand or scheduled, depending on how the site owner configures their Site Audit project. Crawl rate is capped at up to 30 URLs per minute, and verified owners can adjust this in the [Ahrefs](https://ahrefs.com/) tool settings. ## How to Block or Control AhrefsSiteAudit ### To block `AhrefsSiteAudit` via robots.txt: User-agent: `AhrefsSiteAudit` Disallow: / You can also block by IP. [Ahrefs](https://ahrefs.com/) publishes its IP ranges at [https://help.ahrefs.com/en/articles/78658-what-is-the-list-of-your-ip-ranges.](https://help.ahrefs.com/en/articles/78658-what-is-the-list-of-your-ip-ranges.) To verify a request is genuinely from [Ahrefs](https://ahrefs.com/), perform a reverse DNS lookup and confirm the hostname ends with ahrefs.com or ahrefs.net. The bot automatically backs off when it receives 4xx or 5xx responses. One caveat: verified site owners can configure Site Audit to ignore robots.txt for their own domains, so if you own the site in [Ahrefs](https://ahrefs.com/), your robots.txt rules won't necessarily apply to your own audits. ## Common Issues & Troubleshooting Watch out for these common problems when working with AhrefsSiteAudit: - Cloudflare, Incapsula, and ModSecurity often block the [Ahrefs](https://ahrefs.com/) user-agent or IPs by default - A slow or failing robots.txt response can prevent crawling entirely - WordPress security plugins frequently block [Ahrefs](https://ahrefs.com/) bots without warning - Crawl-delay directives may not be honored during JavaScript rendering, since rendering triggers parallel asset requests - Firewall rules blocking the user-agent string cause incomplete Site Audit reports ## Quick Reference ### Platform [Ahrefs](/content/ai-agent-directory?platforms=Ahrefs/index.html) ### Agent Category [SEO](/content/ai-agent-directory?tags=SEO/index.html) ### Growth Value Medium ### Official Documentation [ahrefs.com/robot](https://ahrefs.com/robot) ### User Agent String `ahrefssiteaudit` ### robots.txt Entry `User-agent: ahrefssiteaudit Disallow: /` ## Frequently Asked Questions ### Does AhrefsSiteAudit respect robots.txt? ### Is AhrefsSiteAudit the same as AhrefsBot? ### How do I verify that a request is really from AhrefsSiteAudit? ### Why is AhrefsSiteAudit crawling my site if I don't use Ahrefs? ### Does AhrefsSiteAudit render JavaScript? ### Can I limit how fast AhrefsSiteAudit crawls my site? ## Similar Agents & Bots ### [AhrefsBot](/content/ai-agent-directory/ahrefsbot/index.html) Ahrefs crawler used for backlink and SEO analysis. ### [Barkrowler](/content/ai-agent-directory/barkrowler/index.html) Qwant's SEO crawler for site analysis and indexing. ### [ClarityBot](/content/ai-agent-directory/claritybot/index.html) Microsoft Clarity bot for SEO and verification checks. ### [DataForSeoBot](/content/ai-agent-directory/dataforseobot/index.html) DataForSEO bot collecting ranking and SERP data.