# DataForSeoBot

[SEO](/content/ai-agent-directory?tags=SEO/index.html)

DataForSEO bot collecting ranking and SERP data.

Overall rank

#30 of 51 ▲9 m/m

Category rank

#7 of 10 ▲2 m/m

## What does DataForSeoBot do?

`DataForSeoBot` crawls websites to discover and re-check backlinks and collect page-level metrics. Its data feeds [DataForSEO](https://dataforseo.com/)'s backlink database and OnPage API products, which SEO professionals use for link analysis and on-page audits. It does not drive referral traffic back to your site.

## Should I allow and optimize for DataForSeoBot to drive organic growth?

`DataForSeoBot` does not power any user-facing search or AI assistant product that would surface your content to end users. It feeds a B2B SEO data platform. Allowing it won't generate referral traffic or AI citations. The main reason to allow it is if you or your SEO tools rely on [DataForSEO](https://dataforseo.com/)'s backlink or OnPage data and want accurate results for your own domains.

Here's how to optimize for DataForSeoBot:

- Allow DataForSeoBot only if you or your SEO tools depend on DataForSEO data for your domains
- Set a Crawl-delay value in robots.txt to limit server load from frequent requests
- Use reverse DNS verification (crawling-gateway-*.dataforseo.com) to confirm legitimate requests before blocking
- Check the published IP subnet list at https://dataforseo.com/dataforseo-bot periodically if you use IP-based rules
- Use path-specific Disallow rules to protect sensitive sections while allowing crawling of public pages

## Data Usage & Training

Crawled content populates [DataForSEO](https://dataforseo.com/)'s backlink database and supplies their OnPage API products with page-level metrics. [DataForSEO](https://dataforseo.com/)'s documentation does not state that crawled content is used to train AI models, so any such use is not publicly documented.

## How DataForSeoBot Accesses Content

Here's how DataForSeoBot accesses your site and understands your content:

- Fetches HTML via standard HTTP requests
- Partial JavaScript rendering capability
- Respects robots.txt Disallow and Crawl-delay directives
- Identifies as: Mozilla/5.0 (compatible; DataForSeoBot/1.0; +https://dataforseo.com/dataforseo-bot)
- Can be verified via reverse DNS (crawling-gateway-*.dataforseo.com) and published IP subnets

Continuous crawling with a vendor-default visit interval of approximately 5 seconds between requests. Site owners can adjust this using the Crawl-delay directive in robots.txt.

## How to Block or Control DataForSeoBot

To block `DataForSeoBot` via robots.txt:

User-agent: `DataForSeoBot`  
Disallow: /

To rate-limit instead of fully blocking:

User-agent: `DataForSeoBot`  
Crawl-delay: 10

For IP-based blocking, [DataForSEO](https://dataforseo.com/) publishes IPv4 and IPv6 subnets at [https://dataforseo.com/dataforseo-bot](https://dataforseo.com/dataforseo-bot). You can also filter by user-agent string at the server level, but combine this with reverse DNS verification (crawling-gateway-*.dataforseo.com) to avoid blocking spoofed requests while letting legitimate ones through. If you use [DataForSEO](https://dataforseo.com/)'s OnPage API directly, you can control scanning behavior through the API's custom_robots_txt parameters.

## Common Issues & Troubleshooting

Watch out for these common problems when working with DataForSeoBot:

- Default crawl interval of 5 seconds can cause noticeable server load on smaller sites
- Simple user-agent filtering can be bypassed by spoofed headers; combine with reverse DNS and IP verification
- IP-based blocks may become outdated if [DataForSEO](https://dataforseo.com/) changes their subnets; check their bot page periodically
- Overly broad Disallow rules (e.g., Disallow: /) may accidentally block other crawlers if applied to the wrong user-agent
- Sitemap handling is not explicitly documented, so don't rely on sitemap directives to guide this bot

## Quick Reference

Platform

[DataForSEO](/content/ai-agent-directory?platforms=DataForSEO/index.html)

Agent Category

[SEO](/content/ai-agent-directory?tags=SEO/index.html)

Growth Value

Low

Official Documentation

[dataforseo.com/dataforseo-bot](https://dataforseo.com/dataforseo-bot)

User Agent String

`dataforseobot`

robots.txt Entry

`User-agent: dataforseobot
Disallow: /`