# Directory of AI Agents, Bots & Crawlers

A continuously updated traffic-ranked leaderboard and guide for AI agents, bots & crawlers across the web.

## What you can explore

- AI Search Index bots used to discover and rank web content.
- AI User Initiated agents that fetch content during live user sessions.
- AI Training crawlers that collect public data for model development.
- Traditional SEO and search-engine crawlers that still shape visibility.

### Googlebot  
Primary Google crawler for search indexing.  
`googlebot`  
#1▲2  
[Link](/content/ai-agent-directory/googlebot/index.html)

### Bingbot  
Microsoft Bing web crawler for search indexing.  
`bingbot`  
#2▲2  
[Link](/content/ai-agent-directory/bingbot/index.html)

### ChatGPT-User  
OpenAI browsing agent fetching pages at user request.  
`chatgpt-user`  
#3▲2  
[Link](/content/ai-agent-directory/chatgpt-user/index.html)

### AhrefsBot  
Ahrefs crawler used for backlink and SEO analysis.  
`ahrefsbot`  
#4▲2  
[Link](/content/ai-agent-directory/ahrefsbot/index.html)

### GoogleOther  
Internal Google crawler used for non-public systems.  
`googleother`  
#5▲3  
[Link](/content/ai-agent-directory/googleother/index.html)

### GPTBot  
OpenAI crawler used to gather training data for GPT.  
`gptbot`  
#6▲9  
[Link](/content/ai-agent-directory/gptbot/index.html)

### AhrefsSiteAudit  
Ahrefs site auditing bot that checks technical SEO issues.  
`ahrefssiteaudit`  
#7▼6  
[Link](/content/ai-agent-directory/ahrefssiteaudit/index.html)

### Meta-ExternalAgent  
Meta agent used for AI training data collection.  
`meta-externalagent`  
#8▼6  
[Link](/content/ai-agent-directory/meta-externalagent/index.html)

### ClaudeBot  
Anthropic crawler for collecting training data.  
`claudebot`  
#9▲2  
[Link](/content/ai-agent-directory/claudebot/index.html)

### Bytespider  
ByteDance data collection bot used for AI training.  
`bytespider`  
#10▼3  
[Link](/content/ai-agent-directory/bytespider/index.html)

### OAI-SearchBot  
OpenAI crawler indexing content for answer engines.  
`oai-searchbot`  
#11▲2  
[Link](/content/ai-agent-directory/oai-searchbot/index.html)

### Applebot  
Apple web crawler for indexing and AI training.  
`applebot`  
#12▲2  
[Link](/content/ai-agent-directory/applebot/index.html)

### Amazonbot  
Amazon crawling agent used for AI training and discovery.  
`amazonbot`  
#13▼3  
[Link](/content/ai-agent-directory/amazonbot/index.html)

### DotBot  
Moz crawler for link analysis and SEO metrics.  
`dotbot`  
#14▼2  
[Link](/content/ai-agent-directory/dotbot/index.html)

### Yandexbot  
Yandex crawler for Russian search indexing.  
`yandexbot`  
#15▲2  
[Link](/content/ai-agent-directory/yandexbot/index.html)

### BaiduSpider  
Baidu's primary crawler for web search indexing.  
`baiduspider`  
#16▼7  
[Link](/content/ai-agent-directory/baiduspider/index.html)

### Barkrowler  
Qwant's SEO crawler for site analysis and indexing.  
`barkrowler`  
#17▼1  
[Link](/content/ai-agent-directory/barkrowler/index.html)

### Meta-WebIndexer  
Meta crawler that indexes public web pages.  
`meta-webindexer`  
#18  
[Link](/content/ai-agent-directory/meta-webindexer/index.html)

### Claude-User  
User-initiated fetches triggered by Claude sessions.  
`claude-user`  
#21▲11  
[Link](/content/ai-agent-directory/claude-user/index.html)

### Googlebot Image  
Google image crawler for indexing visual content.  
`googlebot-image`  
#25▲1  
[Link](/content/ai-agent-directory/googlebot-image/index.html)

### DuckAssistBot  
DuckDuckGo assistant fetching content for answers.  
`duckassistbot`  
#26▲1  
[Link](/content/ai-agent-directory/duckassistbot/index.html)

### Google Read Aloud  
Accessibility agent that reads page content aloud.  
`google-read-aloud`  
#28  
[Link](/content/ai-agent-directory/google-read-aloud/index.html)

### Googlebot Video  
Google video crawler for indexing video pages.  
`googlebot-video`  
#43▲4  
[Link](/content/ai-agent-directory/googlebot-video/index.html)

### Naverbot  
Naver search crawler used in Korean search index.  
`yeti`  
[Link](/content/ai-agent-directory/naverbot/index.html)

### ScreamingFrogBot  
Desktop SEO spider used for audits and site maps.  
`screaming frog seo spider`  
[Link](/content/ai-agent-directory/screamingfrogbot/index.html)

### SemrushSiteAudit  
SEMrush crawler for site audits and technical checks.  
`siteauditbot`  
[Link](/content/ai-agent-directory/siteauditbot/index.html)

### Yahoo! Slurp  
Yahoo legacy crawler for web indexing.  
`slurp`  
[Link](/content/ai-agent-directory/yahoo-slurp/index.html)