Directory of AI Agents, Bots & Crawlers

A continuously updated traffic-ranked leaderboard and guide for AI agents, bots & crawlers across the web.

What you can explore

• AI Search Index bots used to discover and rank web content.
• AI User Initiated agents that fetch content during live user sessions.
• AI Training crawlers that collect public data for model development.
• Traditional SEO and search-engine crawlers that still shape visibility.

Googlebot

Primary Google crawler for search indexing.
googlebot
#1▲2
Link

Bingbot

Microsoft Bing web crawler for search indexing.
bingbot
#2▲2
Link

ChatGPT-User

OpenAI browsing agent fetching pages at user request.
chatgpt-user
#3▲2
Link

AhrefsBot

Ahrefs crawler used for backlink and SEO analysis.
ahrefsbot
#4▲2
Link

GoogleOther

Internal Google crawler used for non-public systems.
googleother
#5▲3
Link

GPTBot

OpenAI crawler used to gather training data for GPT.
gptbot
#6▲9
Link

AhrefsSiteAudit

Ahrefs site auditing bot that checks technical SEO issues.
ahrefssiteaudit
#7▼6
Link

Meta-ExternalAgent

Meta agent used for AI training data collection.
meta-externalagent
#8▼6
Link

ClaudeBot

Anthropic crawler for collecting training data.
claudebot
#9▲2
Link

Bytespider

ByteDance data collection bot used for AI training.
bytespider
#10▼3
Link

OAI-SearchBot

OpenAI crawler indexing content for answer engines.
oai-searchbot
#11▲2
Link

Applebot

Apple web crawler for indexing and AI training.
applebot
#12▲2
Link

Amazonbot

Amazon crawling agent used for AI training and discovery.
amazonbot
#13▼3
Link

DotBot

Moz crawler for link analysis and SEO metrics.
dotbot
#14▼2
Link

Yandexbot

Yandex crawler for Russian search indexing.
yandexbot
#15▲2
Link

BaiduSpider

Baidu's primary crawler for web search indexing.
baiduspider
#16▼7
Link

Barkrowler

Qwant's SEO crawler for site analysis and indexing.
barkrowler
#17▼1
Link

Meta-WebIndexer

Meta crawler that indexes public web pages.
meta-webindexer
#18
Link

Claude-User

User-initiated fetches triggered by Claude sessions.
claude-user
#21▲11
Link

Googlebot Image

Google image crawler for indexing visual content.
googlebot-image
#25▲1
Link

DuckAssistBot

DuckDuckGo assistant fetching content for answers.
duckassistbot
#26▲1
Link

Google Read Aloud

Accessibility agent that reads page content aloud.
google-read-aloud
#28
Link

Googlebot Video

Google video crawler for indexing video pages.
googlebot-video
#43▲4
Link

Naverbot

Naver search crawler used in Korean search index.
yeti
Link

ScreamingFrogBot

Desktop SEO spider used for audits and site maps.
screaming frog seo spider
Link

SemrushSiteAudit

SEMrush crawler for site audits and technical checks.
siteauditbot
Link

Yahoo! Slurp

Yahoo legacy crawler for web indexing.
slurp
Link