DuckDuckBot
DuckDuckBot is DuckDuckGo's web crawler that fetches and indexes pages to power DuckDuckGo search results. It runs continuously to keep the search index up to date. Pages indexed by DuckDuckBot appear directly in DuckDuckGo search results with clickable links back to your site, making it a direct source of referral traffic.
Should I allow and optimize for DuckDuckBot to drive organic growth?
DuckDuckGo holds a smaller market share than Google or Bing, but its privacy-focused user base is loyal and growing. Allowing DuckDuckBot means your pages can appear in DuckDuckGo search results with direct citation links. DuckDuckGo users tend to click through to source sites since the platform doesn't personalize or filter results as aggressively as larger engines. Allow DuckDuckBot and treat it like any other search engine crawler to capture this traffic.
Here's how to optimize for DuckDuckBot:
- Allow DuckDuckBot in your robots.txt to ensure your pages are indexed
- Add a Sitemap directive to your robots.txt so DuckDuckBot can discover all your pages
- Ensure important content is rendered server-side, since DuckDuckBot has limited JavaScript rendering
- Use descriptive title tags and meta descriptions for better search result snippets
- Include structured data (JSON-LD) to help DuckDuckGo understand your page content
- Keep page load times fast to avoid timeouts during crawling
Data Usage & Training
DuckDuckGo's official documentation states that DuckDuckBot crawls pages to improve search results, but does not clarify whether crawled content is also used to train AI models. If this distinction matters to you, contact DuckDuckGo directly for clarification.
How DuckDuckBot Accesses Content
Here's how DuckDuckBot accesses your site and understands your content:
- Fetches HTML via standard HTTP requests
- Partial JavaScript rendering capability
- Identifies as
DuckDuckBot/1.1; (+ http://duckduckgo.com/duckduckbot.html) - Crawls from IP addresses published at https://duckduckgo.com/duckduckbot.json
- Reads Sitemap directives from robots.txt
DuckDuckBot crawls continuously on a regular schedule as part of its search indexing process. There is no documented support for the Crawl-delay directive.
How to Block or Control DuckDuckBot
To block DuckDuckBot via robots.txt:
User-agent: DuckDuckBot
Disallow: /
For IP-based blocking, DuckDuckGo publishes its crawler IP prefixes at https://duckduckgo.com/duckduckbot.json. You can use this list in your firewall or server config. Verify requests by matching the source IP against this published list, since user-agent strings can be spoofed. The IP list can change over time, so check it periodically if you rely on IP-based rules.
Common Issues & Troubleshooting
Watch out for these common problems when working with DuckDuckBot:
DuckDuckBothas limited JavaScript rendering, so content behind client-side frameworks may not be indexed- User-agent string matching alone is unreliable for verification; always cross-check against the published IP list
- Third-party bot lists and IP blocklists may contain stale
DuckDuckBotIPs, causing false blocks or missed blocks - Crawl-delay is not supported, so you cannot throttle
DuckDuckBotthrough robots.txt alone - CloudFlare or similar bot protection services may block
DuckDuckBotif not explicitly allowlisted
Quick Reference
Platform: DuckDuckGo
Agent Category: Search Engine Crawler
Growth Value: Medium
Official Documentation: duckduckgo.com/duckduckbot.html
User Agent String: duckduckbot
robots.txt Entry:
User-agent: duckduckbot
Disallow: /