# DuckDuckBot

`DuckDuckBot` is [DuckDuckGo](https://duckduckgo.com/)'s web crawler that fetches and indexes pages to power [DuckDuckGo](https://duckduckgo.com/) search results. It runs continuously to keep the search index up to date. Pages indexed by `DuckDuckBot` appear directly in [DuckDuckGo](https://duckduckgo.com/) search results with clickable links back to your site, making it a direct source of referral traffic.

## Should I allow and optimize for DuckDuckBot to drive organic growth?

[DuckDuckGo](https://duckduckgo.com/) holds a smaller market share than [Google](https://developers.google.com/) or Bing, but its privacy-focused user base is loyal and growing. Allowing `DuckDuckBot` means your pages can appear in [DuckDuckGo](https://duckduckgo.com/) search results with direct citation links. [DuckDuckGo](https://duckduckgo.com/) users tend to click through to source sites since the platform doesn't personalize or filter results as aggressively as larger engines. Allow `DuckDuckBot` and treat it like any other search engine crawler to capture this traffic.

### Here's how to optimize for DuckDuckBot:

- Allow DuckDuckBot in your robots.txt to ensure your pages are indexed
- Add a Sitemap directive to your robots.txt so DuckDuckBot can discover all your pages
- Ensure important content is rendered server-side, since DuckDuckBot has limited JavaScript rendering
- Use descriptive title tags and meta descriptions for better search result snippets
- Include structured data (JSON-LD) to help DuckDuckGo understand your page content
- Keep page load times fast to avoid timeouts during crawling

## Data Usage & Training

[DuckDuckGo](https://duckduckgo.com/)'s official documentation states that `DuckDuckBot` crawls pages to improve search results, but does not clarify whether crawled content is also used to train AI models. If this distinction matters to you, contact [DuckDuckGo](https://duckduckgo.com/) directly for clarification.

## How DuckDuckBot Accesses Content

Here's how DuckDuckBot accesses your site and understands your content:

- Fetches HTML via standard HTTP requests
- Partial JavaScript rendering capability
- Identifies as `DuckDuckBot`/1.1; (+ [http://duckduckgo.com/duckduckbot.html](http://duckduckgo.com/duckduckbot.html))
- Crawls from IP addresses published at [https://duckduckgo.com/duckduckbot.json](https://duckduckgo.com/duckduckbot.json)
- Reads Sitemap directives from robots.txt

`DuckDuckBot` crawls continuously on a regular schedule as part of its search indexing process. There is no documented support for the Crawl-delay directive.

## How to Block or Control DuckDuckBot

To block `DuckDuckBot` via robots.txt:

```
User-agent: DuckDuckBot
Disallow: /
```

For IP-based blocking, [DuckDuckGo](https://duckduckgo.com/) publishes its crawler IP prefixes at [https://duckduckgo.com/duckduckbot.json.](https://duckduckgo.com/duckduckbot.json.) You can use this list in your firewall or server config. Verify requests by matching the source IP against this published list, since user-agent strings can be spoofed. The IP list can change over time, so check it periodically if you rely on IP-based rules.

## Common Issues & Troubleshooting

Watch out for these common problems when working with DuckDuckBot:

- `DuckDuckBot` has limited JavaScript rendering, so content behind client-side frameworks may not be indexed
- User-agent string matching alone is unreliable for verification; always cross-check against the published IP list
- Third-party bot lists and IP blocklists may contain stale `DuckDuckBot` IPs, causing false blocks or missed blocks
- Crawl-delay is not supported, so you cannot throttle `DuckDuckBot` through robots.txt alone
- CloudFlare or similar bot protection services may block `DuckDuckBot` if not explicitly allowlisted

## Quick Reference

**Platform**: [DuckDuckGo](/content/ai-agent-directory?platforms=DuckDuckGo/index.html)  
**Agent Category**: [Search Engine Crawler](/content/ai-agent-directory?tags=Search%20Engine%20Crawler/index.html)  
**Growth Value**: Medium  
**Official Documentation**: [duckduckgo.com/duckduckbot.html](https://duckduckgo.com/duckduckbot.html)  
**User Agent String**: `duckduckbot`  
**robots.txt Entry**: 
```
User-agent: duckduckbot
Disallow: /
```

## Frequently Asked Questions

### Does DuckDuckBot respect robots.txt?  
### How do I verify a request is really from DuckDuckBot?  
### Does DuckDuckBot use my content for AI training?  
### Can DuckDuckBot render JavaScript?  
### Will blocking DuckDuckBot remove my site from DuckDuckGo?  
### How often does DuckDuckBot crawl my site?