# Claude Code

Claude Code agent fetching pages during development or tooling tasks.

## What does Claude Code do?

`Claude Code` is the user-agent [Anthropic](https://claude.com/) uses when the `Claude Code` product fetches web content during development tasks. Unlike other AI bots that only surface content in chat responses, `Claude Code` can autonomously research dev tools, packages, datasets, and documentation on behalf of developers, then decide to integrate them into projects directly. Citations in Claude-powered results can drive referral traffic back to your site.

## Should I allow and optimize for Claude Code to drive organic growth?

`Claude Code` fetches pages during active developer sessions where it researches tools, libraries, APIs, and datasets to help developers build software. Unlike traditional search bots, `Claude Code` doesn't just recommend your content in a response. It can autonomously decide to install a package, use an API, or follow your integration guide based on what it reads. That makes it one of the highest-value bots for developer-facing products: your documentation can directly influence adoption decisions. Claude supports structured citations with clickable links, so referral traffic follows. Blocking `Claude Code` removes your content from these autonomous development workflows entirely.

### Here's how to optimize for Claude Code:

- Allow Claude-Code in your robots.txt so developers using Claude can discover and integrate your tools autonomously
- Use clear, well-structured HTML for documentation and API reference pages
- Include descriptive title tags and meta descriptions on technical content
- Add structured data (JSON-LD) to help the crawler understand page context
- Ensure your robots.txt is at the top level of your domain
- Keep server response times fast to avoid timeouts during on-demand fetches

## Data Usage & Training

Content fetched by `Claude Code` may contribute to model training. [Anthropic](https://claude.com/) uses different bots for different purposes: `ClaudeBot` handles training-focused crawling, `Claude-User` handles conversational retrieval, and `Claude Code` serves terminal and automation workflows. If you want to exclude your content from training, block the relevant tokens in your robots.txt.

## How Claude Code Accesses Content

Here's how Claude Code accesses your site and understands your content:

- Fetches HTML via standard HTTP requests
- Respects robots.txt Disallow and Allow directives
- Supports Crawl-delay directives
- Honors Sitemap directives in robots.txt
- JavaScript rendering capability is unknown

Mixed pattern. `Claude Code` performs on-demand fetches when users initiate retrieval during coding sessions, but may also run scheduled or continuous crawling for search-quality indexing and model development.

## How to Block or Control Claude Code

To block `Claude Code`, add the following to your top-level robots.txt:

User-agent: Claude-Code  
Disallow: /

You can also use Crawl-delay to rate-limit instead of fully blocking:

User-agent: Claude-Code  
Crawl-delay: 10

[Anthropic](https://claude.com/) does not publish IP ranges for this bot, and IP-based blocking is not recommended. [Anthropic](https://claude.com/) warns that IP blocking can prevent bots from reading your robots.txt, which may cause the opposite of what you intend. [Anthropic](https://claude.com/) operates multiple crawlers with different tokens (`ClaudeBot`, `Claude-User`, Claude-Code), so block each one individually if you want to opt out entirely. Contact [Anthropic](https://claude.com/) support if blocking doesn't behave as expected.

## Common Issues & Troubleshooting

Watch out for these common problems when working with Claude Code:

- IP-based blocking can prevent the bot from reading your robots.txt, causing unexpected behavior
- Robots.txt must be at the top level of your domain or it won't be found
- [Anthropic](https://claude.com/) uses multiple bot tokens, so blocking one doesn't block the others
- JavaScript-heavy pages may not render fully (rendering capability is unknown)
- Aggressive rate limiting or bot protection services may block legitimate fetches

## Quick Reference

**Platform**  
[Anthropic](/content/ai-agent-directory?platforms=Anthropic/index.html)

**Agent Category**  
[Coding](/content/ai-agent-directory?tags=Coding/index.html) [AI User Initiated](/content/ai-agent-directory?tags=AI%20User%20Initiated/index.html)

**Growth Value**  
High

**Official Documentation**  
[privacy.claude.com/en/articles/8896518-does-anthropic-crawl-...](https://privacy.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)

**User Agent String**  
`claude-code`

**robots.txt Entry**  
`User-agent: claude-code  
Disallow: /`

## Frequently Asked Questions

### What is Claude Code and how is it different from ClaudeBot?

### Does Claude Code respect robots.txt?

### Does Claude Code use my content for AI training?

### Can Claude Code drive traffic to my site?

### How do I block all Anthropic crawlers at once?

### Should I allow Claude Code on my developer documentation?