# Amazon Q Business `Amazon Q Business`'s web crawler fetches and indexes HTTPS websites (public or internal) to create data sources for `Amazon Q Business`, [Amazon](https://aws.amazon.com/)'s enterprise AI assistant. It extracts HTML and supported attachments like PDFs, DOCX, and PPTX files, then populates a customer-specific [Amazon](https://aws.amazon.com/) Q index used for knowledge-base experiences. Because content is primarily surfaced within private enterprise environments, direct public referral traffic is limited, though source references may appear depending on how the [Amazon](https://aws.amazon.com/) Q application is configured. ## Should I allow and optimize for Amazon Q Business to drive organic growth? `Amazon Q Business` primarily surfaces content within private enterprise assistant experiences, so direct public referral traffic is limited. However, allowing this crawler means your content can appear in [Amazon](https://aws.amazon.com/) Q answers within organizations that have indexed your site. Citation and link display depends on how each [Amazon](https://aws.amazon.com/) Q application is configured. If your site publishes content that enterprise users would search for (documentation, product specs, industry guides), allowing this crawler increases your visibility within those environments. The indirect value through [Amazon](https://aws.amazon.com/)'s broader AI ecosystem also makes allowing it worthwhile for most publishers. Here's how to optimize for Amazon Q Business: - Allow the aws-quick-on-behalf-of- user-agent in your robots.txt - Provide a sitemap.xml to help the crawler discover all relevant pages efficiently - Use standard anchor tags () for navigation instead of JavaScript-driven menus, since the crawler does not simulate clicks or scrolls - Ensure critical content is in the initial HTML response rather than loaded dynamically via JavaScript - Include descriptive page titles and meta descriptions to improve content extraction quality - Add structured data (JSON-LD) to help the crawler understand page content and relationships - Keep server response times fast and avoid aggressive rate limiting on crawler IPs ## Data Usage & Training Whether content crawled by `Amazon Q Business`'s web crawler is used to train [Amazon](https://aws.amazon.com/)'s broader AI models is unclear. AWS Web Crawler documentation indicates crawled content is ingested into a customer-specific [Amazon](https://aws.amazon.com/) Q index, but `Amazonbot`'s developer pages state that crawled content may be used to train [Amazon](https://aws.amazon.com/) AI models. If you want to be cautious, treat content fetched by this crawler as potentially used for training and block it if that's a concern. ## How Amazon Q Business Accesses Content Here's how Amazon Q Business accesses your site and understands your content: - Fetches HTML and supported attachments (PDF, DOCX, PPTX) via HTTPS requests - Partial JavaScript rendering; does not simulate clicks or scrolls - Uses the user-agent string aws-quick-on-behalf-of-, where the UUID identifies the specific customer deployment - Performs an initial full sync followed by incremental syncs on a configurable schedule - Respects robots.txt Allow and Disallow directives, and honors meta robots tags (noindex, nofollow, noarchive) Crawl frequency is configurable by the `Amazon Q Business` customer who set up the web crawler. The typical pattern is an initial full sync followed by incremental syncs on a customer-defined schedule. ## How to Block or Control Amazon Q Business To block this crawler via robots.txt, target the user-agent token. Because each deployment uses a unique UUID, you may need a wildcard approach or block the `Amazonbot` token as a fallback: User-agent: `Amazonbot` Disallow: / You can also use page-level meta robots tags (noindex, nofollow, noarchive) to control indexing on specific pages. For IP-based blocking, [Amazon](https://aws.amazon.com/) publishes IP ranges at [https://developer.amazon.com/amazonbot/searchbot-ip-addresses/.](https://developer.amazon.com/amazonbot/searchbot-ip-addresses/) You can also contact [Amazon](https://aws.amazon.com/) directly at `amazonbot`@amazon.com to request opt-out. AWS customers who configured the crawler can restrict domains and URLs through the AWS console. ## Common Issues & Troubleshooting Watch out for these common problems when working with Amazon Q Business: - The user-agent string includes a unique UUID per deployment (aws-quick-on-behalf-of-), making it harder to write a single robots.txt rule that covers all instances - JavaScript-driven navigation that requires simulated clicks or scrolls will not be followed by the crawler - WAF or rate-limiting rules may inadvertently block legitimate crawl requests - If robots.txt is unavailable (returns a server error), the crawler may fall back to default behavior and crawl more broadly than expected - IP-based blocking may need periodic updates as [Amazon](https://aws.amazon.com/)'s published IP ranges change ## Quick Reference ### Platform [Amazon](/content/ai-agent-directory?platforms=Amazon/index.html) ### Agent Category [AI Training](/content/ai-agent-directory?tags=AI%20Training/index.html) ### Growth Value Medium ### Official Documentation [docs.aws.amazon.com/amazonq/latest/qbusiness-ug/connector-we...](https://docs.aws.amazon.com/amazonq/latest/qbusiness-ug/connector-webcrawler.html) ### User Agent String `amazon-qbusiness` ### robots.txt Entry `User-agent: amazon-qbusiness Disallow: /` ## Frequently Asked Questions ### What is the aws-quick-on-behalf-of user agent in my logs? ### Does Amazon Q Business's crawler respect robots.txt? ### Is content crawled by Amazon Q Business used to train AI models? ### How do I block Amazon Q Business's crawler? ### Will blocking Amazon Q Business's crawler affect my visibility on Amazon search? ### Can Amazon Q Business crawl JavaScript-heavy sites?