Content Scraping

Configure how ChattyBox crawls your website and indexes content for AI responses.

Open your chatbot's Settings tab. Save Website URL under General Settings with Save Changes, then choose a mode and Maximum Pages under Scraping Configuration and select Save Config. Scrape Now starts a job using the current scraping form values and opens its progress page. The dashboard requires a saved Website URL even for Manual URLs mode.

:::warning Public HTML only The crawler fetches HTML; it does not run a browser or render JavaScript. It does not currently read or enforce robots.txt. Only submit content you are authorized to index, and use an explicit source selection rather than robots rules to keep unwanted pages out. Authentication, CAPTCHA solving, and private-network crawling are not supported. :::

For a public-content rollout, see the AI chatbot for website guide for how to choose trustworthy docs, help articles, product pages, and FAQs before testing cited answers.

Scraping Modes

ChattyBox supports four scraping modes:

1. Homepage Only

Scrapes only the saved Website URL, which can be a specific page rather than the site's root. Best for a landing page or a one-page evaluation.

2. Sitemap Mode (Recommended)

Provide a sitemap URL to discover pages, including pages in child sitemaps. Only page URLs on the Website URL's origin are retained, and filtering, page limits, and refresh budgets still apply. A sitemap is a discovery source, not a guarantee that every listed page will be indexed.

https://example.com/sitemap.xml

If Sitemap URL is empty, ChattyBox tries /sitemap.xml, then /sitemap_index.xml. If neither returns URLs, it falls back to the saved Website URL only. An explicitly supplied sitemap that returns no URLs fails the job instead of falling back. Nested sitemap discovery is depth-limited.

3. Crawl Mode

Start from a public URL and let ChattyBox discover pages on the same site. Crawl mode first tries the conventional /sitemap.xml and /sitemap_index.xml locations. If neither yields URLs, it follows same-origin links from the starting page up to the configured depth (default: 3).

If a conventional sitemap returns URLs, crawl mode uses those URLs; it does not follow links to fill gaps in an incomplete sitemap. During link discovery, the crawler skips common account, login, checkout, search, and asset URLs. Review the actual indexed pages rather than treating those heuristics as a privacy boundary.

4. Manual URLs

Specify exact public URLs to scrape, one per line. Use this when you want precise control over which pages are indexed. These pages are not expanded by following links; page and refresh limits still apply.

https://example.com/pricing
https://example.com/features
https://example.com/about
https://example.com/faq

Advanced URL Selection

The dashboard exposes mode, sitemap/manual URLs, maximum pages, and the refresh schedule. It does not expose include/exclude patterns or maximum crawl depth. Configuration as code supports these advanced settings. Include/exclude values are case-sensitive substrings of the URL, not glob patterns or regular expressions; an exclude match wins. During link crawling, filtered-out pages are not traversed, so a rule that excludes the starting page can prevent discovery entirely.

For multilingual sites, explicitly include the public language versions you need. Sitemap discovery and link crawling stay on the same origin, so another language on a subdomain needs its own source selection. Page language is detected from the page and URL; adding a widget locale does not translate or index missing content.

Page Limits

Distinguish stored pages from monthly refresh work. In the forthcoming owner-pooled page rollout, Maximum Pages stores the requested value without clamping it to the base plan. Each run limits outbound discovery and selection to the smaller of that request and the larger of current provider page capacity or known existing project pages. At zero or denied capacity, only known existing pages may refresh; a project with none makes no unknown-page outbound discovery. When omitted, the request defaults to the plan's included account-page allowance. This does not add owner-pool or plan entitlements. Saving the source checks the owner's available page allowance, and a scrape separately rechecks it before saving pages. Existing indexed URLs are prioritized when they are in the discovered set. Changing your source or lowering the limit does not automatically delete older pages.

Plan	Included account pages	Monthly page-refresh budget	Fastest refresh cadence
Free	10	500	Weekly
Starter	150	5,000	Daily
Pro	5,000	100,000	Daily
Business	25,000	250,000	Daily

Pro and Business can enable optional pay-as-you-go (PAYG) page overages in Account & Billing. Free and Starter remain hard-capped. PAYG page allowance does not currently raise the separate monthly refresh budget in the crawler. See plans and limits and your account's current allowance before increasing a run's size.

Refreshing Content

• Automatic Re-scraping is opt-in. Free allows weekly or monthly runs; paid plans also allow daily runs. Monthly scheduling means a 30-day interval, not a particular calendar date.
• Scrape Now is not an unlimited retry button. The minimum interval after the latest scrape job was created is seven days on Free or 24 hours on paid plans, including after a failed job. A pending or running job also blocks another manual scrape.
• New pages and pages whose extracted text changes each consume one unit of the monthly refresh budget. An unchanged page updates its scrape metadata without consuming another unit or triggering a content-change embedding rebuild.
• The refresh budget is shared across the owner's projects (or the legacy workspace) and resets at the start of each UTC calendar month. It is not a separate allowance for every chatbot or every scrape.
• Successfully fetched updates replace stored page text. Failed fetches and rejected block/error pages leave previously stored content in place. Removing a URL from a sitemap or source selection does not remove it from the knowledge base; delete unwanted pages in Content and remove them from future discovery.

Progress and Partial Indexing

Open a job in Content > Scrape Jobs to see status, page counts, timestamps, and an estimated rate/time remaining while pages are being saved. Discovery happens before the final selected-page total is known, so an early zero count does not necessarily mean a stalled job. Counts describe pages saved, including unchanged pages, not completed vector embeddings.

Completed means the run ended, not that every selected URL succeeded. A run may finish with fewer saved pages after fetch failures or a page/refresh limit, and an empty selection can complete with zero pages. Compare the saved count and actual Content list with your expected URLs. The job detail displays at most 200 saved pages; use the paginated Content list for larger collections. It is not a complete per-URL failure report.

Partially indexed means a page's extracted text exceeded the 512 KiB UTF-8 storage limit. Only the beginning is retained, and the UI shows retained versus extracted bytes. A completed job with such pages shows Completed with warnings. That warning is about truncated page text, not every possible failed or skipped URL. Split a long reference into smaller public pages if important sections were cut off.

The fetch layer separately rejects response bodies over 5 MiB and redirect chains beyond five hops. Those fetch failures are different from successfully fetched pages whose extracted text is partially retained.

Changed content queues a separate embedding job. Review the index/rebuild status and test representative answers before launch; a scrape completion badge alone does not validate retrieval quality.

Content Maintenance

Content supports deleting individual pages, Clear All, and Rebuild Index. Clearing pages removes their stored content and embeddings, not the saved source configuration. Later scrapes can add them again.

Rebuild Index rebuilds embeddings from already stored text; it does not fetch your site, restore deleted pages, or recover truncated text. Manual rebuilds have the plan's weekly/daily cooldown, measured from the latest embedding job, and consume the same monthly page-refresh budget for each page reserved for rebuilding. Embeddings automatically queued for changed scraped pages do not charge that work a second time. Config-locked projects reject dashboard scrape, content deletion, and manual rebuild operations; use the configuration deployment workflow.

Content Extraction

ChattyBox reads the page title and prefers text from <main>, then <article>, then an element with role="main", falling back to body text. Within the selected content it retains:

• Page title - The <title> tag
• Headings - <h1> through <h6> text
• Paragraphs - Readable paragraph text
• Lists - <ul> and <ol> items
• Code and tables - Text content, with preformatted code whitespace retained

A limited fallback can fetch a same-origin OpenAPI document referenced by a recognized SwaggerUIBundle configuration when the HTML has too little text. This is not general JavaScript rendering, file upload, or support for every API-reference viewer.

Ignored Content

Extraction excludes common non-content elements, including:

• Navigation menus
• Footer content
• Scripts and styles
• Elements marked hidden, aria-hidden="true", or with inline display: none / visibility: hidden
• Recognized cookie/consent banner markup and dialog/navigation roles
• Iframes, canvas, SVG, and embedded objects

These are HTML-based heuristics, not a computed browser layout. Do not assume every visually hidden element or every banner will be detected.

---

Troubleshooting

Pages not being scraped?

• Check that the URL is publicly accessible
• Check that the server allows ordinary HTML requests without a login, CAPTCHA, or bot challenge
• Verify the page isn't behind authentication
• Check the mode, origin, selected URLs, page allowance, and refresh budget; robots rules are not consulted
• For link-crawl fallback, confirm important links are reachable from the start URL within the depth limit

Content missing from responses?

• Re-scrape after publishing changes, once the refresh cooldown allows it
• Serve important text in the initial HTML; content loaded only by JavaScript is not rendered
• Ensure the content is in the main body, not in iframes
• Check for partial-indexing warnings and wait for embedding work before retesting

Need more control?

For unsupported sources or persistent extraction failures, contact support@chattybox.ai with a public example URL. Do not send login credentials or assume support can enable authenticated or JavaScript-rendered crawling.

For launch issues after scraping, use the launch checklist and troubleshooting guide.