Content Scraping

Configure how ChattyBox crawls your website and indexes content for AI responses.

Open Knowledge → Import & refresh, connect your website, review discovered pages, and explicitly import or refresh your selection. Website and advanced source settings save automatically; check the save status and resolve validation errors or retry a failed save. Discovery and autosave do not themselves import pages. A selected-page import does not replace the saved source configuration. The separate advanced Scrape Now action first saves valid source settings and then runs the configured mode, not the picker selection. The dashboard requires a saved Website URL even for Manual URLs mode.

:::warning Public HTML only The crawler fetches HTML; it does not run a browser or render JavaScript. ChattyBox does not enforce robots.txt crawl directives; automatic sitemap detection may read the file for same-origin sitemap declarations when conventional sitemap locations do not return URLs. The Crawl mode's link fallback does not use those declarations. Do not treat robots.txt as access control. Only submit content you are authorized to index, and use an explicit source selection rather than robots rules to keep unwanted pages out. Authentication, CAPTCHA solving, and private-network crawling are not supported. :::

For a public-content rollout, see the AI chatbot for website guide for how to choose trustworthy docs, help articles, product pages, and FAQs before testing cited answers.

Scraping Modes

ChattyBox supports four scraping modes:

1. Homepage Only

Scrapes only the saved Website URL, which can be a specific page rather than the site's root. Best for a landing page or a one-page evaluation.

2. Sitemap Mode (Recommended)

Provide a sitemap URL to discover pages, including pages in child sitemaps. Only page URLs on the Website URL's origin are retained, and filtering, page limits, and refresh budgets still apply. A sitemap is a discovery source, not a guarantee that every listed page will be indexed.

https://example.com/sitemap.xml

If Sitemap URL is empty, ChattyBox first tries any detected sitemap, then /sitemap.xml and /sitemap_index.xml. If none returns URLs, it falls back to the saved Website URL only. An explicitly supplied sitemap that returns no URLs fails the job instead of falling back. Nested sitemap discovery is depth-limited.

3. Crawl Mode

Start from a public URL and let ChattyBox discover pages on the same site. Crawl mode first tries the conventional /sitemap.xml and /sitemap_index.xml locations. If neither yields URLs, it follows same-origin links from the starting page up to the configured depth (default: 3).

If a conventional sitemap returns URLs, crawl mode uses those URLs; it does not follow links to fill gaps in an incomplete sitemap. During link discovery, the crawler skips common account, login, checkout, search, and asset URLs. Review the actual indexed pages rather than treating those heuristics as a privacy boundary.

4. Manual URLs

Specify exact public URLs to scrape, one per line. Use this when you want precise control over which pages are indexed. These pages are not expanded by following links; page and refresh limits still apply.

https://example.com/pricing
https://example.com/features
https://example.com/about
https://example.com/faq

Advanced URL Selection

The dashboard exposes mode, sitemap/manual URLs, maximum pages, and the refresh schedule. It does not expose include/exclude patterns or maximum crawl depth. Configuration as code supports these advanced settings. Include/exclude values are case-sensitive substrings of the URL, not glob patterns or regular expressions; an exclude match wins. During link crawling, filtered-out pages are not traversed, so a rule that excludes the starting page can prevent discovery entirely.

For multilingual sites, explicitly include the public language versions you need. Sitemap discovery and link crawling stay on the same origin, so another language on a subdomain needs its own source selection. Page language is detected from the page and URL; adding a widget locale does not translate or index missing content.

Page Limits

Distinguish stored pages from monthly refresh work. The saved Maximum Pages is a requested per-run selection limit, not an entitlement; the dashboard editor also validates against its displayed limit. When omitted, the request defaults to the plan's included account-page allowance. Before a run, ChattyBox rechecks provider capacity, prioritizes known existing project URLs, and limits admission of new URLs to remaining owner-wide capacity. New page slots are checked again when pages are saved. Discovery may make network requests before the final selection is known; do not use capacity limits as a privacy or outbound-request boundary. The Knowledge picker separately caps a selected import at 500 URLs and rechecks capacity before starting. Changing the source or lowering the limit does not automatically delete older pages. Included stored pages are shared across the owner's chatbots; see Pricing for documented allowances and Account & Billing for current usage and available allowance.

Plan	Included account pages	Monthly page-refresh budget	Fastest refresh cadence
Free	10	500	Weekly
Starter	150	5,000	Daily
Pro	5,000	100,000	Daily
Business	25,000	250,000	Daily

Pro and Business can enable optional pay-as-you-go (PAYG) page overages in Account & Billing. Free and Starter remain hard-capped. PAYG page allowance does not currently raise the separate monthly refresh budget in the crawler. See plans and limits and your account's current allowance before increasing a run's size.

Refreshing Content

• Automatic Re-scraping is opt-in. Free allows weekly or monthly runs; paid plans also allow daily runs. Monthly scheduling means a 30-day interval, not a particular calendar date.
• Scrape Now is not an unlimited retry button. The minimum interval after the latest scrape job was created is seven days on Free or 24 hours on paid plans, including after a failed job. A pending or running job also blocks another manual scrape.
• New pages and pages whose stored extracted text changes each consume one unit of the monthly refresh budget. An unchanged page updates metadata without another scrape refresh unit; embedding repair may still be queued if needed.
• The refresh budget is shared across the owner's projects and resets at the start of each UTC calendar month. It is not a separate allowance for every chatbot or every scrape.
• Successfully fetched updates replace stored page text. Failed fetches and rejected block/error pages leave previously stored content in place. Removing a URL from a sitemap or source selection does not remove it from the knowledge base; delete unwanted pages in Knowledge → Pages and remove them from future discovery.

Progress and Partial Indexing

Open Knowledge → Pages, expand Import history when jobs exist, and select a job under Scrape Jobs to see status, page counts, timestamps, and an estimated rate/time remaining while pages are being saved. Discovery happens before the final selected-page total is known, so an early zero count does not necessarily mean a stalled job. Counts describe pages saved, including unchanged pages, not completed vector embeddings.

Completed means the run ended, not that every selected URL succeeded. A run may finish with fewer saved pages after fetch failures or a page/refresh limit, and an empty selection can complete with zero pages. Compare the saved count and actual pages list with your expected URLs. The job detail displays at most 8 saved pages; use the paginated Knowledge → Pages list for larger collections. It is not a complete per-URL failure report.

Partially indexed means a page's extracted text exceeded the 512 KiB UTF-8 storage limit. Only the beginning is retained, and the UI shows retained versus extracted bytes. A completed job with such pages shows Completed with warnings. That warning is about truncated page text, not every possible failed or skipped URL. Split a long reference into smaller public pages if important sections were cut off.

The fetch layer separately rejects response bodies over 5 MiB and redirect chains beyond five hops. Those fetch failures are different from successfully fetched pages whose extracted text is partially retained.

Changed content queues a separate embedding job. Review the index/rebuild status and test representative answers before launch; a scrape completion badge alone does not validate retrieval quality.

Content Maintenance

Knowledge → Pages supports deleting individual pages, Clear All, and Rebuild Index. Deselecting a page in the picker does not delete stored content. Deleting an individual page also removes it from a saved Manual URLs list, but sitemap and crawl sources can rediscover it; adjust those sources as well. Clear All cancels active scrape and embedding work and removes stored pages and embeddings in background batches, but leaves the saved source configuration intact. Later scrapes can add pages again.

Rebuild Index rebuilds embeddings from already stored text; it does not fetch your site, restore deleted pages, or recover truncated text. Manual rebuilds have the plan's weekly/daily cooldown, measured from the latest embedding job, and consume the same monthly page-refresh budget for each page reserved for rebuilding. Embeddings automatically queued for changed scraped pages do not charge that work a second time. Config-locked projects reject dashboard scrape, content deletion, and manual rebuild operations; use the configuration deployment workflow.

Content Extraction

ChattyBox reads the page title and prefers text from <main>, then <article>, then an element with role="main", falling back to body text. Within the selected content it retains:

• Page title - The <title> tag
• Headings - <h1> through <h6> text
• Paragraphs - Readable paragraph text
• Lists - <ul> and <ol> items
• Code and tables - Text content, with preformatted code whitespace retained

A limited fallback can fetch a same-origin OpenAPI document referenced by a recognized SwaggerUIBundle configuration when the HTML has too little text. This is not general JavaScript rendering, file upload, or support for every API-reference viewer.

Ignored Content

Extraction excludes common non-content elements, including:

• Navigation menus
• Footer content
• Scripts and styles
• Elements marked hidden, aria-hidden="true", or with inline display: none / visibility: hidden
• Recognized cookie/consent banner markup and dialog/navigation roles
• Iframes, canvas, SVG, and embedded objects

These are HTML-based heuristics, not a computed browser layout. Do not assume every visually hidden element or every banner will be detected.

---

Troubleshooting

Pages not being scraped?

• Check that the URL is publicly accessible
• Check that the server allows ordinary HTML requests without a login, CAPTCHA, or bot challenge
• Verify the page isn't behind authentication
• Check the mode, origin, selected URLs, page allowance, and refresh budget; robots crawl directives are not enforced
• For link-crawl fallback, confirm important links are reachable from the start URL within the depth limit

Content missing from responses?

• Re-scrape after publishing changes, once the refresh cooldown allows it
• Serve important text in the initial HTML; content loaded only by JavaScript is not rendered
• Ensure the content is in the main body, not in iframes
• Check for partial-indexing warnings and wait for embedding work before retesting

Need more control?

For unsupported sources or persistent extraction failures, contact support@chattybox.ai with a public example URL. Do not send login credentials or assume support can enable authenticated or JavaScript-rendered crawling.

For launch issues after scraping, use the launch checklist and troubleshooting guide.