Website crawl

Goal: Discover pages on your website, choose which to index, and add them to a knowledge base.

Prerequisites

A knowledge base — Create a knowledge base
A publicly reachable website (no login required for crawled pages)

Step 1 — Start a crawl

Open your knowledge base → Website crawl.
Enter the start URL (usually your homepage or docs root).
Set crawl scope:

Setting	Guidance
Same domain only	Recommended — stays on your site
Max depth	How many link hops from the start URL
Max pages	Cap discovery to stay within plan document limits

Click Discover links.

Step 2 — Select pages

Chatevo lists discovered URLs. Review and select the pages you want indexed:

Include help articles, FAQs, and product pages visitors ask about
Exclude login, checkout, account, and duplicate listing pages
Set classification per page or use the KB default

Step 3 — Index selected pages

Click Index selected. Each page is fetched, snapshotted, and queued through the standard pipeline: extract → chunk → embed → Qdrant. See Processing and indexing.

Step 4 — Monitor status

Status	Meaning
Queued	Waiting to fetch
Processing	Extracting and embedding
Indexed	Available for retrieval
Failed	Blocked, empty, or unreachable — retry or exclude

Keep content fresh

On Standard, Pro, and Enterprise plans, enable Auto-retrain to re-crawl and re-index when source pages change. Free and Starter plans re-crawl manually.

Tips

Start with a focused section (e.g. /help/) instead of the entire marketing site
Ensure robots.txt allows Chatevo’s crawler
After indexing, test with real visitor questions in Test chat