Websites
Scrape and index your website content so the AI assistant can reference it when answering questions.
Adding a website source
Section titled “Adding a website source”- Navigate to AI > Websites in the dashboard.
- Click Add website.
- Enter a name and the URL to scrape.
Configuration
Section titled “Configuration”| Setting | Description |
|---|---|
| Name | Display name for this source (max 50 characters) |
| URL | Page URL to scrape (must be http or https) |
| Sync frequency | How often to re-crawl: Never, Once a day, Once a week, or Once a month |
| Crawl website | When enabled, crawls the entire site instead of a single page |
| Exclude paths | Path patterns to skip when crawling (e.g. /blog*, /login) |
When Crawl website is enabled, the URL sets the crawl scope. Use a bare domain (e.g. https://example.com) to crawl the whole site, or a subpath (e.g. https://example.com/docs/) to limit crawling to that section and its sub-pages. You can add up to 20 exclude path patterns, each supporting a trailing * wildcard.
Status
Section titled “Status”Each source shows its current status: Draft, Pending, Indexing, Live, Warning, or Error. Content is split into chunks and indexed for semantic search.
Managing sources
Section titled “Managing sources”- Re-sync — re-crawl the page to update content.
- Delete — remove the source and its indexed content.