Adding your website.
How Askbridge reads your website, how it treats robots.txt and sitemaps, and how page limits work.
Your website is usually the main source of answers. Askbridge reads the pages, splits them into short passages, and searches those passages when a visitor asks a question.
Add a website source
- Open Knowledge, choose Add source, then Website.
- Enter the address of your home page, for example
https://www.example.co.uk. - Optionally open Advanced options to set the most pages to read and which parts of the site to include or leave out (see below).
- Choose Add website. The source shows Updating while pages are read, then Ready with the number of pages.
You can keep working while it runs. Small sites usually take a few minutes; large sites can take longer, because Askbridge reads pages slowly on purpose so it never puts load on your server.
What Askbridge reads
- It starts from your sitemap if you have one (usually
/sitemap.xml, or the one named in robots.txt), then follows links within the same site. - It respects your robots.txt. Pages you disallow for crawlers are not read.
- It reads the visible text of each page, with its title and headings. It skips navigation, cookie banners and scripts where it can.
- It stays on the address you entered. Links to other websites are not followed.
Pages that need a login, or that only appear after a visitor clicks something, cannot be read. Put that information into an FAQ or an uploaded file instead.
Include and exclude
Use include and exclude rules to keep answers focused. For example:
| Rule | Effect |
|---|---|
Include /services/ | Read only pages under /services/ |
Exclude /blog/ | Skip the blog |
Exclude /careers/ | Skip job adverts you do not want quoted |
Excluding old or off-topic pages often improves answers more than adding new ones.
Page limits
Each plan has a page allowance across all your websites: 500 pages on Starter, 3,000 on Growth and 15,000 on Scale. You can also set a lower maximum for a single source. When the limit is reached, Askbridge stops reading new pages and tells you on the Knowledge page; pages already read keep working.
Keeping it up to date
When your website changes, choose Refresh content on the source. Each page has a fingerprint (a checksum), so only pages that have changed are read again, and pages removed from your site stop being used. To read a single page again, open the source and choose Refresh content next to that page.
Turning a source off
Each source has a Used in answers switch. When it is off, the source shows Disabled and its pages are not used in answers straight away, but nothing is deleted, so you can switch it back on at any time without reading the site again. Remove a source to delete its pages completely.
Source status
- Ready: pages are read and used in answers.
- Updating: Askbridge is reading the site now.
- Needs attention: something went wrong, for example robots.txt blocked the crawl, the site could not be reached, or the page limit was reached. The source says what happened and what to do.
- Disabled: switched off, so not used in answers.
The number of passages each source is split into is under Technical details. You do not need it for everyday use.
If pages are missing
- Check robots.txt does not block the pages.
- Check the pages are linked from your sitemap or from other pages.
- Check you have not hit your page limit.
- See Troubleshooting for more.
Still stuck? Email support@vitalos.co.uk and tell us your workspace name.