web-spider

Web Spider

The Web Spider lets you crawl a website and automatically index its content so that Ariadne's agents can search and reference it. This is ideal for building a knowledge base from a documentation site, product catalogue, or any publicly accessible web content.

Accessing the Web Spider

Click Tools → Web Spider in the top navigation.

How It Works

The spider performs a breadth-first crawl starting from a URL you provide:

1. It fetches the starting page
2. It follows links within the same domain
3. It indexes the text content of each page as a searchable document
4. It respects a configurable depth limit so it doesn't crawl indefinitely

Indexed content becomes available to agents via the blob file search tool, so agents can search across everything the spider collected.

Starting a Crawl

1. Go to Tools → Web Spider
2. Enter the Start URL (e.g. https://docs.example.com)
3. Set the Maximum Depth — how many link-hops from the start page to follow (default is 2–3 levels)
4. Click Start Crawl

The crawl runs in the background. You can monitor progress and see how many pages have been indexed.

Viewing Crawl Results

Each completed crawl shows:

• Number of pages indexed
• The URLs that were crawled
• Any pages that were skipped (e.g. non-HTML content, external domains)

Using Indexed Content

Once indexed, the crawled content is searchable by agents. You can ask:

"What does the documentation say about rate limiting?"

The agent will search across all indexed pages and cite the relevant passages.

Managing Crawls

• Re-crawl – run the spider again on the same URL to pick up new or updated content
• Delete – remove a crawl index when it's no longer needed

Tips

• Start with a specific sub-path (e.g. https://docs.example.com/api/) rather than the whole domain if you only need a subset of pages
• Use a lower depth limit for large sites to avoid indexing thousands of unrelated pages
• Re-crawl periodically to keep the index fresh if the source content changes