URL Connector

Use the URL connector when you want to ingest exactly one page.

This is the best connector for a first web-content test because it keeps the crawl surface tiny and makes debugging much easier than starting with a full site crawl.

Use this connector when

• you want one documentation page, policy page, or runbook
• the page is publicly reachable
• you want the fastest “create source -> sync -> query” loop

If you need multiple pages, use web crawler or sitemap instead.

Create the source

curl -X POST "https://api.retaindb.com/v1/projects/proj_123/sources" \
  -H "Authorization: Bearer $RETAINDB_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Auth Runbook",
    "connector_type": "url",
    "config": {
      "url": "https://docs.acme.com/runbooks/auth"
    }
  }'

Start sync

curl -X POST "https://api.retaindb.com/v1/sources/src_123/sync" \
  -H "Authorization: Bearer $RETAINDB_API_KEY"

Check status

curl "https://api.retaindb.com/v1/sources/src_123/status" \
  -H "Authorization: Bearer $RETAINDB_API_KEY"

What success looks like

A healthy first run usually looks like this:

1. source creation returns immediately
2. sync is queued
3. source status progresses toward READY
4. a query against the same project can retrieve the page content

Common mistakes

Starting with a login-protected page

The URL connector is best for public or directly accessible pages. If the page requires a browser session or complex rendering, it may not behave the way you expect.

Expecting a whole site from one URL

This connector is intentionally narrow. It is for one page, not discovery across a domain.

Testing retrieval in the wrong project

If the page synced but search is empty, check project scope before blaming the connector.

Good first test

Choose a page with a unique phrase, sync it, then query that phrase from the same project. That proves the connector, indexing, and retrieval loop all work together.

Next step

If you need more than one page, move to web crawler or sitemap. If you prefer the dashboard flow for the same setup, use sources: add, sync, and troubleshoot.