Get better results from your bot with these tips.

## Structuring large websites

For sites with 100+ pages, consider splitting into multiple data sources:

- **By section:** Docs, Blog, Product pages as separate data sources
- **By update frequency:** Static pages vs frequently updated content
- **By importance:** Core pages synced daily, archives synced weekly

This lets you:

- Sync only what changed (saves storage tokens)
- Use different CSS selectors per section
- Troubleshoot issues more easily

## CSS selectors for common platforms

The right CSS selector captures your main content and excludes headers, footers, and navigation.

| Platform         | Recommended selector               |
| -----------------| ---------------------------------- |
| **WordPress**    | `article` or `.entry-content`     |
| **Webflow**      | `.main-content` or `main`         |
| **Squarespace**  | `.main-content`                    |
| **Shopify**      | `main` or `.product-description`   |
| **GitBook**      | `.markdown-body`                   |
| **Notion (public)** | `.notion-page-content`         |
| **Generic**      | `main` or `article`               |

💡 **Tip:** Inspect your page in browser DevTools to find the best selector. Test on one page before syncing your whole site.

## Crawl vs Sitemap vs Manual URLs

| Method        | Best for                                                |
| --------------| ----------------------------------------------------- |
| **Crawl**     | Discovering all pages on a site (up to 600)          |
| **Sitemap**   | Large sites with existing sitemaps                     |
| **Manual URLs**| Specific pages only, or when crawl misses pages       |
| **Bulk URLs** | Adding a known list of pages quickly                   |

## Sync frequency recommendations

| Content type                 | Recommended schedule          |
| ---------------------------- | ----------------------------- |
| News/blog (daily posts)     | Daily                         |
| Documentation                | Daily or weekly               |
| Marketing pages              | Weekly                        |
| Product pages                | Daily (if prices/stock change)|
| Static content               | Weekly or manual              |

## Reducing storage token usage

- Use CSS selectors to exclude unnecessary content
- Split large data sources so you only re-sync what changed
- Remove pages you don't need from the data source
- Use auto-sync with diffs (only syncs changed content)

## Notion tips

- Add top-level pages only - child pages are included automatically
- Don't add the same page multiple times
- If permissions change, disconnect and reconnect Notion in account settings

## YouTube tips

- Transcripts must be enabled on videos
- Add playlists to bulk-add related videos
- Auto-generated captions work, but manual captions are more accurate