algolia-crawler
Installation
SKILL.md
Algolia Crawler
Turn web pages into a RAG-optimized Algolia index with the Algolia Crawler, driven entirely by the Algolia CLI (algolia crawler …). The Crawler visits URLs, extracts content with a JavaScript recordExtractor, and writes Algolia records on a schedule — no scraping code to maintain.
The single most important idea: crawling for RAG is not the same as crawling for site search. A RAG record is a chunk of text that will be dropped into an LLM's context window in isolation. That changes how you shape every record (see The RAG record mental model).
When to use this skill vs. the others
| Need | Skill |
|---|---|
| Crawl/ingest web pages into an index (this workflow) | algolia-crawler |
| Build the chatbot / agent / RAG app on top of the index | algobot-cli |
| Raw record/settings/synonym/rule ops on an existing index | algolia-cli |
| Read-only search, analytics, recommendations | algolia-mcp |
| Frontend search UI (autocomplete, results, facets) | instantsearch |
Rule of thumb: if the user wants to get web content into Algolia, you're in the right skill. Once the index exists and they want to ask questions over it, hand off to algobot-cli.