AutoGineer
Generated banner for Firecrawl Knowledge Base
Generated logo for Firecrawl Knowledge Base

Firecrawl Knowledge Base

FirecrawlOfficialCreated by firecrawl

  • Skill
  • Workflow
  • Automation and integrations
  • ISC
  • No issues found
Download ZIP

From the author

Build a knowledge base from web content with Firecrawl. Use for local reference docs, RAG-ready chunks, fine-tuning datasets, documentation mirrors, topic corpora, or LLM-ready markdown organized from web sources.
— the description by firecrawl

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

The Firecrawl Knowledge Base skill automates the process of gathering, cleaning, and formatting web content for various AI-driven applications. By leveraging Firecrawl, it converts URLs or topics into organized markdown files, making it suitable for building RAG systems, fine-tuning datasets, creating documentation mirrors, or establishing local reference libraries. The workflow handles the entire lifecycle from discovery and scraping to structuring the output for specific use cases.

What it does

  • Maps documentation sites and searches for topic-based corpora.
  • Scrapes web pages into clean, LLM-ready markdown format.
  • Preserves essential code examples and tables from source content.
  • Organizes output into structured directories with metadata manifests.
  • Supports multiple output modes including RAG, training data, and documentation mirrors.
  • Removes boilerplate navigation elements to improve content quality.

When to use it

  • When you need to create a RAG-ready dataset from existing web documentation.
  • When building a fine-tuning dataset for a custom model.
  • When creating a local, offline mirror of a website for reference.
  • When aggregating information from multiple web sources into a unified corpus.

Advantages

  • Automates the complex task of web scraping and content cleaning.
  • Produces structured, machine-readable output formats.
  • Maintains source attribution and metadata for traceability.
  • Flexible output modes tailored to specific AI development needs.

Usage examples

“Create a RAG-ready knowledge base from the official documentation of a specific framework.”
The agent will map the documentation site, scrape the pages into markdown, generate chunk files, and produce a manifest.json for immediate use in a RAG pipeline.
“Build a training dataset from a series of technical blog posts on a specific topic.”
The agent will collect the content, clean the markdown, and output the data in a training-data.jsonl format along with associated metadata.

Install

  1. Download the skill as a ZIP:

    https://mp-staging.autogineer.com/agent-skills/firecrawl/firecrawl-knowledge-base/download

  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip firecrawl-knowledge-base.zip -d .agents/skills

Documentation by firecrawl

Firecrawl Knowledge Base

Use this to turn URLs or topics into organized LLM-ready content.

Onboarding Interview

Infer the source, goal, depth, and output location from context. If the source and goal are clear, proceed immediately.

Ask at most 1-3 concise questions only if blocked, such as the source URL/topic, whether the output is reference/RAG/training/docs, or training format if training is requested.

Firecrawl Collection Plan

Use Firecrawl map for documentation sites, search for topic-based corpora, scrape pages into markdown, and preserve code examples and tables.

For files, follow the Firecrawl download-style convention:

.firecrawl/
  <hostname>/
    <path>/
      index.md

Parallel Work

If appropriate, use sub-agents or equivalent parallel task runners:

  • one docs section per researcher

  • official docs, tutorials, community discussions, and references by source type

  • source scraping vs chunk generation vs manifest generation

Output Modes

  • Reference: markdown files, index.md, and sources.json.

  • RAG: markdown files plus chunk files and manifest.json.

  • Training: scraped source files plus training-data.jsonl and training-metadata.json.

  • Docs mirror: complete markdown mirror with a table of contents.

Final Deliverable

# Knowledge Base: [Source]

## Summary
[What was collected and why]

## Output Structure
[Files/directories created]

## Coverage
[Sections, source types, counts]

## Usage Notes
[How to use in RAG, docs, training, or agent context]

## Sources
[URLs collected]

## Rerun Inputs
workflow: firecrawl-knowledge-base
source: [url/topic]
goal: [reference/rag/train/docs]
depth: [quick/thorough/exhaustive]
output_dir: [.firecrawl/]

Quality Bar

  • Preserve code examples and formatting.

  • Remove boilerplate navigation where possible.

  • Include source URLs in frontmatter or metadata.

Files

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Security

No issues found by our static scan· No issues found

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by
firecrawl
Publisher
Firecrawl (Official)
Source
firecrawl/skills · skills/workflows/firecrawl-knowledge-base · commit 458d9d0
Licence
ISC License (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced

AutoGineer is not affiliated with or endorsed by firecrawl.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download firecrawl-knowledge-base.zip