AutoGineer
Generated banner for Firecrawl Crawl
Generated logo for Firecrawl Crawl

Firecrawl Crawl

FirecrawlOfficialPublished by Firecrawl

  • Skill
  • Tool integration
  • Automation and integrations
  • ISC
  • No issues found
Download ZIP

From the author

Bulk-extract many pages from one site or section. Use for "crawl", "everything under /docs", or content spanning linked pages.
— the description by Firecrawl

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

The firecrawl-crawl skill provides a command-line interface for performing bulk web data extraction. It allows users to crawl entire websites or specific sub-sections by following links up to a defined depth or page limit. The tool supports both synchronous and asynchronous execution, enabling users to either wait for results or manage long-running jobs via job IDs. It is designed to integrate into workflows where structured content from multiple pages is required for further processing.

What it does

  • Performs bulk extraction of web pages from a specified URL.
  • Supports path-based scoping to limit crawls to specific site sections.
  • Allows configuration of crawl depth and total page limits.
  • Provides options for synchronous waiting or asynchronous job management.
  • Saves crawl results directly to local JSON files.
  • Includes progress tracking for active crawl operations.

When to use it

  • When you need to extract content from multiple pages across a website.
  • When you want to target specific documentation or site sections.
  • When you need to automate the collection of data spanning linked pages.
  • When you prefer a command-line interface for managing large-scale web scraping tasks.

Advantages

  • Efficiently handles large-scale data extraction across multiple pages.
  • Flexible scoping options prevent unnecessary data collection.
  • Supports both immediate and background job processing.
  • Integrates easily into terminal-based automation workflows.

Usage examples

“Crawl the documentation section of example.com and save to a file.”
The agent executes the crawl command with the --include-paths /docs flag, limiting the scope and saving the output to the specified JSON file.
“Perform a deep crawl of a website with a limit of 50 pages.”
The agent runs the crawl with --max-depth and --limit parameters, providing progress updates until the task completes.

Install

  1. Download the skill as a ZIP:

    https://mp-staging.autogineer.com/agent-skills/firecrawl/firecrawl-crawl/download

  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip firecrawl-crawl.zip -d .agents/skills

Documentation by Firecrawl

firecrawl crawl

Bulk extract content from a website. Crawls pages following links up to a depth/limit.

Prerequisite: crawl requires authentication (no keyless free tier); without credentials the CLI prompts an interactive login.

Quick start

# Crawl a docs section
firecrawl crawl "<url>" --include-paths /docs --limit 50 --wait -o .firecrawl/crawl.json

# Full crawl with depth limit
firecrawl crawl "<url>" --max-depth 3 --wait --progress -o .firecrawl/crawl.json

# Check status of a running crawl
firecrawl crawl <job-id>

Run firecrawl crawl --help for the full option list.

Done when: the crawl reaches a terminal status and the saved output under .firecrawl/ contains the expected pages.

Tips

  • Use --wait when you need the results immediately. It has no default timeout; use --timeout <seconds> to bound polling. Without --wait, crawl returns a job ID for async polling.

  • Scope crawls with --include-paths whenever the request names a section — crawl only the pages you need.

  • Crawl consumes credits per page. Check firecrawl credit-usage before large crawls (credit-usage requires authentication).

See also

  • firecrawl-scrape — scrape individual pages

  • firecrawl-map — discover URLs before deciding to crawl

  • firecrawl-download — download site to local files (uses map + scrape)

  • firecrawl-build-scrape — building bulk extraction into an app instead of running it here

Files

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Declares allowed tools

Security

No issues found by our static scan· No issues found

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by
Firecrawl
Publisher
Firecrawl (Official)
Source
firecrawl/skills · skills/core/firecrawl-crawl · commit 458d9d0
Licence
ISC License (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced

AutoGineer is not affiliated with or endorsed by Firecrawl.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download firecrawl-crawl.zip