AutoGineer
Generated banner for Firecrawl Crawl
Generated logo for Firecrawl Crawl

Firecrawl Crawl

FirecrawlOfficialPublished by Firecrawl

  • Skill
  • Tool integration
  • Automation and integrations
  • ISC
  • No issues found
Download ZIP

From the author

Bulk-extract many pages from one site or section. Use for "crawl", "everything under /docs", or content spanning linked pages.
— the description by Firecrawl

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

The firecrawl-crawl skill provides a command-line interface for performing bulk web data extraction. It allows users to crawl entire websites or specific sub-sections by following links up to a defined depth or page limit. The tool supports both synchronous and asynchronous execution, enabling users to either wait for results or manage long-running jobs via job IDs. It is designed to integrate into workflows where structured content from multiple pages is required for further processing.

What it does

  • Performs bulk extraction of web pages from a specified URL.
  • Supports path-based scoping to limit crawls to specific site sections.
  • Allows configuration of crawl depth and total page limits.
  • Provides options for synchronous waiting or asynchronous job management.
  • Saves crawl results directly to local JSON files.
  • Includes progress tracking for active crawl operations.

When to use it

  • When you need to extract content from multiple pages across a website.
  • When you want to target specific documentation or site sections.
  • When you need to automate the collection of data spanning linked pages.
  • When you prefer a command-line interface for managing large-scale web scraping tasks.

Advantages

  • Efficiently handles large-scale data extraction across multiple pages.
  • Flexible scoping options prevent unnecessary data collection.
  • Supports both immediate and background job processing.
  • Integrates easily into terminal-based automation workflows.

Usage examples

“Crawl the documentation section of example.com and save to a file.”
The agent executes the crawl command with the --include-paths /docs flag, limiting the scope and saving the output to the specified JSON file.
“Perform a deep crawl of a website with a limit of 50 pages.”
The agent runs the crawl with --max-depth and --limit parameters, providing progress updates until the task completes.

Install

  1. Download the skill as a ZIP:

    https://mp-staging.autogineer.com/agent-skills/firecrawl/firecrawl-crawl/download

  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip firecrawl-crawl.zip -d .agents/skills

Documentation by Firecrawl

firecrawl crawl

Bulk extract content from a website. Crawls pages following links up to a depth/limit.

Prerequisite: crawl requires authentication (no keyless free tier); without credentials the CLI prompts an interactive login.

Quick start

# Crawl a docs section
firecrawl crawl "<url>" --include-paths /docs --limit 50 --wait -o .firecrawl/crawl.json

# Full crawl with depth limit
firecrawl crawl "<url>" --max-depth 3 --wait --progress -o .firecrawl/crawl.json

# Check status of a running crawl
firecrawl crawl <job-id>

Run firecrawl crawl --help for the full option list.

Done when: the crawl reaches a terminal status and the saved output under .firecrawl/ contains the expected pages.

Tips

  • Use --wait when you need the results immediately. It has no default timeout; use --timeout <seconds> to bound polling. Without --wait, crawl returns a job ID for async polling.

  • Scope crawls with --include-paths whenever the request names a section — crawl only the pages you need.

  • Crawl consumes credits per page. Check firecrawl credit-usage before large crawls (credit-usage requires authentication).

See also

  • firecrawl-scrape — scrape individual pages

  • firecrawl-map — discover URLs before deciding to crawl

  • firecrawl-download — download site to local files (uses map + scrape)

  • firecrawl-build-scrape — building bulk extraction into an app instead of running it here

Files

LICENSE

ISC License

Copyright (c) Firecrawl

Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.

THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH
REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT,
INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM
LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR
OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
PERFORMANCE OF THIS SOFTWARE.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Declares allowed tools

Security

No issues found by our static scan· No issues found

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by
Firecrawl
Publisher
Firecrawl (Official)
Source
firecrawl/skills · skills/core/firecrawl-crawl · commit 458d9d0
Licence
ISC License (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced

AutoGineer is not affiliated with or endorsed by Firecrawl.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download firecrawl-crawl.zip