Basin
CloudflareOfficialPublished by Cloudflare
- Skill
- Tool integration
- Data and AI
- Apache-2.0
- No issues found
AutoGineer summary
Written by AutoGineer from the skill’s own files, not by the author
Basin is a suite of tools for ingesting, transforming, and querying data using Apache Iceberg tables stored in Cloudflare R2. It comprises three main components: Basin Pipelines for event ingestion and transformation, Basin Catalog for managing Iceberg metadata and table maintenance, and Basin SQL for performing analytical queries. This skill provides guidance on setting up these components, configuring connections for engines like Spark or Python, and troubleshooting common operational issues.
What it does
- Stream events and transform data rows for delivery to R2 storage.
- Manage Iceberg table metadata, maintenance tasks, and engine access.
- Execute analytical SQL queries against Iceberg tables.
- Configure and authenticate connections for Basin Pipelines, Catalog, and SQL.
- Provide guidance on using Wrangler commands for Basin resource management.
When to use it
- When setting up new data ingestion pipelines that target R2 storage.
- When you need to manage Iceberg table metadata or perform table maintenance.
- When querying large datasets stored in R2 using analytical SQL.
- When migrating from legacy Cloudflare Data Platform or R2 Data Catalog workflows.
Advantages
- Unified management of data ingestion, cataloging, and querying.
- Native integration with Cloudflare R2 and Apache Iceberg standards.
- Supports standard analytical SQL for querying object storage data.
- Provides clear paths for both control-plane administration and engine-specific operations.
Usage examples
- “How do I set up a new Basin Pipeline to ingest events into R2?”
- The agent will guide you through the Basin Pipelines configuration, including defining event transformations and setting up the delivery destination in R2.
- “What are the steps to connect PyIceberg to my Basin Catalog?”
- The agent will provide the necessary configuration steps, including retrieving the Catalog URI and Warehouse name, and setting up the required credentials.
Install
Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/cloudflare/basin/downloadUnzip it into the cross-client skills folder of your project:
mkdir -p .agents/skills && unzip basin.zip -d .agents/skills
Documentation by Cloudflare
Cloudflare Basin
Basin ingests and transforms events, manages Apache Iceberg tables in R2, and queries them with distributed SQL. Start with the Basin overview and retrieve the relevant current docs before implementing changes, especially for limits, pricing, permissions, and CLI syntax. While PR #33850 remains open, use the proposed pages in that PR if the new URLs have not been published.
Rebrand
Cloudflare Data Platform is now Basin. Cloudflare Pipelines, R2 Data Catalog, and R2 SQL are now Basin Pipelines, Basin Catalog, and Basin SQL. Existing resources and configurations continue to work. The prior Wrangler command families also continue to work: wrangler pipelines, wrangler r2 bucket catalog, and wrangler r2 sql. Use the new Basin commands and documentation URLs in new guidance. Legacy API identifiers and metrics names may still use the old terms; check the current docs for deprecation details. The migration is documented in cloudflare-docs PR #33850.
Choose the workflow
| Need | Product | Reference | Current docs |
|---|---|---|---|
| Receive events, transform rows, and deliver to R2 | Basin Pipelines | Pipelines guide | Basin Pipelines |
| Manage Iceberg metadata, table maintenance, and engine access | Basin Catalog | Catalog guide | Basin Catalog |
| Query Iceberg tables with analytical SQL | Basin SQL | SQL guide | Basin SQL |
Typical flow: Basin Pipelines → Basin Catalog tables in R2 → Basin SQL or a compatible external engine. Begin with the Basin getting started guide for an end-to-end setup.
The proposed Wrangler command families are wrangler basin pipelines, wrangler basin catalog, and wrangler basin sql. Basin SQL uses WRANGLER_BASIN_SQL_AUTH_TOKEN; its REST query path is /basin-sql/query/{BUCKET}. Check the installed Wrangler version and current reference pages before running commands because the migration is ongoing.
Skill feedback
If the user reports that this skill gave incorrect or outdated guidance, or is missing information, read references/skill-feedback.md.
Files
- ATTRIBUTION.mdAdded by AutoGineer846 B
- LICENSELicence file added by AutoGineer11.1 KiB
- SKILL.md2.9 KiB
- references/catalog/README.md1.2 KiB
- references/catalog/api.md3.3 KiB
- references/catalog/configuration.md2.1 KiB
- references/catalog/gotchas.md2.2 KiB
- references/catalog/patterns.md1.7 KiB
- references/pipelines/README.md4.5 KiB
- references/pipelines/api.md5.0 KiB
- references/pipelines/configuration.md6.1 KiB
- references/pipelines/gotchas.md3.0 KiB
- references/pipelines/patterns.md4.6 KiB
- references/skill-feedback.md2.7 KiB
- references/sql/README.md3.4 KiB
- references/sql/api.md4.8 KiB
- references/sql/configuration.md1.7 KiB
- references/sql/gotchas.md2.0 KiB
- references/sql/patterns.md4.3 KiB
references/pipelines/api.md
Basin Pipelines API Reference
Code templates and verified behavior. For the full SQL function set and HTTP status semantics, pull https://developers.cloudflare.com/basin-pipelines/sql-reference/index.md and the streams docs.
Worker Binding Interface
// from cloudflare:pipelines / @cloudflare/workers-types
interface Pipeline<T = any> { send(records: T[]): Promise<void>; }
interface Env { MY_STREAM: Pipeline; }
export default {
async fetch(req: Request, env: Env, ctx: ExecutionContext): Promise<Response> {
await env.MY_STREAM.send([{ event_id: crypto.randomUUID(), amount: 29.99 }]);
return new Response("OK");
}
} satisfies ExportedHandler<Env>;send()takes an array, returnsPromise<void>(no confirmation payload).Throws on network errors — wrap in try/catch or use
ctx.waitUntil()for fire-and-forget.Validation errors are not thrown here (deferred during processing — see gotchas.md).
Payload/rate limits apply — check
https://developers.cloudflare.com/basin-pipelines/platform/limits/index.mdbefore sizing batches.
HTTP Ingest
https://{stream-id}.ingest.cloudflare.comGet {stream-id} from npx wrangler basin pipelines streams list.
# Batch (preferred)
curl -X POST https://{stream-id}.ingest.cloudflare.com \
-H "Content-Type: application/json" \
-d '[{"event_id":"evt-1","amount":29.99},{"event_id":"evt-2","amount":14.99}]'
# Single event — auto-wrapped in an array
curl -X POST https://{stream-id}.ingest.cloudflare.com \
-H "Content-Type: application/json" -d '{"event_id":"evt-3","amount":9.99}'If stream auth is enabled, add -H "Authorization: Bearer $TOKEN" (token needs Workers Pipelines Send). Standard HTTP status codes apply (400 invalid, 401 auth, 413 too large, 429 rate-limited, 5xx retry).
JSON only — no Avro, Protobuf, or CSV input.
REST Management API
Base: https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/pipelines/v1
# List
curl -s "$BASE_URL/streams" -H "Authorization: Bearer $API_TOKEN"
curl -s "$BASE_URL/sinks" -H "Authorization: Bearer $API_TOKEN"
curl -s "$BASE_URL/pipelines" -H "Authorization: Bearer $API_TOKEN"
# Get one (pipeline GET includes status + failure_reason — useful for debugging)
curl -s "$BASE_URL/pipelines/{pipeline-id}" -H "Authorization: Bearer $API_TOKEN"
# Delete in reverse order: pipeline → sink → stream
curl -X DELETE "$BASE_URL/pipelines/{id}" -H "Authorization: Bearer $API_TOKEN"
curl -X DELETE "$BASE_URL/sinks/{id}" -H "Authorization: Bearer $API_TOKEN"
curl -X DELETE "$BASE_URL/streams/{id}" -H "Authorization: Bearer $API_TOKEN"
wrangler basin pipelines deletedefaults to "no" non-interactively — use the REST API for automated cleanup. Deleting a stream removes buffered events and dependent pipelines.
Pipeline Lifecycle States
| Status | Meaning |
|---|---|
running | Active, processing events |
initializing | Starting up (minutes after creation or recovery) |
failed | Stopped on error — check failure_reason (expired token, deleted bucket, disabled catalog) |
A
GETon a sink showsschema.fields: []— expected. The sink inherits schema from the stream via the pipeline SQL.
Pipeline SQL (Transforms)
Row-level only — no GROUP BY/aggregation. CTEs (WITH) and UNNEST are supported. Full function list: https://developers.cloudflare.com/basin-pipelines/sql-reference/index.md.
-- Passthrough / filter / enrich
INSERT INTO my_sink SELECT * FROM my_stream;
INSERT INTO my_sink SELECT * FROM my_stream WHERE amount > 10;
INSERT INTO my_sink
SELECT event_id, UPPER(category) AS category, amount * 1.1 AS amount_with_tax
FROM my_stream;
-- CTE
WITH filtered AS (SELECT event_id, amount FROM my_stream WHERE amount > 50)
INSERT INTO my_sink SELECT * FROM filtered;
-- UNNEST arrays (one per SELECT)
SELECT UNNEST(tags) AS tag FROM my_stream;Supported categories: string, regex, hashing (sha256), JSON extraction, timestamp conversion, conditional (CASE), CAST, COALESCE, math/comparison operators.
Verifying End-to-End Data Flow
# 1. Pipeline running (not initializing/failed)?
curl -s "$BASE_URL/pipelines/{id}" -H "Authorization: Bearer $API_TOKEN"
# 2. Table created yet? (3–7 min on first flush)
curl -s "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/r2-catalog/$BUCKET/namespaces/my_ns/tables" \
-H "Authorization: Bearer $API_TOKEN"
# 3. Data present? (Basin SQL)
curl -s -X POST \
"https://api.sql.cloudflarestorage.com/api/v1/accounts/$ACCOUNT_ID/basin-sql/query/$BUCKET" \
-H "Authorization: Bearer $API_TOKEN" -H "Content-Type: application/json" \
-d '{"query": "SELECT COUNT(*) AS total FROM my_ns.my_table"}'Expect 3–7 minutes from first send to first queryable data. Subsequent flushes are much faster.
See Also
configuration.md — creating resources · patterns.md — producers, Logpush, observability
sql/api.md — querying results
Requirements and compatibility
Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta
“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.
Security
No issues found by our static scan· No issues found
A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.
Attribution and licence
- Published by
- Cloudflare
- Publisher
- Cloudflare (Official)
- Source
- cloudflare/skills · skills/basin · commit 6eb6c78
- Licence
- Apache License 2.0 (redistribution allowed)
- LICENSE (added by AutoGineer from LICENSE)
- ATTRIBUTION.md (written by AutoGineer)
- Synced
AutoGineer is not affiliated with or endorsed by Cloudflare.
Download ZIP
The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download basin.zip