Earth2studio Create Datasource

NVIDIAOfficialCreated by NVIDIA Earth-2 Team

Skill
Workflow
Software development
Apache-2.0
Medium risk
Has scripts

From the author

Create and validate Earth2Studio data source wrappers (DataSource, ForecastSource, DataFrameSource, ForecastFrameSource) from remote stores. Do NOT use for fetching data with existing sources, model inference, or installation tasks.

— the description by NVIDIA Earth-2 Team

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured, 14-step procedure for developers to integrate new remote data sources into the Earth2Studio framework. It covers the entire lifecycle from initial analysis of the remote store, through implementation of the appropriate source wrapper (DataSource, ForecastSource, DataFrameSource, or ForecastFrameSource), to testing, documentation, and PR submission. The workflow emphasizes asynchronous data fetching patterns, proper lexicon mapping, and rigorous validation using sanity-check plots to ensure data integrity before final submission.

What it does

Determines the correct Earth2Studio source wrapper type based on data characteristics.
Guides the implementation of asynchronous data fetching logic using fsspec.
Facilitates the creation of lexicon mappings to standardize variable names.
Automates the generation of skeleton files for source code and unit tests.
Provides templates for variable validation and sanity-check visualization.
Assists with PR creation and automated code review triage.

When to use it

When you need to connect a new remote data store (S3, GCS, Azure, HTTP, HuggingFace) to Earth2Studio.
When implementing custom data ingestion logic for gridded or sparse climate data.
When you need to ensure your new data source meets project standards for testing, linting, and documentation.
When you require a systematic approach to validate variable coverage and data quality.

Advantages

Ensures consistency across all Earth2Studio data source implementations.
Promotes high code quality through mandatory linting, formatting, and 90% test coverage.
Reduces development time by providing standardized skeletons and async utility patterns.
Prevents common errors by enforcing a clear validation and sanity-check process.

Usage examples

“Add a new data source for the NOAA GFS analysis stored on S3.”: The agent will initiate the 14-step workflow, starting by analyzing the S3 bucket, determining the appropriate DataSource type, and guiding you through the implementation, testing, and PR submission process.

Install

Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/nvidia/earth2studio-create-datasource/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip earth2studio-create-datasource.zip -d .agents/skills

Documentation by NVIDIA Earth-2 Team

Create and Validate Data Source

Purpose

End-to-end workflow for implementing a new Earth2Studio data source wrapper that connects a remote data store (S3, GCS, Azure, HTTP, HuggingFace) to Earth2Studio's async data fetching infrastructure — from analysis through implementation, testing, validation, and PR submission.

Prerequisites

Earth2Studio dev environment with uv (uv run python must work)
Git configured with fork (origin) and upstream (upstream) remotes
Access to the target remote data store (credentials if private)
Python 3.10+

Workspace

Use the directory containing pyproject.toml. For Harbor evals, write to /workspace/output/ preserving paths. Never read evals/targets/.

Instructions

Python Environment: Always use uv run python or the local .venv. Never use the system Python directly.

Follow every step in order.

[CONFIRM] gates: Only Step 1 (Source Type) and Step 12 (Sanity-Check Plots) require explicit user approval. All other [CONFIRM] markers are advisory — present decisions inline and proceed without blocking.
Deliverables first: Write the source file and test file (Steps 6–7) before extended exploration, documentation, registration, CHANGELOG, or PR work. Skip Steps 8–14 when the user asks for implementation only.
Before you finish: Run verification commands in the repo root so results appear in the session log:
uv run pytest test/data/test_<source>.py -x
make format && make lint
Be concise: Avoid long architecture reports; summarize decisions in a few sentences and move on to file writes.
Hangs or User Feedback If agent becomes stuck or user provides a correction during this skills use, conservatively review relevant part of the skill and improve. Be concise.
One source type per invocation. Invoke again for companion types.

Reference Files

Load these on demand during the relevant steps:

File	Content	Load at
`references/implementation-guide.py`	Skeleton source with FILL comments	Steps 3–10
`references/testing-guide.py`	Test skeleton with FILL comments	Step 11
`references/validation-guide.md`	Plot templates, PR body template, Greptile handling	Steps 12–14 (optional, for templates)

Workflow Overview

Step 0: Obtain reference → Step 1: Determine type → Step 2: Dependencies
→ Step 3: Add deps → Step 4: Create lexicon → Step 5: Update vocab/schema
→ Step 6: Create skeleton → Step 7: Implement source → Step 8: Register
→ Step 9: Documentation → Step 10: CHANGELOG → Step 11: Tests
→ Step 12: Validate & plots (user confirms) → Step 13: PR + sanity comment
→ Step 14: Greptile review

Step 0 — Obtain Remote Data Store Reference

If $ARGUMENTS is provided, use it (URL → WebFetch; file path → read).

If empty, ask:

Please provide a URL, API documentation link, or description of the remote data store. This will be used to understand storage format, access pattern, variable inventory, temporal/spatial resolution.

Step 1 — Determine Source Type

Protocol	Returns	Has `lead_time`?	Use
DataSource	`xr.DataArray`	No	Gridded analysis/reanalysis
ForecastSource	`xr.DataArray`	Yes	Gridded forecast
DataFrameSource	`pd.DataFrame`	No	Sparse/station obs
ForecastFrameSource	`pd.DataFrame`	Yes	Sparse forecast obs

Key factors: gridded vs sparse → DataArray vs DataFrame; analysis vs forecast → Source vs ForecastSource.

[CONFIRM — Source Type]

Present recommended type with justification. Ask for confirmation.

Step 2 — Examine Remote Store & Propose Dependencies

Analyze: storage backend, file format, authentication, access pattern, temporal/spatial resolution, variable inventory.

Prefer fsspec:

Backend	Preferred	Avoid
AWS S3	`s3fs` (core dep)	`boto3` directly
GCS	`gcsfs` (core dep)	`google-cloud-storage`
Azure	`adlfs`	`azure-storage-blob`
HTTP	`fsspec` (core dep)	`requests`
HuggingFace	`huggingface_hub` (core dep)	custom scripts

Only fall back to dedicated libraries when fsspec cannot access the store.

Check pyproject.toml — only propose packages not already present. Core deps include: s3fs, gcsfs, fsspec, zarr, netCDF4, h5py, pygrib, huggingface-hub, pandas, pyarrow.

[CONFIRM — Dependencies & Access Pattern]

Present: backend, fsspec filesystem, new packages (with license), auth method.

Step 3 — Add Dependencies

Load references/implementation-guide.py from here through Step 10.

If new packages needed:

uv add --extra data <package>
uv lock
Add optional dependency imports using OptionalDependencyFailure pattern

Step 4 — Create Lexicon Class

Create earth2studio/lexicon/<source_name>.py with:

metaclass=LexiconType
VOCAB: dict[str, str] mapping E2S names → remote keys
get_item(cls, val) returning tuple[str, Callable]
Use :: separator for structured keys

Map remote variables against E2STUDIO_VOCAB (282 entries in earth2studio/lexicon/base.py).

[CONFIRM — Lexicon & Variable Mapping]

Present: class name, key format, full mapping table, modifiers, reference URL.

Step 5 — Update E2STUDIO_VOCAB / SCHEMA (if needed)

New vocab: surface = descriptive abbrev; pressure = {name}{level}
New schema fields: DataFrame sources only, check E2STUDIO_SCHEMA

[CONFIRM — Vocabulary & Schema Updates]

Skip if no updates needed.

Step 6 — Create Skeleton Data Source File

Follow canonical method ordering:

Class constants
SCHEMA
__init__
_async_init
__call__
fetch
_create_tasks
fetch_wrapper
fetch_array
_validate_time
Helpers
cache property
available classmethod

Use async task dataclass pattern for parallel execution.

[CONFIRM — Skeleton]

Present: class name, file path, skeleton code, task dataclass.

Step 7 — Implement the Data Source & Tests

The test file is a co-equal deliverable. Create test/data/test_<filename>.py alongside the source. Add test_<source>_call_mock for async sources.

Sync sources: Use prep_data_inputs/prep_forecast_inputs, direct __call__.

Async sources: See references/implementation-guide.py for required patterns: _sync_async, managed_session, gather_with_concurrency, async_retry, pure async I/O, try/finally cleanup. Constructor params: cache=True, verbose=True, async_timeout=600, async_workers=16, retries=3. DataFrame sources add time_tolerance.

Step 8 — Register the Source

earth2studio/data/__init__.py — alphabetical import
earth2studio/lexicon/__init__.py — alphabetical import
Verify pyproject.toml deps

Step 9 — Update Documentation

Add to correct RST file (datasources_analysis.rst / _forecast.rst / _dataframe.rst)
Class docstring: Parameters, Warning (download size), Note (reference URLs), Badges (last)
All public methods: NumPy-style docstrings

Step 10 — Update CHANGELOG.md

Add entry under the current unreleased version. See references/implementation-guide.py REGISTRATION CHECKLIST for the format.

One line per source. Do NOT add separate lexicon entries.

Step 11 — Verify Style & Expand Tests

Run make format && make lint && make license. Load references/testing-guide.py for test skeletons. Required tests: test_<source>_fetch (slow), _cache (slow), _call_mock, _exceptions, _available. Target 90%+ coverage with --slow.

[CONFIRM — Tests]

Present test file, functions, coverage.

Step 12 — Validate Variables & Sanity-Check

Validate all lexicon vars against real data (run script, do NOT commit)
Remove variables with < 10% valid data
Create sanity-check plot (gridded or sparse template)
Tell user the plot path and ask for visual confirmation

[CONFIRM — Sanity-Check Plots]

User MUST visually inspect plots. Do not proceed without confirmation.

Step 13 — Branch, Commit & Open PR

Create branch feat/data-source-<name>
Commit (do NOT add sanity-check script/images)
Push to fork
gh pr create --repo NVIDIA/earth2studio
Immediately post sanity-check validation as PR comment with:
- Variable coverage table (name, count, range, unit)
- Data validation summary (regions, storms/stations, time range)
- Key findings (physically reasonable values, conversions verified)
- Full validation script in <details> block
- Image placeholder:

[CONFIRM — Ready to Submit]

Verify all steps complete before creating PR.

Step 14 — Automated Code Review

Poll for Greptile review (5 min timeout)
Categorize feedback (bug/style/perf/docs/suggestion/false-positive)
Present triage table to user
Implement accepted fixes
Respond to PR comments
Push

[CONFIRM — Review Triage]

User approves which comments to address.

Examples

User: Add a data source for the NOAA GFS analysis on S3
Agent: [loads skill, proceeds through Steps 0–14]

Limitations

One source type per invocation
Requires network access for validation (Step 12)
SPDX license headers required in all files

Reminders

DO: uv run python, loguru.logger, alphabetical order in __init__.py/RST/CHANGELOG, canonical method ordering, async utilities (managed_session, gather_with_concurrency, async_retry), pure async I/O, reference URLs in docstrings, try/finally cleanup.

AVOID: asyncio.to_thread, bare tqdm.gather, xarray for loading, full file downloads.

NEVER: loop.set_default_executor(), commit secrets, commit sanity-check scripts/images.

Files

ATTRIBUTION.mdAdded by AutoGineer961 B
BENCHMARK.md2.4 KiB
LICENSELicence file added by AutoGineer11.2 KiB
SKILL.md10.4 KiB
evals/config.yml1.1 KiB
evals/environment/Dockerfile867 B
evals/environment/setup/bootstrap.shScript934 B
evals/evals.json6.1 KiB
evals/targets/eval_1_target.pyScript2.5 KiB
evals/targets/eval_2_target.pyScript2.9 KiB
evals/targets/eval_3_target.pyScript3.8 KiB
evals/targets/eval_4_target.pyScript11.5 KiB
evals/targets/test/eval_1_test_target.pyScript2.4 KiB
evals/targets/test/eval_2_test_target.pyScript3.1 KiB
evals/targets/test/eval_3_test_target.pyScript3.7 KiB
evals/targets/test/eval_4_test_target.pyScript4.5 KiB
references/implementation-guide.pyScript17.3 KiB
references/testing-guide.pyScript10.1 KiB
references/validation-guide.md9.5 KiB
skill-card.md2.6 KiB
skill.oms.sig7.9 KiB

references/validation-guide.md

Validation Guide

Reference for post-implementation validation, PR submission, and automated review. Load this after implementation and tests are complete.

Run Tests
Validate Variables
Sanity-Check Plots
Branch, Commit and Open PR
Automated Code Review (Greptile)

Run Tests

Run the new test file

uv run python -m pytest test/data/test_<filename>.py -v --timeout=60

All tests must pass (or xfail for network tests).

Run coverage with `--slow`

The source file must achieve at least 90% line coverage:

uv run python -m pytest test/data/test_<filename>.py -v \
    --slow --timeout=300 \
    --cov=earth2studio/data/<filename> \
    --cov-report=term-missing \
    --cov-fail-under=90

If below 90%, add tests for:

Error handling branches
Edge cases in parsing
Cache property paths (cache=True vs cache=False)
resolve_fields with different input types

Full data test suite (optional)

make pytest TOX_ENV=test-data

Validate Variables

Every variable in the lexicon must be validated against real data.

Run a validation script (do NOT commit):

"""Variable validation for <SourceName>."""
from datetime import datetime
from earth2studio.data import SourceName
from earth2studio.lexicon import SourceNameLexicon

ds = SourceName(cache=True)
time = ...  # pick a recent valid time
all_vars = list(SourceNameLexicon.VOCAB.keys())
df = ds(time, all_vars)

print(f"{'Variable':<16} {'Obs Count':>10} {'Valid %':>8} {'Min':>10} {'Max':>10}")
print("-" * 60)
for var in sorted(all_vars):
    sub = df[df["variable"] == var]
    n_total = len(sub)
    n_valid = sub["observation"].notna().sum()
    pct = (n_valid / n_total * 100) if n_total > 0 else 0
    vmin = sub["observation"].min() if n_valid > 0 else float("nan")
    vmax = sub["observation"].max() if n_valid > 0 else float("nan")
    flag = " *** REMOVE" if pct < 10 else ""
    print(f"{var:<16} {n_total:>10} {pct:>7.1f}% {vmin:>10.2f} {vmax:>10.2f}{flag}")

Actions:

Variables with < 10% valid data: Remove from lexicon VOCAB, E2STUDIO_VOCAB, and document
Variables with 10-50% valid data: Keep but document low coverage reason
After removing, re-run make lint and tests

Summarize to user

Present a summary table covering:

All valid variables — name, description, observation count, value range
Removed variables — name, reason
Valid time range
Typical data density

Sanity-Check Plots

Create a standalone script (do NOT commit).

Gridded template (DataSource / ForecastSource)

"""Sanity-check plot for <SourceName>."""
import cartopy.crs as ccrs
import cartopy.feature as cfeature
import matplotlib.pyplot as plt
import numpy as np
from earth2studio.data import SourceName

ds = SourceName(cache=False)
time = ...
variables = ["t2m", "msl", "u10m"]
data = ds(time, variables)

fig, axes = plt.subplots(
    1, len(variables), figsize=(6 * len(variables), 5),
    subplot_kw={"projection": ccrs.PlateCarree()},
)
if len(variables) == 1:
    axes = [axes]

for ax, var in zip(axes, variables):
    arr = data.sel(variable=var).isel(time=0)
    ax.set_global()
    ax.add_feature(cfeature.COASTLINE, linewidth=0.5)
    ax.add_feature(cfeature.BORDERS, linewidth=0.3, linestyle=":")
    im = ax.pcolormesh(
        arr.lon, arr.lat, arr.values,
        cmap="turbo", transform=ccrs.PlateCarree(),
    )
    ax.set_title(f"{var}")
    plt.colorbar(im, ax=ax, shrink=0.6, orientation="horizontal", pad=0.05)

plt.suptitle(f"<SourceName> — {time}", y=1.02)
plt.tight_layout()
plt.savefig("sanity_check_<source_name>.png", dpi=150, bbox_inches="tight")

Sparse template (DataFrameSource / ForecastFrameSource)

"""Sanity-check plot for <SourceName>."""
import cartopy.crs as ccrs
import cartopy.feature as cfeature
import matplotlib.pyplot as plt
import numpy as np
from earth2studio.data import SourceName

ds = SourceName(cache=False)
time = ...
variables = ["var1", "var2"]
df = ds(time, variables)

df["lon_plt"] = df["lon"].where(df["lon"] <= 180, df["lon"] - 360)

fig, axes = plt.subplots(
    1, len(variables), figsize=(8 * len(variables), 5),
    subplot_kw={"projection": ccrs.Robinson()},
)
if len(variables) == 1:
    axes = [axes]

for ax, var in zip(axes, variables):
    subset = df[df["variable"] == var]
    obs = subset["observation"].values
    vmin, vmax = np.percentile(obs[np.isfinite(obs)], [2, 98])
    ax.set_global()
    ax.add_feature(cfeature.COASTLINE, linewidth=0.5)
    sc = ax.scatter(
        subset["lon_plt"], subset["lat"], c=obs, s=2, cmap="turbo",
        alpha=0.8, vmin=vmin, vmax=vmax, edgecolors="none",
        transform=ccrs.PlateCarree(),
    )
    ax.set_title(f"{var} ({len(subset)} obs)")
    plt.colorbar(sc, ax=ax, shrink=0.6, orientation="horizontal", pad=0.05)

plt.tight_layout()
plt.savefig("sanity_check_<source_name>.png", dpi=150, bbox_inches="tight")

User confirmation required

Tell the user the absolute path to the plot and ask them to visually confirm:

Data points visible (not blank/empty)
Geographic coverage matches expectations
Colorbar values in physically reasonable ranges
No obvious artifacts

Do not proceed until user confirms.

Branch, Commit and Open PR

Create branch and commit

git checkout -b feat/data-source-<name>
git add earth2studio/data/<filename>.py \
        earth2studio/data/__init__.py \
        earth2studio/lexicon/<filename>.py \
        earth2studio/lexicon/__init__.py \
        earth2studio/lexicon/base.py \
        test/data/test_<filename>.py \
        docs/modules/datasources_*.rst \
        pyproject.toml \
        CHANGELOG.md
git commit -m "feat: add <SourceName> data source

Add <SourceName> <source_type> for <brief description>.
Includes lexicon, unit tests, and documentation."

Do NOT add sanity-check script or images.

Push to fork

git remote -v  # identify fork remote
git push -u <fork-remote> feat/data-source-<name>

Open PR (fork → NVIDIA/earth2studio)

gh pr create \
  --repo NVIDIA/earth2studio \
  --base main \
  --head <fork-owner>:feat/data-source-<name> \
  --title "feat: add <SourceName> data source" \
  --body "..."

PR body template

You MUST include all sections below. The PR body is the primary record of data licensing and dependency changes for legal review.

## Description

Add `<ClassName>` <source_type> for <brief description>.

### Data source details

| Property | Value |
|---|---|
| **Source type** | DataSource / ForecastSource / DataFrameSource / ForecastFrameSource |
| **Remote store** | <URL> |
| **Format** | GRIB2 / NetCDF / Zarr / etc. |
| **Spatial resolution** | X deg x Y deg (or "Point observations" for sparse) |
| **Temporal resolution** | Hourly / 6-hourly / daily |
| **Date range** | YYYY-MM-DD to present |
| **Authentication** | Anonymous / API key |

### Data licensing

> **License**: <Name> (e.g., Public Domain, CC-BY-4.0, Apache-2.0)
> **URL**: <Link to license or data policy page>
>
> <Brief summary of permissions/restrictions>

### Dependencies added

<!-- If no new dependencies, write: "No new dependencies required. Uses existing `<pkg1>` and `<pkg2>`." -->

| Package | Version | License | License URL | Reason |
|---|---|---|---|---|
| `<pkg>` | `>=X.Y` | <License> | [link](<URL>) | <reason> |

## Checklist

- [x] New or existing tests cover these changes.
- [x] The documentation is up to date.
- [x] The CHANGELOG.md is up to date.
- [ ] Assess and address Greptile feedback.

Post sanity-check as PR comment

Post immediately after creating PR (before Greptile review). Use gh pr comment:

gh pr comment <PR_NUMBER> --repo NVIDIA/earth2studio --body "..."

Required content:

Variable coverage table — name, obs count, value range, unit
Data validation summary — regions/stations, time range, key statistics
Key findings — physically reasonable values, unit conversions verified
Full validation script in <details> block
Image placeholder for user to drag-and-drop:

### Sanity-Check Plot

<!-- Drag and drop sanity-check image here -->

Automated Code Review (Greptile)

Wait for review

Poll every 30s for up to 5 minutes:

for i in $(seq 1 10); do
  REVIEW_ID=$(gh api repos/NVIDIA/earth2studio/pulls/<PR_NUMBER>/reviews \
    --jq '.[] | select(.user.login == "greptile-apps[bot]") | .id' 2>/dev/null)
  if [ -n "$REVIEW_ID" ]; then break; fi
  sleep 30
done

Categorize feedback

Category	Default action
Bug / correctness	Fix
Style / convention	Fix if valid
Performance	Evaluate
Documentation	Fix
Suggestion	User decides
False positive	Dismiss

Present to user

Show summary table with file, line, category, summary, and proposed action. Ask user to confirm which comments to address.

Implement fixes

Make code changes
Run make format && make lint
Run tests
Commit: fix: address code review feedback (Greptile)

Respond to comments

Fixed:

gh api repos/NVIDIA/earth2studio/pulls/<PR_NUMBER>/comments/<COMMENT_ID>/replies \
  -f body="Fixed in <commit_sha>. <description>"

Dismissed:

gh api repos/NVIDIA/earth2studio/pulls/<PR_NUMBER>/comments/<COMMENT_ID>/replies \
  -f body="Won't fix — <justification>"

Push and report

git push origin <branch>

Report to user: comments fixed, dismissed, and any remaining open threads.

Requirements and compatibility

Written for: Claude Code

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Claude Code frontmatter, Names Claude Code tools, Needs Python

Security

What our scanner noticed· Medium risk

Large base64-encoded blobs
Scripts that your agent may run

This skill includes scripts that your agent may run. Read them before you use it.

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by

NVIDIA Earth-2 Team

Publisher

NVIDIA (Official)

Source

NVIDIA/skills · skills/earth2studio-create-datasource · commit fce3cca

Licence

Apache License 2.0 (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE-APACHE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-08

AutoGineer is not affiliated with or endorsed by NVIDIA Earth-2 Team or NVIDIA.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download earth2studio-create-datasource.zip