Earth2studio Create Datasource

NVIDIAOfficialCreated by NVIDIA Earth-2 Team

Skill
Workflow
Software development
Apache-2.0
Medium risk
Has scripts

From the author

Create and validate Earth2Studio data source wrappers (DataSource, ForecastSource, DataFrameSource, ForecastFrameSource) from remote stores. Do NOT use for fetching data with existing sources, model inference, or installation tasks.

— the description by NVIDIA Earth-2 Team

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured, 14-step procedure for developers to integrate new remote data sources into the Earth2Studio framework. It covers the entire lifecycle from initial analysis of the remote store, through implementation of the appropriate source wrapper (DataSource, ForecastSource, DataFrameSource, or ForecastFrameSource), to testing, documentation, and PR submission. The workflow emphasizes asynchronous data fetching patterns, proper lexicon mapping, and rigorous validation using sanity-check plots to ensure data integrity before final submission.

What it does

Determines the correct Earth2Studio source wrapper type based on data characteristics.
Guides the implementation of asynchronous data fetching logic using fsspec.
Facilitates the creation of lexicon mappings to standardize variable names.
Automates the generation of skeleton files for source code and unit tests.
Provides templates for variable validation and sanity-check visualization.
Assists with PR creation and automated code review triage.

When to use it

When you need to connect a new remote data store (S3, GCS, Azure, HTTP, HuggingFace) to Earth2Studio.
When implementing custom data ingestion logic for gridded or sparse climate data.
When you need to ensure your new data source meets project standards for testing, linting, and documentation.
When you require a systematic approach to validate variable coverage and data quality.

Advantages

Ensures consistency across all Earth2Studio data source implementations.
Promotes high code quality through mandatory linting, formatting, and 90% test coverage.
Reduces development time by providing standardized skeletons and async utility patterns.
Prevents common errors by enforcing a clear validation and sanity-check process.

Usage examples

“Add a new data source for the NOAA GFS analysis stored on S3.”: The agent will initiate the 14-step workflow, starting by analyzing the S3 bucket, determining the appropriate DataSource type, and guiding you through the implementation, testing, and PR submission process.

Install

Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/nvidia/earth2studio-create-datasource/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip earth2studio-create-datasource.zip -d .agents/skills

Documentation by NVIDIA Earth-2 Team

Create and Validate Data Source

Purpose

End-to-end workflow for implementing a new Earth2Studio data source wrapper that connects a remote data store (S3, GCS, Azure, HTTP, HuggingFace) to Earth2Studio's async data fetching infrastructure — from analysis through implementation, testing, validation, and PR submission.

Prerequisites

Earth2Studio dev environment with uv (uv run python must work)
Git configured with fork (origin) and upstream (upstream) remotes
Access to the target remote data store (credentials if private)
Python 3.10+

Workspace

Use the directory containing pyproject.toml. For Harbor evals, write to /workspace/output/ preserving paths. Never read evals/targets/.

Instructions

Python Environment: Always use uv run python or the local .venv. Never use the system Python directly.

Follow every step in order.

[CONFIRM] gates: Only Step 1 (Source Type) and Step 12 (Sanity-Check Plots) require explicit user approval. All other [CONFIRM] markers are advisory — present decisions inline and proceed without blocking.
Deliverables first: Write the source file and test file (Steps 6–7) before extended exploration, documentation, registration, CHANGELOG, or PR work. Skip Steps 8–14 when the user asks for implementation only.
Before you finish: Run verification commands in the repo root so results appear in the session log:
uv run pytest test/data/test_<source>.py -x
make format && make lint
Be concise: Avoid long architecture reports; summarize decisions in a few sentences and move on to file writes.
Hangs or User Feedback If agent becomes stuck or user provides a correction during this skills use, conservatively review relevant part of the skill and improve. Be concise.
One source type per invocation. Invoke again for companion types.

Reference Files

Load these on demand during the relevant steps:

File	Content	Load at
`references/implementation-guide.py`	Skeleton source with FILL comments	Steps 3–10
`references/testing-guide.py`	Test skeleton with FILL comments	Step 11
`references/validation-guide.md`	Plot templates, PR body template, Greptile handling	Steps 12–14 (optional, for templates)

Workflow Overview

Step 0: Obtain reference → Step 1: Determine type → Step 2: Dependencies
→ Step 3: Add deps → Step 4: Create lexicon → Step 5: Update vocab/schema
→ Step 6: Create skeleton → Step 7: Implement source → Step 8: Register
→ Step 9: Documentation → Step 10: CHANGELOG → Step 11: Tests
→ Step 12: Validate & plots (user confirms) → Step 13: PR + sanity comment
→ Step 14: Greptile review

Step 0 — Obtain Remote Data Store Reference

If $ARGUMENTS is provided, use it (URL → WebFetch; file path → read).

If empty, ask:

Please provide a URL, API documentation link, or description of the remote data store. This will be used to understand storage format, access pattern, variable inventory, temporal/spatial resolution.

Step 1 — Determine Source Type

Protocol	Returns	Has `lead_time`?	Use
DataSource	`xr.DataArray`	No	Gridded analysis/reanalysis
ForecastSource	`xr.DataArray`	Yes	Gridded forecast
DataFrameSource	`pd.DataFrame`	No	Sparse/station obs
ForecastFrameSource	`pd.DataFrame`	Yes	Sparse forecast obs

Key factors: gridded vs sparse → DataArray vs DataFrame; analysis vs forecast → Source vs ForecastSource.

[CONFIRM — Source Type]

Present recommended type with justification. Ask for confirmation.

Step 2 — Examine Remote Store & Propose Dependencies

Analyze: storage backend, file format, authentication, access pattern, temporal/spatial resolution, variable inventory.

Prefer fsspec:

Backend	Preferred	Avoid
AWS S3	`s3fs` (core dep)	`boto3` directly
GCS	`gcsfs` (core dep)	`google-cloud-storage`
Azure	`adlfs`	`azure-storage-blob`
HTTP	`fsspec` (core dep)	`requests`
HuggingFace	`huggingface_hub` (core dep)	custom scripts

Only fall back to dedicated libraries when fsspec cannot access the store.

Check pyproject.toml — only propose packages not already present. Core deps include: s3fs, gcsfs, fsspec, zarr, netCDF4, h5py, pygrib, huggingface-hub, pandas, pyarrow.

[CONFIRM — Dependencies & Access Pattern]

Present: backend, fsspec filesystem, new packages (with license), auth method.

Step 3 — Add Dependencies

Load references/implementation-guide.py from here through Step 10.

If new packages needed:

uv add --extra data <package>
uv lock
Add optional dependency imports using OptionalDependencyFailure pattern

Step 4 — Create Lexicon Class

Create earth2studio/lexicon/<source_name>.py with:

metaclass=LexiconType
VOCAB: dict[str, str] mapping E2S names → remote keys
get_item(cls, val) returning tuple[str, Callable]
Use :: separator for structured keys

Map remote variables against E2STUDIO_VOCAB (282 entries in earth2studio/lexicon/base.py).

[CONFIRM — Lexicon & Variable Mapping]

Present: class name, key format, full mapping table, modifiers, reference URL.

Step 5 — Update E2STUDIO_VOCAB / SCHEMA (if needed)

New vocab: surface = descriptive abbrev; pressure = {name}{level}
New schema fields: DataFrame sources only, check E2STUDIO_SCHEMA

[CONFIRM — Vocabulary & Schema Updates]

Skip if no updates needed.

Step 6 — Create Skeleton Data Source File

Follow canonical method ordering:

Class constants
SCHEMA
__init__
_async_init
__call__
fetch
_create_tasks
fetch_wrapper
fetch_array
_validate_time
Helpers
cache property
available classmethod

Use async task dataclass pattern for parallel execution.

[CONFIRM — Skeleton]

Present: class name, file path, skeleton code, task dataclass.

Step 7 — Implement the Data Source & Tests

The test file is a co-equal deliverable. Create test/data/test_<filename>.py alongside the source. Add test_<source>_call_mock for async sources.

Sync sources: Use prep_data_inputs/prep_forecast_inputs, direct __call__.

Async sources: See references/implementation-guide.py for required patterns: _sync_async, managed_session, gather_with_concurrency, async_retry, pure async I/O, try/finally cleanup. Constructor params: cache=True, verbose=True, async_timeout=600, async_workers=16, retries=3. DataFrame sources add time_tolerance.

Step 8 — Register the Source

earth2studio/data/__init__.py — alphabetical import
earth2studio/lexicon/__init__.py — alphabetical import
Verify pyproject.toml deps

Step 9 — Update Documentation

Add to correct RST file (datasources_analysis.rst / _forecast.rst / _dataframe.rst)
Class docstring: Parameters, Warning (download size), Note (reference URLs), Badges (last)
All public methods: NumPy-style docstrings

Step 10 — Update CHANGELOG.md

Add entry under the current unreleased version. See references/implementation-guide.py REGISTRATION CHECKLIST for the format.

One line per source. Do NOT add separate lexicon entries.

Step 11 — Verify Style & Expand Tests

Run make format && make lint && make license. Load references/testing-guide.py for test skeletons. Required tests: test_<source>_fetch (slow), _cache (slow), _call_mock, _exceptions, _available. Target 90%+ coverage with --slow.

[CONFIRM — Tests]

Present test file, functions, coverage.

Step 12 — Validate Variables & Sanity-Check

Validate all lexicon vars against real data (run script, do NOT commit)
Remove variables with < 10% valid data
Create sanity-check plot (gridded or sparse template)
Tell user the plot path and ask for visual confirmation

[CONFIRM — Sanity-Check Plots]

User MUST visually inspect plots. Do not proceed without confirmation.

Step 13 — Branch, Commit & Open PR

Create branch feat/data-source-<name>
Commit (do NOT add sanity-check script/images)
Push to fork
gh pr create --repo NVIDIA/earth2studio
Immediately post sanity-check validation as PR comment with:
- Variable coverage table (name, count, range, unit)
- Data validation summary (regions, storms/stations, time range)
- Key findings (physically reasonable values, conversions verified)
- Full validation script in <details> block
- Image placeholder:

[CONFIRM — Ready to Submit]

Verify all steps complete before creating PR.

Step 14 — Automated Code Review

Poll for Greptile review (5 min timeout)
Categorize feedback (bug/style/perf/docs/suggestion/false-positive)
Present triage table to user
Implement accepted fixes
Respond to PR comments
Push

[CONFIRM — Review Triage]

User approves which comments to address.

Examples

User: Add a data source for the NOAA GFS analysis on S3
Agent: [loads skill, proceeds through Steps 0–14]

Limitations

One source type per invocation
Requires network access for validation (Step 12)
SPDX license headers required in all files

Reminders

DO: uv run python, loguru.logger, alphabetical order in __init__.py/RST/CHANGELOG, canonical method ordering, async utilities (managed_session, gather_with_concurrency, async_retry), pure async I/O, reference URLs in docstrings, try/finally cleanup.

AVOID: asyncio.to_thread, bare tqdm.gather, xarray for loading, full file downloads.

NEVER: loop.set_default_executor(), commit secrets, commit sanity-check scripts/images.

Files

ATTRIBUTION.mdAdded by AutoGineer961 B
BENCHMARK.md2.4 KiB
LICENSELicence file added by AutoGineer11.2 KiB
SKILL.md10.4 KiB
evals/config.yml1.1 KiB
evals/environment/Dockerfile867 B
evals/environment/setup/bootstrap.shScript934 B
evals/evals.json6.1 KiB
evals/targets/eval_1_target.pyScript2.5 KiB
evals/targets/eval_2_target.pyScript2.9 KiB
evals/targets/eval_3_target.pyScript3.8 KiB
evals/targets/eval_4_target.pyScript11.5 KiB
evals/targets/test/eval_1_test_target.pyScript2.4 KiB
evals/targets/test/eval_2_test_target.pyScript3.1 KiB
evals/targets/test/eval_3_test_target.pyScript3.7 KiB
evals/targets/test/eval_4_test_target.pyScript4.5 KiB
references/implementation-guide.pyScript17.3 KiB
references/testing-guide.pyScript10.1 KiB
references/validation-guide.md9.5 KiB
skill-card.md2.6 KiB
skill.oms.sig7.9 KiB

LICENSE

Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.


                                 Apache License
                           Version 2.0, January 2004
                        http://www.apache.org/licenses/

   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION

   1. Definitions.

      "License" shall mean the terms and conditions for use, reproduction,
      and distribution as defined by Sections 1 through 9 of this document.

      "Licensor" shall mean the copyright owner or entity authorized by
      the copyright owner that is granting the License.

      "Legal Entity" shall mean the union of the acting entity and all
      other entities that control, are controlled by, or are under common
      control with that entity. For the purposes of this definition,
      "control" means (i) the power, direct or indirect, to cause the
      direction or management of such entity, whether by contract or
      otherwise, or (ii) ownership of fifty percent (50%) or more of the
      outstanding shares, or (iii) beneficial ownership of such entity.

      "You" (or "Your") shall mean an individual or Legal Entity
      exercising permissions granted by this License.

      "Source" form shall mean the preferred form for making modifications,
      including but not limited to software source code, documentation
      source, and configuration files.

      "Object" form shall mean any form resulting from mechanical
      transformation or translation of a Source form, including but
      not limited to compiled object code, generated documentation,
      and conversions to other media types.

      "Work" shall mean the work of authorship, whether in Source or
      Object form, made available under the License, as indicated by a
      copyright notice that is included in or attached to the work
      (an example is provided in the Appendix below).

      "Derivative Works" shall mean any work, whether in Source or Object
      form, that is based on (or derived from) the Work and for which the
      editorial revisions, annotations, elaborations, or other modifications
      represent, as a whole, an original work of authorship. For the purposes
      of this License, Derivative Works shall not include works that remain
      separable from, or merely link (or bind by name) to the interfaces of,
      the Work and Derivative Works thereof.

      "Contribution" shall mean any work of authorship, including
      the original version of the Work and any modifications or additions
      to that Work or Derivative Works thereof, that is intentionally
      submitted to Licensor for inclusion in the Work by the copyright owner
      or by an individual or Legal Entity authorized to submit on behalf of
      the copyright owner. For the purposes of this definition, "submitted"
      means any form of electronic, verbal, or written communication sent
      to the Licensor or its representatives, including but not limited to
      communication on electronic mailing lists, source code control systems,
      and issue tracking systems that are managed by, or on behalf of, the
      Licensor for the purpose of discussing and improving the Work, but
      excluding communication that is conspicuously marked or otherwise
      designated in writing by the copyright owner as "Not a Contribution."

      "Contributor" shall mean Licensor and any individual or Legal Entity
      on behalf of whom a Contribution has been received by Licensor and
      subsequently incorporated within the Work.

   2. Grant of Copyright License. Subject to the terms and conditions of
      this License, each Contributor hereby grants to You a perpetual,
      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
      copyright license to reproduce, prepare Derivative Works of,
      publicly display, publicly perform, sublicense, and distribute the
      Work and such Derivative Works in Source or Object form.

   3. Grant of Patent License. Subject to the terms and conditions of
      this License, each Contributor hereby grants to You a perpetual,
      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
      (except as stated in this section) patent license to make, have made,
      use, offer to sell, sell, import, and otherwise transfer the Work,
      where such license applies only to those patent claims licensable
      by such Contributor that are necessarily infringed by their
      Contribution(s) alone or by combination of their Contribution(s)
      with the Work to which such Contribution(s) was submitted. If You
      institute patent litigation against any entity (including a
      cross-claim or counterclaim in a lawsuit) alleging that the Work
      or a Contribution incorporated within the Work constitutes direct
      or contributory patent infringement, then any patent licenses
      granted to You under this License for that Work shall terminate
      as of the date such litigation is filed.

   4. Redistribution. You may reproduce and distribute copies of the
      Work or Derivative Works thereof in any medium, with or without
      modifications, and in Source or Object form, provided that You
      meet the following conditions:

      (a) You must give any other recipients of the Work or
          Derivative Works a copy of this License; and

      (b) You must cause any modified files to carry prominent notices
          stating that You changed the files; and

      (c) You must retain, in the Source form of any Derivative Works
          that You distribute, all copyright, patent, trademark, and
          attribution notices from the Source form of the Work,
          excluding those notices that do not pertain to any part of
          the Derivative Works; and

      (d) If the Work includes a "NOTICE" text file as part of its
          distribution, then any Derivative Works that You distribute must
          include a readable copy of the attribution notices contained
          within such NOTICE file, excluding those notices that do not
          pertain to any part of the Derivative Works, in at least one
          of the following places: within a NOTICE text file distributed
          as part of the Derivative Works; within the Source form or
          documentation, if provided along with the Derivative Works; or,
          within a display generated by the Derivative Works, if and
          wherever such third-party notices normally appear. The contents
          of the NOTICE file are for informational purposes only and
          do not modify the License. You may add Your own attribution
          notices within Derivative Works that You distribute, alongside
          or as an addendum to the NOTICE text from the Work, provided
          that such additional attribution notices cannot be construed
          as modifying the License.

      You may add Your own copyright statement to Your modifications and
      may provide additional or different license terms and conditions
      for use, reproduction, or distribution of Your modifications, or
      for any such Derivative Works as a whole, provided Your use,
      reproduction, and distribution of the Work otherwise complies with
      the conditions stated in this License.

   5. Submission of Contributions. Unless You explicitly state otherwise,
      any Contribution intentionally submitted for inclusion in the Work
      by You to the Licensor shall be under the terms and conditions of
      this License, without any additional terms or conditions.
      Notwithstanding the above, nothing herein shall supersede or modify
      the terms of any separate license agreement you may have executed
      with Licensor regarding such Contributions.

   6. Trademarks. This License does not grant permission to use the trade
      names, trademarks, service marks, or product names of the Licensor,
      except as required for reasonable and customary use in describing the
      origin of the Work and reproducing the content of the NOTICE file.

   7. Disclaimer of Warranty. Unless required by applicable law or
      agreed to in writing, Licensor provides the Work (and each
      Contributor provides its Contributions) on an "AS IS" BASIS,
      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
      implied, including, without limitation, any warranties or conditions
      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
      PARTICULAR PURPOSE. You are solely responsible for determining the
      appropriateness of using or redistributing the Work and assume any
      risks associated with Your exercise of permissions under this License.

   8. Limitation of Liability. In no event and under no legal theory,
      whether in tort (including negligence), contract, or otherwise,
      unless required by applicable law (such as deliberate and grossly
      negligent acts) or agreed to in writing, shall any Contributor be
      liable to You for damages, including any direct, indirect, special,
      incidental, or consequential damages of any character arising as a
      result of this License or out of the use or inability to use the
      Work (including but not limited to damages for loss of goodwill,
      work stoppage, computer failure or malfunction, or any and all
      other commercial damages or losses), even if such Contributor
      has been advised of the possibility of such damages.

   9. Accepting Warranty or Additional Liability. While redistributing
      the Work or Derivative Works thereof, You may choose to offer,
      and charge a fee for, acceptance of support, warranty, indemnity,
      or other liability obligations and/or rights consistent with this
      License. However, in accepting such obligations, You may act only
      on Your own behalf and on Your sole responsibility, not on behalf
      of any other Contributor, and only if You agree to indemnify,
      defend, and hold each Contributor harmless for any liability
      incurred by, or claims asserted against, such Contributor by reason
      of your accepting any such warranty or additional liability.

   END OF TERMS AND CONDITIONS

   APPENDIX: How to apply the Apache License to your work.

      To apply the Apache License to your work, attach the following
      boilerplate notice, with the fields enclosed by brackets "[]"
      replaced with your own identifying information. (Don't include
      the brackets!)  The text should be enclosed in the appropriate
      comment syntax for the file format. We also recommend that a
      file or class name and description of purpose be included on the
      same "printed page" as the copyright notice for easier
      identification within third-party archives.

   Copyright [yyyy] [name of copyright owner]

   Licensed under the Apache License, Version 2.0 (the "License");
   you may not use this file except in compliance with the License.
   You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.

Requirements and compatibility

Written for: Claude Code

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Claude Code frontmatter, Names Claude Code tools, Needs Python

Security

What our scanner noticed· Medium risk

Large base64-encoded blobs
Scripts that your agent may run

This skill includes scripts that your agent may run. Read them before you use it.

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by

NVIDIA Earth-2 Team

Publisher

NVIDIA (Official)

Source

NVIDIA/skills · skills/earth2studio-create-datasource · commit fce3cca

Licence

Apache License 2.0 (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE-APACHE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-08

AutoGineer is not affiliated with or endorsed by NVIDIA Earth-2 Team or NVIDIA.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download earth2studio-create-datasource.zip