Data Catalog

MicrosoftOfficialPublished by Microsoft

Skill
Content generator
Data and AI
CC-BY-4.0
Low risk
Has scripts

From the author

Create and enrich durable data catalogs using the native DS_CATALOG_V1 Markdown contract, declared entity relationships, privacy citation fields, and stable relationship IDs. Use when inventorying engagement data, recording semantic relationships, or preparing a catalog for ERD rendering.

— the description by Microsoft

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

The data-catalog skill provides a standardized procedure for inventorying business entities and their relationships. It utilizes the DS_CATALOG_V1 Markdown contract to ensure that data catalogs are both human-readable and machine-validatable. The workflow emphasizes data integrity by requiring stable identifiers, explicit confidence levels for relationships, and clear separation between semantic facts and narrative descriptions. It is designed to support data science and engineering engagements by providing a reliable, versioned record of data assets, lineage, and privacy classifications without embedding sensitive credentials or duplicating profile data.

What it does

Defines a structured Markdown and YAML format for data catalogs.
Requires stable, unique identifiers for all entities and relationships.
Enforces explicit confidence levels (confirmed, inferred, assumed) for all semantic links.
Validates catalog structure against a predefined JSON schema.
Separates metadata and facts from narrative documentation.
Provides a mechanism for tracking privacy standards and lineage.

When to use it

When you need to inventory engagement data for a data science or engineering project.
When recording semantic relationships between business entities for ERD rendering.
When preparing a durable, versioned data catalog that must be machine-readable.
When you need to document privacy classifications and lineage in a standardized way.

Advantages

Ensures consistency across different data catalogs using a strict contract.
Prevents the accidental leakage of credentials by prohibiting embedded secrets.
Maintains high data quality by requiring explicit evidence for relationships.
Supports automated validation to catch errors before review.
Provides a clear, audit-ready history of data relationships.

Usage examples

“Create a new data catalog for the customer engagement project.”: The agent initializes a new Markdown file using the DS_CATALOG_V1 template and guides the user through inventorying entities and defining their relationships.
“Validate the current data catalog file.”: The agent executes the validation script to ensure the YAML frontmatter adheres to the DS_CATALOG_V1 schema and reports any structural issues.

Install

Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/microsoft/data-catalog/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip data-catalog.zip -d .agents/skills

Documentation by Microsoft

Data Catalog Workflow

Goal

Produce a customer-readable Markdown catalog whose YAML frontmatter is a valid DS_CATALOG_V1 machine contract. Preserve uncertainty explicitly so inferred or assumed relationships never appear confirmed.

Flow

Confirm the engagement name and the caller-approved durable output path.
Inventory entities at business grain. Record source access, tier, volume, profile pointer, classification, lineage, and open questions without copying column-level profile data.
Assign every relationship a stable rel-* identifier. Record endpoints, maximum cardinality, both endpoint minimums, one or more paired join-key fields, confidence, and evidence basis.
Reconcile coverage counts with the entity and relationship records.
Render the human-readable sections from the YAML facts, ending with the canonical Data Science and Engineering Coaching disclaimer footer. Narrative can explain facts but cannot redefine them.
Validate the artifact with scripts/validate_catalog.py before treating it as ready for review.

Inputs

Engagement context and a caller-approved output path
Data source inventory and access status
Business entity names, grain, and declared relationships
Existing per-dataset profile paths, when available
Privacy classifications or standards citations produced by the owning privacy workflow

Success criteria

The frontmatter declares exactly catalog_version: DS_CATALOG_V1 and validates against assets/ds-catalog-v1.schema.json.
Entity IDs and relationship IDs are unique and stable. Every endpoint and lineage reference resolves.
Every relationship declares cardinality as its maximum multiplicity plus from_minimum and to_minimum as zero or one.
Join keys use one string on both sides or paired arrays of equal length, and record field names only without primary-key, foreign-key, or uniqueness roles.
Relationship confidence is one of confirmed, inferred, or assumed, and every relationship records its basis.
Classification uses the privacy-standards citation-field names. The catalog does not invent standards identifiers.
Column statistics and feature metadata remain behind profile_ref rather than being copied into the catalog.
Every customer-facing catalog ends with the canonical Data Science and Engineering Coaching disclaimer footer.

Constraints

Treat catalog enrichment as user-driven. Offer enrichment when a source is missing, but do not change the catalog silently.
Record source locations as paths or connection references, never embedded credentials.
Keep DCAT alignment non-binding. The catalog is native YAML, not RDF, and makes no DCAT conformance claim.
Keep tier recording separate from tier behavior. data-catalog records the tier; dataops owns what that tier means.
Keep rendering separate from semantic authority. Diagram tools consume declared relationships and do not infer new ones.
End every customer-facing catalog with the canonical Data Science and Engineering Coaching disclaimer from the disclaimer-language skill.

Stop rules

Stop and request clarification when an entity grain, relationship endpoint, join-key pairing, or endpoint minimum is ambiguous.
Stop and retain inferred or assumed confidence when evidence does not support confirmed.
Stop before a durable write when the caller has not confirmed the destination.
Stop and route privacy interpretation to privacy-standards or the Privacy Planner when citation values or DPIA status are unknown.

Package resources

Resource	Use
catalog-contract.md	Read for the authoritative field, identity, multiplicity, and body-generation rules
dcat-crosswalk.md	Read when planning a DCAT, DCAT-AP, or catalog-platform export
provenance.md	Read for standards selection, licensing, and non-conformance boundaries
ds-catalog-v1.md	Copy when starting a new catalog
northwind-catalog.md	Read as a complete valid example with scalar and composite join keys
`assets/ds-catalog-v1.schema.json`	Use as the structural JSON Schema for parsed frontmatter
`scripts/validate_catalog.py`	Execute with `uv run python scripts/validate_catalog.py <catalog.md>` to validate a catalog

Attribution

The native contract, workflow, schema, template, examples, and validator are repository-original content licensed CC BY 4.0.

The crosswalk paraphrases selected concepts from W3C DCAT 3, W3C PROV-O, and Frictionless Table Schema v2. Standard names and term identifiers are factual citations, the crosswalk prose and the paired-array join-key contract are independently authored, and no upstream schema, example, table, or substantial excerpt is reproduced. Because no upstream expression is redistributed, the package remains solely CC BY 4.0. See provenance.md.

Files

ATTRIBUTION.mdAdded by AutoGineer975 B
LICENSEAdded by AutoGineer16.6 KiB
SKILL.md6.1 KiB
assets/ds-catalog-v1.schema.json5.9 KiB
examples/northwind-catalog.md8.4 KiB
pyproject.toml698 B
references/catalog-contract.md11.1 KiB
references/dcat-crosswalk.md5.4 KiB
references/provenance.md3.1 KiB
scripts/validate_catalog.pyScript15.0 KiB
templates/ds-catalog-v1.md5.3 KiB
tests/corpus/0_valid_frontmatter90 B
tests/corpus/1_empty_frontmatter8 B
tests/corpus/2_unclosed_sequence20 B
tests/corpus/3_duplicate_key24 B
tests/corpus/4_no_frontmatter15 B
tests/corpus/README.md1.2 KiB
tests/fuzz_harness.pyScript1.2 KiB
tests/test_validate_catalog.pyScript24.0 KiB
uv.lock107.7 KiB

examples/northwind-catalog.md

catalog_version: DS_CATALOG_V1 engagement: northwind-modernization generated_at: "2026-08-01T09:00:00Z" last_enriched: "2026-08-02T14:30:00Z" entities:

id: customer name: Customer Master description: Canonical customer account record source: system: crm location: connections/crm-readonly format: table access_confirmed: true tier: silver grain: One row per customer account volume: row_estimate: 120000 period_covered: "2018-01-01/2026-07-31" update_frequency: daily profile_ref: outputs/data-profile-customer-2026-07-28.json classification: sensitivity: confidential contains_personal_data: true data_categories: - contact-data gdpr_article: "Art. 6(1)(b)" ccpa_section: null nist_pf_category: ID.IM-P nistir8062_objective: null owasp_privacy_id: null dpia_ref: docs/privacy/customer-data-plan.md lineage: derived_from: [] transform_ref: src/customer/clean.py open_questions: []
id: sales-order-line name: Sales Order Line description: Individual line items recorded on sales orders source: system: erp location: lake/bronze/sales-order-line format: parquet access_confirmed: true tier: bronze grain: One row per order and line number volume: row_estimate: 9000000 period_covered: "2020-01-01/2026-07-31" update_frequency: hourly profile_ref: outputs/data-profile-sales-order-line-2026-08-01.json classification: sensitivity: internal contains_personal_data: false data_categories: [] gdpr_article: null ccpa_section: null nist_pf_category: null nistir8062_objective: null owasp_privacy_id: null dpia_ref: null lineage: derived_from: - customer transform_ref: null open_questions:
- Confirm whether order lines can move between accounts
id: product name: Product Catalogue description: Sellable product definitions and their commercial attributes source: system: erp location: lake/silver/product format: parquet access_confirmed: true tier: silver grain: One row per product code volume: row_estimate: 4200 period_covered: null update_frequency: daily profile_ref: null classification: sensitivity: internal contains_personal_data: false data_categories: [] gdpr_article: null ccpa_section: null nist_pf_category: null nistir8062_objective: null owasp_privacy_id: null dpia_ref: null lineage: derived_from: [] transform_ref: null open_questions: []
id: support-ticket name: Support Ticket description: Customer support requests raised through the service desk source: system: service-desk location: connections/service-desk-export format: jsonl access_confirmed: false tier: bronze grain: One row per support ticket volume: row_estimate: null period_covered: null update_frequency: unknown profile_ref: null classification: sensitivity: confidential contains_personal_data: true data_categories: - contact-data gdpr_article: "Art. 6(1)(f)" ccpa_section: null nist_pf_category: null nistir8062_objective: null owasp_privacy_id: null dpia_ref: null lineage: derived_from: [] transform_ref: null open_questions:
- Confirm the identifier that links tickets to customer accounts relationships:
id: rel-customer-order-line from: customer to: sales-order-line cardinality: one-to-many from_minimum: one to_minimum: zero join_keys: from_field: - tenant_id - customer_id to_field: - tenant_id - customer_id confidence: confirmed basis: Confirmed by the CRM and ERP data owners
id: rel-product-order-line from: product to: sales-order-line cardinality: one-to-many from_minimum: one to_minimum: one join_keys: from_field: product_code to_field: product_code confidence: inferred basis: Every sampled order line resolves to exactly one product code
id: rel-customer-support-ticket from: customer to: support-ticket cardinality: one-to-many from_minimum: zero to_minimum: zero join_keys: from_field: customer_id to_field: account_ref confidence: assumed basis: Proposed during the discovery workshop; service-desk access is not yet granted coverage: entities_catalogued: 4 entities_access_confirmed: 3 entities_classified: 4 relationships_confirmed: 1 relationships_inferred: 1

Northwind modernization data catalog

Overview and engagement context

This catalog records the confirmed customer, order-line, product, and support entities for modernization planning, along with the relationships that remain unconfirmed.

Entity summary

Entity	Grain	Tier	Sensitivity	Access
Customer Master	One row per customer account	Silver	Confidential	Confirmed
Sales Order Line	One row per order and line number	Bronze	Internal	Confirmed
Product Catalogue	One row per product code	Silver	Internal	Confirmed
Support Ticket	One row per support ticket	Bronze	Confidential	Unconfirmed

Entity relationship diagram

The declared relationships are ready for catalog-driven ERD rendering. One relationship uses a composite key, two use scalar keys, and only one is confirmed.

Relationship	Endpoints	Cardinality	Minimums	Join keys	Confidence
`rel-customer-order-line`	customer to sales-order-line	one-to-many	from `one`, to `zero`	`tenant_id`, `customer_id` to `tenant_id`, `customer_id`	confirmed
`rel-product-order-line`	product to sales-order-line	one-to-many	from `one`, to `one`	`product_code` to `product_code`	inferred
`rel-customer-support-ticket`	customer to support-ticket	one-to-many	from `zero`, to `zero`	`customer_id` to `account_ref`	assumed

Entity details

Customer Master

The CRM record is the canonical customer account source and points to a dated profile.

Sales Order Line

The ERP extract uses tenant and customer identifiers as a composite relationship key.

Product Catalogue

The curated product table supplies the product code that order lines reference. The pairing is supported by sampling rather than an owner confirmation.

Support Ticket

The service-desk export is not yet accessible, so its link to customer accounts remains an assumption from the discovery workshop.

Coverage summary

Four entities are catalogued and classified. Three have confirmed access. Of three declared relationships, one is confirmed, one is inferred, and one is assumed.

Open questions and access gaps

Confirm whether order lines can move between accounts
Confirm the identifier that links tickets to customer accounts
Obtain service-desk access so the support-ticket relationship can be evidenced

Disclaimer

[!CAUTION] Disclaimer: This agent is an assistive data-science and data-engineering coaching tool only. It does not validate customer data, execute production pipelines, establish model fitness, or replace data owners, privacy and Responsible AI reviewers, engineering review, or business decision authority. Catalogs, feasibility findings, analyses, experiments, tests, and operational recommendations generated with this tool may be incomplete or inaccurate and must be independently reviewed against approved data sources, stakeholder evidence, and organizational controls before use. Outputs from this tool do not constitute data approval, feasibility sign-off, model approval, privacy or Responsible AI approval, or production readiness.

🤖 Crafted with precision by ✨Copilot following brilliant human instruction, then carefully refined by our team of discerning human reviewers.

Requirements and compatibility

Written for: GitHub Copilot, Claude Code

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Claude Code frontmatter, Needs Python

Security

What our scanner noticed· Low risk

Scripts that your agent may run

This skill includes scripts that your agent may run. Read them before you use it.

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by

Microsoft

Publisher

Microsoft (Official)

Source

microsoft/hve-core · .github/skills/data-science-engineering/data-catalog · commit f425e94

Licence

Creative Commons Attribution 4.0 International (redistribution allowed)

LICENSE (written by AutoGineer)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-08

AutoGineer is not affiliated with or endorsed by Microsoft.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download data-catalog.zip