Generated banner for Data Science Engineering Foundation
Generated logo for Data Science Engineering Foundation

Data Science Engineering Foundation

MicrosoftOfficialPublished by Microsoft

  • Skill
  • Meta (authoring, orchestration, planning, memory)
  • Agent tooling
  • CC-BY-4.0
  • No issues found

From the author

State, resume, reconstruction, job-lifecycle, transition, and flow-state mechanics for the Data Science and Engineering Coach. Loaded by the coach; not a user entry point.
— the description by Microsoft

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

The Data Science and Engineering Foundation serves as the core orchestration layer for the Data Science and Engineering Coach. It provides a centralized set of protocols for managing session state, job lifecycles, and flow-state mechanics. This foundation ensures consistency across different coaching sessions and job transitions by defining how the coach should initialize, resume, and navigate between various tasks. It acts as an internal reference index rather than a user-facing tool, enforcing rules for when to interrupt, how to handle durable writes, and how to maintain session integrity.

What it does

  • Defines protocols for session initialization, state reconstruction, and resumption.
  • Provides a registry for routing tasks to specific skills and specialists.
  • Implements a flow-state protocol to manage interruptions and user gates.
  • Enforces a durable-write gate to scan for sensitive content before creating artifacts.
  • Maintains clear boundaries between general orchestration and job-specific logic.
  • Standardizes the lifecycle management of coaching jobs.

When to use it

  • When the coach needs to initialize or resume a session.
  • When transitioning between different data science or engineering jobs.
  • Before performing a durable write of a customer artifact.
  • When validating the state of a coaching session.
  • When determining the appropriate skill or specialist to handle a specific task.

Advantages

  • Ensures consistent behavior across different coaching sessions.
  • Provides a structured approach to handling sensitive data during artifact creation.
  • Decouples orchestration logic from job-specific implementation details.
  • Improves reliability by enforcing clear state and lifecycle rules.
  • Facilitates modularity by routing tasks to specialized skills.

Usage examples

“Initialize the coaching session for a new data engineering project.”
The coach loads the session-state protocol, validates the project identity, and prepares the environment for the selected job.
“I want to save this new data catalog document.”
The coach triggers the flow-state protocol, performs a sensitive content scan on the document, and proceeds with the write only if the scan passes.
“Switch from the current feasibility study to a new pipeline design task.”
The coach follows the transition protocol to pause the current job, confirm the switch with the user, and route to the dataops skill.

Install

  1. Download the skill as a ZIP:

    https://mp-staging.autogineer.com/agent-skills/microsoft/data-science-engineering-foundation/download
  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip data-science-engineering-foundation.zip -d .agents/skills

Documentation by Microsoft

Data Science and Engineering Foundation

Goal

Keep Data Science and Engineering Coach orchestration consistent across job changes and sessions without duplicating job-specific guidance. The coach loads this index at initialization and resume, then reads the reference for the current coaching moment.

Reference index

ReferenceWhen to read
job-registry.mdBefore offering or selecting a job, routing to a skill or specialist, or naming a durable output
lifecycle-classes.mdWhen starting, pausing, resuming, completing, or re-invoking a job
transition-protocol.mdWhen a topic shift, explicit request, or completion suggests moving between jobs
session-state.mdBefore initialization, validation, mutation, resume, recovery, or reconstruction of coaching state
flow-state.mdBefore interrupting work, crossing a gate, writing a durable artifact, or offering post-job choices

Success criteria

  • Each orchestration rule has one owner in this package.

  • State and lifecycle mechanics remain independent of job-specific methods.

  • The coach reads the applicable reference before acting on its contract.

Constraints

  • Keep catalog guidance in data-catalog, the durable data-catalog workflow for entities, declared relationships, lineage, and ERD-ready model semantics.

  • Keep feasibility guidance in feasibility, the evidence-led data and ML feasibility-study workflow with lifecycle and interchange traceability.

  • Keep pipeline and testing guidance in dataops, the DataOps reference for tier behavior, pipeline invariants, validation placement, tests, drift, and operational signals.

  • Keep general experiment guidance in experiment-design, the reusable workflow for candidate selection, hypotheses, vetting, minimum scope, and result evaluation.

  • Keep ML-specific experiment guidance in ml-experimentation, the reference for ML environments, reproducibility, tracking, evaluation, abstractions, and production readiness.

  • Keep notebook and dashboard guidance in analysis-authoring, the reference for EDA notebook and analytical dashboard composition, visualization selection, and dashboard validation.

  • Keep AI-system evaluation guidance in evaluation-design, the reference for evaluation dataset design, difficulty balance, metric selection, and tooling fit.

  • Treat this package as internal foundation knowledge, not a user-selectable workflow.

  • Preserve the skill authority boundaries and cross-cutting concerns defined in the job registry.

Stop rules

  • Stop before a state mutation when the session-state protocol has not been loaded or project identity is uncertain.

  • Stop before a job switch when the user has not confirmed the transition.

  • Stop and report the missing owner when requested behavior belongs to neither this foundation nor a registered job capability.

Attribution

This package is repository-original orchestration guidance licensed under CC BY 4.0. It does not reproduce or summarize an external standard.

Files

references/job-registry.md


title: Data Science and Engineering Job Registry description: Authoritative job names, lifecycle classes, skill and specialist routes, and durable outputs for Data Science and Engineering Coach sessions

Data Science and Engineering Job Registry

Purpose

Use this registry before offering or selecting work. A job is a user-confirmed unit of focus inside one continuing coaching session. The registry routes work; it does not duplicate the methods owned by a skill.

Registry

JobClassPrimary routeOptional supporting routeDurable output or completion evidence
catalogcontinuousdata-catalog: durable catalog entities, declared relationships, lineage, and model semanticsanalysis-authoring when a presented dataset profile is the requested outputCaller-approved data catalog
model-diagramepisodicdata-catalog: authoritative declared entities and relationshipsarchitecture-diagrams: render the declared model without becoming semantic authorityMermaid or ASCII diagram
problem-framingepisodicexperiment-design: candidate data-science problem classes for a stated business outcome, with entry-test reasoningNoneCandidate framing set with reasoning, unknowns, and gaps
feasibilityboundedfeasibility: evidence-led feasibility studies with durable traceabilityNoneCaller-approved feasibility study
pipelineepisodicdataops: tier behavior, pipeline invariants, validation, tests, and driftNoneTransformation, validation, or pipeline code
analysisepisodicanalysis-authoring: EDA notebook and analytical dashboard authoring and validationdata-catalog for column semantics; dataops for derived-dataset persistenceNotebook, dashboard, or analysis deliverable
evaluationepisodicevaluation-design: AI-system evaluation dataset design, metrics, and toolingrpi-research only for a demonstrated live evaluator-catalog fact gapEvaluation dataset with curation, metric, and tooling documents
experimentepisodicexperiment-design: experiment selection, hypotheses, vetting, scope, and evaluationml-experimentation: ML reproducibility, tracking, evaluation, abstractions, and production readinessHypothesis, experiment assets, and result disposition
testingepisodicdataops: DataOps and DS/MLOps test techniquesanalysis-authoring for dashboard validation techniqueTest code and assertions
observabilityepisodicdataops: data/model signals and validation-versus-drift guidanceNoneInstrumentation code and signal recommendations

Optional supporting routes in the registry are job-specific skill support and do not limit the separate segment-level Research eligibility defined by the RPI depth matrix.

RPI depth matrix

RPI is a user-directed depth route inside the confirmed job, not a job or lifecycle. Keep simple or adequately evidenced work on the primary route. No RPI segment auto-activates. Before each eligible segment, state the demonstrated need, purpose, expected artifact, expected interaction cost, limits, and direct path available if the user skips it.

JobResearch postureExecution-loop eligibilityRPI does not own
catalogNone by defaultNone by defaultCatalog identity, semantics, enrichment, or continuous lifecycle
model-diagramNone by defaultNone by defaultDeclared relationships, diagram format, or episodic completion
problem-framinganalysis mode for a named current-fact or prior-art gap; preserve alternatives and unknownsNone by defaultSelecting, ranking, or scoring a problem class
feasibilityBounded current-fact evidence that the feasibility study cannot establish locallyNone by defaultFeasibility verdict, study lifecycle, or recommendation authority
pipelineCurrent platform, compatibility, standard, or prior-art facts identified by dataopsFull Plan, Implement, and Review for substantial deliveryTier semantics, pipeline invariants, validation placement, or scan authority
analysisCurrent library, platform, accessibility, or analytical-method facts identified by analysis-authoringFull Plan, Implement, and Review for substantial deliveryAnalytical intent, visualization choice, dashboard validation, or scan authority
evaluationCurrent evaluator names, availability, preview state, compatibility, and prerequisites identified by evaluation-designFull Plan, Implement, and Review for substantial deliveryMetric selection, dataset design, tooling fit, or scan authority
experimentInherit the Experiment Designer convergence Research contractInherit the Experiment Designer post-design execution loopHypotheses, criteria, experiment outcome, or partner commitments
testingCurrent tool, framework, compatibility, or prior-art facts identified by the owning skillFull Plan, Implement, and Review for substantial deliveryTest intent, assertion authority, or scan authority
observabilityCurrent platform, telemetry, compatibility, or prior-art facts identified by the owning skillFull Plan, Implement, and Review for substantial deliverySignal-selection intent, telemetry vocabulary, drift interpretation, or scan authority

Substantial delivery means multi-step code or artifact production whose dependencies, validation, or interruption risk make direct episodic execution unreliable. The owning skill first accepts the domain design. RPI then plans and executes delivery and reviews conformance without changing the job, class, owning skill, coach state authority, output root, or durable-write gate.

RPI Researcher may run only as an internal helper to an active rpi-research phase for one bounded source-gathering lane. Its return is not an independent job, invocation, state record, evidence authority, or recommendation. The active Research phase verifies selected sources and exclusively owns canonical evidence IDs, findings, recommendations, and the primary artifact.

The cross-cutting concerns below are evaluated in addition to the routes in this table. They are not listed per job, because they apply by trigger rather than by job name.

Cross-cutting concerns

Privacy, Responsible AI, and telemetry are evaluated across every job rather than being optional per-job routes. Evaluate each concern whenever its trigger is present, name the concern to the user, and route to its owning skill for the contribution that skill owns.

ConcernTrigger to evaluateOwning skillContribution
PrivacyPersonal, sensitive, or regulated data is catalogued, moved, transformed, sampled, analyzed, presented, or persistedprivacy-standardsSensitivity classification, citation fields, DPIA thresholds, and data-flow reasoning
Responsible AIAn AI or ML system is designed, evaluated, or experimented onrai-standards, then rai-planner when a risk surfacesRisk framing and standards mapping, plus a scoped assessment when a surfaced risk warrants one
TelemetrySignals, metrics, traces, or logs are emitted, named, or instrumented, including drift and pipeline health signalstelemetry-foundationsMetric names, instruments, units, cardinality limits, and PII-safe telemetry conventions

A cross-cutting concern always surfaces an observation and offers concrete choices. It never blocks a durable write, never edits an artifact silently, and never decides for the user. The durable-write scan gate remains the only condition that blocks a write.

When a concern triggers, state which concern applies and why, name its owning skill, offer the contribution that skill can make, and let the user choose whether to take it. When the owning skill is unavailable in the installed collection, say so and continue without improvising its authority.

Skill authority boundaries

The Data Science and Engineering Coach routes to the data science and engineering skills below, plus the cross-cutting skills above. Keep their authority separate even when one job loads more than one skill.

Exact skill nameCapability descriptionDoes not own
data-catalogDurable catalog entities, declared relationships, identity, coverage, lineage, and attached dataset profilesPipeline tier behavior, inferred diagram relationships, or feasibility verdicts
dataopsDataOps tier semantics, pipeline invariants, validation, tests, drift, signals, and derived-dataset persistenceCatalog meaning, experiment selection, ML tracking setup, or sensitivity classification
feasibilityEvidence-led feasibility studies, lifecycle, recommendations, and interchange traceabilityFunctional requirement numbering, downstream writeback, catalog semantics, or implementation
analysis-authoringEDA notebook and analytical dashboard composition, visualization selection, and dashboard validationColumn semantics, persistence format, accessibility conformance, or evaluation dataset design
evaluation-designAI-system evaluation dataset design, difficulty balance, metric selection, and tooling fitTrained-model evaluation, Responsible AI approval, or session and durable-write mechanics
experiment-designProblem-class framing for a stated business outcome, plus experiment selection, hypotheses, vetting, minimum scope, and result evaluationML infrastructure, production implementation, or pipeline mechanics
ml-experimentationML environments, reproducibility, tracking, abstractions, evaluation, and readinessGeneral hypothesis framing, pipeline replay, or Responsible AI approval
privacy-standardsSensitivity classification, data-flow reasoning, standards mapping, and DPIA thresholdsCatalog semantics, pipeline mechanics, or durable-write authority
rai-standardsResponsible AI standards, risk framing, and standards mappingEvaluation dataset design, model performance, or approval authority
rai-plannerScoped Responsible AI assessment when a surfaced risk warrants oneData science method authority, session mechanics, or approval authority
telemetry-foundationsMetric names, instruments, units, cardinality, and PII-safe telemetry conventionsSignal selection intent, pipeline behavior, or drift interpretation

The two Responsible AI authorities are sequential, not alternatives. rai-standards provides risk framing and standards alignment and is evaluated first whenever an AI or ML system is in scope. rai-planner provides a scoped assessment of one surfaced risk and is reached only after that risk is identified.

problem-framing and experiment both route to experiment-design and stay distinct. problem-framing asks what kind of data-science problem a stated business outcome is, and completes without choosing one. experiment assumes a direction and selects what to test.

Problem-framing completion evidence

This job has two completion shapes and neither is an abandonment.

  • Normal completion records the candidate problem classes, the grounded-versus- original reasoning behind each, unresolved discriminating questions marked unknown, and gaps carrying an answer holder and a coordinator. No candidate is selected, ranked, or scored.

  • An unstated business outcome completes with a single outcome-definition gap, its answer holder, its coordinator, and an explicit statement that no candidate analysis was performed. Record it as completed work, not discarded-cleanly.

Completion returns control to the coach. A feasibility or experiment job starts only after a separate user-confirmed transition.

The two evaluation authorities are adjacent and must not be blended. evaluation-design covers systems whose output is a response, such as assistants and agents. ml-experimentation covers a trained model's predictive performance.

Selection rules

  1. Present the relevant jobs with their class and expected output.

  2. Ask the user to choose; do not infer or activate a job silently.

  3. Record the confirmed job and its class using the session-state protocol.

  4. Load the primary skill before job-specific guidance.

  5. Load a supporting route only for the concern it owns, then reconcile the result into the session artifact list without transferring authority to it.

  6. When a capability is unavailable in the installed collection, state the gap and offer an available route. Do not improvise a missing capability.

Routing boundary

A supporting route contributes within its own authority. The coach retains session identity, job selection, transition decisions, state mutation, pause and resume, durable-write gates, and completion choices. Loading a supporting skill never changes the active job without user confirmation.

The RPI depth matrix is the only job-specific Research and execution-routing authority. Every route remains subordinate to the active job's primary skill, lifecycle class, coach state, transition protocol, and durable-write gate.

Provenance

This registry is repository-original guidance licensed under CC BY 4.0. It does not reproduce or summarize an external standard.

Requirements and compatibility

Written for: GitHub Copilot, Claude Code

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Claude Code frontmatter

Security

No issues found by our static scan· No issues found

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by
Microsoft
Publisher
Microsoft (Official)
Source
microsoft/hve-core · .github/skills/data-science-engineering/data-science-engineering-foundation · commit f425e94
Licence
Creative Commons Attribution 4.0 International (redistribution allowed)
  • LICENSE (written by AutoGineer)
  • ATTRIBUTION.md (written by AutoGineer)
Synced

AutoGineer is not affiliated with or endorsed by Microsoft.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download data-science-engineering-foundation.zip