Synapse Migration

MicrosoftOfficialPublished by Microsoft

Skill
Workflow
Software development
MIT
Low risk
Has scripts

From the author

Converts Synapse workspaces to Fabric: Spark items, Lake Databases, Linked Services, and Dedicated SQL Pool schema/procedures for Lakehouse or Warehouse. During Pool-to-Lakehouse migration only, can migrate approved dependent procedure callers after notebook readiness. Excludes standalone pipelines and Warehouse administration. Triggers: Synapse workspace to Fabric, Dedicated SQL Pool to Lakehouse, Dedicated SQL Pool to Fabric Warehouse.

— the description by Microsoft

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

The synapse-migration skill provides a structured, phased workflow for transitioning Synapse Analytics workspaces to Microsoft Fabric. It covers the migration of Spark items, Lake Databases, Linked Services, and Dedicated SQL Pool schema and stored procedures. The skill emphasizes a dependency-ordered approach, requiring gap assessment and explicit approval before conversion. It distinguishes between migrating to a Fabric Lakehouse (schema and code only, no data movement) and a Fabric Warehouse, while providing specific routing for dependent pipeline callers.

What it does

Discovers and classifies Synapse workspace artifacts.
Converts Dedicated SQL Pool schema and stored procedures to Spark SQL notebooks or Warehouse objects.
Migrates Spark items, including notebooks and Spark Job Definitions, to Fabric equivalents.
Maps Linked Services to Fabric Data Connections or OneLake Shortcuts.
Manages the migration of approved dependent pipeline callers.
Validates persisted definitions and artifacts against established contracts.

When to use it

When migrating a full Synapse workspace to Microsoft Fabric.
When converting Dedicated SQL Pool schema and procedures to a Lakehouse or Warehouse.
When porting Spark pools, notebooks, or Spark Job Definitions to Fabric.
When replacing Synapse Linked Services with Fabric-native connectivity.

Advantages

Ensures dependency-ordered migration to maintain system integrity.
Provides clear routing for different workload types (Lakehouse vs. Warehouse).
Enforces strict validation and approval gates for behavior-changing decisions.
Supports incremental migration rather than high-risk big-bang cutovers.
Standardizes the conversion of T-SQL procedures to executable Spark SQL notebooks.

Usage examples

“Migrate this Dedicated SQL Pool DACPAC to a Fabric Lakehouse and convert the procedures.”: The agent will assess gaps, obtain approval for procedure-to-notebook mappings, create a migration manifest, convert schema/code artifacts, and validate the results.
“Migrate this Dedicated SQL Pool to Fabric Warehouse.”: The agent will assess DDL compatibility, remove unsupported distribution/index declarations, obtain provisioning consent, and prepare for potential data migration.

Install

Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/microsoft/synapse-migration/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip synapse-migration.zip -d .agents/skills

Documentation by Microsoft

Telemetry — MANDATORY. Every api.fabric.microsoft.com call must carry x-ms-fabric-skill: synapse-migration (az rest: --headers "x-ms-fabric-skill=synapse-migration"), including every LRO poll, fabric_lro and retry. Snippets omit it — add it anyway.

OFFLINE LAKEHOUSE ARTIFACT FAST PATH — DECIDE FIRST. For a no-live-call request with complete supplied inputs and contracts, load only the fast path; do not load migration-planning-reference.md, dedicated-pool-conversion.md, or implementation scripts. Use one generator for artifacts and final-byte hashes, then one verifier. Do not inspect generated files or run post-pass spot checks. For every optional non-null source default, set missingValueBehavior to UseSourceDefault and restore that default before validation. Do not use this path for a Warehouse target; incomplete and large-procedure requests use their own routes.

CRITICAL NOTES
To find workspace details (including its ID) from a workspace name: list all workspaces, then use JMESPath filtering.
To find item details (including its ID) from workspace ID, item type, and item name: list all items of that type in that workspace, then use JMESPath filtering.
Treat a source Dedicated SQL Pool as read-only. Do not run source DDL, DML, stored procedures, or create sample objects.
Dedicated Pool-to-Lakehouse data movement is out of scope. Migrate schema and executable code artifacts only; do not copy source rows or run row-level/business-result equivalence queries. The separate Warehouse route may move data only from a separately approved restored export copy under its own consent gate and restored-source contract.
For Dedicated Pool-to-Lakehouse requests, complete gap assessment, explicit mapping approval, manifest creation, conversion, deployment, and validation in order. Read dedicated-pool-to-lakehouse.md once as the router, then load only the current phase reference.
Stored-procedure 1:1, N:1, and N:N mappings never apply to views. Views remain schema objects; procedures become traceable Spark SQL notebooks with executable %%sql transformation cells.
Dependent ADF/Synapse callers are conditional scope, not automatic scope. Discover them during assessment, require a separate complete-closure approval, and migrate them only after notebook readiness. Route standalone pipeline requests to pipeline-migration.
Before executing a specialized Dedicated Pool workflow, read the matching contract in dedicated-pool-execution-contracts.md. Its validation, hashing, request-body, resume, and completion-language requirements are mandatory.
Keep Lakehouse and Warehouse paths separate. If the target is unclear, ask before loading path-specific resources. Lakehouse conversion never moves source rows; Warehouse data migration requires its own explicit consent gate and restored-source contract.
For a Fabric Warehouse target, explicitly state that the complete DISTRIBUTION and CLUSTERED COLUMNSTORE INDEX declarations are removed because Fabric manages distribution, storage, and indexing; a bare mention of either source clause is insufficient.
Run setup DDL and CETAS only against the separately approved restored export copy after proving it is not the original production source.

Synapse Analytics to Microsoft Fabric Migration

Core Workflow

Discover the Synapse workspace and classify every source artifact.
Choose targets using workload-routing.md.
Assess gaps before conversion, including runtime, library, connectivity, T-SQL, security, and capacity constraints.
Obtain approval for behavior-changing decisions, target placement, and every stored-procedure-to-notebook mapping.
Create or update migration-manifest.json and persist approval, per-object status, hashes, and checkpoints atomically.
Convert and deploy in dependency order, loading only the current phase resource.
Validate persisted definitions and artifacts without executing generated notebooks or pipelines unless a separately approved workflow explicitly requires execution.
Report completed, blocked, skipped, and pending objects without exposing credentials or sensitive values.

For full-workspace orchestration, read migration-orchestrator.md. For a Dedicated Pool-to-Lakehouse migration, use dedicated-pool-to-lakehouse.md as the phase router. For a Warehouse target, load only the matching dw-* resource.

Scope Routing

Request	Action	Load
Full Synapse workspace migration	Run phased discovery and migration	migration-orchestrator.md
Dedicated SQL Pool schema/code to Lakehouse	Run the strict assessment-to-validation workflow. For guidance-only notebook publication, explicitly state: "The stored procedures were converted from T-SQL to Spark SQL before publication."	dedicated-pool-to-lakehouse.md
Guidance-only Dedicated Pool incremental deployment hardening	Answer directly from the focused checklist; do not search scripts or load phase resources. State: "For columns with special characters, enable Delta `columnMapping` selectively and only with approval."	dedicated-pool-execution-contracts.md
Large or complex stored procedure	For a complete offline contract, read only the audit resource once, use one run-local generator and one verifier, do not inspect implementation scripts or generated outputs, and stop after successful verification	dedicated-pool-large-procedure-audit.md
Spark pool	Convert configuration and libraries to a Fabric Environment	spark-pool-migration.md
Lake Database or HMS	Create or bind the target Lakehouse	lake-database-migration.md or external-hms-migration.md
Notebook or Spark Job Definition	Port code, bindings, parameters, and runtime settings	spark-item-migration.md
Linked Service	Choose a Fabric Data Connection for external services or a OneLake Shortcut for ADLS Gen2/Blob	connectivity-migration.md
Approved dependent procedure callers	Run the internal closure-preserving sub-flow after notebook readback	dedicated-pool-dependent-pipelines.md
Standalone pipeline migration	Delegate	`pipeline-migration`
Dedicated SQL Pool to Fabric Warehouse	Run the source extraction, DDL compatibility, approval, optional data, security, and validation route	dw-source-and-extraction.md, then the matching `dw-*` phase resource
Standalone Fabric Warehouse administration	Delegate	`sqldw-cli`
Synapse Link	Use Fabric Mirroring; not covered here	—

Always report the Lake Database/HMS phase. If none is discovered, mark it NotApplicable (none discovered) while retaining the required target Lakehouse binding.

Load References On Demand

Do not preload all files. Load only the references needed for the current phase:

Need	Reference
Routing, phase order, API audiences, quick examples, troubleshooting, or post-migration handoff	workload-routing.md
Dedicated Pool execution variants and response contracts	dedicated-pool-execution-contracts.md
Dedicated Pool discovery	dedicated-pool-discovery.md
Gap assessment and mapping approval	dedicated-pool-gap-assessment.md
Procedure conversion and notebook shape	dedicated-pool-conversion.md
Schema/notebook deployment	dedicated-pool-deployment.md
Dedicated Pool validation	dedicated-pool-validation.md
Runtime/library compatibility	feature-parity.md and library-compatibility.md
`mssparkutils` and connector refactoring	utility-api-mapping.md, connector-refactoring.md, and code-patterns.md
Capacity planning	capacity-sizing.md
Dedicated Pool-to-Warehouse source discovery and extraction	dw-source-and-extraction.md
Warehouse DDL compatibility and deployment	dw-ddl-compatibility.md
Separately approved Warehouse data migration	dw-data-migration.md
Warehouse security and validation	dw-security-validation.md
Validation, reporting, and production readiness	validation-testing.md, migration-report.md, and security-governance.md

Use COMMON-CLI.md only for shared authentication and az rest mechanics not already covered by the phase resource. Use COMMON-CORE.md only for shared Fabric REST patterns.

Must

Preserve every stored-procedure input contract, source default, source decision, and approved target mapping.
Block conversion until gap dispositions, notebook names, dependency grouping, and target workspace placement are explicitly approved.
Use SqlPackage/DMVs for source discovery, Fabric REST for item management, and Fabric Livy statements for Lakehouse schema and Delta execution.
Run the maintained validators required by the selected execution contract; never replace them with ad hoc JSON or cell-shape checks.
Choose the Lakehouse or Warehouse target before loading implementation resources; never apply Warehouse data-movement rules to Lakehouse artifact conversion.
Before persisting any dependent-pipeline source definition, recursively inspect it for secret-bearing or uncertain properties; block the complete selected closure without writing the artifact or mutating a target when any are found.
In every dependent-pipeline response, including guidance-only and pre-approval explanations, state: "Use a TridentNotebook activity with exact target identities typeProperties.notebookId and typeProperties.workspaceId, and pass inputs through typeProperties.notebookParameters." Then state: "After deployment, read back each pipeline with POST /v1/workspaces/{workspaceId}/items/{dataPipelineId}/getDefinition and validate its canonical hash and structural bindings."
For every Linked Service response, first state both replacement classes: Fabric Data Connections for external databases/services and OneLake Shortcuts for ADLS Gen2/Blob storage; then identify which class applies.
Replace mssparkutils with notebookutils, Linked Services with the correct Fabric connection class, and spark.read.synapsesql() with a supported OneLake or JDBC pattern.
Externalize workspace and item IDs through parameters or Variable Libraries.
Keep phase status concise, emit bounded heartbeats for long operations, and persist per-object evidence in the manifest.

Prefer

OneLake Shortcuts over copying data.
Fabric Environments for reproducible libraries and Spark settings.
Incremental, dependency-ordered migration over big-bang cutover.
Parameterized notebooks and Variable Libraries for environment promotion.
Lakehouse SQL Endpoint for compatible Serverless SQL Pool read workloads.

Avoid

Do not execute generated notebooks or pipelines as part of artifact migration.
Do not broaden a selected dependent-pipeline closure or route that internal sub-flow to pipeline-migration.
Do not use sqlcmd against the target Fabric Warehouse; use the registered SQL endpoint tooling required by the Warehouse resource.
Do not preserve Synapse distribution/storage hints, PolyBase DDL, Linked Service connection strings, or pool-level library installation patterns verbatim.
Do not use target notebooks to orchestrate the migration itself.
Do not hardcode workspace IDs, item IDs, credentials, connection strings, or environment-specific paths.

Examples

Prompt: "Migrate this Dedicated SQL Pool DACPAC to a Fabric Lakehouse and convert the procedures."

Response behavior: Assess gaps first, present and obtain approval for the complete procedure-to-notebook mapping and workspace placement, create the manifest, convert approved schema/code artifacts without moving source rows, validate persisted definitions, and report the resolved Lakehouse and artifact evidence.

Prompt: "Migrate this ADF pipeline to Fabric."

Response behavior: Route to pipeline-migration. Keep it here only when the pipeline is an explicitly approved dependent caller discovered inside an active Dedicated Pool-to-Lakehouse migration.

Prompt: "Migrate this Dedicated SQL Pool to Fabric Warehouse."

Response behavior: Keep this inside synapse-migration, assess DDL compatibility, remove unsupported distribution and columnstore declarations, obtain provisioning consent, and ask separately before any restored-source data migration.

For code transformations and edge cases, load workload-routing.md and the phase-specific resource rather than expanding the main instructions.

Files

ATTRIBUTION.mdAdded by AutoGineer879 B
LICENSELicence file added by AutoGineer1.1 KiB
SKILL.md14.3 KiB
apm.yml597 B
references/dedicated-pool-execution-contracts.md14.5 KiB
references/workload-routing.md6.0 KiB
resources/capacity-sizing.md4.1 KiB
resources/code-patterns.md8.4 KiB
resources/connectivity-migration.md6.4 KiB
resources/connector-refactoring.md13.8 KiB
resources/dedicated-pool-conversion.md39.2 KiB
resources/dedicated-pool-dependent-pipelines.md23.3 KiB
resources/dedicated-pool-deployment.md20.0 KiB
resources/dedicated-pool-discovery.md17.8 KiB
resources/dedicated-pool-gap-assessment.md17.4 KiB
resources/dedicated-pool-large-procedure-audit.md24.5 KiB
resources/dedicated-pool-to-lakehouse.md17.9 KiB
resources/dedicated-pool-validation.md17.2 KiB
resources/dw-data-migration.md13.7 KiB
resources/dw-ddl-compatibility.md9.4 KiB
resources/dw-security-validation.md15.6 KiB
resources/dw-source-and-extraction.md8.0 KiB
resources/external-hms-migration.md20.2 KiB
resources/feature-parity.md5.4 KiB
resources/lake-database-migration.md34.8 KiB
resources/library-compatibility.md7.2 KiB
resources/migration-gotchas.md16.1 KiB
resources/migration-orchestrator.md63.0 KiB
resources/migration-planning-reference.md16.3 KiB
resources/migration-report.md16.9 KiB
resources/security-governance.md47.4 KiB
resources/spark-item-migration.md19.2 KiB
resources/spark-pool-migration.md16.3 KiB
resources/utility-api-mapping.md5.4 KiB
resources/validation-testing.md14.5 KiB
scripts/Invoke-DedicatedPoolTool.ps1Script941 B
scripts/dedicated-pool-tool/DedicatedPoolMessageSanitizer.csScript2.1 KiB
scripts/dedicated-pool-tool/DedicatedPoolTool.csproj346 B
scripts/dedicated-pool-tool/Program.csScript31.4 KiB
scripts/dedicated_pool_runtime.pyScript116.9 KiB
scripts/validate_dedicated_pool_spark.pyScript24.8 KiB

resources/spark-pool-migration.md

Synapse Spark Pool → Fabric Environment Migration

Migrate Synapse Spark pool configurations (compute settings, libraries, Spark properties) to Fabric Environments via REST APIs.

Execute as Phase 0 — before Lake Databases, Notebooks, and SJDs — so that Environments exist for notebook and SJD binding in later phases.
Auth tokens (see COMMON-CLI.md § Authentication Recipes for commands):
Synapse ARM audience: https://management.azure.com
Fabric audience: https://api.fabric.microsoft.com

Migration Workflow

Phase 0: Spark Pool → Fabric Environment
├── Step 1: Inventory Synapse Spark pools (ARM API)
├── Step 2: Decide per pool — Starter Pool or Custom Environment?
├── Step 3: Create Fabric Environment (if needed)
├── Step 4: Build definition parts (Sparkcompute.yml + libraries)
├── Step 5: Upload definition and publish
├── Step 6: Record pool → Environment mapping for Phase 2/3
└── Step 7: (Optional) Validate before proceeding to Phase 1

Step 1: Inventory Synapse Spark Pools

Spark pools use the ARM (management plane) API, not the Synapse data-plane API.

List All Pools

Endpoint:

GET https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Synapse/workspaces/{workspaceName}/bigDataPools?api-version=2021-06-01

Response: { "value": [ ...BigDataPoolResourceInfo[] ] } — paginated via nextLink.

Get Single Pool

Endpoint:

GET https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Synapse/workspaces/{workspaceName}/bigDataPools/{poolName}?api-version=2021-06-01

Key Properties to Extract

Property	Field Path	Example
Pool name	`name`	`"MySparkPool"`
Spark version	`properties.sparkVersion`	`"3.3"`, `"3.4"`, `"3.5"`
Node size	`properties.nodeSize`	`Small`, `Medium`, `Large`, `XLarge`, `XXLarge`
Node size family	`properties.nodeSizeFamily`	`MemoryOptimized`, `HardwareAcceleratedGPU`
Node count	`properties.nodeCount`	`4`
Autoscale	`properties.autoScale`	`{ enabled, minNodeCount, maxNodeCount }`
Auto-pause	`properties.autoPause`	`{ enabled, delayInMinutes }`
Dynamic executor alloc	`properties.dynamicExecutorAllocation`	`{ enabled, minExecutors, maxExecutors }`
Library requirements	`properties.libraryRequirements`	`{ content, filename }` — pip/conda requirements.txt
Custom libraries	`properties.customLibraries[]`	`{ name, type, path, containerName }` — uploaded JARs, wheels
Spark config properties	`properties.sparkConfigProperties`	`{ content, filename, configurationType }` — Spark .conf/.yml
Compute isolation	`properties.isComputeIsolationEnabled`	`true` / `false`
Autotune	`properties.isAutotuneEnabled`	`true` / `false`

Step 2: Decision — Starter Pool vs. Environment

For each Synapse pool, decide whether a Fabric Environment is needed:

Synapse Pool:
├── Has GPU nodes (nodeSizeFamily == "HardwareAcceleratedGPU")?
│   └── YES → MIGRATION BLOCKER — GPU not supported in Fabric. Refactor to CPU or keep on Synapse.
│
├── Has custom libraries OR custom Spark config properties?
│   ├── YES → Create Environment (libraries + config only; compute can still be Starter Pool)
│   └── NO
│       ├── Has non-default node size (not Medium)?
│       │   ├── YES → Create Environment WITH Custom Pool definition
│       │   └── NO  → No Environment needed — use Starter Pool
│       └── Has Managed Private Endpoint requirements?
│           ├── YES → Create Environment WITH Custom Pool (MPE requires Custom Pool)
│           └── NO  → No Environment needed — use Starter Pool

Decision Summary Table

Synapse Pool Has...	Environment Needed?	Custom Pool Needed?	Action
Default (Medium), no libs, no config	No	No	Use Starter Pool — nothing to create
Custom libraries only	Yes	No	Environment with libraries; compute = Starter Pool
Custom Spark config only	Yes	No	Environment with Spark properties; compute = Starter Pool
Custom libs + Spark config	Yes	No	Environment with both; compute = Starter Pool
Non-default node size (Small, Large, XLarge, XXLarge)	Yes	Yes	Environment with Custom Pool definition
Node size + libs + config	Yes	Yes	Environment with Custom Pool + libs + config
GPU nodes	N/A	N/A	Migration blocker — no Fabric equivalent
Managed Private Endpoints required	Yes	Yes	Custom Pool required for MPE in Fabric

Pools that need no Environment: Record them in the mapping as "StarterPool" so that Phase 2/3 knows no Environment binding is needed for their notebooks/SJDs.

Step 3: Create Fabric Environment

Endpoint: POST https://api.fabric.microsoft.com/v1/workspaces/{workspaceId}/environments

{
  "displayName": "{poolName}_env",
  "description": "Migrated from Synapse Spark pool: {poolName}"
}

Response: HTTP 201 (or 202 LRO). Capture the Environment id for subsequent steps.

Step 4: Build Definition Parts

The Fabric Environment definition consists of file-based parts, each base64-encoded.

Part 1: `Setting/Sparkcompute.yml` — Compute & Spark Properties

Build this YAML from the Synapse pool properties. The schema follows the canonical Fabric Environment definition — see common/ITEM-DEFINITIONS-CORE.md § Environment for the source of truth.

# Sparkcompute.yml
instance_pool_id: "{customPoolId}"      # GUID of a Fabric Custom Pool created at workspace level (omit for Starter Pool)
runtime_version: "1.3"                  # See Spark Version Mapping below

# Spark configuration carried over from Synapse sparkConfigProperties
spark_conf:
  spark.sql.shuffle.partitions: "200"
  spark.sql.adaptive.enabled: "true"
  spark.executor.memory: "8g"
  spark.driver.memory: "4g"
  # ... all valid key=value pairs from Synapse pool config

# Optional — dynamic executor allocation (carried over from Synapse pool)
dynamic_executor_allocation:
  enabled: {dynamicExecutorAllocation.enabled}
  min_executors: {dynamicExecutorAllocation.minExecutors}
  max_executors: {dynamicExecutorAllocation.maxExecutors}

# Optional — Native Execution Engine (Fabric-only, no Synapse equivalent)
enable_native_execution_engine: false

Pool sizing: Synapse nodeFamily/nodeSize/autoScale (min/max node count) are not part of the Sparkcompute.yml schema. Instead, create a Fabric Custom Pool at the workspace level with the equivalent node family/size and autoscale settings, then reference its GUID via instance_pool_id. See notebookutils.fabricClient Custom Pool APIs or the workspace settings UI.

If no Custom Pool needed (Starter Pool scenario): Omit the instance_pool_id field entirely. Only include runtime_version and spark_conf.

Spark Version Mapping

Synapse `sparkVersion`	Fabric Runtime	Status
`"3.3"`	Runtime 1.1	EOL — upgrade to 1.3
`"3.4"`	Runtime 1.2	EOL — upgrade to 1.3
`"3.5"`	Runtime 1.3	GA (recommended)

If the Synapse pool uses Spark 3.3 or 3.4, upgrade to Runtime 1.3 (Spark 3.5). Test for deprecated API warnings after migration.

Spark Config Migration Rules

Direct carryover — copy these as-is to spark_conf:

Config Key	Notes
`spark.sql.shuffle.partitions`	Works identically
`spark.executor.memory`	Works identically
`spark.executor.cores`	Works identically
`spark.driver.memory`	Works identically
`spark.driver.cores`	Works identically
`spark.dynamicAllocation.*`	Prefer the `dynamic_executor_allocation` section instead
`spark.sql.adaptive.enabled`	Enabled by default in Fabric
`spark.sql.adaptive.coalescePartitions.enabled`	Works identically
`spark.sql.sources.default`	Usually "delta"
`spark.databricks.delta.*`	Works identically
`spark.sql.parquet.*`	Works identically
`spark.serializer`	Works identically
`spark.kryoserializer.*`	Works identically
`spark.sql.broadcastTimeout`	Works identically
`spark.network.timeout`	Works identically
`spark.rpc.message.maxSize`	Works identically
`spark.hadoop.fs.azure.account.auth.type.*`	Review — may need updated credentials
`spark.hadoop.fs.azure.account.oauth.*`	Review — may need new service principal values

Strip/remove — these are Synapse-specific and will cause warnings or errors:

Config Key	Reason
`spark.synapse.linkedService.*`	Linked services don't exist in Fabric
`spark.storage.synapse.*`	Synapse internal storage config
`spark.synapse.pool.*`	Synapse pool-specific settings
`spark.synapse.workspace.*`	Synapse workspace internals
`spark.hadoop.fs.azure.account.oauth.provider.type` = `...LinkedServiceBasedTokenProvider`	Replace with `ClientCredsTokenProvider` (see connectivity-migration.md)

Add (Fabric-only optimizations) — consider adding these:

Config Key	Value	Benefit
`spark.native.enabled`	`"true"`	Enables Native Execution Engine (Velox/Gluten) — up to 4x on TPC-DS
`spark.fabric.delta.vorder.enabled`	`"true"`	Enables V-Order write optimization for Power BI Direct Lake

Part 2: `Libraries/PublicLibraries/environment.yml` — PyPI/Conda Packages

Convert the Synapse libraryRequirements.content (requirements.txt format) to Fabric's environment.yml format:

Synapse (requirements.txt format):

pandas==2.1.0
scikit-learn==1.3.0
azure-storage-blob==12.19.0

Fabric (environment.yml format):

dependencies:
  - pip:
    - pandas==2.1.0
    - scikit-learn==1.3.0
    - azure-storage-blob==12.19.0

If the Synapse pool has no libraryRequirements.content (empty string), skip this part.

Part 3: `Libraries/CustomLibraries/*` — JARs, Wheels, Tars

For each entry in properties.customLibraries[]:

Download the library file from the Synapse workspace storage (the path field gives the blob path)
Base64-encode the file content
Include as a definition part with path Libraries/CustomLibraries/{filename}

Supported file types:

.jar — Java/Scala libraries
.whl — Python wheels
.tar.gz — Python/R packages
.py — Python files

Downloading custom libraries: The path and containerName fields in customLibraries[] point to the Synapse workspace's linked storage account. Use the Azure Storage SDK or az storage blob download to retrieve each file before uploading to Fabric.

Step 5: Upload Definition and Publish

Update Environment Definition

Endpoint: POST https://api.fabric.microsoft.com/v1/workspaces/{workspaceId}/environments/{environmentId}/updateDefinition

Important: The endpoint is /updateDefinition — do NOT add a /staging/ prefix. The /staging/ prefix is only used for the publish endpoint. Using /staging/updateDefinition returns 404.

Race condition: After creating an Environment (POST /environments → 201), there is a brief provisioning window during which the updateDefinition endpoint may return 404. If updateDefinition returns 404 immediately after creation, wait 2–5 seconds and retry (up to 3 attempts). This is a known Fabric API timing issue — the item exists but its definition endpoints are not yet ready.

Retry Helper

import time, requests

def update_env_definition(workspace_id, env_id, definition_payload, headers, max_retries=3):
    """Upload Environment definition with retry for post-creation race condition."""
    url = (f"https://api.fabric.microsoft.com/v1/workspaces/{workspace_id}"
           f"/environments/{env_id}/updateDefinition")
    for attempt in range(1, max_retries + 1):
        resp = requests.post(url, headers=headers, json=definition_payload)
        if resp.status_code == 404 and attempt < max_retries:
            time.sleep(3 * attempt)  # 3s, 6s backoff
            continue
        resp.raise_for_status()
        return resp
    resp.raise_for_status()  # raise on final 404

Definition Payload

{
  "definition": {
    "parts": [
      {
        "path": "Setting/Sparkcompute.yml",
        "payload": "{base64-encoded-sparkcompute-yml}",
        "payloadType": "InlineBase64"
      },
      {
        "path": "Libraries/PublicLibraries/environment.yml",
        "payload": "{base64-encoded-environment-yml}",
        "payloadType": "InlineBase64"
      },
      {
        "path": "Libraries/CustomLibraries/mylib.jar",
        "payload": "{base64-encoded-jar}",
        "payloadType": "InlineBase64"
      }
    ]
  }
}

Returns HTTP 202 (LRO). Poll Location header until status == "Succeeded".

Publish Environment

Critical: Environment changes are staged until published. You must explicitly publish:

Endpoint: POST https://api.fabric.microsoft.com/v1/workspaces/{workspaceId}/environments/{environmentId}/staging/publish

Returns HTTP 202 (LRO). Poll until status == "Succeeded". Publishing can take several minutes for environments with many libraries.

Verify Publish Status

Endpoint: GET https://api.fabric.microsoft.com/v1/workspaces/{workspaceId}/environments/{environmentId}

Check properties.publishDetails.state — should be "Success".

Step 6: Record Pool → Environment Mapping

Produce a mapping table for use in Phase 2 (Notebooks) and Phase 3 (SJDs):

{
  "poolMappings": {
    "MySparkPool": {
      "fabricEnvironmentId": "5b218778-e7a5-4d73-8187-f10824047715",
      "fabricEnvironmentName": "MySparkPool_env",
      "mode": "customPool"
    },
    "DevPool": {
      "fabricEnvironmentId": null,
      "fabricEnvironmentName": null,
      "mode": "starterPool"
    },
    "ETLPool": {
      "fabricEnvironmentId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
      "fabricEnvironmentName": "ETLPool_env",
      "mode": "librariesOnly"
    }
  }
}

This mapping is consumed by:

Phase 2 (Notebooks): When a Synapse notebook references bigDataPool.referenceName: "MySparkPool", bind the Fabric notebook to MySparkPool_env
Phase 3 (SJDs): When a Synapse SJD references targetBigDataPool.referenceName: "ETLPool", bind the Fabric SJD to ETLPool_env

Step 7: (Optional) Validate Before Proceeding to Phase 1

Run these checks before moving to Lake Database migration. Catching issues here avoids cascading failures in later phases.

Check	How	Pass Criteria
Publish status	`GET /v1/workspaces/{wsId}/environments/{envId}` → `properties.publishDetails.state`	`"Success"` for every Environment
Library imports	Run a test notebook that `import`s each required library	No `ImportError`
Spark config	Compare `spark_conf` in Environment definition against Synapse pool config	All carried-over keys present; no stripped keys
Pool mapping file	Review the JSON mapping from Step 6	Every Synapse pool has an entry (Environment ID or `"starterPool"`)

Do not proceed to Phase 1 until all Environments publish successfully and libraries import without errors. Notebooks and SJDs in later phases will bind to these Environments — a broken Environment means every notebook fails.

See validation-testing.md → V1: Environment Validation for detailed scripts.

Migration Blockers & Limitations

Synapse Feature	Fabric Status	Action
GPU pools (`HardwareAcceleratedGPU`)	Not supported	Refactor to CPU-based alternatives or keep on Synapse
Compute isolation (`isComputeIsolationEnabled`)	Not available	Remove; rely on capacity-based isolation
Auto-pause	Not needed	Fabric Starter Pool has no idle cost; Custom Pools auto-release
Session-level packages (`sessionLevelPackagesEnabled`)	Supported via `%pip install` in notebooks	Not recommended for production — use Environment libraries
Workspace-level libraries	Deprecated in Fabric	Migrate all to Environment artifacts
Autotune (`isAutotuneEnabled`)	Not available	Remove; manually tune or rely on defaults

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs Python

Security

What our scanner noticed· Low risk

Instructions that use the network
Scripts that your agent may run

This skill includes scripts that your agent may run. Read them before you use it.

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by

Microsoft

Publisher

Microsoft (Official)

Source

microsoft/skills-for-fabric · skills/synapse-migration · commit f8fac11

Licence

MIT License (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-08

Also distributed in

microsoft/skills-for-fabric

AutoGineer is not affiliated with or endorsed by Microsoft.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download synapse-migration.zip