Synapse Migration
MicrosoftOfficialPublished by Microsoft
- Skill
- Workflow
- Software development
- MIT
- Low risk
- Has scripts
AutoGineer summary
Written by AutoGineer from the skill’s own files, not by the author
The synapse-migration skill provides a structured, phased workflow for transitioning Synapse Analytics workspaces to Microsoft Fabric. It covers the migration of Spark items, Lake Databases, Linked Services, and Dedicated SQL Pool schema and stored procedures. The skill emphasizes a dependency-ordered approach, requiring gap assessment and explicit approval before conversion. It distinguishes between migrating to a Fabric Lakehouse (schema and code only, no data movement) and a Fabric Warehouse, while providing specific routing for dependent pipeline callers.
What it does
- Discovers and classifies Synapse workspace artifacts.
- Converts Dedicated SQL Pool schema and stored procedures to Spark SQL notebooks or Warehouse objects.
- Migrates Spark items, including notebooks and Spark Job Definitions, to Fabric equivalents.
- Maps Linked Services to Fabric Data Connections or OneLake Shortcuts.
- Manages the migration of approved dependent pipeline callers.
- Validates persisted definitions and artifacts against established contracts.
When to use it
- When migrating a full Synapse workspace to Microsoft Fabric.
- When converting Dedicated SQL Pool schema and procedures to a Lakehouse or Warehouse.
- When porting Spark pools, notebooks, or Spark Job Definitions to Fabric.
- When replacing Synapse Linked Services with Fabric-native connectivity.
Advantages
- Ensures dependency-ordered migration to maintain system integrity.
- Provides clear routing for different workload types (Lakehouse vs. Warehouse).
- Enforces strict validation and approval gates for behavior-changing decisions.
- Supports incremental migration rather than high-risk big-bang cutovers.
- Standardizes the conversion of T-SQL procedures to executable Spark SQL notebooks.
Usage examples
- “Migrate this Dedicated SQL Pool DACPAC to a Fabric Lakehouse and convert the procedures.”
- The agent will assess gaps, obtain approval for procedure-to-notebook mappings, create a migration manifest, convert schema/code artifacts, and validate the results.
- “Migrate this Dedicated SQL Pool to Fabric Warehouse.”
- The agent will assess DDL compatibility, remove unsupported distribution/index declarations, obtain provisioning consent, and prepare for potential data migration.
Install
Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/microsoft/synapse-migration/downloadUnzip it into the cross-client skills folder of your project:
mkdir -p .agents/skills && unzip synapse-migration.zip -d .agents/skills
Documentation by Microsoft
Telemetry — MANDATORY. Every
api.fabric.microsoft.comcall must carryx-ms-fabric-skill: synapse-migration(az rest:--headers "x-ms-fabric-skill=synapse-migration"), including every LRO poll,fabric_lroand retry. Snippets omit it — add it anyway.
OFFLINE LAKEHOUSE ARTIFACT FAST PATH — DECIDE FIRST. For a no-live-call request with complete supplied inputs and contracts, load only the fast path; do not load
migration-planning-reference.md,dedicated-pool-conversion.md, or implementation scripts. Use one generator for artifacts and final-byte hashes, then one verifier. Do not inspect generated files or run post-pass spot checks. For every optional non-null source default, setmissingValueBehaviortoUseSourceDefaultand restore that default before validation. Do not use this path for a Warehouse target; incomplete and large-procedure requests use their own routes.
CRITICAL NOTES
To find workspace details (including its ID) from a workspace name: list all workspaces, then use JMESPath filtering.
To find item details (including its ID) from workspace ID, item type, and item name: list all items of that type in that workspace, then use JMESPath filtering.
Treat a source Dedicated SQL Pool as read-only. Do not run source DDL, DML, stored procedures, or create sample objects.
Dedicated Pool-to-Lakehouse data movement is out of scope. Migrate schema and executable code artifacts only; do not copy source rows or run row-level/business-result equivalence queries. The separate Warehouse route may move data only from a separately approved restored export copy under its own consent gate and restored-source contract.
For Dedicated Pool-to-Lakehouse requests, complete gap assessment, explicit mapping approval, manifest creation, conversion, deployment, and validation in order. Read dedicated-pool-to-lakehouse.md once as the router, then load only the current phase reference.
Stored-procedure
1:1,N:1, andN:Nmappings never apply to views. Views remain schema objects; procedures become traceable Spark SQL notebooks with executable%%sqltransformation cells.Dependent ADF/Synapse callers are conditional scope, not automatic scope. Discover them during assessment, require a separate complete-closure approval, and migrate them only after notebook readiness. Route standalone pipeline requests to
pipeline-migration.Before executing a specialized Dedicated Pool workflow, read the matching contract in dedicated-pool-execution-contracts.md. Its validation, hashing, request-body, resume, and completion-language requirements are mandatory.
Keep Lakehouse and Warehouse paths separate. If the target is unclear, ask before loading path-specific resources. Lakehouse conversion never moves source rows; Warehouse data migration requires its own explicit consent gate and restored-source contract.
For a Fabric Warehouse target, explicitly state that the complete
DISTRIBUTIONandCLUSTERED COLUMNSTORE INDEXdeclarations are removed because Fabric manages distribution, storage, and indexing; a bare mention of either source clause is insufficient.Run setup DDL and CETAS only against the separately approved restored export copy after proving it is not the original production source.
Synapse Analytics to Microsoft Fabric Migration
Core Workflow
Discover the Synapse workspace and classify every source artifact.
Choose targets using workload-routing.md.
Assess gaps before conversion, including runtime, library, connectivity, T-SQL, security, and capacity constraints.
Obtain approval for behavior-changing decisions, target placement, and every stored-procedure-to-notebook mapping.
Create or update
migration-manifest.jsonand persist approval, per-object status, hashes, and checkpoints atomically.Convert and deploy in dependency order, loading only the current phase resource.
Validate persisted definitions and artifacts without executing generated notebooks or pipelines unless a separately approved workflow explicitly requires execution.
Report completed, blocked, skipped, and pending objects without exposing credentials or sensitive values.
For full-workspace orchestration, read migration-orchestrator.md. For a Dedicated Pool-to-Lakehouse migration, use dedicated-pool-to-lakehouse.md as the phase router. For a Warehouse target, load only the matching dw-* resource.
Scope Routing
| Request | Action | Load |
|---|---|---|
| Full Synapse workspace migration | Run phased discovery and migration | migration-orchestrator.md |
| Dedicated SQL Pool schema/code to Lakehouse | Run the strict assessment-to-validation workflow. For guidance-only notebook publication, explicitly state: "The stored procedures were converted from T-SQL to Spark SQL before publication." | dedicated-pool-to-lakehouse.md |
| Guidance-only Dedicated Pool incremental deployment hardening | Answer directly from the focused checklist; do not search scripts or load phase resources. State: "For columns with special characters, enable Delta columnMapping selectively and only with approval." | dedicated-pool-execution-contracts.md |
| Large or complex stored procedure | For a complete offline contract, read only the audit resource once, use one run-local generator and one verifier, do not inspect implementation scripts or generated outputs, and stop after successful verification | dedicated-pool-large-procedure-audit.md |
| Spark pool | Convert configuration and libraries to a Fabric Environment | spark-pool-migration.md |
| Lake Database or HMS | Create or bind the target Lakehouse | lake-database-migration.md or external-hms-migration.md |
| Notebook or Spark Job Definition | Port code, bindings, parameters, and runtime settings | spark-item-migration.md |
| Linked Service | Choose a Fabric Data Connection for external services or a OneLake Shortcut for ADLS Gen2/Blob | connectivity-migration.md |
| Approved dependent procedure callers | Run the internal closure-preserving sub-flow after notebook readback | dedicated-pool-dependent-pipelines.md |
| Standalone pipeline migration | Delegate | pipeline-migration |
| Dedicated SQL Pool to Fabric Warehouse | Run the source extraction, DDL compatibility, approval, optional data, security, and validation route | dw-source-and-extraction.md, then the matching dw-* phase resource |
| Standalone Fabric Warehouse administration | Delegate | sqldw-cli |
| Synapse Link | Use Fabric Mirroring; not covered here | — |
Always report the Lake Database/HMS phase. If none is discovered, mark it NotApplicable (none discovered) while retaining the required target Lakehouse binding.
Load References On Demand
Do not preload all files. Load only the references needed for the current phase:
| Need | Reference |
|---|---|
| Routing, phase order, API audiences, quick examples, troubleshooting, or post-migration handoff | workload-routing.md |
| Dedicated Pool execution variants and response contracts | dedicated-pool-execution-contracts.md |
| Dedicated Pool discovery | dedicated-pool-discovery.md |
| Gap assessment and mapping approval | dedicated-pool-gap-assessment.md |
| Procedure conversion and notebook shape | dedicated-pool-conversion.md |
| Schema/notebook deployment | dedicated-pool-deployment.md |
| Dedicated Pool validation | dedicated-pool-validation.md |
| Runtime/library compatibility | feature-parity.md and library-compatibility.md |
mssparkutils and connector refactoring | utility-api-mapping.md, connector-refactoring.md, and code-patterns.md |
| Capacity planning | capacity-sizing.md |
| Dedicated Pool-to-Warehouse source discovery and extraction | dw-source-and-extraction.md |
| Warehouse DDL compatibility and deployment | dw-ddl-compatibility.md |
| Separately approved Warehouse data migration | dw-data-migration.md |
| Warehouse security and validation | dw-security-validation.md |
| Validation, reporting, and production readiness | validation-testing.md, migration-report.md, and security-governance.md |
Use COMMON-CLI.md only for shared authentication and az rest mechanics not already covered by the phase resource. Use COMMON-CORE.md only for shared Fabric REST patterns.
Must
Preserve every stored-procedure input contract, source default, source decision, and approved target mapping.
Block conversion until gap dispositions, notebook names, dependency grouping, and target workspace placement are explicitly approved.
Use SqlPackage/DMVs for source discovery, Fabric REST for item management, and Fabric Livy statements for Lakehouse schema and Delta execution.
Run the maintained validators required by the selected execution contract; never replace them with ad hoc JSON or cell-shape checks.
Choose the Lakehouse or Warehouse target before loading implementation resources; never apply Warehouse data-movement rules to Lakehouse artifact conversion.
Before persisting any dependent-pipeline source definition, recursively inspect it for secret-bearing or uncertain properties; block the complete selected closure without writing the artifact or mutating a target when any are found.
In every dependent-pipeline response, including guidance-only and pre-approval explanations, state: "Use a
TridentNotebookactivity with exact target identitiestypeProperties.notebookIdandtypeProperties.workspaceId, and pass inputs throughtypeProperties.notebookParameters." Then state: "After deployment, read back each pipeline withPOST /v1/workspaces/{workspaceId}/items/{dataPipelineId}/getDefinitionand validate its canonical hash and structural bindings."For every Linked Service response, first state both replacement classes: Fabric Data Connections for external databases/services and OneLake Shortcuts for ADLS Gen2/Blob storage; then identify which class applies.
Replace
mssparkutilswithnotebookutils, Linked Services with the correct Fabric connection class, andspark.read.synapsesql()with a supported OneLake or JDBC pattern.Externalize workspace and item IDs through parameters or Variable Libraries.
Keep phase status concise, emit bounded heartbeats for long operations, and persist per-object evidence in the manifest.
Prefer
OneLake Shortcuts over copying data.
Fabric Environments for reproducible libraries and Spark settings.
Incremental, dependency-ordered migration over big-bang cutover.
Parameterized notebooks and Variable Libraries for environment promotion.
Lakehouse SQL Endpoint for compatible Serverless SQL Pool read workloads.
Avoid
Do not execute generated notebooks or pipelines as part of artifact migration.
Do not broaden a selected dependent-pipeline closure or route that internal sub-flow to
pipeline-migration.Do not use
sqlcmdagainst the target Fabric Warehouse; use the registered SQL endpoint tooling required by the Warehouse resource.Do not preserve Synapse distribution/storage hints, PolyBase DDL, Linked Service connection strings, or pool-level library installation patterns verbatim.
Do not use target notebooks to orchestrate the migration itself.
Do not hardcode workspace IDs, item IDs, credentials, connection strings, or environment-specific paths.
Examples
Prompt: "Migrate this Dedicated SQL Pool DACPAC to a Fabric Lakehouse and convert the procedures."
Response behavior: Assess gaps first, present and obtain approval for the complete procedure-to-notebook mapping and workspace placement, create the manifest, convert approved schema/code artifacts without moving source rows, validate persisted definitions, and report the resolved Lakehouse and artifact evidence.
Prompt: "Migrate this ADF pipeline to Fabric."
Response behavior: Route to pipeline-migration. Keep it here only when the pipeline is an explicitly approved dependent caller discovered inside an active Dedicated Pool-to-Lakehouse migration.
Prompt: "Migrate this Dedicated SQL Pool to Fabric Warehouse."
Response behavior: Keep this inside synapse-migration, assess DDL compatibility, remove unsupported distribution and columnstore declarations, obtain provisioning consent, and ask separately before any restored-source data migration.
For code transformations and edge cases, load workload-routing.md and the phase-specific resource rather than expanding the main instructions.
Files
- ATTRIBUTION.mdAdded by AutoGineer879 B
- LICENSELicence file added by AutoGineer1.1 KiB
- SKILL.md14.3 KiB
- apm.yml597 B
- references/dedicated-pool-execution-contracts.md14.5 KiB
- references/workload-routing.md6.0 KiB
- resources/capacity-sizing.md4.1 KiB
- resources/code-patterns.md8.4 KiB
- resources/connectivity-migration.md6.4 KiB
- resources/connector-refactoring.md13.8 KiB
- resources/dedicated-pool-conversion.md39.2 KiB
- resources/dedicated-pool-dependent-pipelines.md23.3 KiB
- resources/dedicated-pool-deployment.md20.0 KiB
- resources/dedicated-pool-discovery.md17.8 KiB
- resources/dedicated-pool-gap-assessment.md17.4 KiB
- resources/dedicated-pool-large-procedure-audit.md24.5 KiB
- resources/dedicated-pool-to-lakehouse.md17.9 KiB
- resources/dedicated-pool-validation.md17.2 KiB
- resources/dw-data-migration.md13.7 KiB
- resources/dw-ddl-compatibility.md9.4 KiB
- resources/dw-security-validation.md15.6 KiB
- resources/dw-source-and-extraction.md8.0 KiB
- resources/external-hms-migration.md20.2 KiB
- resources/feature-parity.md5.4 KiB
- resources/lake-database-migration.md34.8 KiB
- resources/library-compatibility.md7.2 KiB
- resources/migration-gotchas.md16.1 KiB
- resources/migration-orchestrator.md63.0 KiB
- resources/migration-planning-reference.md16.3 KiB
- resources/migration-report.md16.9 KiB
- resources/security-governance.md47.4 KiB
- resources/spark-item-migration.md19.2 KiB
- resources/spark-pool-migration.md16.3 KiB
- resources/utility-api-mapping.md5.4 KiB
- resources/validation-testing.md14.5 KiB
- scripts/Invoke-DedicatedPoolTool.ps1Script941 B
- scripts/dedicated-pool-tool/DedicatedPoolMessageSanitizer.csScript2.1 KiB
- scripts/dedicated-pool-tool/DedicatedPoolTool.csproj346 B
- scripts/dedicated-pool-tool/Program.csScript31.4 KiB
- scripts/dedicated_pool_runtime.pyScript116.9 KiB
- scripts/validate_dedicated_pool_spark.pyScript24.8 KiB
resources/external-hms-migration.md
External Hive Metastore → Fabric Lakehouse Migration
Migrate Synapse workspaces that use an external Hive Metastore (backed by Azure SQL Database or Azure Database for MySQL) to Fabric Lakehouses.
When to use this guide: Your Synapse Spark pools are configured with
spark.hadoop.javax.jdo.option.ConnectionURLpointing to an external database. If your workspace uses the built-in HMS (no external connection configured), use lake-database-migration.md instead.Deprecation notice: External HMS support in Synapse is deprecated after Spark 3.4. Fabric does not support connecting to an external Hive Metastore — all metadata must be migrated into the Fabric Lakehouse catalog.
Auth tokens needed (see COMMON-CLI.md § Authentication Recipes for commands):
Synapse ARM audience:
https://management.azure.com(for detection)HMS database: JDBC credentials (SQL auth or Entra ID) to query the external HMS
Fabric audience:
https://api.fabric.microsoft.com
Migration Workflow
External HMS Migration:
├── Step 0: Detect external HMS configuration
├── Step 1: Connect to HMS database and inventory databases & tables
├── Step 2: Select databases to migrate and choose mapping mode
├── Step 3: Create Fabric Lakehouse(es)
├── Step 4: Create schemas (if using schema mapping)
├── Step 5: Create OneLake shortcuts (Delta → Tables/, non-Delta → Files/)
├── Step 6: Handle non-Delta tables — convert to Delta or retain original format
├── Step 7: Validate
└── Step 8: (Optional) Validate before proceeding to Phase 2Step 0: Detect External HMS Configuration
Read the Spark pool configuration from the ARM API to determine if an external HMS is configured.
Endpoint:
GET https://management.azure.com/subscriptions/{subId}/resourceGroups/{rg}/providers/Microsoft.Synapse/workspaces/{ws}/bigDataPools/{poolName}?api-version=2021-06-01In the response, check properties.sparkConfigProperties.content for these keys:
| Spark Config Key | Present? | Meaning |
|---|---|---|
spark.hadoop.javax.jdo.option.ConnectionURL | Yes | External HMS — this guide applies |
spark.hadoop.javax.jdo.option.ConnectionDriverName | Yes | JDBC driver (e.g., com.microsoft.sqlserver.jdbc.SQLServerDriver for Azure SQL, org.mariadb.jdbc.Driver for MySQL) |
spark.hadoop.javax.jdo.option.ConnectionUserName | Yes | HMS database username |
spark.hadoop.javax.jdo.option.ConnectionPassword | Yes | HMS database password (may reference Key Vault) |
spark.sql.hive.metastore.version | Optional | HMS version (e.g., 3.1.0) |
spark.sql.hive.metastore.jars | Optional | Path to HMS client JARs |
If none of these keys are present: The workspace uses the built-in HMS → use lake-database-migration.md instead.
Extract Connection Details
# Parse the JDBC connection URL from pool config
# Azure SQL DB example: jdbc:sqlserver://myserver.database.windows.net:1433;database=hive_metastore;...
# MySQL example: jdbc:mysql://myserver.mysql.database.azure.com:3306/hive_metastore?...
connection_url = spark_config["spark.hadoop.javax.jdo.option.ConnectionURL"]
driver = spark_config["spark.hadoop.javax.jdo.option.ConnectionDriverName"]
username = spark_config["spark.hadoop.javax.jdo.option.ConnectionUserName"]
# password may be a Key Vault reference — resolve before connectingStep 1: Inventory — Query the HMS Database
Connect to the external HMS database via JDBC and extract metadata. The Hive Metastore uses a standard schema (same structure for both Azure SQL DB and MySQL).
Namespace: The external HMS has a flat namespace —
Database → Table. There are no inner schemas. This simplifies mapping to Fabric.
JDBC Connection Troubleshooting
If the JDBC connection to the external HMS database fails, check these common causes:
| Error | Cause | Fix |
|---|---|---|
Login failed for user / Access denied for user | Wrong credentials or expired password | Verify username/password; check if Key Vault secret has rotated |
Cannot open server ... requested by the login | Database name is wrong or database has been deleted | Verify the database name in the JDBC URL |
Connection timed out / No route to host | Firewall blocks access from the machine running the migration | Add client IP to Azure SQL / MySQL firewall rules; check VNet/Private Endpoint settings |
SSL handshake failed / certificate verify failed | TLS configuration mismatch | Add encrypt=true;trustServerCertificate=true (Azure SQL) or useSSL=true&requireSSL=false (MySQL) to JDBC URL |
com.microsoft.sqlserver.jdbc.SQLServerException: TCP/IP connection ... has failed | SQL Server is paused or stopped (serverless) | Resume the Azure SQL DB in the Portal |
Communications link failure | Network-level connectivity issue (DNS, proxy, VPN) | Test connectivity with Test-NetConnection -ComputerName {server} -Port {port} |
1a. List Databases
SELECT
d.DB_ID,
d.NAME AS database_name,
d.DB_LOCATION_URI AS location,
d.OWNER_NAME AS owner
FROM DBS d
ORDER BY d.NAME;1b. List Tables with Storage Info
SELECT
d.NAME AS database_name,
t.TBL_NAME AS table_name,
t.TBL_TYPE AS table_type, -- MANAGED_TABLE or EXTERNAL_TABLE
s.LOCATION AS data_location,
s.INPUT_FORMAT AS input_format,
s.OUTPUT_FORMAT AS output_format,
tp_provider.PARAM_VALUE AS spark_provider, -- 'delta', 'parquet', etc.
tp_delta.PARAM_VALUE AS is_delta -- non-null if Delta table
FROM TBLS t
JOIN DBS d ON t.DB_ID = d.DB_ID
JOIN SDS s ON t.SD_ID = s.SD_ID
LEFT JOIN TABLE_PARAMS tp_provider
ON t.TBL_ID = tp_provider.TBL_ID AND tp_provider.PARAM_KEY = 'spark.sql.sources.provider'
LEFT JOIN TABLE_PARAMS tp_delta
ON t.TBL_ID = tp_delta.TBL_ID AND tp_delta.PARAM_KEY = 'delta.lastCommitTimestamp'
ORDER BY d.NAME, t.TBL_NAME;Detecting table format:
| How to detect | Condition | Format |
|---|---|---|
spark.sql.sources.provider = 'delta' | Preferred | Delta |
delta.lastCommitTimestamp is not null | Fallback | Delta |
INPUT_FORMAT contains parquet and no delta markers | — | Parquet |
INPUT_FORMAT contains orc | — | ORC |
INPUT_FORMAT contains Text or csv | — | CSV/Text |
spark.sql.sources.provider = 'parquet' | — | Parquet |
spark.sql.sources.provider = 'orc' | — | ORC |
1c. List Columns
SELECT
d.NAME AS database_name,
t.TBL_NAME AS table_name,
c.COLUMN_NAME,
c.TYPE_NAME,
c.INTEGER_IDX AS ordinal_position
FROM COLUMNS_V2 c
JOIN SDS s ON c.CD_ID = s.CD_ID
JOIN TBLS t ON t.SD_ID = s.SD_ID
JOIN DBS d ON t.DB_ID = d.DB_ID
ORDER BY d.NAME, t.TBL_NAME, c.INTEGER_IDX;1d. List Partition Keys
SELECT
d.NAME AS database_name,
t.TBL_NAME AS table_name,
pk.PKEY_NAME AS partition_column,
pk.PKEY_TYPE AS partition_type,
pk.INTEGER_IDX AS ordinal_position
FROM PARTITION_KEYS pk
JOIN TBLS t ON pk.TBL_ID = t.TBL_ID
JOIN DBS d ON t.DB_ID = d.DB_ID
ORDER BY d.NAME, t.TBL_NAME, pk.INTEGER_IDX;1e. List Partitions (for non-Delta tables)
SELECT
d.NAME AS database_name,
t.TBL_NAME AS table_name,
p.PART_ID,
s.LOCATION AS partition_location,
p.CREATE_TIME
FROM PARTITIONS p
JOIN TBLS t ON p.TBL_ID = t.TBL_ID
JOIN DBS d ON t.DB_ID = d.DB_ID
JOIN SDS s ON p.SD_ID = s.SD_ID
ORDER BY d.NAME, t.TBL_NAME, p.CREATE_TIME;Delta tables: Skip partition enumeration — Delta handles partitions internally via
_delta_log. Only query partitions for non-Delta tables that needMSCK REPAIR TABLEafter migration.
1f. Summary Output
After running the queries, produce an inventory summary:
External HMS Inventory:
HMS Database: jdbc:sqlserver://myserver.database.windows.net;database=hive_metastore
Total databases: 5
Total tables: 142
Database: sales (37 tables)
Delta tables: 30 (24 managed, 6 external)
Parquet tables: 5 (all external)
ORC tables: 2 (all managed)
Partitioned tables: 8
Database: marketing (22 tables)
Delta tables: 22 (all managed)
Partitioned tables: 3
Database: staging (45 tables)
...Step 2: Select Databases and Choose Mapping Mode
Database Selection
If the external HMS is shared with other platforms (HDInsight, Databricks), not all databases may be Synapse-owned. Ask the user which databases to migrate:
{
"databasesToMigrate": ["sales", "marketing", "staging"],
"databasesToSkip": ["hdinsight_etl", "databricks_ml"]
}If the HMS is Synapse-only (being decommissioned), migrate all databases.
Mapping Mode
Choose one of two modes:
Mode A: Schemas in One Lakehouse (Default)
All selected HMS databases → schemas within one target Lakehouse.
| HMS Database | Fabric Target |
|---|---|
sales | Schema sales in target Lakehouse |
marketing | Schema marketing in target Lakehouse |
staging | Schema staging in target Lakehouse |
default | Schema dbo (Lakehouse default) |
Advantages: Fewer items to manage; cross-schema queries via 2-part names; single SQL endpoint.
Disadvantages: Less isolation; shared OneLake path; harder to assign per-database permissions.
No schema collision risk: The external HMS has a flat namespace (Database → Table), so each database name maps directly to a Fabric schema name with no composite naming needed.
Mode B: Separate Lakehouses
Each selected HMS database → its own Fabric Lakehouse.
| HMS Database | Fabric Target |
|---|---|
sales | Lakehouse sales |
marketing | Lakehouse marketing |
staging | Lakehouse staging |
Advantages: Strong isolation; independent security (OneLake RBAC per Lakehouse); independent SQL endpoints.
Disadvantages: More items to manage; cross-database queries require 3-part names.
Recommended when: The HMS is shared with other platforms and you want clear isolation for migrated databases, or different databases are owned by different teams.
Emit Mapping Report
Before creating anything, show the user the planned mapping:
External HMS Migration Plan:
Source: jdbc:sqlserver://myserver.database.windows.net;database=hive_metastore
Mode: A (schemas in one Lakehouse)
Target Lakehouse: MigratedData_Lakehouse
HMS sales.customers → MigratedData_Lakehouse.sales.customers (Delta, shortcut)
HMS sales.orders → MigratedData_Lakehouse.sales.orders (Delta, shortcut)
HMS sales.legacy_archive → MigratedData_Lakehouse.sales.legacy_archive (Parquet, Files/)
HMS marketing.campaigns → MigratedData_Lakehouse.marketing.campaigns (Delta, shortcut)
HMS staging.raw_events → MigratedData_Lakehouse.staging.raw_events (Delta, shortcut)Step 3: Create Fabric Lakehouse(es)
Endpoint: POST https://api.fabric.microsoft.com/v1/workspaces/{workspaceId}/items
{
"displayName": "{lakehouseName}",
"type": "Lakehouse",
"description": "Migrated from external Hive Metastore",
"creationPayload": {
"enableSchemas": true
}
}Mode A: Create one Lakehouse with
enableSchemas: trueMode B: Create one Lakehouse per selected database (still use
enableSchemas: true)
Returns HTTP 202 (LRO). Poll
Locationheader untilstatus == "Succeeded". Capture theidfor subsequent steps.
Step 4: Create Schemas (Mode A Only)
For each selected HMS database, create the corresponding Fabric schema:
CREATE SCHEMA IF NOT EXISTS sales;
CREATE SCHEMA IF NOT EXISTS marketing;
CREATE SCHEMA IF NOT EXISTS staging;
-- 'default' database maps to the built-in 'dbo' schema — no creation neededExecute via:
SQL endpoint: Connect to the Lakehouse SQL endpoint
Fabric notebook: Run in a notebook cell attached to the Lakehouse
Livy session:
POST /v1/workspaces/{workspaceId}/lakehouses/{lakehouseId}/livySessions
Step 5: Create OneLake Shortcuts
For each table in the selected databases, create a shortcut based on format and type.
Shortcut Target Decision
| Table Format | Shortcut Location | Catalog Registration |
|---|---|---|
| Delta | Tables/{schema}/{tableName} | Auto-registers in Lakehouse catalog |
| Parquet / ORC / CSV / Avro | Files/{schema}/{tableName} | Not auto-registered — handle in Step 6 |
Parse Storage Location
The SDS.LOCATION (or data_location from Step 1b) provides the ADLS Gen2 path. Parse it into shortcut parameters:
Managed tables (typical warehouse directory):
abfss://{container}@{storage}.dfs.core.windows.net/synapse/workspaces/{workspace}/warehouse/{database}.db/{tableName}location:https://{storage}.dfs.core.windows.netsubpath:/{container}/synapse/workspaces/{workspace}/warehouse/{database}.db/{tableName}
Note: External HMS managed tables may use
.dbsuffix in the warehouse directory (e.g.,sales.db/customers), unlike built-in HMS which omits it. Check the actualSDS.LOCATIONvalue.
External tables (arbitrary ADLS path):
abfss://{container}@{storage}.dfs.core.windows.net/{custom/path/to/table}location:https://{storage}.dfs.core.windows.netsubpath:/{container}/{custom/path/to/table}
Create Shortcut API
Endpoint: POST https://api.fabric.microsoft.com/v1/workspaces/{workspaceId}/items/{lakehouseId}/shortcuts
Note: The endpoint uses
/items/{lakehouseId}/shortcuts, NOT/lakehouses/{lakehouseId}/shortcuts(which returns 404).
{
"name": "{tableName}",
"path": "Tables/{schemaName}",
"target": {
"type": "AdlsGen2",
"adlsGen2": {
"location": "https://{storageAccount}.dfs.core.windows.net",
"subpath": "/{container}/{path/to/table}",
"connectionId": "{connectionId}"
}
}
}
connectionIdis required. See lake-database-migration.md § Step 4b for how to discover or create the ADLS connection.
For non-Delta tables, change
"path"to"Files/{schemaName}".
Authentication for Shortcuts
The Fabric workspace identity (or creating user) must have Storage Blob Data Reader on the target ADLS Gen2 storage account.
Step 6: Handle Non-Delta Tables
For non-Delta tables (Parquet, ORC, CSV, Avro), the shortcut is created under Files/ but the table is not auto-registered in the Lakehouse catalog. Choose one option:
Option A: Convert to Delta (Recommended)
# Read from the shortcut path
df = spark.read.format("{originalFormat}").load("Files/{schemaName}/{tableName}")
# Write as Delta to the Tables section
df.write.format("delta").mode("overwrite").saveAsTable("{schemaName}.{tableName}")For partitioned tables (identified in Step 1d):
df = spark.read.format("parquet").load("Files/{schemaName}/{tableName}")
df.write.format("delta") \
.partitionBy("{partitionCol}") \
.mode("overwrite") \
.saveAsTable("{schemaName}.{tableName}")Advantages: Full Lakehouse catalog registration, SQL endpoint queries, Power BI Direct Lake, V-Order, ACID transactions.
Option B: Retain Original Format
Keep tables in their legacy format under Files/. Register them in the catalog manually:
CREATE TABLE IF NOT EXISTS {schemaName}.{tableName}
USING {format}
LOCATION 'Files/{schemaName}/{tableName}';For Hive-style partitioned tables, recover partition metadata:
CREATE TABLE IF NOT EXISTS {schemaName}.{tableName}
USING PARQUET
PARTITIONED BY ({partitionCols})
LOCATION 'Files/{schemaName}/{tableName}';
MSCK REPAIR TABLE {schemaName}.{tableName};Why
MSCK REPAIR TABLE? Non-Delta tables rely on HMS partition metadata for partition pruning. Without this step, queries scan all files instead of pruning to relevant partitions.
Comparison
| Capability | Delta (Option A) | Original Format (Option B) |
|---|---|---|
| Lakehouse catalog registration | Yes | Yes (after CREATE TABLE) |
| SQL endpoint queries | Yes | Yes |
| Power BI Direct Lake | Yes | No — requires Delta |
| V-Order optimization | Yes | No |
| ACID / time travel | Yes | No |
| Partition pruning | Automatic | Requires MSCK REPAIR TABLE |
| Data duplication | Yes (new copy) | No (zero-copy via shortcut) |
Step 7: Validate
After migration, verify:
Lakehouse catalog: Check that tables appear in the Lakehouse Explorer UI under Tables
SQL endpoint: Query migrated tables via the SQL endpoint to confirm schema and data
Row counts: Compare row counts between HMS (via JDBC or Spark SQL on Synapse) and Fabric
-- On Synapse (or via JDBC to HMS + Spark) SELECT COUNT(*) FROM sales.customers; -- On Fabric SELECT COUNT(*) FROM sales.customers;Shortcut health: Verify shortcuts are accessible
notebookutils.fs.ls("Tables/sales/") notebookutils.fs.ls("Files/sales/") # if non-Delta tables existPartition coverage (for Option B non-Delta tables): Verify partition counts match
SHOW PARTITIONS {schemaName}.{tableName};
Step 8: (Optional) Validate Before Proceeding to Phase 2
Run these checks before migrating Notebooks. Notebooks rely on Lakehouses being healthy — missing shortcuts or unregistered tables cause immediate runtime failures.
| Check | How | Pass Criteria |
|---|---|---|
| Shortcut health | notebookutils.fs.ls("Tables/{schema}/") and Files/{schema}/ | All shortcuts resolve; no PathNotFound errors |
| Row counts | Compare SELECT COUNT(*) on HMS source (via JDBC/Spark) vs. Fabric for each table | Counts match |
| Schema comparison | Compare column names and types from HMS COLUMNS_V2 vs. Fabric DESCRIBE TABLE | Exact match |
| Non-Delta registration | SHOW TABLES IN {schema} | All Option B tables appear in catalog |
| Partition coverage | SHOW PARTITIONS {schema}.{table} for Option B partitioned tables | Partition count matches HMS PARTITIONS table |
Do not proceed to Phase 2 until all shortcuts are healthy and row counts match. A notebook that reads from a missing or broken shortcut will fail silently or produce wrong results.
See validation-testing.md → V2: Data Validation for detailed scripts.
Limitations and Considerations
| Limitation | Impact | Mitigation |
|---|---|---|
| External HMS is deprecated after Spark 3.4 | No new development; should migrate sooner rather than later | This guide helps you migrate off it |
| Fabric cannot connect to an external HMS | Must copy metadata into Lakehouse catalog — no live connection | This guide extracts and recreates all metadata |
| HMS functions (UDFs) are not migrated | Custom functions stored in HMS are not extracted by these queries | Recreate manually via spark.udf.register() or CREATE FUNCTION |
| HMS views are not migrated | Views stored in HMS are not extracted | Extract view definitions (see query below), then recreate as Spark SQL views in Fabric |
| Shared HMS — other platforms still using it | Migrating databases to Fabric doesn't remove them from the external HMS | Coordinate with other platform teams; HMS remains untouched |
| Large catalogs (10K+ tables) | JDBC queries scale well, but shortcut creation is sequential (one API call per table) | Batch shortcut creation; consider parallel requests (respect API rate limits) |
| Managed table data in Synapse storage | Workspace-internal storage requires Fabric identity to have read access | Grant Storage Blob Data Reader before creating shortcuts |
Extracting HMS view definitions:
SELECT
d.NAME AS database_name,
t.TBL_NAME AS view_name,
vt.PARAM_VALUE AS view_sql
FROM TBLS t
JOIN DBS d ON t.DB_ID = d.DB_ID
JOIN TABLE_PARAMS vt ON t.TBL_ID = vt.TBL_ID AND vt.PARAM_KEY = 'view.query.text'
WHERE t.TBL_TYPE = 'VIRTUAL_VIEW'
AND vt.PARAM_VALUE IS NOT NULL
ORDER BY d.NAME, t.TBL_NAME;The view SQL is stored in the
view.query.textparameter. Recreate in Fabric:CREATE OR REPLACE VIEW {schema}.{view_name} AS {view_sql}.
Requirements and compatibility
Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta
“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.
Needs: Needs Python
Security
What our scanner noticed· Low risk
- Instructions that use the network
- Scripts that your agent may run
This skill includes scripts that your agent may run. Read them before you use it.
A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.
Attribution and licence
- Published by
- Microsoft
- Publisher
- Microsoft (Official)
- Source
- microsoft/skills-for-fabric · skills/synapse-migration · commit f8fac11
- Licence
- MIT License (redistribution allowed)
- LICENSE (added by AutoGineer from LICENSE)
- ATTRIBUTION.md (written by AutoGineer)
- Synced
- Also distributed in
- microsoft/skills-for-fabric
AutoGineer is not affiliated with or endorsed by Microsoft.
Download ZIP
The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download synapse-migration.zip