Synapse Migration

Migration Troubleshooting Guide

Common migration failures, edge cases, and resolution steps. Items are flagged during migration and surfaced in the migration report with links back to the relevant section here.

Philosophy: The legacy notebook and Spark job definition issues in this guide do not block artifact copying; affected items are copied as-is, failures are logged, and this guide is linked from the report for post-migration remediation. This does not override fail-closed conversion gates in other migration paths. In particular, Dedicated SQL Pool objects classified as ManualReviewRequired are not converted or deployed until the required decision or approval is recorded.

G1: spark.read.synapsesql() — No Direct Fabric Equivalent

Flag ID: SYNAPSESQL_NO_EQUIVALENT

When it fires: Code audit detects synapsesql pattern in a notebook or SJD.

Symptom: After migration, running the notebook fails with:

AnalysisException: Failed to find data source: synapsesql

Root cause: The spark.read.synapsesql() connector reads from Synapse Dedicated SQL Pools. This connector does not exist in Fabric.

Resolution Options

Code Example — Option B (JDBC)

# ❌ BEFORE — Synapse
df = spark.read.synapsesql("mypool.dbo.FactSales")

# ✅ AFTER — Fabric (JDBC to SQL endpoint)
jdbc_url = "jdbc:sqlserver://myserver.sql.azuresynapse.net:1433;database=mypool"
token = notebookutils.credentials.getToken("https://database.windows.net/")

df = (spark.read
    .format("jdbc")
    .option("url", jdbc_url)
    .option("dbtable", "dbo.FactSales")
    .option("accessToken", token)
    .option("encrypt", "true")
    .option("hostNameInCertificate", "*.sql.azuresynapse.net")
    .load())

See also: connector-refactoring.md § spark.read.synapsesql() for the full before/after pattern.

G2: Custom Library Version Conflicts with Fabric Runtime

Flag ID: LIBRARY_VERSION_CONFLICT

When it fires: Environment publish fails, or a notebook import fails with version resolution errors. Also flagged when Synapse pool libraries specify versions incompatible with Fabric Runtime 1.3 built-ins.

Symptom: After Environment publish or notebook execution:

ResolvePackageNotFound: ... conflicts with runtime built-in version

or

ImportError: cannot import name 'X' from 'Y' (wrong version installed)

Root cause: Synapse Spark pools may pin library versions (e.g., pandas==1.4.3) that conflict with Fabric Runtime 1.3 built-in versions (e.g., pandas==2.1.1). Fabric does not allow downgrading built-in libraries.

Resolution Steps

1. Identify conflicts — compare libraryRequirements.content from the Synapse pool against Fabric Runtime 1.3 built-in versions:

   Copy

   # In a Fabric notebook, list built-in versions:
   import pkg_resources
   for pkg in sorted(pkg_resources.working_set, key=lambda p: p.key):
       print(f"{pkg.key}=={pkg.version}")

2. Remove pins for built-in libraries — if the Synapse requirements.txt pins a version that Fabric already provides at a newer version, remove the pin:

   Copy

   # Synapse requirements.txt:
   pandas==1.4.3          ← REMOVE (Fabric has 2.1.1)
   scikit-learn==1.2.0    ← REMOVE (Fabric has 1.3.2)
   my-custom-lib==1.0.0   ← KEEP (not a built-in)

3. Test for breaking API changes — if the version jump is major (e.g., pandas 1.x → 2.x), code may break. Common issues:

   • pandas.DataFrame.append() removed in pandas 2.x → use pd.concat()

   • sklearn.externals.joblib removed → use import joblib directly

   • numpy dtype changes (e.g., np.int → np.int64)

4. If a specific older version is required — use %pip install in the notebook as a last resort:

   Copy

   %pip install pandas==1.4.3 --force-reinstall

   This overrides the built-in version for the session only. Not recommended for production — may cause instability with other built-in libraries that depend on the newer version.

See also: library-compatibility.md for a complete list of Synapse-vs-Fabric library gaps.

G3: Delta Lake Protocol Version Incompatibility

Flag ID: DELTA_PROTOCOL_MISMATCH

When it fires: A Lakehouse shortcut points to Delta tables written by Synapse using a newer or incompatible Delta protocol version.

Symptom: Reading the shortcutted table fails with:

DeltaErrors: The table has reader version X, which is not supported by this version of Delta Lake

or

Unsupported reader/writer feature: deletionVectors / columnMapping / v2Checkpoints

Root cause: Synapse Spark pools may write Delta tables using protocol features that the selected Fabric Runtime or downstream reader does not support, or the target DDL may contain logical column names that require Delta column mapping.

Resolution Steps

1. Check the protocol version of the source Delta table:

   Copy

   # In a Synapse notebook (before migration):
   from delta.tables import DeltaTable
   dt = DeltaTable.forPath(spark, "abfss://[email protected]/path/to/table")
   detail = dt.detail().select("minReaderVersion", "minWriterVersion").collect()[0]
   print(f"Reader: {detail.minReaderVersion}, Writer: {detail.minWriterVersion}")

2. Check the selected Fabric Runtime and every downstream reader against current Microsoft documentation. Do not use a hardcoded runtime/version matrix.

3. Keep the source read-only. If the source shortcut protocol is unsupported, record ManualReviewRequired; do not rewrite, drop, rename, or downgrade the source table as part of this migration.

4. Plan new target DDL selectively with plan_delta_column_mapping:

   • safe logical names: normal Delta DDL;

   • restricted names such as spaces, commas, semicolons, braces, parentheses, newlines, tabs, or equals signs: require downstream compatibility approval, then emit TBLPROPERTIES ('delta.columnMapping.mode' = 'name');

   • existing compatible target or shortcut: validate and reuse it without protocol changes;

   • existing incompatible or uncertain target: block for manual review rather than altering it.

5. Record protocol and mapping evidence in expected-schema.json and migration-manifest.json, including offending logical names, current mode, decision, and approval.

Pre-migration check: Inventory Delta protocol features and column mapping for every existing target or shortcut, and validate downstream compatibility before any target mutation.

G4: Synapse Security Model — Managed Identities & IP Firewall

Flag ID: SECURITY_MODEL_INCOMPATIBLE

When it fires: The Synapse workspace uses security features that don't have direct Fabric equivalents.

G4a: Managed Identity Differences

Symptom: After migration, notebooks fail with 403 Forbidden when accessing external resources (ADLS, Key Vault, SQL).

Steps:

1. Enable Fabric Workspace Identity: Workspace Settings → Identity → Enable

2. Grant the workspace identity the same roles the Synapse MSI had:

   • Storage Blob Data Reader/Contributor on ADLS Gen2 accounts

   • Key Vault Secrets User on Key Vault instances

   • SQL db_datareader on SQL databases (via Entra ID admin)

3. Update notebook code that used mssparkutils.credentials.getToken() — this works in Fabric with notebookutils.credentials.getToken(), but the identity backing it is now the Fabric workspace identity, not the Synapse MSI

G4b: IP Firewall Rules

Symptom: Synapse workspace had IP-based access control. Fabric does not support IP firewall rules at the workspace level.

Steps:

1. Document existing IP rules from the Synapse workspace: Portal → Synapse workspace → Firewalls

2. Create Entra Conditional Access policy: Entra ID → Security → Conditional Access → New policy

   • Target: Microsoft Fabric (app ID)

   • Condition: Named locations (replicate the IP ranges from Synapse firewall)

   • Grant: Allow access only from named locations

3. For private connectivity: Set up Managed Private Endpoints in a Fabric Environment with Custom Pool. See security-governance.md § S6

See also: security-governance.md for the full security migration guide.

G5: GPU Pool Migration Blocker

Flag ID: GPU_POOL_UNSUPPORTED

When it fires: Synapse Spark pool has nodeSizeFamily == "HardwareAcceleratedGPU".

Symptom: No Fabric equivalent — GPU-accelerated Spark is not available in Fabric.

Resolution Options

G6: .NET for Spark (C#/F#) SJD Blocker

Flag ID: DOTNET_SPARK_UNSUPPORTED

When it fires: SJD has language == "dotnet" or language == "csharp".

Symptom: Fabric does not support .NET for Apache Spark.

Resolution

Rewrite the SJD in Python or Scala. Common patterns:

G7: bigDataPool / targetBigDataPool Field is null (Not Missing)

Flag ID: NULLABLE_POOL_REFERENCE

When it fires: Code reads properties.bigDataPool.referenceName (notebooks) or properties.targetBigDataPool.referenceName (SJDs) and crashes with AttributeError: 'NoneType' object has no attribute 'get'.

Root cause: The Synapse API returns "bigDataPool": null (field present with value null) when no pool is assigned — instead of omitting the field entirely. The common pattern .get("bigDataPool", {}).get("referenceName") fails because dict.get() returns None (the actual value), not {} (the default).

Resolution

Use the or {} defensive pattern:

# ❌ WRONG — crashes when bigDataPool is null
pool_name = nb.get("bigDataPool", {}).get("referenceName", "")

# ✅ CORRECT — handles null, missing, and present cases
pool_name = (nb.get("bigDataPool") or {}).get("referenceName", "")

Same pattern applies to SJDs:

pool_name = (sjd.get("targetBigDataPool") or {}).get("referenceName", "")

G8: Notebook Session Configuration (%%configure)

Flag ID: SESSION_CONFIG_IGNORED

When it fires: Notebook contains %%configure magic with Synapse-specific driver/executor settings.

Symptom: %%configure is supported in Fabric but some Synapse-specific keys are silently ignored.

Synapse-Specific Keys to Remove

// ❌ These keys are Synapse-specific — remove from %%configure:
{
  "conf": {
    "spark.synapse.pool.name": "MyPool",          // → remove
    "spark.storage.synapse.linkedServiceName": "", // → remove
    "spark.synapse.workspace.name": ""             // → remove
  },
  "driverMemory": "28g",     // ✅ kept — works in Fabric
  "executorMemory": "28g",   // ✅ kept — works in Fabric
  "numExecutors": 4          // ✅ kept — works in Fabric
}

G9: Shortcut Creation Failures (ADLS Connection Issues)

Flag ID: SHORTCUT_CONNECTION_FAILED

When it fires: Phase 1 shortcut creation fails with 403, 400 (credential), or connection reset errors.

Symptom: Lakehouse is created but shortcuts are missing — tables are not accessible. The migration script aborts remaining shortcuts after the first fatal error.

Common Causes

Resolution

1. 403 (permission):

   • Identify the connection's identity (Service Principal or user) from the Fabric Portal → Manage connections

   • Grant Storage Blob Data Reader on the storage account (or container) via Azure Portal → IAM

   • Re-run the migration — it will resume from where it left off (existing shortcuts return 409)

2. 400 Stored Credential:

   • Open Fabric Portal → Settings → Manage connections and gateways

   • Find the ADLS connection → Edit → Re-authenticate with valid credentials

   • Alternatively, create a new connection and delete the stale one

   • Re-run the migration

3. Connection reset:

   • Retry after a few minutes — this is typically transient

   • If persistent, check VNet/private endpoint configuration between Fabric and the storage account

Note: The migration script uses an early-abort pattern for these errors — it stops creating shortcuts after the first fatal error because all subsequent shortcuts to the same storage account will fail for the same reason. See lake-database-migration.md § Shortcut Creation Error Cascade for the implementation pattern.

Flag ID Summary — Quick Reference

Use these IDs in the migration report to link to the corresponding section: