Databricks Migration

MicrosoftOfficialPublished by Microsoft

Skill
Workflow
Cloud and DevOps
MIT
Low risk

From the author

Ports existing Databricks notebooks and jobs to Fabric, covering dbutils to notebookutils, secret scopes to Key Vault, DBFS to OneLake, Unity Catalog mapping to schema-enabled Lakehouses, Jobs and Delta Live Tables to Spark Job Definitions and Pipelines, and Photon to the Native Execution Engine. Use whenever Databricks code has to be converted. For Fabric notebook code that is not being migrated, use spark-cli.

— the description by Microsoft

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

The databricks-migration skill provides a structured framework for porting Databricks assets to Microsoft Fabric. It covers the conversion of core components including dbutils to notebookutils, secret scopes to Azure Key Vault, DBFS to OneLake, and Unity Catalog structures to schema-enabled Lakehouses. The skill emphasizes a systematic approach, requiring inventory assessment before prescribing topology, and provides specific mappings for Spark job definitions, MLflow experiments, and cluster configurations to ensure a smooth transition to Fabric's native execution environment.

What it does

Maps dbutils API calls to their notebookutils equivalents.
Provides strategies for migrating Unity Catalog namespaces to Fabric Lakehouse schemas.
Translates Databricks Jobs and Delta Live Tables into Fabric Spark Job Definitions and Pipelines.
Guides the transition from Databricks cluster configurations to Fabric Spark Pools and Environments.
Facilitates the migration of MLflow tracking and model registry workflows.
Defines a multi-notebook migration protocol to manage complex workspace transitions.

When to use it

When porting existing Databricks notebooks, jobs, or data pipelines to Microsoft Fabric.
When needing to resolve namespace collisions during catalog migration.
When replacing proprietary Databricks features like DLT or DBFS with Fabric-native alternatives.
When configuring security and secret management for migrated workloads in Fabric.

Advantages

Ensures architectural alignment with Fabric's native execution engine.
Provides clear, actionable mappings for deprecated Databricks-specific APIs.
Reduces manual effort through structured migration patterns and protocols.
Promotes best practices for library management and environment configuration in Fabric.

Usage examples

“How do I migrate my dbutils.fs.ls calls to Fabric?”: The agent will provide the equivalent notebookutils.fs.ls syntax and explain the necessary path adjustments from DBFS to OneLake.
“I need to move my Databricks Unity Catalog to Fabric. What is the recommended approach?”: The agent will guide you through mapping Unity Catalog catalogs to schema-enabled Lakehouses and explain how to handle namespace preservation.
“How should I handle Databricks Jobs that use Delta Live Tables?”: The agent will explain that DLT has no direct equivalent and will provide a plan to rewrite the logic as parameterized notebooks orchestrated by Fabric Pipelines.
“What is the Fabric equivalent for my Databricks secret scopes?”: The agent will instruct you to use notebookutils.credentials.getSecret, mapping the Databricks scope to an Azure Key Vault URL.

Install

Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/microsoft/databricks-migration/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip databricks-migration.zip -d .agents/skills

Documentation by Microsoft

Telemetry — MANDATORY. Every api.fabric.microsoft.com call must carry x-ms-fabric-skill: databricks-migration (az rest: --headers "x-ms-fabric-skill=databricks-migration"), including every LRO poll, fabric_lro and retry. Snippets omit it — add it anyway.

CRITICAL NOTES
To find workspace details (including its ID) from a workspace name: list all workspaces, then use JMESPath filtering
To find item details (including its ID) from workspace ID, item type, and item name: list all items of that type in that workspace, then use JMESPath filtering
dbutils.widgets has no direct equivalent in Fabric — use notebook parameters (cell tag parameters); notebookutils.runtime.context is execution metadata, not parameter storage. If showing context fields, use documented names such as currentWorkspaceId, currentWorkspaceName, currentNotebookId, currentNotebookName, isForPipeline, and isForInteractive; activityId is the Livy job ID
dbutils.library (runtime library install) has no equivalent — use Fabric Environments for reproducible library management
Map each Unity Catalog catalog to a schema-enabled Lakehouse by default. This preserves the source schema.table hierarchy, with the Lakehouse representing the catalog; collisions arise only if multiple catalogs are intentionally consolidated into one Lakehouse
For an under-specified workspace-wide migration, ask focused questions about inventory, workload topology, security, data locations, and runtime constraints before recommending a Fabric topology
A completed Fabric migration must not retain executable dbutils.* calls in dual-runtime branches or try/except guards — replace the calls and Databricks paths outright

Databricks → Microsoft Fabric Migration

Prerequisite Knowledge

Read these companion documents before executing migration tasks:

COMMON-CORE.md — Fabric REST API patterns, authentication, token audiences, item discovery
COMMON-CLI.md — az rest, az login, token acquisition, Fabric REST via CLI
SPARK-AUTHORING-CORE.md — Notebook deployment, lakehouse creation, Spark job execution

For notebook and Lakehouse creation, see spark-cli. For Fabric Warehouse DDL/DML authoring, see sqldw-cli.

Topic	Reference
Migration Orchestrator	migration-orchestrator.md
Migration Workload Map	§ Migration Workload Map
Complete `dbutils` → `notebookutils` Mapping	dbutils-to-notebookutils.md
Unity Catalog → Fabric Lakehouse Schemas	catalog-migration.md
Before/After Code Patterns	code-patterns.md
Cluster Config → Fabric Spark Pools	§ Cluster Config → Fabric Spark Pools
Databricks Jobs → Spark Job Definitions	§ Databricks Jobs → Spark Job Definitions
Delta Sharing → Fabric External Data Sharing and OneLake Shortcuts	§ Delta Sharing → Fabric External Data Sharing and OneLake Shortcuts
MLflow → Fabric ML Experiments	§ MLflow → Fabric ML Experiments
Post-Migration Validation & Testing	validation-testing.md
Migration Gotchas & Troubleshooting	migration-gotchas.md
Multi-Notebook Migration Protocol	§ Multi-Notebook Migration Protocol
Failure Reporting	§ Failure Reporting
Must / Prefer / Avoid	§ Must / Prefer / Avoid
Authentication & Token Acquisition	COMMON-CORE.md § Authentication
Lakehouse Management	SPARK-AUTHORING-CORE.md § Lakehouse Management
Notebook Management	SPARK-AUTHORING-CORE.md § Notebook Management

Context Loading Guide

IMPORTANT — Load only what you need. Do NOT read all resource files upfront. Load the specific file for the phase you are executing:

When	Read This File
User asks to migrate a workspace (full orchestration)	migration-orchestrator.md
Applying code transforms (dbutils, namespaces, paths)	dbutils-to-notebookutils.md + code-patterns.md
Resolving Unity Catalog namespace collisions	catalog-migration.md
Post-migration verification	validation-testing.md
Troubleshooting failures or known issues	migration-gotchas.md

Migration Workload Map

Databricks Component	Fabric Target	Severity	Notes
All-purpose cluster (notebooks, REPL)	Fabric Notebook (Starter Pool or Custom Pool)	Info	No persistent cluster — Fabric provisions compute on session start
Job cluster (automated jobs)	Spark Job Definition (SJD)	Info	SJD maps one-to-one with Databricks Jobs on job clusters
Unity Catalog	Fabric Lakehouse (schema-enabled, one per catalog)	Info	Schema-enabled Lakehouse preserves `schema.table`; default one-Lakehouse-per-catalog has no collision — see catalog-migration.md
Databricks Repos (Git-backed notebooks)	Fabric Git Integration	Info	Connect workspace to Azure DevOps or GitHub; notebooks are synced
Delta Live Tables (DLT)	Fabric Notebooks + Data Pipelines	Blocker	No DLT equivalent — rewrite DLT datasets as parameterized notebook cells with pipeline orchestration
Databricks SQL Warehouses	Fabric Warehouse or Lakehouse SQL Endpoint	Info	SQL warehouse sessions → Warehouse (for write) or SQL Endpoint (for read-only)
MLflow Tracking	Fabric ML Experiments	Info	MLflow SDK is supported in Fabric — see § MLflow
Delta Sharing	OneLake Shortcuts + Fabric external data sharing	Warning	See § Delta Sharing → Fabric External Data Sharing and OneLake Shortcuts
Databricks Feature Store	Feature engineering on Lakehouse/Delta tables + MLflow	Warning	Fabric has no drop-in managed Feature Store; recreate feature tables as Delta tables in a Lakehouse and manage features via notebooks/MLflow. Verify current Fabric feature-store roadmap before committing an approach
dbutils (all sub-modules)	`notebookutils` (most sub-modules)	Info	See dbutils-to-notebookutils.md for full mapping
Scala notebooks	Fabric Notebook (Spark/Scala)	Warning	Scala is supported; swap cell magic `%scala` → `%%spark` and rewrite Databricks-specific APIs/libraries
R notebooks	Fabric Notebook (SparkR)	Warning	SparkR is supported; swap cell magic `%r` → `%%sparkr`, validate package availability, rewrite Databricks-specific APIs

Severity Definitions

Level	Meaning	Action
Blocker	Cannot run in Fabric without redesign or user decision	Stop — surface to user, require resolution
Warning	Migratable but requires validation or architectural decision	Migrate with review flag
Info	Direct substitution-level change	Auto-migrate

`dbutils` → `notebookutils` Quick Reference

The complete side-by-side API table is in dbutils-to-notebookutils.md. The key mappings are:

`dbutils` Call	`notebookutils` Equivalent	Compatibility Note
`dbutils.fs.ls(path)`	`notebookutils.fs.ls(path)`	Direct replacement
`dbutils.fs.cp(src, dest)`	`notebookutils.fs.cp(src, dest)`	Direct replacement
`dbutils.fs.mv(src, dest)`	`notebookutils.fs.mv(src, dest, create_path, overwrite=False)`	⚠️ Signature differs — see dbutils-to-notebookutils.md
`dbutils.fs.rm(path, recurse)`	`notebookutils.fs.rm(path, recurse)`	Direct replacement
`dbutils.fs.mkdirs(path)`	`notebookutils.fs.mkdirs(path)`	Direct replacement
`dbutils.fs.put(path, contents)`	`notebookutils.fs.put(path, contents)`	Direct replacement
`dbutils.fs.head(path, maxBytes)`	`notebookutils.fs.head(path, max_bytes)`	⚠️ Default differs — Python/Scala 100 KB, R 64 KB. See dbutils-to-notebookutils.md
`dbutils.fs.mount(...)`	`notebookutils.fs.mount(source, mountPoint, extraConfigs=None)`	✅ Supported — Microsoft Entra (default), `accountKey`, or `sasToken` auth. For cross-workspace / persistent sharing, prefer OneLake Shortcuts
`dbutils.secrets.get(scope, key)`	`notebookutils.credentials.getSecret(keyVaultUrl, secretName)`	Scope → Key Vault URL; key → secret name
`dbutils.notebook.run(path, timeout, args)`	`notebookutils.notebook.run(name, timeout, args)`	`path` → notebook `name` (relative to workspace)
`dbutils.notebook.exit(value)`	`notebookutils.notebook.exit(value)`	Direct replacement
`dbutils.widgets.get(name)`	See § Widgets Migration	No direct equivalent
`dbutils.library.install(...)`	Not available at runtime — use Fabric Environments	`dbutils.library.restartPython()` → `notebookutils.session.restartPython()`
`dbutils.data.summarize(df)`	`display(df.summary())`	Use `display()` or pandas `describe()`

Widgets Migration

dbutils.widgets has no direct equivalent in Fabric. Use these patterns instead:

Use Case	Fabric Pattern
Pass parameter from parent notebook	Mark a cell in the child notebook as a parameters cell (notebook UI: cell "..." menu → "Mark cell as parameters"). The parent calls `notebookutils.notebook.run("child", arguments={"param": "value"})` — at runtime the engine inserts a new cell beneath the parameters cell that overrides the defaults
Pipeline-driven parameterization	Same parameters-cell mechanism; the Fabric Pipeline notebook activity supplies override values via its Base parameters setting
Centralized cross-notebook config	Use `notebookutils.variableLibrary.getLibrary("<name>")` to read values from a Variable Library item (deployment pipelines activate the right value set per stage)
Interactive selection in notebook	Use `display()` with input cells, IPython widgets (Python only), or Fabric Data Activator

Note: notebookutils.runtime.context does not expose parameter values. It's for execution metadata (workspace/notebook/activity/user IDs, pipeline-vs-interactive flags, etc.). See dbutils-to-notebookutils.md § Runtime Context.

Cluster Config → Fabric Spark Pools

Databricks Cluster Concept	Fabric Spark Equivalent	Notes
All-purpose cluster (interactive)	Starter Pool	Auto-provisioned; no config; ideal for notebooks
Job cluster (single-use for jobs)	Custom Pool (or Starter Pool) attached to SJD	Configure node size, autoscale in Fabric capacity settings
Node type (e.g., `Standard_DS3_v2`)	Fabric node size (Small/Medium/Large/X-Large/XX-Large)	Map by vCore/memory ratio
Autoscale min/max workers	Custom Pool min/max node settings	Available in workspace Spark settings
`spark.conf` in cluster settings	Fabric Environment Spark properties	Move to Environment item; attach to workspace or notebook
`init_scripts` (cluster init)	Fabric Environment install script	Not fully equivalent — only library installs are supported
Databricks Runtime version	Fabric Runtime (1.1 = Spark 3.3, 1.2 = Spark 3.4, 1.3 = Spark 3.5)	Choose matching Spark version; test deprecated APIs
Photon accelerator	Fabric Native Execution Engine (NEE)	Enable in workspace Spark settings; vectorized execution similar to Photon

Databricks Jobs → Spark Job Definitions

Databricks Jobs Concept	Fabric SJD Equivalent	Notes
Job with single notebook task	SJD referencing a notebook	Attach a default Lakehouse; pass parameters via SJD args
Multi-task job (DAG of tasks)	Fabric Data Pipeline orchestrating multiple SJDs/notebooks	Pipeline activities map to job tasks; dependencies = activity dependencies
Job schedule (cron)	Pipeline schedule trigger	Cron expression → recurrence trigger in pipeline
Job parameters	SJD default arguments or notebook cell parameters	Parameters cell in notebook is injected at runtime
Job clusters per task	Pool attached to SJD	Each SJD can specify its Spark pool independently
Databricks Workflows	Fabric Data Pipelines	Full DAG orchestration with conditions, loops, and failure branches

Delegate to spark-cli for SJD creation and notebook deployment.

Databricks Delta Sharing Pattern	Fabric Equivalent
Provider publishes a Delta share	Fabric external data sharing for cross-tenant Fabric data, or a OneLake Shortcut to ADLS Gen2 where the Delta data resides
Recipient reads shared data	Accept the external data share into a Lakehouse (Fabric creates a read-only OneLake Shortcut), or create a direct OneLake Shortcut to accessible ADLS Gen2 data
Cross-workspace table sharing within org	OneLake Shortcuts pointing to another workspace's Lakehouse tables — no data copy
Cross-tenant sharing	Fabric external data sharing — live, read-only, in-place access through a shortcut in the recipient tenant

When producing a migration workload map, include both paths: direct OneLake Shortcuts for accessible ADLS or same-tenant OneLake data, and Fabric external data sharing for native cross-tenant recipient sharing.

MLflow → Fabric ML Experiments

Fabric ML Experiments are built on the MLflow SDK — most code is directly portable:

Databricks MLflow Pattern	Fabric Equivalent	Migration Action
`mlflow.set_tracking_uri("databricks")`	Remove — Fabric tracking is automatic	Delete this line in Fabric notebooks
`mlflow.set_experiment("/path/exp")`	`mlflow.set_experiment("experiment_name")`	Use name only (not path); Fabric creates the Experiment item
`mlflow.log_metric(...)`	`mlflow.log_metric(...)` — identical	No change
`mlflow.log_artifact(...)`	`mlflow.log_artifact(...)` — identical	No change
`mlflow.autolog()`	`mlflow.autolog()` — identical	No change
`mlflow.register_model(...)`	`mlflow.register_model(...)` — identical	Model Registry is available in Fabric ML
Databricks Model Serving	Azure ML Online Endpoints or Fabric Data Activator	No direct Fabric model serving yet — use Azure ML

Must / Prefer / Avoid

MUST DO

Inventory before prescribing a workspace topology — for workspace-wide requests that omit workload inventory, dependencies, security requirements, data locations, or runtime constraints, ask focused clarifying questions and present conditional choices before selecting a Fabric design
Replace all dbutils.* calls using the mapping in dbutils-to-notebookutils.md — dbutils is not available in Fabric notebooks
Migrate dbutils.fs.mount() to notebookutils.fs.mount() (✅ supported — Microsoft Entra default, or accountKey / sasToken from Key Vault). For cross-workspace or persistent sharing, prefer OneLake Shortcuts instead. Always pair mount() with unmount() in try/finally — Fabric mounts are not released automatically on session end
Replace dbutils.secrets.get(scope, key) with notebookutils.credentials.getSecret(keyVaultUrl, secretName) — secret scopes map to Azure Key Vault URLs
Redesign widget-based parameter passing using notebook parameters cells (cell "..." menu → "Mark cell as parameters"); use notebookutils.variableLibrary for centralized cross-notebook config. notebookutils.runtime.context does not expose parameter values
Replace dbutils.library.install*() with Fabric Environments — runtime library installs are not supported in production. dbutils.library.restartPython() maps to notebookutils.session.restartPython() (Python / PySpark only)
Map Unity Catalog namespaces deliberately — default to one schema-enabled Lakehouse per catalog so schema.table is preserved; require a user-approved naming policy only when consolidating multiple catalogs into one Lakehouse. See catalog-migration.md
Map Databricks cluster init scripts to Fabric Environments — cluster-level library installs must move to Environment items

PREFER

Fabric Native Execution Engine (NEE) as the Photon equivalent — enable in workspace Spark settings for vectorized execution on Delta Lake
OneLake Shortcuts over data copy for Delta tables that already exist in ADLS Gen2 — point directly without re-ingesting
Fabric Git Integration as the replacement for Databricks Repos — connect workspace to ADO or GitHub for notebook version control
Fabric ML Experiments for direct MLflow continuity — tracking code requires minimal changes (remove set_tracking_uri)
Medallion architecture when restructuring migrated Databricks catalogs — align bronze, silver, gold Unity Catalog schemas to separate Fabric Lakehouses
Starter Pool for migrating interactive notebook workflows — eliminates cluster startup time that was a common pain point in Databricks job clusters

AVOID

Do not prescribe a one-size-fits-all workspace topology when the source inventory and migration constraints are missing
Do not import dbutils or attempt dbutils = ... assignments in Fabric notebooks — import attempts fail with ModuleNotFoundError, while unresolved dbutils references raise NameError; always use notebookutils
Do not retain dbutils.* calls behind runtime-detection guards (try/except, if IS_DATABRICKS) — replace the calls and Databricks paths outright with notebookutils and Fabric paths
Do not assume Unity Catalog governance policies transfer automatically — RBAC, row-level security, and column masking must be reconfigured in Fabric using workspace roles and Lakehouse permissions
Do not use %pip install in production Fabric notebooks at runtime — use Fabric Environments for stable, versioned library management
Do not attempt to port Delta Live Tables (DLT) pipelines verbatim — DLT has no Fabric equivalent; rewrite as parameterized notebooks orchestrated by Fabric Pipelines
Do not rely on Databricks-specific Spark configurations (e.g., spark.databricks.*) — these are proprietary and will be silently ignored or raise errors in Fabric
Do not use DBFS paths (dbfs:/...) — there is no DBFS in Fabric; all paths must use OneLake abfss:// or Lakehouse-relative paths

Multi-Notebook Migration Protocol

For workspaces with >3 notebooks or individual notebooks >5KB, process one notebook at a time (enumerate → export → transform → summarize → deploy → release) to avoid context overflow. Track each notebook through a status lifecycle (inventory → analyzed → converted → deployed → validated, or failed).

Full protocol, status definitions, and per-notebook summary schema: migration-orchestrator.md § Phase 2.

Failure Reporting

When migration cannot complete (permission failures, unresolvable Blockers such as DLT or OS-level init scripts, namespace collisions with no user-chosen policy, or repeated API failures), emit a structured failure report — do not abandon silently. The report captures phase_reached, blockers[] (item / pattern / reason / recommendation), partial_success counts, and next_steps.

Full report schema and stopping conditions: migration-orchestrator.md § Failure Reporting.

Examples

See dbutils-to-notebookutils.md and code-patterns.md for the full mapping. Key quick references:

dbutils.fs → notebookutils.fs

# Databricks
dbutils.fs.ls("/mnt/bronze/orders/")
dbutils.fs.cp("/mnt/raw/file.csv", "/mnt/archive/file.csv")

# Fabric (replace DBFS/mount paths with OneLake relative paths)
notebookutils.fs.ls("Files/bronze/orders/")
notebookutils.fs.cp("Files/raw/file.csv", "Files/archive/file.csv")

dbutils.secrets → notebookutils.credentials

# Databricks
pwd = dbutils.secrets.get(scope="prod", key="db-password")

# Fabric (scope → Key Vault URL, key → secret name)
pwd = notebookutils.credentials.getSecret("https://myvault.vault.azure.net/", "db-password")

Unity Catalog namespace → Lakehouse schema

# Databricks
df = spark.read.table("prod.silver.customers")

# Fabric (ProdLakehouse represents the source catalog and is attached as context)
df = spark.read.table("silver.customers")

Files

ATTRIBUTION.mdAdded by AutoGineer885 B
LICENSELicence file added by AutoGineer1.1 KiB
SKILL.md22.1 KiB
apm.yml460 B
resources/catalog-migration.md6.5 KiB
resources/code-patterns.md6.7 KiB
resources/dbutils-to-notebookutils.md21.2 KiB
resources/migration-gotchas.md9.3 KiB
resources/migration-orchestrator.md17.4 KiB
resources/validation-testing.md12.4 KiB

resources/dbutils-to-notebookutils.md

`dbutils` → `notebookutils` Complete API Mapping

Exhaustive side-by-side reference for porting Databricks dbutils calls to Microsoft Fabric notebookutils.

Quick Compatibility Summary

`dbutils` Module	`notebookutils` Status	Action
`dbutils.fs` (file ops)	✅ Full equivalent — `notebookutils.fs`	Direct namespace swap; OneLake/ABFSS paths instead of DBFS
`dbutils.fs` (mount/unmount/mounts)	✅ Supported — `notebookutils.fs.mount/unmount/mounts/getMountPath`	Same API shape; OneLake Shortcuts are the higher-level alternative for sharing data
`dbutils.secrets`	✅ Equivalent — `notebookutils.credentials`	Scope→Key Vault URL; key→secret name. Also exposes `getToken`, `putSecret`, `isValidToken`
`dbutils.notebook`	✅ Superset — `notebookutils.notebook`	`run`/`exit` direct swap. Adds `runMultiple` (DAG), `validateDAG`, cross-workspace runs, and CRUD (`create`/`get`/`list`/`update`/`delete`)
`dbutils.widgets`	⚠️ No direct equivalent	Mark a cell as a parameters cell (pipeline / parent-notebook injection writes a new cell beneath it that overrides the defaults). For centralized cross-notebook config, use `notebookutils.variableLibrary`
`dbutils.library.restartPython()`	✅ Supported — `notebookutils.session.restartPython()`	Direct replacement
`dbutils.library.install*()`	❌ Not available at runtime	Use Fabric Environments for library management (preferred) or in-session `%pip install` + `notebookutils.session.restartPython()`
`dbutils.data.summarize()`	❌ Not available	Use `display(df.summary())` or pandas `.describe()`
`dbutils.jobs`	⚠️ Different model	Fabric uses Pipelines / Spark Job Definitions; runtime/job context via `notebookutils.runtime.context`
`dbutils.secrets.listScopes()` / `.list()`	❌ Not available	Key Vault URLs are referenced directly; enumerate secrets via Azure CLI / Key Vault SDK
`dbutils.fs.refreshMounts()`	❌ Not available	Not needed — `notebookutils.fs.mount()` already attaches the mount to the driver and all worker nodes in one call, so there's no separate mount-registry propagation step

`dbutils.fs` → `notebookutils.fs`

Most fs methods are direct replacements — only paths change (DBFS → OneLake/ABFSS). A few have minor signature differences (called out below).

`dbutils.fs` Method	`notebookutils.fs` Equivalent	Notes
`dbutils.fs.ls(path)`	`notebookutils.fs.ls(path)`	Returns `FileInfo` list (`name`, `path`, `size`, `isDir`, `isFile`)
`dbutils.fs.cp(src, dest, recurse=False)`	`notebookutils.fs.cp(src, dest, recurse=False)`	Identical. In Python notebooks, internally uses azcopy (same as `fastcp`)
`dbutils.fs.mv(src, dest, recurse=False)`	`notebookutils.fs.mv(src, dest, create_path=…, overwrite=False)`	⚠️ Signature differs: no `recurse`; uses `create_path` and `overwrite`. `create_path` default is `False` in PySpark/Scala/R and `True` in Python notebooks — set it explicitly
`dbutils.fs.rm(path, recurse=False)`	`notebookutils.fs.rm(path, recurse=False)`	Identical
`dbutils.fs.mkdirs(path)`	`notebookutils.fs.mkdirs(path)`	Identical
`dbutils.fs.put(path, contents, overwrite=False)`	`notebookutils.fs.put(path, contents, overwrite=False)`	Identical. No atomicity guarantees for concurrent writes
`dbutils.fs.head(path, maxBytes=65536)`	`notebookutils.fs.head(path, max_bytes=102400)`	⚠️ Default differs: Python/Scala `102400` (100 KB), R `65535` (Scala uses `maxBytes`)
`dbutils.fs.append(path, contents, createIfNotExists)`	`notebookutils.fs.append(file, content, createFileIfNotExists=False)`	Equivalent. Add `time.sleep(0.5)` between writes in loops (async flush)
`dbutils.fs.mount(source, mountPoint, …)`	`notebookutils.fs.mount(source, mountPoint, extraConfigs=None)`	✅ Supported. Microsoft Entra (default), `accountKey`, or `sasToken` auth. `extraConfigs` also accepts `timeout` (default 30 s — increase for mounts under high executor count) and `fileCacheTimeout`. (docs)
`dbutils.fs.unmount(mountPoint)`	`notebookutils.fs.unmount(mountPoint)`	✅ Supported. Not automatic on session end — call it explicitly
`dbutils.fs.mounts()`	`notebookutils.fs.mounts()`	✅ Supported. Returns array of `MountPointInfo`
`dbutils.fs.refreshMounts()`	❌ Not available	Not needed — `notebookutils.fs.mount()` attaches the mount to the driver and all worker nodes in one call, so there's no separate mount-registry propagation step. (Note: file-content cache freshness is a different concern — control that via `fileCacheTimeout` in the `mount()` `extraConfigs`.)
`dbutils.fs.updateMount(...)`	❌ Not available	`unmount()` then `mount()` again with new config
`dbutils.fs.help()`	`notebookutils.fs.help()`	Identical
(no dbutils equivalent)	`notebookutils.fs.exists(path)`	New — prefer over try/except for existence checks
(no dbutils equivalent)	`notebookutils.fs.fastcp(src, dest, recurse=True, extraConfigs=None)`	New — azcopy-backed; use for large/bulk copies. Doesn't work across OneLake regions — fall back to `cp`. For S3/GCS-typed OneLake Shortcuts use a mounted path instead of the `abfss://` path. (docs)
(no dbutils equivalent)	`notebookutils.fs.getMountPath(mountPoint, scope="")`	New — returns the local FS path (`/synfs/notebook/{sessionId}/...`) for a mount
(no dbutils equivalent)	`notebookutils.fs.getProperties(path)`	New — Python notebooks only (not in PySpark/Scala/R). Returns a map of file properties (e.g. size, timestamps)

Path Migration: DBFS → OneLake

# Databricks: DBFS path
dbutils.fs.ls("dbfs:/mnt/mydata/bronze/")
dbutils.fs.ls("/mnt/mydata/bronze/")         # shorthand

# Fabric: OneLake path (Files section of attached Lakehouse)
notebookutils.fs.ls("Files/mydata/bronze/")  # relative path

# Or explicit OneLake path
notebookutils.fs.ls(
    "abfss://[email protected]/BronzeLakehouse.Lakehouse/Files/mydata/bronze/"
)

Migrating `dbutils.fs.mount()`

Fabric supports two patterns for accessing remote storage. Pick the one that fits your scenario:

Option A — notebookutils.fs.mount() (direct equivalent of dbutils.fs.mount())

Use this when you need a local file-system path (e.g. for libraries that don't speak ABFSS), when you're porting code that already relies on a mount, or for ad-hoc mounts inside a session.

# BEFORE — Databricks
dbutils.fs.mount(
    source="abfss://[email protected]/",
    mount_point="/mnt/mydata",
    extra_configs={
        "fs.azure.account.oauth2.client.secret": dbutils.secrets.get("myScope", "clientSecret"),
        # ...other OAuth configs
    }
)

# AFTER — Fabric: Microsoft Entra (default, recommended) — no credentials in code
notebookutils.fs.mount(
    "abfss://[email protected]",
    "/mydata"
)

try:
    # Access the mounted data:
    local_path = notebookutils.fs.getMountPath("/mydata")      # /synfs/notebook/{sessionId}/mydata
    with open(f"{local_path}/file.csv", "r") as f:
        data = f.read()
finally:
    # Mounts are NOT released automatically when the session ends — always unmount.
    notebookutils.fs.unmount("/mydata")

Alternative auth modes — pick one of these in place of the Entra mount() call above (don't mount the same point twice without an unmount() in between):

# Auth via account key from Key Vault
account_key = notebookutils.credentials.getSecret("https://my-kv.vault.azure.net/", "storageKey")
notebookutils.fs.mount(
    "abfss://[email protected]",
    "/mydata",
    {"accountKey": account_key}
)

# Or auth via SAS token from Key Vault
sas_token = notebookutils.credentials.getSecret("https://my-kv.vault.azure.net/", "storageSas")
notebookutils.fs.mount(
    "abfss://[email protected]",
    "/mydata",
    {"sasToken": sas_token}
)

Fabric mounts are job-level, not durable workspace-level shortcuts. Unmounting is not automatic when a session ends — mount points stay on the node until you call unmount() explicitly, so always pair mount() with unmount() (a try/finally block is the safe pattern).

Option B — OneLake Shortcuts (preferred for cross-workspace / persistent data sharing)

Use this when the same external location is consumed by many notebooks/pipelines, or when you want the storage to appear permanently inside a Lakehouse without per-session setup.

# Create a OneLake Shortcut once (Portal or REST API):
# https://learn.microsoft.com/fabric/onelake/onelake-shortcuts
# Then access directly — no runtime mounting required:
df = spark.read.parquet("Files/mydata/bronze/customers/")

`dbutils.secrets` → `notebookutils.credentials`

The concept maps directly — Databricks secret scopes correspond to Azure Key Vault instances.

`dbutils.secrets` Method	`notebookutils.credentials` Equivalent	Notes
`dbutils.secrets.get(scope, key)`	`notebookutils.credentials.getSecret(keyVaultUrl, secretName)`	`scope` → Key Vault URL; `key` → secret name. Secrets are auto-redacted in notebook output
`dbutils.secrets.getBytes(scope, key)`	`notebookutils.credentials.getSecret(keyVaultUrl, secretName)`	Returns string; encode to bytes if needed
`dbutils.secrets.list(scope)`	❌ Not available	Use Azure CLI / Key Vault SDK to list secrets
`dbutils.secrets.listScopes()`	❌ Not available	Key Vault URLs are referenced directly
`dbutils.secrets.help()`	`notebookutils.credentials.help()`

notebookutils.credentials also exposes capabilities that have no dbutils.secrets equivalent:

Method	Description
`notebookutils.credentials.getToken(audience)`	Get a Microsoft Entra token for `storage`, `pbi`, `keyvault`, or `kusto` audiences. Use to call Azure Storage, Fabric/Power BI REST, Key Vault, and Kusto without managing credentials
`notebookutils.credentials.putSecret(keyVaultUrl, secretName, secretValue)`	Write/update a secret in Azure Key Vault (requires Set permission). Not available in the public Scala API
`notebookutils.credentials.isValidToken(token)`	Check whether a token is unexpired before reusing it in long-running jobs. Not available in the public Scala API

# BEFORE — Databricks
password = dbutils.secrets.get(scope="prod-secrets", key="db-password")
api_key  = dbutils.secrets.get(scope="prod-secrets", key="api-key")

# AFTER — Fabric
password = notebookutils.credentials.getSecret(
    "https://my-keyvault.vault.azure.net/",
    "db-password"
)
api_key = notebookutils.credentials.getSecret(
    "https://my-keyvault.vault.azure.net/",
    "api-key"
)

The Fabric notebook's managed identity (or the signed-in user) must have Key Vault Secrets User role on the Key Vault.

`dbutils.notebook` → `notebookutils.notebook`

notebookutils.notebook is a superset of dbutils.notebook. run/exit are direct replacements, and it also adds DAG-based parallel execution, cross-workspace runs, and CRUD operations on notebook artifacts.

`dbutils.notebook` Method	`notebookutils.notebook` Equivalent	Notes
`dbutils.notebook.run(path, timeout, arguments)`	`notebookutils.notebook.run(path, timeout_seconds=90, arguments=None, workspace="")`	`path` is the child notebook name. `workspace` is the workspace ID for cross-workspace runs (Runtime ≥ 1.2). For workspace name or ID, use `runMultiple()` with the activity-level `workspace` field. ⚠️ Lakehouse-binding rule: the child must use the parent's default lakehouse, inherit it, or have none — otherwise the run is blocked. Pass `useRootDefaultLakehouse=True` in `arguments` to bypass. (docs)
`dbutils.notebook.exit(value)`	`notebookutils.notebook.exit(value)`	Identical. Don't wrap in `try/except` — the exit signal won't propagate
`dbutils.notebook.help()`	`notebookutils.notebook.help()`	Identical

notebookutils.notebook also adds:

Method	Description
`runMultiple(dag, config=None)`	Run notebooks in parallel with a DAG. Root keys: `activities`, `timeoutInSeconds`, `concurrency`. Per-activity keys: `name`, `path`, `args`, `workspace` (name or ID), `timeoutPerCellInSeconds`, `retry`, `retryIntervalInSeconds`, `dependencies`. Returns `{activity: {exitVal, exception}}`. Replaces hand-rolled Threads/Futures in Databricks
`validateDAG(dag)`	Validates DAG structure (duplicate names, missing/circular deps) before `runMultiple`
`create / get / getDefinition / update / updateDefinition / delete / list`	Notebook artifact CRUD — useful for CI/CD and templating workflows. No `dbutils` equivalent

# BEFORE — Databricks
result = dbutils.notebook.run(
    "/path/to/silver_transform",
    timeout=600,
    arguments={"input_date": "2024-01-01", "env": "prod"}
)
dbutils.notebook.exit("completed")

# AFTER — Fabric: single notebook
result = notebookutils.notebook.run(
    "silver_transform",          # notebook name in the same workspace
    600,
    {"input_date": "2024-01-01", "env": "prod"}
)

# AFTER — Fabric: cross-workspace (pass the workspace ID)
result = notebookutils.notebook.run(
    "silver_transform",
    600,
    {"input_date": "2024-01-01"},
    "fe0a6e2a-a909-4aa3-a698-0a651de790aa"   # workspace ID (single run() requires ID)
)

notebookutils.notebook.exit("completed")

# AFTER — Fabric: parallel DAG (replaces Databricks Threads/Futures patterns)
DAG = {
    "activities": [
        {"name": "Extract",   "path": "bronze_ingest", "args": {"date": "2024-01-01"}},
        {"name": "Transform", "path": "silver_transform",
         "args": {"in": "@activity('Extract').exitValue()"},
         "dependencies": ["Extract"], "retry": 2, "retryIntervalInSeconds": 30},
    ],
    "concurrency": 4,
    "timeoutInSeconds": 3600,
}
notebookutils.notebook.validateDAG(DAG)
results = notebookutils.notebook.runMultiple(DAG)

`dbutils.widgets` — No Direct Equivalent

Databricks widgets are interactive UI controls. Fabric uses parameter cells (mark a cell as "parameters" in the UI — values are overridden at runtime), with notebookutils.variableLibrary for centralized configuration.

Pattern 1: Parameter cell (most common — works for both pipeline injection and parent→child runs)

# BEFORE — Databricks: define widget, read in notebook
dbutils.widgets.text("input_date", "2024-01-01", "Input Date")
input_date = dbutils.widgets.get("input_date")

# AFTER — Fabric: mark a cell as a parameters cell. In the notebook UI, open the cell's
# More commands ("...") and select "Mark cell as parameters" (older UI: "Toggle parameter cell").
# Declare defaults — at runtime, the engine adds a NEW cell beneath the parameters cell that
# overrides these values. Override sources:
#   - Fabric Pipeline notebook activity (Base parameters)
#   - notebookutils.notebook.run(..., arguments={...}) from a parent notebook
input_date = "2024-01-01"
env = "dev"

Pattern 2: Centralized config via Variable Library

# AFTER — Fabric: pull environment-specific values from a Variable Library
# (deployment pipelines activate the right value set per stage)
cfg = notebookutils.variableLibrary.getLibrary("app_config")
input_date = cfg.input_date
api_endpoint = cfg.api_endpoint

⚠️ Caveats (per docs): notebookutils.variableLibrary is same-workspace only — cross-workspace reads (including child notebooks in a cross-workspace reference run) are not supported. Service Principal (SPN) identity is not supported for variable library calls, so this pattern can't drive centralized config in SPN-authenticated CI/CD pipelines today.

Pattern 3: Parent notebook passes parameters to child

# AFTER — Fabric: parent calls child with explicit args
result = notebookutils.notebook.run(
    "child_notebook",
    300,
    {"input_date": "2024-01-01", "table_name": "fact_orders"}
)
# In child_notebook, the parameter cell values are replaced with the passed args at runtime

`dbutils.library` → Fabric Environments + `notebookutils.session`

dbutils.library.install*() runtime install isn't available in Fabric — use Fabric Environments for reproducible library management. The interpreter restart helper, however, is available as notebookutils.session.restartPython().

`dbutils.library` Method	Fabric Equivalent	Notes
`dbutils.library.installPyPI(pkg, version)`	Fabric Environment item (preferred) — add the package to its pip list; or `%pip install <pkg>==<version>` in-session	Environment attach gives reproducible, workspace-scoped library sets
`dbutils.library.install("dbfs:/...whl")`	Upload the wheel as a custom library in a Fabric Environment, or `%pip install /lakehouse/default/Files/mylib.whl`
`dbutils.library.restartPython()`	`notebookutils.session.restartPython()`	✅ Direct replacement (Python and PySpark notebooks only — not available in Scala/R). In PySpark, restarts only the Python interpreter and keeps the Spark context. Import new packages in the next cell
`dbutils.library.list()`	❌ Not available	Inspect Environment configuration in the Fabric Portal / REST API

# BEFORE — Databricks: runtime library install
dbutils.library.installPyPI("scikit-learn", version="1.3.0")
dbutils.library.install("dbfs:/mnt/libs/mylib-1.0.whl")
dbutils.library.restartPython()

# AFTER — Fabric option A (preferred): Fabric Environment
# 1. Create a Fabric Environment item in the workspace.
# 2. Add `scikit-learn==1.3.0` to the Environment's pip packages.
# 3. For custom .whl files, upload as a custom library on the Environment item.
# 4. Attach the Environment to the notebook (or set as workspace default).
# No code change needed — the library is available on session start.

# AFTER — Fabric option B (ad-hoc, session-scoped): %pip + restartPython
# Cell 1: install and restart
%pip install scikit-learn==1.3.0
notebookutils.session.restartPython()

# Cell 2 (a NEW cell — code after restartPython() in the same cell does NOT run):
import sklearn

Also available: stop the interactive session

notebookutils.session.stop()        # async; releases Spark resources to the pool
# PySpark/Scala/R: stop() accepts an optional `detach` parameter (Python notebook doesn't).
#   - Default behavior (detach=True): on a high-concurrency session, detaches the session
#     instead of stopping it entirely.
#   - stop(detach=False): fully stops a high-concurrency session.

`dbutils.data` — Use display() or pandas

# BEFORE — Databricks: summarize a DataFrame
dbutils.data.summarize(df)

# AFTER — Fabric: use built-in display or pandas
display(df.summary())           # Spark summary stats in Fabric display
display(df.describe())          # Alternative
df.toPandas().describe()        # Full pandas stats (for small DataFrames only)

Runtime Context

notebookutils.runtime.context returns a read-only map. Property names use the current* / default* / root* prefix — they are not the same names as Databricks tags. The sample below shows the most common keys; the full key list (including defaultLakehouseWorkspaceId, currentRunId/parentRunId/rootRunId, rootNotebookName, clusterId, poolName, environmentWorkspaceId, currentKernel, productType, hcReplId, etc.) is on the runtime context docs page.

# BEFORE — Databricks: access job context
ctx = dbutils.notebook.entry_point.getDbutils().notebook().getContext()
workspace = ctx.tags().get("orgId").get()
job_id    = ctx.tags().get("jobId").get()

# AFTER — Fabric
ctx = notebookutils.runtime.context

workspace_id        = ctx["currentWorkspaceId"]
workspace_name      = ctx["currentWorkspaceName"]
notebook_id         = ctx["currentNotebookId"]
notebook_name       = ctx["currentNotebookName"]

# Default lakehouse (only populated when one is attached)
lakehouse_id        = ctx.get("defaultLakehouseId")
lakehouse_name      = ctx.get("defaultLakehouseName")

# Execution mode flags — useful for pipeline-vs-interactive branching
is_pipeline         = ctx["isForPipeline"]
is_interactive      = ctx["isForInteractive"]
is_reference_run    = ctx["isReferenceRun"]

# Job / activity identity
activity_id         = ctx["activityId"]        # Livy job ID for current activity (closest to Databricks jobId)
environment_id      = ctx.get("environmentId")
user_id             = ctx["userId"]
user_name           = ctx["userName"]

The runtime context does not expose pipeline parameters under a parameters key — parameter values arrive via parameter-cell overrides (see the dbutils.widgets section).

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs Python, Needs Node.js

Security

What our scanner noticed· Low risk

Instructions that use the network

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by

Microsoft

Publisher

Microsoft (Official)

Source

microsoft/skills-for-fabric · skills/databricks-migration · commit f8fac11

Licence

MIT License (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-08

Also distributed in

microsoft/skills-for-fabric

AutoGineer is not affiliated with or endorsed by Microsoft.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download databricks-migration.zip