---

name: databricks-migration description: "Ports existing Databricks notebooks and jobs to Fabric, covering dbutils to notebookutils, secret scopes to Key Vault, DBFS to OneLake, Unity Catalog mapping to schema-enabled Lakehouses, Jobs and Delta Live Tables to Spark Job Definitions and Pipelines, and Photon to the Native Execution Engine. Use whenever Databricks code has to be converted. For Fabric notebook code that is not being migrated, use spark-cli."

Telemetry — MANDATORY. Every api.fabric.microsoft.com call must carry x-ms-fabric-skill: databricks-migration (az rest: --headers "x-ms-fabric-skill=databricks-migration"), including every LRO poll, fabric_lro and retry. Snippets omit it — add it anyway.

CRITICAL NOTES

1. To find workspace details (including its ID) from a workspace name: list all workspaces, then use JMESPath filtering
2. To find item details (including its ID) from workspace ID, item type, and item name: list all items of that type in that workspace, then use JMESPath filtering
3. dbutils.widgets has no direct equivalent in Fabric — use notebook parameters (cell tag parameters); notebookutils.runtime.context is execution metadata, not parameter storage. If showing context fields, use documented names such as currentWorkspaceId, currentWorkspaceName, currentNotebookId, currentNotebookName, isForPipeline, and isForInteractive; activityId is the Livy job ID
4. dbutils.library (runtime library install) has no equivalent — use Fabric Environments for reproducible library management
5. Map each Unity Catalog catalog to a schema-enabled Lakehouse by default. This preserves the source schema.table hierarchy, with the Lakehouse representing the catalog; collisions arise only if multiple catalogs are intentionally consolidated into one Lakehouse
6. For an under-specified workspace-wide migration, ask focused questions about inventory, workload topology, security, data locations, and runtime constraints before recommending a Fabric topology
7. A completed Fabric migration must not retain executable dbutils.* calls in dual-runtime branches or try/except guards — replace the calls and Databricks paths outright

Databricks → Microsoft Fabric Migration

Prerequisite Knowledge

Read these companion documents before executing migration tasks:

• COMMON-CORE.md — Fabric REST API patterns, authentication, token audiences, item discovery
• COMMON-CLI.md — az rest, az login, token acquisition, Fabric REST via CLI
• SPARK-AUTHORING-CORE.md — Notebook deployment, lakehouse creation, Spark job execution

For notebook and Lakehouse creation, see spark-cli. For Fabric Warehouse DDL/DML authoring, see sqldw-cli.

---

Table of Contents

| Topic | Reference | |---|---| | Migration Orchestrator | migration-orchestrator.md | | Migration Workload Map | § Migration Workload Map | | Complete dbutils → notebookutils Mapping | dbutils-to-notebookutils.md | | Unity Catalog → Fabric Lakehouse Schemas | catalog-migration.md | | Before/After Code Patterns | code-patterns.md | | Cluster Config → Fabric Spark Pools | § Cluster Config → Fabric Spark Pools | | Databricks Jobs → Spark Job Definitions | § Databricks Jobs → Spark Job Definitions | | Delta Sharing → Fabric External Data Sharing and OneLake Shortcuts | § Delta Sharing → Fabric External Data Sharing and OneLake Shortcuts | | MLflow → Fabric ML Experiments | § MLflow → Fabric ML Experiments | | Post-Migration Validation & Testing | validation-testing.md | | Migration Gotchas & Troubleshooting | migration-gotchas.md | | Multi-Notebook Migration Protocol | § Multi-Notebook Migration Protocol | | Failure Reporting | § Failure Reporting | | Must / Prefer / Avoid | § Must / Prefer / Avoid | | Authentication & Token Acquisition | COMMON-CORE.md § Authentication | | Lakehouse Management | SPARK-AUTHORING-CORE.md § Lakehouse Management | | Notebook Management | SPARK-AUTHORING-CORE.md § Notebook Management |

Context Loading Guide

IMPORTANT — Load only what you need. Do NOT read all resource files upfront. Load the specific file for the phase you are executing:

| When | Read This File | |---|---| | User asks to migrate a workspace (full orchestration) | migration-orchestrator.md | | Applying code transforms (dbutils, namespaces, paths) | dbutils-to-notebookutils.md + code-patterns.md | | Resolving Unity Catalog namespace collisions | catalog-migration.md | | Post-migration verification | validation-testing.md | | Troubleshooting failures or known issues | migration-gotchas.md |

---

Migration Workload Map

| Databricks Component | Fabric Target | Severity | Notes | |---|---|---|---| | All-purpose cluster (notebooks, REPL) | Fabric Notebook (Starter Pool or Custom Pool) | Info | No persistent cluster — Fabric provisions compute on session start | | Job cluster (automated jobs) | Spark Job Definition (SJD) | Info | SJD maps one-to-one with Databricks Jobs on job clusters | | Unity Catalog | Fabric Lakehouse (schema-enabled, one per catalog) | Info | Schema-enabled Lakehouse preserves schema.table; default one-Lakehouse-per-catalog has no collision — see catalog-migration.md | | Databricks Repos (Git-backed notebooks) | Fabric Git Integration | Info | Connect workspace to Azure DevOps or GitHub; notebooks are synced | | Delta Live Tables (DLT) | Fabric Notebooks + Data Pipelines | Blocker | No DLT equivalent — rewrite DLT datasets as parameterized notebook cells with pipeline orchestration | | Databricks SQL Warehouses | Fabric Warehouse or Lakehouse SQL Endpoint | Info | SQL warehouse sessions → Warehouse (for write) or SQL Endpoint (for read-only) | | MLflow Tracking | Fabric ML Experiments | Info | MLflow SDK is supported in Fabric — see § MLflow | | Delta Sharing | OneLake Shortcuts + Fabric external data sharing | Warning | See § Delta Sharing → Fabric External Data Sharing and OneLake Shortcuts | | Databricks Feature Store | Feature engineering on Lakehouse/Delta tables + MLflow | Warning | Fabric has no drop-in managed Feature Store; recreate feature tables as Delta tables in a Lakehouse and manage features via notebooks/MLflow. Verify current Fabric feature-store roadmap before committing an approach | | dbutils (all sub-modules) | notebookutils (most sub-modules) | Info | See dbutils-to-notebookutils.md for full mapping | | Scala notebooks | Fabric Notebook (Spark/Scala) | Warning | Scala is supported; swap cell magic %scala → %%spark and rewrite Databricks-specific APIs/libraries | | R notebooks | Fabric Notebook (SparkR) | Warning | SparkR is supported; swap cell magic %r → %%sparkr, validate package availability, rewrite Databricks-specific APIs |

Severity Definitions

| Level | Meaning | Action | |---|---|---| | Blocker | Cannot run in Fabric without redesign or user decision | Stop — surface to user, require resolution | | Warning | Migratable but requires validation or architectural decision | Migrate with review flag | | Info | Direct substitution-level change | Auto-migrate |

---

dbutils → notebookutils Quick Reference

The complete side-by-side API table is in dbutils-to-notebookutils.md. The key mappings are:

| dbutils Call | notebookutils Equivalent | Compatibility Note | |---|---|---| | dbutils.fs.ls(path) | notebookutils.fs.ls(path) | Direct replacement | | dbutils.fs.cp(src, dest) | notebookutils.fs.cp(src, dest) | Direct replacement | | dbutils.fs.mv(src, dest) | notebookutils.fs.mv(src, dest, create_path, overwrite=False) | ⚠️ Signature differs — see dbutils-to-notebookutils.md | | dbutils.fs.rm(path, recurse) | notebookutils.fs.rm(path, recurse) | Direct replacement | | dbutils.fs.mkdirs(path) | notebookutils.fs.mkdirs(path) | Direct replacement | | dbutils.fs.put(path, contents) | notebookutils.fs.put(path, contents) | Direct replacement | | dbutils.fs.head(path, maxBytes) | notebookutils.fs.head(path, max_bytes) | ⚠️ Default differs — Python/Scala 100 KB, R 64 KB. See dbutils-to-notebookutils.md | | dbutils.fs.mount(...) | notebookutils.fs.mount(source, mountPoint, extraConfigs=None) | ✅ Supported — Microsoft Entra (default), accountKey, or sasToken auth. For cross-workspace / persistent sharing, prefer OneLake Shortcuts | | dbutils.secrets.get(scope, key) | notebookutils.credentials.getSecret(keyVaultUrl, secretName) | Scope → Key Vault URL; key → secret name | | dbutils.notebook.run(path, timeout, args) | notebookutils.notebook.run(name, timeout, args) | path → notebook name (relative to workspace) | | dbutils.notebook.exit(value) | notebookutils.notebook.exit(value) | Direct replacement | | dbutils.widgets.get(name) | See § Widgets Migration | No direct equivalent | | dbutils.library.install(...) | Not available at runtime — use Fabric Environments | dbutils.library.restartPython() → notebookutils.session.restartPython() | | dbutils.data.summarize(df) | display(df.summary()) | Use display() or pandas describe() |

Widgets Migration

dbutils.widgets has no direct equivalent in Fabric. Use these patterns instead:

| Use Case | Fabric Pattern | |---|---| | Pass parameter from parent notebook | Mark a cell in the child notebook as a parameters cell (notebook UI: cell "..." menu → "Mark cell as parameters"). The parent calls notebookutils.notebook.run("child", arguments={"param": "value"}) — at runtime the engine inserts a new cell beneath the parameters cell that overrides the defaults | | Pipeline-driven parameterization | Same parameters-cell mechanism; the Fabric Pipeline notebook activity supplies override values via its Base parameters setting | | Centralized cross-notebook config | Use notebookutils.variableLibrary.getLibrary("<name>") to read values from a Variable Library item (deployment pipelines activate the right value set per stage) | | Interactive selection in notebook | Use display() with input cells, IPython widgets (Python only), or Fabric Data Activator |

Note: notebookutils.runtime.context does not expose parameter values. It's for execution metadata (workspace/notebook/activity/user IDs, pipeline-vs-interactive flags, etc.). See dbutils-to-notebookutils.md § Runtime Context.

---

Cluster Config → Fabric Spark Pools

| Databricks Cluster Concept | Fabric Spark Equivalent | Notes | |---|---|---