---

name: e2e-medallion-architecture description: "Plans and builds end-to-end Fabric data platforms on the Bronze, Silver, and Gold medallion pattern: ingestion to analytics, PySpark and Delta layers, pipeline orchestration, per-layer Spark tuning, Materialized Lake View versus notebook choices for Silver and Gold, and Direct Lake handoff. Use for layered architecture design, planning and recommendations. For questions over an existing Power BI report use fabriciq."

Telemetry — MANDATORY. Every api.fabric.microsoft.com call must carry x-ms-fabric-skill: e2e-medallion-architecture (az rest: --headers "x-ms-fabric-skill=e2e-medallion-architecture"), including every LRO poll, fabric_lro and retry. Snippets omit it — add it anyway.

CRITICAL NOTES

1. To find the workspace details (including its ID) from workspace name: list all workspaces and, then, use JMESPath filtering
2. To find the item details (including its ID) from workspace ID, item type, and item name: list all items of that type in that workspace and, then, use JMESPath filtering

End-to-End Medallion Architecture

Prerequisite Knowledge

Read these companion documents — they contain the foundational context this skill depends on:

• COMMON-CORE.md — Fabric REST API patterns, authentication, token audiences, item discovery
• COMMON-CLI.md — az rest, az login, token acquisition, Fabric REST via CLI
• SPARK-AUTHORING-CORE.md — Notebook deployment, lakehouse creation, job execution
• notebook-api-operations.md — Required for notebook creation — .ipynb structure requirements, cell format, getDefinition/updateDefinition workflow

For Spark-specific optimization details, see data-engineering-patterns.md.

---

Architecture Overview

Medallion Architecture is a data lakehouse pattern with three progressive layers:

| Layer | Purpose | Optimization Profile | Use Case | |-------|---------|---------------------|----------| | Bronze (Raw) | Land raw data exactly as received | Write-optimized, append-only, partitioned by ingestion date | Audit trail, reprocessing, lineage | | Silver (Cleaned) | Deduplicated, validated, conformed data | Balanced read/write, partitioned by business date | Feature engineering, operational reporting | | Gold (Aggregated) | Pre-calculated metrics for analytics | Read-optimized (ZORDER, compaction), partitioned by month/year | Power BI reports, dashboards, ad-hoc analytics via SQL endpoint |

• Bronze: Schema-on-read — flexible schema, Delta time travel supports audit and rollback
• Silver: Schema enforcement — reject non-conforming writes; handle schema evolution with mergeSchema when sources change
• Gold: Strict schema governance — curated, business-approved datasets only

---

Must/Prefer/Avoid

MUST DO

• Choose lakehouse architecture based on schema-enabled availability (see infrastructure-orchestration.md):
  • Preferred: Schema-enabled lakehouse → create ONE workspace + ONE lakehouse with bronze, silver, gold schemas
  • Legacy: Non-schema-enabled → create separate workspaces per layer (Bronze, Silver, Gold) for governance and access control
• Use Livy API for schema and table creation — to create schemas and tables in a schema-enabled lakehouse, submit Spark SQL statements via Livy sessions (POST /livyApi/versions/2023-12-01/sessions → POST .../statements). This is the only programmatic REST path for DDL operations (CREATE SCHEMA, CREATE TABLE) in Fabric lakehouses.
• Add metadata columns in Bronze: ingestion timestamp, source file, batch ID
• Apply data quality rules in the Bronze-to-Silver transformation (deduplication, null handling, range validation)
• Use Delta Lake format for all medallion layer tables
• Use partition-aware overwrite in Silver/Gold writes to avoid reprocessing unchanged data
• Include validation steps after each layer (row counts, schema checks, anomaly detection)
• Follow the .ipynb validation + Fabric nuances in notebook-api-operations.md when creating notebooks via REST API — every code cell must include "outputs": [] and "execution_count": null
• Complete the full end-to-end flow — do not stop after creating notebooks; always bind lakehouses, execute notebooks sequentially (Bronze → Silver → Gold), verify results, and connect Power BI to the Gold layer unless the user explicitly requests a partial setup
• In every MLV-versus-notebook recommendation, state that MLVs require a schema-enabled lakehouse. Hand off MLV definition and incremental-refresh review to spark-cli authoring mode; hand off scheduling, refresh, monitoring, and failure diagnosis to spark-cli mlv mode.
• For recurring MLV refresh, give the exact interactive path Lakehouse → Materialized lake views → Manage → Schedules. For automation, use POST /workspaces/{workspaceId}/lakehouses/{lakehouseId}/jobs/refreshMaterializedLakeViews/schedules; never invent /mlvRefreshSchedules or route recurring refresh through notebook scheduling.

PREFER

• Incremental processing (watermark pattern) over full refresh
• Separate notebooks per layer for independent testing and debugging
• ZORDER on frequently filtered columns in Gold tables
• Running OPTIMIZE after writes in Silver and Gold layers
• Environment-specific Spark configs (write-heavy for Bronze, balanced for Silver, read-heavy for Gold)
• OneLake shortcuts to expose Gold data to consumer workspaces without duplication
• Clear layer ownership: engineers own Bronze/Silver, analysts own Gold
• Fabric Variable Libraries to centralize paths and configuration across layers
• Multi-workspace deployment patterns for medium/high governance requirements (Bronze/Silver/Gold in separate workspaces)
• Use Materialized Lake Views (MLVs) for Silver/Gold tables when the transformation is expressible in Spark SQL and benefits from declarative refresh semantics. See spark-cli — Materialized Lake View patterns and MLV incremental refresh patterns.
• Treat "materialized view", "spark materialized view", and "MLV" as the same Fabric feature.

AVOID

• Storing all layers in a single lakehouse WITHOUT schemas — non-schema lakehouses require notebook init cells or Environment configuration to enable OneLake Spark Catalog for RLS/CLS and MLVs. Use separate lakehouses for isolation if schemas aren't available.
• Creating 3 separate lakehouses when schema-enabled lakehouse is available — use schemas within one lakehouse instead (cleaner, no boilerplate init cells, more efficient for MLV cross-schema transformations)
• Skipping the Silver layer and going directly from Bronze to Gold
• Hardcoded workspace IDs, lakehouse IDs, or FQDNs — discover via REST API
• SELECT * without LIMIT on Bronze tables (they grow unboundedly)
• Running VACUUM without checking downstream dependencies
• Chaining OneLake shortcuts between medallion layers (Bronze→Silver→Gold) — each layer must be physically materialized for lineage and governance
• Copying complete implementation code into skills — guide the LLM to generate instead
• Reading from external HTTP/HTTPS URLs directly in Spark — Fabric Spark cannot access arbitrary external URLs; land data in lakehouse Files/ first (via curl, OneLake API, or Fabric pipeline Copy activity), then read from the lakehouse path
• Creating notebooks via REST API without validating .ipynb structure — missing execution_count: null or outputs: [] on code cells causes silent failures or "Job instance failed without detail error"

---

Workspace Setup Guidance

When setting up a medallion workspace, choose your architecture pattern first (see infrastructure-orchestration.md for detailed guidance):

Option A: Schema-Enabled Lakehouse (Preferred)

1. Create single workspace: {project}-{env}
2. Create one lakehouse with schemas: {project}_lakehouse
3. Create schemas within the lakehouse:
   • bronze schema for raw ingestion
   • silver schema for cleaned/validated data
   • gold schema for aggregated analytics
4. Choose transformation approach:
   • Option 4a: Use notebooks for each layer (PySpark or Spark SQL transformations)
   • Option 4b: Use Materialized Lake Views (Spark SQL) for declarative transformations with incremental refresh (when query is IR-eligible) — see materialized-lake-view-patterns.md and mlv-incremental-refresh-patterns.md
   • Note: PySpark MLVs exist but use full refresh only (no incremental) — use when you need UDFs/complex Python logic
   • MLV benefit: OneLake Spark Catalog is automatically enabled for schema-enabled lakehouses — MLVs work out-of-box with no notebook init cells or Environment configuration required
5. RBAC (optional): Use row-level security and column masking within schemas for fine-grained access control (also requires OneLake Spark Catalog)

Option B: Separate Lakehouses (Legacy)

1. Create three workspaces:
   • {project}-bronze-{env}
   • {project}-silver-{env}
   • {project}-gold-{env}
2. Create one lakehouse per workspace:
   • Bronze workspace → {project}_bronze lakehouse
   • Silver workspace → {project}_silver lakehouse
   • Gold workspace → {project}_gold lakehouse
3. Assign RBAC per layer workspace:
   • Bronze: ingestion/engineering write permissions
   • Silver: engineering/data quality permissions
   • Gold: analytics/BI consumer access with stricter curation controls
4. Enable OneLake Spark Catalog for non-schema lakehouses (required for RLS/CLS and catalog-backed access patterns):
   • Primary: Set spark.sql.fabric.catalog.enable-schemaless-lakehouses=true in an Environment and attach it to notebooks.
   • Alternative: Omit default lakehouse binding from notebooks. Use four-part fully-qualified references (workspace.lakehouse.schema.table). OneLake Spark Catalog auto-enables when no default lakehouse is set.
   • Alternative (internal/unsupported): Add this as the first cell in every notebook:

   %%pyspark
   !echo "spark.sql.fabric.catalog.enable-schemaless-lakehouses=true" >> /home/trusted-service-user/.trident-context
   

   • ⚠️ Note: This workaround uses an internal runtime configuration path that may change in future Fabric releases. Prefer schema-enabled lakehouses for stable, documented OneLake Spark Catalog support.
   • With this configuration, non-schema lakehouses support:
     • ✅ Row-level security (RLS) and column-level security (CLS)
   • Note: MLVs require schema-enabled lakehouses (Option A). For non-schema lakehouses, use notebooks with Delta tables.

Common Steps (Both Options)

After completing Option A or Option B above, perform these steps:

1. Create notebooks for each layer (one per transformation stage) — follow .ipynb validation + Fabric nuances
2. Bind each notebook to its lakehouse — set metadata.dependencies.lakehouse with the correct lakehouse ID (see [notebook-api-operations.md § Default Lakehouse Binding](../spark-cli/references/authoring/resources/notebook-api-operations.md#default-lakehouse-binding