9 min read
Event-Driven Refresh for Materialized Lake Views: Schedules Still Win
Event-triggered refresh for Fabric materialized lake views trades predictable capacity and clean ordering for freshness. Where events fit and where they don't.
Lakehouse
31 articles
9 min read
Materialized Lake Views as a Silver Flow: What to Hand Over
Now that Fabric materialized lake views are GA: which raw-to-silver steps I'd move onto them, how to keep rejected rows, and where notebooks still win.
8 min read
Changing a Shared Fabric Table Without Breaking Its Notebooks
How to version a Fabric lakehouse table contract once several notebooks depend on it: change classes, a writer guard, pinned readers and safe renames.
8 min read
Fabric Warehouse Tables or Lakehouse Shortcuts: Where Each Table Lives
A placement rule for Fabric: when data belongs in a Warehouse table you own, and when reading a lakehouse shortcut through the SQL analytics endpoint is enough.
8 min read
Ingestion vs Curation in Fabric: Where I Draw the Workspace Line
Why I split Fabric ingestion and curation at the landing table, when that means separate workspaces joined by shortcuts, and when one lakehouse is enough.
8 min read
When a Slow Fabric Lakehouse Is Really a Raw Zone Problem
Why slow Fabric silver loads often trace back to the raw zone: landing layout, header checks, small files and bronze table maintenance, with PySpark examples.
8 min read
Break Your Silver Layer on Purpose: Bad-Input Tests in Fabric
Why I keep a library of deliberately broken raw files for every Fabric bronze-to-silver transform, and how to run them as plain PySpark tests before release.
9 min read
Table Contracts in a Fabric Lakehouse: Where Raw Becomes a Product
Four decisions for a messy Fabric raw zone: contracts at silver, Delta constraints, quarantined rows with failed batches, and lakehouse schemas.
7 min read
Delta Lake in Fabric: Not Every Engine Reads Every Table
Every Fabric engine speaks Delta Lake, but not the same features. How to use the interoperability matrix so Spark tables still work in SQL and Power BI.
1 min read
Building Data Lakehouses: Medallion Architecture Best Practices
Each layer serves a specific purpose in the data refinement pipeline. Bronze captures raw data, silver cleans and conforms it, and gold delivers…
1 min read
Delta Lake Deep Dive: Time Travel and Schema Evolution in Fabric
Every write operation to a Delta table creates a new version. This transaction log enables point-in-time queries and rollback capabilities essential for…
1 min read
Implementing Data Quality Checks in Microsoft Fabric Lakehouses
Poor data quality leads to incorrect business decisions, failed ML models, and eroded trust in analytics platforms. A proactive approach to data validation…
1 min read
Implementing Retrieval-Augmented Generation with Microsoft Fabric
Use Fabric notebooks to create interactive RAG applications that combine data exploration with AI-powered question answering over enterprise datasets.
1 min read
Microsoft Fabric Lakehouse: Medallion Architecture Implementation
Bronze holds raw ingested data. Silver contains cleansed and conformed data. Gold presents business-level aggregates ready for consumption.
1 min read
Medallion Architecture in Microsoft Fabric: Bronze, Silver, Gold Explained
Bronze tables store data exactly as received from source systems, preserving the original format for replayability and debugging.
1 min read
Apache Iceberg on Microsoft Fabric: Open Table Format Integration
Iceberg provides ACID transactions, schema evolution, time travel, and partition evolution for data lakes. Its format-agnostic design works with Spark…
1 min read
Microsoft Fabric Lakehouse: Unified Analytics for the Modern Data Stack
A Lakehouse stores data in open formats like Delta Lake while providing SQL query capabilities, ACID transactions, and schema enforcement. In Microsoft…
1 min read
Lakehouse Federation in Microsoft Fabric
1. Define clear domains : Each lakehouse should own specific data 2. Minimize cross domain joins : Materialize frequently joined data 3. Implement…
2 min read
Microsoft Fabric Data Engineering: Best Practices After 6 Months in Production
Avoid over-partitioning. If partitions have < 1GB of data, consolidate. Use Data Pipelines, Not Just Notebooks
8 min read
Laying Out OneLake: Workspaces, Lakehouses and Shortcuts
How I decide OneLake layout in Fabric: workspaces as security boundaries, a lakehouse per layer, a flat Tables folder, and shortcuts where copies hurt.
1 min read
Data Platform Evolution: Where We're Heading
Having lived through the shift from monolithic warehouses to lakehouses, the pattern I'm seeing now is convergence: open formats, governance, and…
1 min read
Lakehouse vs Warehouse in Fabric: Making the Right Choice
Choosing Lakehouse vs Warehouse is an architectural trade-off: for open-format, large-scale analytics and ML, Lakehouse is the better fit; for classic T-SQL…
1 min read
Fabric Architecture Patterns: Designing for Scale
Designing for scale in Fabric is about decomposition and clear contracts between layers. The medallion architecture works well: small, composable…
1 min read
Delta Tables in Microsoft Fabric: Deep Dive
The theoretical benefits of Delta Lake — ACID transactions, time travel, schema enforcement — are easy to describe, but the practical payoff becomes clear…
1 min read
Creating Your First Fabric Lakehouse: Step-by-Step Guide
A Lakehouse combines the best of data lakes and data warehouses: Once created, your Lakehouse has two main sections: The Files section is for unstructured…
1 min read
Delta Lake in Microsoft Fabric: ACID Transactions for Data Lakes
Delta Lake isn't a bolt-on feature in Microsoft Fabric — it's the foundational table format for everything. Every table you write through a Spark notebook…
1 min read
Microsoft Fabric Lakehouse Architecture: Deep Dive
A week after Build 2023, I'm still unpacking what Microsoft Fabric actually means for how we design data platforms. The Lakehouse is the centrepiece — but…
1 min read
Building with Fabric Lakehouse: The Best of Data Lakes and Warehouses
The Lakehouse is where most of your Fabric data engineering work will happen. Tomorrow, I will cover Data Factory in Fabric for orchestrating data movement…
1 min read
Data Engineering Patterns That Defined 2022
2022 established these patterns as the foundation of modern data engineering. The lakehouse became the default architecture for new data platforms. Data…
2 min read
Getting Started with Delta Lake on Azure Databricks
Data lakes have a long-standing reputation problem: cheap to fill, painful to trust. Schema drift, half-written files from failed jobs, "is this row a…
1 min read
Databricks Delta Lake: Advanced Patterns
Delta Lake: the foundation of the modern lakehouse.