7 min read
Schema Evolution: Surviving the Day Upstream Renames a Column
Every source schema changes eventually. Which changes Delta accepts silently, which it rejects, a contract per layer, and the changes no schema check sees.
Delta Lake
46 articles
7 min read
Slowly Changing Dimensions Without Corrupting Your Facts
Should last year's revenue move when a customer changes segment? Pick an SCD type per column, write a replay-safe Type 2 MERGE, and join facts correctly.
6 min read
The Small Files Problem: How Tables End Up With Two Million Files
Small files come from simple arithmetic, and every reader pays a cost per file. Where they come from, how to measure them, and how to compact them safely.
7 min read
Partition Pruning: The Fastest Data Is the Data You Never Read
A query for one day should not read three years. How partition pruning works, the five ways it silently breaks, and how to pick partition columns.
7 min read
Data Skew: Why 199 Tasks Finish in Seconds and One Takes an Hour
One hot key can make one task do a third of a job, and more nodes will not help. How to spot skew in the Spark UI, find the keys, and fix it properly.
8 min read
Fabric Runtime 2.0 Is GA but Not Default: An Upgrade Plan
Fabric Runtime 2.0 is GA and set to become the default in late September. A plan to inventory, test libraries and Delta features, and pin runtimes first.
8 min read
Idempotent Writes, and Why Retries Duplicate Data
Every pipeline gets retried eventually, and a non-idempotent write turns each retry into duplicates. Four patterns that make a write safe to repeat.
9 min read
Event-Driven Refresh for Materialized Lake Views: Schedules Still Win
Event-triggered refresh for Fabric materialized lake views trades predictable capacity and clean ordering for freshness. Where events fit and where they don't.
9 min read
Materialized Lake Views as a Silver Flow: What to Hand Over
Now that Fabric materialized lake views are GA: which raw-to-silver steps I'd move onto them, how to keep rejected rows, and where notebooks still win.
7 min read
Write-Audit-Publish in Fabric: Stop Bad Data Before It Ships
Why many data quality root-cause fixes come down to where the check runs, and how to build a write-audit-publish boundary on a Fabric lakehouse.
8 min read
Changing a Shared Fabric Table Without Breaking Its Notebooks
How to version a Fabric lakehouse table contract once several notebooks depend on it: change classes, a writer guard, pinned readers and safe renames.
8 min read
When a Slow Fabric Lakehouse Is Really a Raw Zone Problem
Why slow Fabric silver loads often trace back to the raw zone: landing layout, header checks, small files and bronze table maintenance, with PySpark examples.
8 min read
Two Tickets per Data Incident: Contain First, Fix the Cause Second
Why every data quality incident should produce two pieces of work, a containment action and a root-cause fix, and how to run both on a Fabric lakehouse.
8 min read
Fabric Data Movement: Design for the Half-Finished Copy
How to design Fabric Copy jobs and copy activities so a failed, partial or repeated copy never leaves duplicates, gaps or a watermark that moved too far.
9 min read
Table Contracts in a Fabric Lakehouse: Where Raw Becomes a Product
Four decisions for a messy Fabric raw zone: contracts at silver, Delta constraints, quarantined rows with failed batches, and lakehouse schemas.
7 min read
Delta Lake in Fabric: Not Every Engine Reads Every Table
Every Fabric engine speaks Delta Lake, but not the same features. How to use the interoperability matrix so Spark tables still work in SQL and Power BI.
8 min read
Running Data Pipelines in Production: Reruns, Late Data and Drift
Eight years of data pipelines taught me to plan for reruns, late data and schema drift. The habits that matter, with Delta MERGE and Fabric examples.
1 min read
Building Data Lakehouses: Medallion Architecture Best Practices
Each layer serves a specific purpose in the data refinement pipeline. Bronze captures raw data, silver cleans and conforms it, and gold delivers…
1 min read
Delta Lake Deep Dive: Time Travel and Schema Evolution in Fabric
Every write operation to a Delta table creates a new version. This transaction log enables point-in-time queries and rollback capabilities essential for…
1 min read
Microsoft Fabric Lakehouse: Medallion Architecture Implementation
Bronze holds raw ingested data. Silver contains cleansed and conformed data. Gold presents business-level aggregates ready for consumption.
1 min read
Medallion Architecture in Microsoft Fabric: Bronze, Silver, Gold Explained
Bronze tables store data exactly as received from source systems, preserving the original format for replayability and debugging.
1 min read
Delta Lake Optimization: VACUUM, OPTIMIZE, and Z-ORDER Explained
VACUUM removes data files no longer referenced by the Delta log. Without regular vacuuming, your storage costs grow unbounded as old file versions accumulate.
1 min read
Microsoft Fabric Lakehouse: Unified Analytics for the Modern Data Stack
A Lakehouse stores data in open formats like Delta Lake while providing SQL query capabilities, ACID transactions, and schema enforcement. In Microsoft…
1 min read
Data Lakehouse Architecture with Microsoft Fabric: Medallion Pattern Implementation
Use Delta Lake constraints and expectations to ensure data quality at each layer. Fabric's Data Quality monitoring provides visibility into quality metrics…
1 min read
Z-Order vs V-Order: Choosing the Right Data Ordering
1. Always enable V Order : Low overhead, universal benefits 2. Add Z Order for specific needs : Known filter patterns 3. Limit Z Order columns : Maximum 3 4…
1 min read
Optimized Writes in Delta Lake
1. Enable by default : For most analytical workloads 2. Choose appropriate bin size : 128MB general, 256MB for Direct Lake 3. Combine with auto compact :…
1 min read
File Compaction Strategies in Delta Lake
1. Schedule regular compaction : Daily for high volume tables 2. Use partition aware compaction : Avoid rewriting entire large tables 3. Enable auto…
1 min read
Parquet File Optimization for Analytics
1. Target 128 256MB files : Optimal for most query engines 2. Use Snappy or ZSTD : Best compression/speed balance 3. Sort by filter columns : Improves…
1 min read
Delta Lake Everywhere: The Universal Table Format
Delta Lake provides a reliable, performant foundation for modern data platforms. Its ACID transactions, time travel, and schema evolution capabilities make…
1 min read
Microsoft Fabric and Databricks: Coexistence Strategies
1. Use shared storage : ADLS as the common data layer 2. Standardize on Delta Lake : Compatible format for both 3. Optimize for readers : Both platforms…
1 min read
Open Mirroring in Fabric: Build Custom Data Sources
Open Mirroring provides a standardized way to: Write Delta Lake tables directly to a mirrored database location Maintain the mirrored database experience…
1 min read
V-Order Optimization: Deep Dive into Fabric's Secret Weapon
V-Order feels like a secret weapon because it's a write-time optimisation with outsized read-time benefits. In practice I enable V-Order on wide, heavily…
1 min read
Delta Tables in Microsoft Fabric: Deep Dive
The theoretical benefits of Delta Lake — ACID transactions, time travel, schema enforcement — are easy to describe, but the practical payoff becomes clear…
1 min read
Creating Your First Fabric Lakehouse: Step-by-Step Guide
A Lakehouse combines the best of data lakes and data warehouses: Once created, your Lakehouse has two main sections: The Files section is for unstructured…
1 min read
Delta Lake in Microsoft Fabric: ACID Transactions for Data Lakes
Delta Lake isn't a bolt-on feature in Microsoft Fabric — it's the foundational table format for everything. Every table you write through a Spark notebook…
1 min read
Microsoft Fabric Lakehouse Architecture: Deep Dive
A week after Build 2023, I'm still unpacking what Microsoft Fabric actually means for how we design data platforms. The Lakehouse is the centrepiece — but…
1 min read
Building with Fabric Lakehouse: The Best of Data Lakes and Warehouses
The Lakehouse is where most of your Fabric data engineering work will happen. Tomorrow, I will cover Data Factory in Fabric for orchestrating data movement…
1 min read
Data Lakehouse Patterns: Best of Both Worlds
The lakehouse architecture proved its value in 2021. Organizations are consolidating their data warehouses and data lakes into unified lakehouses, reducing…
1 min read
Data Engineering Trends That Defined 2021
Data engineering in 2021 matured from a support function to a strategic capability. The tools improved, patterns solidified, and the role gained the…
1 min read
Mastering Delta Lake MERGE Operations for Upserts and SCD
The MERGE statement combines INSERT, UPDATE, and DELETE operations in a single atomic transaction: Handle complex business logic with conditional updates:…
2 min read
Delta Lake Time Travel - Versioned Data Lake Queries
Delta Lake time travel is the feature that makes "oops" survivable. An engineer runs a DELETE with a typo in the WHERE clause. A batch job writes corrupt…
7 min read
Wiring a Medallion Pipeline on Azure Databricks with Auto Loader
How to connect Auto Loader, Structured Streaming and Delta MERGE into a bronze, silver and gold pipeline that restarts safely and doesn't run up the bill.
7 min read
Databricks SQL Analytics in Preview: What to Test Before You Commit
SQL Analytics on Azure Databricks is in public preview. Here is what it is, how to pilot it properly, and when Synapse or Power BI alone is the better fit.
2 min read
Getting Started with Delta Lake on Azure Databricks
Data lakes have a long-standing reputation problem: cheap to fill, painful to trust. Schema drift, half-written files from failed jobs, "is this row a…
1 min read
Databricks Delta Lake: Advanced Patterns
Delta Lake: the foundation of the modern lakehouse.
1 min read
Introduction to Delta Lake on Azure Databricks
Anyone who's run a "data lake" for any length of time has hit the same wall: parquet files everywhere, no transactional guarantees, partial-write disasters…