1 min read
Building Data Lakehouses: Medallion Architecture Best Practices
Each layer serves a specific purpose in the data refinement pipeline. Bronze captures raw data, silver cleans and conforms it, and gold delivers…
Delta Lake
28 articles
1 min read
Delta Lake Deep Dive: Time Travel and Schema Evolution in Fabric
Every write operation to a Delta table creates a new version. This transaction log enables point-in-time queries and rollback capabilities essential for…
1 min read
Microsoft Fabric Lakehouse: Medallion Architecture Implementation
Bronze holds raw ingested data. Silver contains cleansed and conformed data. Gold presents business-level aggregates ready for consumption.
1 min read
Medallion Architecture in Microsoft Fabric: Bronze, Silver, Gold Explained
Bronze tables store data exactly as received from source systems, preserving the original format for replayability and debugging.
1 min read
Delta Lake Optimization: VACUUM, OPTIMIZE, and Z-ORDER Explained
VACUUM removes data files no longer referenced by the Delta log. Without regular vacuuming, your storage costs grow unbounded as old file versions accumulate.
1 min read
Microsoft Fabric Lakehouse: Unified Analytics for the Modern Data Stack
A Lakehouse stores data in open formats like Delta Lake while providing SQL query capabilities, ACID transactions, and schema enforcement. In Microsoft…
1 min read
Data Lakehouse Architecture with Microsoft Fabric: Medallion Pattern Implementation
Use Delta Lake constraints and expectations to ensure data quality at each layer. Fabric's Data Quality monitoring provides visibility into quality metrics…
1 min read
Z-Order vs V-Order: Choosing the Right Data Ordering
1. Always enable V Order : Low overhead, universal benefits 2. Add Z Order for specific needs : Known filter patterns 3. Limit Z Order columns : Maximum 3 4…
1 min read
Optimized Writes in Delta Lake
1. Enable by default : For most analytical workloads 2. Choose appropriate bin size : 128MB general, 256MB for Direct Lake 3. Combine with auto compact :…
1 min read
File Compaction Strategies in Delta Lake
1. Schedule regular compaction : Daily for high volume tables 2. Use partition aware compaction : Avoid rewriting entire large tables 3. Enable auto…
1 min read
Parquet File Optimization for Analytics
1. Target 128 256MB files : Optimal for most query engines 2. Use Snappy or ZSTD : Best compression/speed balance 3. Sort by filter columns : Improves…
1 min read
Delta Lake Everywhere: The Universal Table Format
Delta Lake provides a reliable, performant foundation for modern data platforms. Its ACID transactions, time travel, and schema evolution capabilities make…
1 min read
Microsoft Fabric and Databricks: Coexistence Strategies
1. Use shared storage : ADLS as the common data layer 2. Standardize on Delta Lake : Compatible format for both 3. Optimize for readers : Both platforms…
1 min read
Open Mirroring in Fabric: Build Custom Data Sources
Open Mirroring provides a standardized way to: Write Delta Lake tables directly to a mirrored database location Maintain the mirrored database experience…
1 min read
V-Order Optimization: Deep Dive into Fabric's Secret Weapon
V-Order feels like a secret weapon because it's a write-time optimisation with outsized read-time benefits. In practice I enable V-Order on wide, heavily…
1 min read
Delta Tables in Microsoft Fabric: Deep Dive
The theoretical benefits of Delta Lake — ACID transactions, time travel, schema enforcement — are easy to describe, but the practical payoff becomes clear…
1 min read
Creating Your First Fabric Lakehouse: Step-by-Step Guide
A Lakehouse combines the best of data lakes and data warehouses: Once created, your Lakehouse has two main sections: The Files section is for unstructured…
1 min read
Delta Lake in Microsoft Fabric: ACID Transactions for Data Lakes
Delta Lake isn't a bolt-on feature in Microsoft Fabric — it's the foundational table format for everything. Every table you write through a Spark notebook…
1 min read
Microsoft Fabric Lakehouse Architecture: Deep Dive
A week after Build 2023, I'm still unpacking what Microsoft Fabric actually means for how we design data platforms. The Lakehouse is the centrepiece — but…
1 min read
Building with Fabric Lakehouse: The Best of Data Lakes and Warehouses
The Lakehouse is where most of your Fabric data engineering work will happen. Tomorrow, I will cover Data Factory in Fabric for orchestrating data movement…
1 min read
Data Lakehouse Patterns: Best of Both Worlds
The lakehouse architecture proved its value in 2021. Organizations are consolidating their data warehouses and data lakes into unified lakehouses, reducing…
1 min read
Data Engineering Trends That Defined 2021
Data engineering in 2021 matured from a support function to a strategic capability. The tools improved, patterns solidified, and the role gained the…
1 min read
Mastering Delta Lake MERGE Operations for Upserts and SCD
The MERGE statement combines INSERT, UPDATE, and DELETE operations in a single atomic transaction: Handle complex business logic with conditional updates:…
2 min read
Delta Lake Time Travel - Versioned Data Lake Queries
Delta Lake time travel is the feature that makes "oops" survivable. An engineer runs a DELETE with a typo in the WHERE clause. A batch job writes corrupt…
2 min read
Building Data Pipelines with Delta Lake on Azure Databricks
Delta Lake has gone from "interesting open source project" to the default storage layer for data lakes inside about eighteen months. ACID, schema…
2 min read
Getting Started with Delta Lake on Azure Databricks
Data lakes have a long-standing reputation problem: cheap to fill, painful to trust. Schema drift, half-written files from failed jobs, "is this row a…
1 min read
Databricks Delta Lake: Advanced Patterns
Delta Lake: the foundation of the modern lakehouse.
1 min read
Introduction to Delta Lake on Azure Databricks
Anyone who's run a "data lake" for any length of time has hit the same wall: parquet files everywhere, no transactional guarantees, partial-write disasters…