7 min read
Schema Evolution: Surviving the Day Upstream Renames a Column
Every source schema changes eventually. Which changes Delta accepts silently, which it rejects, a contract per layer, and the changes no schema check sees.
Spark
34 articles
6 min read
The Small Files Problem: How Tables End Up With Two Million Files
Small files come from simple arithmetic, and every reader pays a cost per file. Where they come from, how to measure them, and how to compact them safely.
7 min read
Partition Pruning: The Fastest Data Is the Data You Never Read
A query for one day should not read three years. How partition pruning works, the five ways it silently breaks, and how to pick partition columns.
7 min read
Reading a Spark Query Plan: Ten Minutes That Explain a Two-Hour Job
The physical plan shows what Spark will actually do: shuffles, join strategy, pruning, Python UDFs. How to read one, and which plans should worry you.
7 min read
Data Skew: Why 199 Tasks Finish in Seconds and One Takes an Hour
One hot key can make one task do a third of a job, and more nodes will not help. How to spot skew in the Spark UI, find the keys, and fix it properly.
7 min read
Late-Arriving Data: Deciding When a Day Is Actually Done
Late data is a distribution you can measure, not an edge case. How to set a waiting policy, handle what arrives after it, and say when a number is final.
8 min read
Fabric Runtime 2.0 Is GA but Not Default: An Upgrade Plan
Fabric Runtime 2.0 is GA and set to become the default in late September. A plan to inventory, test libraries and Delta features, and pin runtimes first.
7 min read
Event Time vs Processing Time: Know Which Clock Your Numbers Are On
Every record has three timestamps: when it happened, when it arrived, and when it was processed. Group by the wrong one and past days keep changing.
8 min read
Idempotent Writes, and Why Retries Duplicate Data
Every pipeline gets retried eventually, and a non-idempotent write turns each retry into duplicates. Four patterns that make a write safe to repeat.
7 min read
Delta Lake in Fabric: Not Every Engine Reads Every Table
Every Fabric engine speaks Delta Lake, but not the same features. How to use the interoperability matrix so Spark tables still work in SQL and Power BI.
1 min read
Delta Lake Optimization: VACUUM, OPTIMIZE, and Z-ORDER Explained
VACUUM removes data files no longer referenced by the Delta log. Without regular vacuuming, your storage costs grow unbounded as old file versions accumulate.
1 min read
Native Execution Engine in Microsoft Fabric
Native execution bypasses traditional JVM-based Spark execution to run queries using native code optimized for modern CPUs.
1 min read
Query Performance Tuning in Microsoft Fabric
1. Filter early : Push filters as close to data source as possible 2. Use appropriate joins : Broadcast small tables 3. Enable AQE : Adaptive Query…
2 min read
Microsoft Fabric Data Engineering: Best Practices After 6 Months in Production
Avoid over-partitioning. If partitions have < 1GB of data, consolidate. Use Data Pipelines, Not Just Notebooks
7 min read
Tracing a Slow Fabric Report Back Through Every Layer
A diagnose-first workflow for slow Fabric reports: check capacity, the visual, DAX, Direct Lake fallback, Delta file layout and Spark, in that order.
7 min read
Fabric's December 2023 Update: What I'd Act On in January
A practitioner's triage of the Microsoft Fabric December 2023 update: %%configure, Git conflict resolution, KQL in OneLake and what to leave alone.
1 min read
Query Performance Optimization in Microsoft Fabric
Query performance is where UX and cost meet. My practical approach is to measure user-facing latency, inspect query plans, and prioritise optimisations that…
1 min read
Data Science Workflows in Microsoft Fabric
Building end-to-end data science workflows using Microsoft Fabric's integrated capabilities.
1 min read
Spark ML Patterns for Production Systems
Spark ML provides battle-tested patterns for production machine learning. From feature engineering to model persistence, these patterns ensure reliable ML…
1 min read
SynapseML: Distributed AI at Scale
SynapseML democratizes distributed machine learning. Train on massive datasets, tune hyperparameters in parallel, and deploy models at scale - all with…
3 min read
Azure Data Factory Mapping Data Flows: ETL Patterns for the Modern Data Platform
Think of it as the best of both worlds: visual design for maintainability, Spark for scale.
2 min read
Photon Engine: Accelerating Spark Workloads on Databricks
Photon is Databricks' native vectorised query execution engine written in C++—a replacement for the JVM-based Apache Spark execution engine for SQL and…
1 min read
Synapse Spark Pools: Optimization and Best Practices
Properly optimized Spark pools can process petabytes of data efficiently and cost-effectively.
2 min read
Job Clusters vs All-Purpose Clusters: Choosing the Right Approach
The job cluster versus all-purpose cluster decision in Databricks is primarily a cost decision: all-purpose clusters stay running between tasks (you pay for…
2 min read
Azure Databricks Cluster Configuration Best Practices
Azure Databricks cluster configuration is where cost and performance trade-offs become very concrete: the wrong cluster type for a workload is either money…
1 min read
Getting Started with Apache Spark 3.0 in Azure Synapse Analytics
First, let us create a new Spark pool with version 3.0 in Azure Synapse: One of the most significant improvements in Spark 3.0 is Adaptive Query Execution.…
1 min read
Real-Time Data Processing with Spark Structured Streaming
The appeal of Spark Structured Streaming is that you write it almost identically to a batch Spark job. Same DataFrame API, same transformations, same Spark…
1 min read
Visual ETL with Azure Synapse Data Flows
Data flows in Synapse run on Spark clusters and offer: Visual drag and drop interface 80+ built in transformations Schema drift handling Data preview and…
1 min read
Enterprise Big Data Analytics with Azure HDInsight
HDInsight supports multiple cluster types for different workloads: Apache Spark Fast, general purpose cluster computing Apache Hadoop Batch processing with…
2 min read
Big Data Processing with Azure Synapse Spark Pools
Synapse Spark pools are the feature that stops the "should I use Databricks or Synapse?" question from being purely a product choice and makes it an…
2 min read
Data Transformation with Azure Data Factory Mapping Data Flows
Mapping Data Flows are the feature that ended most of my "should I use PySpark or Spark SQL for this?" debates. Visual, yes—but reviewable in JSON and…
8 min read
Synapse Link for Cosmos DB: Is It Time to Retire the Nightly ETL?
When Azure Synapse Link can replace a Cosmos DB export pipeline, the limits to check first, and how to query the analytical store from SQL and Spark.
1 min read
Databricks Structured Streaming: Real-Time Data Processing
Incrementally process files as they arrive. Structured Streaming makes real-time processing accessible with familiar DataFrame semantics.
1 min read
Azure Synapse Spark Pools (Preview): Big Data Processing
Spark on Synapse is the same managed Spark story most cloud providers offer now, but it's done with the parts of Azure I already use — same workspace as the…