1 min read
Delta Lake Optimization: VACUUM, OPTIMIZE, and Z-ORDER Explained
VACUUM removes data files no longer referenced by the Delta log. Without regular vacuuming, your storage costs grow unbounded as old file versions accumulate.
Spark
22 articles
1 min read
Native Execution Engine in Microsoft Fabric
Native execution bypasses traditional JVM-based Spark execution to run queries using native code optimized for modern CPUs.
1 min read
Query Performance Tuning in Microsoft Fabric
1. Filter early : Push filters as close to data source as possible 2. Use appropriate joins : Broadcast small tables 3. Enable AQE : Adaptive Query…
2 min read
Microsoft Fabric Data Engineering: Best Practices After 6 Months in Production
Avoid over-partitioning. If partitions have < 1GB of data, consolidate. Use Data Pipelines, Not Just Notebooks
1 min read
Microsoft Fabric Performance Tuning: From Notebooks to Reports
When diagnosing Fabric performance, I've found the root cause can be anywhere from Spark configs to report visuals. This guide consolidates tuning…
1 min read
Query Performance Optimization in Microsoft Fabric
Query performance is where UX and cost meet. My practical approach is to measure user-facing latency, inspect query plans, and prioritise optimisations that…
1 min read
Data Science Workflows in Microsoft Fabric
Building end-to-end data science workflows using Microsoft Fabric's integrated capabilities.
1 min read
Spark ML Patterns for Production Systems
Spark ML provides battle-tested patterns for production machine learning. From feature engineering to model persistence, these patterns ensure reliable ML…
1 min read
SynapseML: Distributed AI at Scale
SynapseML democratizes distributed machine learning. Train on massive datasets, tune hyperparameters in parallel, and deploy models at scale - all with…
3 min read
Azure Data Factory Mapping Data Flows: ETL Patterns for the Modern Data Platform
Think of it as the best of both worlds: visual design for maintainability, Spark for scale.
2 min read
Photon Engine: Accelerating Spark Workloads on Databricks
Photon is Databricks' native vectorised query execution engine written in C++—a replacement for the JVM-based Apache Spark execution engine for SQL and…
1 min read
Synapse Spark Pools: Optimization and Best Practices
Properly optimized Spark pools can process petabytes of data efficiently and cost-effectively.
2 min read
Job Clusters vs All-Purpose Clusters: Choosing the Right Approach
The job cluster versus all-purpose cluster decision in Databricks is primarily a cost decision: all-purpose clusters stay running between tasks (you pay for…
2 min read
Azure Databricks Cluster Configuration Best Practices
Azure Databricks cluster configuration is where cost and performance trade-offs become very concrete: the wrong cluster type for a workload is either money…
1 min read
Getting Started with Apache Spark 3.0 in Azure Synapse Analytics
First, let us create a new Spark pool with version 3.0 in Azure Synapse: One of the most significant improvements in Spark 3.0 is Adaptive Query Execution.…
1 min read
Real-Time Data Processing with Spark Structured Streaming
The appeal of Spark Structured Streaming is that you write it almost identically to a batch Spark job. Same DataFrame API, same transformations, same Spark…
1 min read
Visual ETL with Azure Synapse Data Flows
Data flows in Synapse run on Spark clusters and offer: Visual drag and drop interface 80+ built in transformations Schema drift handling Data preview and…
1 min read
Enterprise Big Data Analytics with Azure HDInsight
HDInsight supports multiple cluster types for different workloads: Apache Spark Fast, general purpose cluster computing Apache Hadoop Batch processing with…
2 min read
Big Data Processing with Azure Synapse Spark Pools
Synapse Spark pools are the feature that stops the "should I use Databricks or Synapse?" question from being purely a product choice and makes it an…
2 min read
Data Transformation with Azure Data Factory Mapping Data Flows
Mapping Data Flows are the feature that ended most of my "should I use PySpark or Spark SQL for this?" debates. Visual, yes—but reviewable in JSON and…
1 min read
Databricks Structured Streaming: Real-Time Data Processing
Incrementally process files as they arrive. Structured Streaming makes real-time processing accessible with familiar DataFrame semantics.
1 min read
Azure Synapse Spark Pools (Preview): Big Data Processing
Spark on Synapse is the same managed Spark story most cloud providers offer now, but it's done with the parts of Azure I already use — same workspace as the…