7 min read
Schema Evolution: Surviving the Day Upstream Renames a Column
Every source schema changes eventually. Which changes Delta accepts silently, which it rejects, a contract per layer, and the changes no schema check sees.
Data Engineering
154 articles
7 min read
Slowly Changing Dimensions Without Corrupting Your Facts
Should last year's revenue move when a customer changes segment? Pick an SCD type per column, write a replay-safe Type 2 MERGE, and join facts correctly.
6 min read
The Small Files Problem: How Tables End Up With Two Million Files
Small files come from simple arithmetic, and every reader pays a cost per file. Where they come from, how to measure them, and how to compact them safely.
9 min read
Fabric's Data Engineering Agent: Set Guardrails Before Autonomy
Fabric's preview data engineering agent runs with the launching engineer's permissions. What to delegate first, which settings to pick, and what stays human.
7 min read
Partition Pruning: The Fastest Data Is the Data You Never Read
A query for one day should not read three years. How partition pruning works, the five ways it silently breaks, and how to pick partition columns.
7 min read
Reading a Spark Query Plan: Ten Minutes That Explain a Two-Hour Job
The physical plan shows what Spark will actually do: shuffles, join strategy, pruning, Python UDFs. How to read one, and which plans should worry you.
7 min read
Data Skew: Why 199 Tasks Finish in Seconds and One Takes an Hour
One hot key can make one task do a third of a job, and more nodes will not help. How to spot skew in the Spark UI, find the keys, and fix it properly.
7 min read
Late-Arriving Data: Deciding When a Day Is Actually Done
Late data is a distribution you can measure, not an edge case. How to set a waiting policy, handle what arrives after it, and say when a number is final.
8 min read
Fabric Runtime 2.0 Is GA but Not Default: An Upgrade Plan
Fabric Runtime 2.0 is GA and set to become the default in late September. A plan to inventory, test libraries and Delta features, and pin runtimes first.
7 min read
Event Time vs Processing Time: Know Which Clock Your Numbers Are On
Every record has three timestamps: when it happened, when it arrived, and when it was processed. Group by the wrong one and past days keep changing.
6 min read
The Grain of a Table: Declare It Before the First Join
Grain is what one row of a table means. Most wrong numbers come from a join or aggregation that quietly changed it. How to declare grain and test it.
8 min read
Idempotent Writes, and Why Retries Duplicate Data
Every pipeline gets retried eventually, and a non-idempotent write turns each retry into duplicates. Four patterns that make a write safe to repeat.
9 min read
Event-Driven Refresh for Materialized Lake Views: Schedules Still Win
Event-triggered refresh for Fabric materialized lake views trades predictable capacity and clean ordering for freshness. Where events fit and where they don't.
8 min read
Synapse Serverless varchar(8000): Fixing Strings for Spark and Power BI
Why Synapse serverless SQL infers strings as varchar(8000), why Spark-created lake tables behave differently, and where Power BI string contracts belong.
8 min read
Rerun From Failed Activity in Fabric: Design for the Second Run
What a Fabric pipeline rerun actually runs again, why utcNow() and run IDs break recovery, and how to make rerun from failed activity safe.
9 min read
Materialized Lake Views as a Silver Flow: What to Hand Over
Now that Fabric materialized lake views are GA: which raw-to-silver steps I'd move onto them, how to keep rejected rows, and where notebooks still win.
7 min read
Write-Audit-Publish in Fabric: Stop Bad Data Before It Ships
Why many data quality root-cause fixes come down to where the check runs, and how to build a write-audit-publish boundary on a Fabric lakehouse.
7 min read
Does Your LLM Eval Score Predict Anything the Business Cares About?
Before an eval score becomes a product metric, join its verdicts to real outcomes like escalations and re-asks, and check whether failing answers do worse.
9 min read
Fabric Pipeline Deadlines: Alert on a Late Run Before Anyone Asks
How to catch a slipping Fabric pipeline before its deadline: alert on missing and late runs, not just failures, using job events, KQL and the jobs API.
8 min read
Changing a Shared Fabric Table Without Breaking Its Notebooks
How to version a Fabric lakehouse table contract once several notebooks depend on it: change classes, a writer guard, pinned readers and safe renames.
8 min read
Readable Fabric Pipelines: Names, Boundaries and Config Off the Canvas
How to keep Fabric pipelines readable as they grow: naming before the first Git commit, child pipelines with clear contracts, and variable libraries for config.
8 min read
Ingestion vs Curation in Fabric: Where I Draw the Workspace Line
Why I split Fabric ingestion and curation at the landing table, when that means separate workspaces joined by shortcuts, and when one lakehouse is enough.
7 min read
Failure Budgets for Data Products: SLOs for Fresh, Correct Tables
Measure data product reliability in good hours, catch bad data at the publish boundary, and let a failure budget decide when pipeline changes stop.
9 min read
Store Changes, Not Readings: Shaping Noisy Streams in an Eventhouse
Use update policies, materialized views and a KQL state-change function in a Fabric Eventhouse to strip duplicates, chatter and flapping before anyone queries.
8 min read
OneLake Shortcuts Are Dependencies: Inventory Them Before Scale
Treat every OneLake shortcut as a cross-team dependency: placement, identity, write paths, environment variables and a REST inventory of what points where.
8 min read
When a Slow Fabric Lakehouse Is Really a Raw Zone Problem
Why slow Fabric silver loads often trace back to the raw zone: landing layout, header checks, small files and bronze table maintenance, with PySpark examples.
7 min read
Pruning Data Quality Checks: Keep, Demote, or Delete
How to cut an inherited data quality rule set down to the checks that change decisions, using failure history, ownership and an explicit retirement path.
8 min read
Fabric Pipeline Failure Paths: Catch the Error, Then Fail the Run
How dependency conditions decide a Fabric pipeline's final status, why try-catch hides failures, and how to log, alert and still fail the run honestly.
8 min read
Keeping OneLake Clean: Design the Workspace Lifecycle Before Scale
How to stop Fabric workspace and item sprawl before it starts: lifecycle states, domains, tags, endorsement, retention settings and a weekly hygiene audit.
8 min read
Two Tickets per Data Incident: Contain First, Fix the Cause Second
Why every data quality incident should produce two pieces of work, a containment action and a root-cause fix, and how to run both on a Fabric lakehouse.
8 min read
Fabric Data Movement: Design for the Half-Finished Copy
How to design Fabric Copy jobs and copy activities so a failed, partial or repeated copy never leaves duplicates, gaps or a watermark that moved too far.
9 min read
Who Gets Told? Owners, Error Codes and Runbooks in Fabric
A table contract is half the job: assigning owners per layer, collapsing load outcomes to three, and mapping Fabric pipeline failures to runbooks.
7 min read
Five Data Quality Signals for the Data-to-AI Handoff
Why I'd track five quality signals per feature table at the data engineering to AI handoff, and how to compute them in a Fabric notebook without a new tool.
9 min read
Fabric Daily Loads: Gate on Data Readiness, Not the Clock
How to remove brittle dependencies from daily Fabric loads with one orchestrating pipeline, readiness checks, idempotent writes and deliberate retries.
8 min read
OneLake Table Contracts: Owners, Schemas and Failure Modes Up Front
Why OneLake governance has to be designed before scale: table contracts, schema-enabled lakehouses, explicit owners and deliberate failure modes in Fabric.
9 min read
Table Contracts in a Fabric Lakehouse: Where Raw Becomes a Product
Four decisions for a messy Fabric raw zone: contracts at silver, Delta constraints, quarantined rows with failed batches, and lakehouse schemas.
9 min read
Leaving ADF and Databricks for Fabric: What Held Up
What consolidating Azure Data Factory, Databricks, Azure SQL and Power BI onto Microsoft Fabric delivered in production, and where the seams still show.
8 min read
Running Data Pipelines in Production: Reruns, Late Data and Drift
Eight years of data pipelines taught me to plan for reruns, late data and schema drift. The habits that matter, with Delta MERGE and Fabric examples.
2 min read
Data Engineering Career Retrospective: What Changed in 2025
The data engineering profession evolved dramatically in 2025. As the year ends, here's my retrospective on the skills, tools, and mindsets that defined…
1 min read
Ignite 2025 Day 2: Microsoft Fabric Innovations and Real-Time Intelligence
These Fabric updates demonstrate Microsoft's commitment to building a truly unified data platform that handles real-time and batch workloads while…
1 min read
Real-Time Data Streaming with Microsoft Fabric Eventstreams
Batch processing introduces latency between data generation and insight availability. For fraud detection, operational monitoring, and customer engagement…
1 min read
Implementing Feature Stores with Microsoft Fabric OneLake
Without a feature store, teams often duplicate feature engineering work across projects. Features computed for training may differ from inference, causing…
1 min read
Delta Lake Deep Dive: Time Travel and Schema Evolution in Fabric
Every write operation to a Delta table creates a new version. This transaction log enables point-in-time queries and rollback capabilities essential for…
1 min read
Implementing Data Quality Checks in Microsoft Fabric Lakehouses
Poor data quality leads to incorrect business decisions, failed ML models, and eroded trust in analytics platforms. A proactive approach to data validation…
1 min read
Building Production-Ready AI Pipelines with Azure Machine Learning
Many organizations struggle with the gap between experimentation and production. Models that work in notebooks often fail in real-world scenarios due to…
1 min read
Implementing Retrieval-Augmented Generation with Microsoft Fabric
Use Fabric notebooks to create interactive RAG applications that combine data exploration with AI-powered question answering over enterprise datasets.
1 min read
Microsoft Fabric Lakehouse: Medallion Architecture Implementation
Bronze holds raw ingested data. Silver contains cleansed and conformed data. Gold presents business-level aggregates ready for consumption.
1 min read
Microsoft Fabric Real-Time Analytics: Streaming Data Pipelines
Eventhouse is the storage layer optimized for time-series and streaming data. It automatically indexes data for lightning-fast queries.
1 min read
Building Data Pipelines with Microsoft Fabric Data Factory
Data Factory pipelines use a visual designer with activities that can be chained together.
1 min read
Medallion Architecture in Microsoft Fabric: Bronze, Silver, Gold Explained
Bronze tables store data exactly as received from source systems, preserving the original format for replayability and debugging.
1 min read
Data Quality for AI: Ensuring Clean Data for ML and LLM Applications
Quality encompasses completeness, accuracy, consistency, timeliness, and validity. Each dimension requires specific checks.
1 min read
Real-Time Data Processing with Microsoft Fabric Eventstreams
Eventstreams support multiple source types including Azure Event Hubs, Azure IoT Hub, and custom applications. The visual designer makes it easy to…
1 min read
Apache Iceberg on Microsoft Fabric: Open Table Format Integration
Iceberg provides ACID transactions, schema evolution, time travel, and partition evolution for data lakes. Its format-agnostic design works with Spark…
1 min read
Delta Lake Optimization: VACUUM, OPTIMIZE, and Z-ORDER Explained
VACUUM removes data files no longer referenced by the Delta log. Without regular vacuuming, your storage costs grow unbounded as old file versions accumulate.
1 min read
Microsoft Fabric Lakehouse: Unified Analytics for the Modern Data Stack
A Lakehouse stores data in open formats like Delta Lake while providing SQL query capabilities, ACID transactions, and schema enforcement. In Microsoft…
1 min read
Data Lakehouse Architecture with Microsoft Fabric: Medallion Pattern Implementation
Use Delta Lake constraints and expectations to ensure data quality at each layer. Fabric's Data Quality monitoring provides visibility into quality metrics…
1 min read
Microsoft Fabric Real-Time Intelligence: Streaming Analytics at Scale
Microsoft Fabric's Real-Time Intelligence workload has matured significantly since GA. Today I'm exploring how to build streaming analytics pipelines that…
3 min read
Microsoft Fabric Real-Time Intelligence: Patterns for Streaming Analytics
Microsoft Fabric's Real-Time Intelligence workload has matured significantly since GA. We've implemented several production streaming scenarios, and I want…
1 min read
AI in Data Pipelines: Intelligent ETL and ELT
AI-enhanced pipelines handle complex transformations and quality issues automatically.
2 min read
Microsoft Fabric Copilot Now Available for All Paid SKUs
This works on F2 and above now. For small businesses, this transforms how reports are built.
1 min read
dbt and AI: Intelligent Data Transformations
AI supercharges dbt workflows with intelligent generation and optimization.
1 min read
AI Data Pipeline Patterns: Intelligent ETL and Processing
AI-powered pipelines transform raw data into intelligent, enriched datasets. Design for both batch and streaming scenarios.
1 min read
Streaming Inference Patterns: Processing Data as It Flows
Streaming inference brings AI insights to real-time data. Design for throughput, latency, and reliability from the start.
1 min read
AI in Data Pipelines: Intelligent ETL and Data Processing
AI transforms data pipelines from rigid rule-based systems to adaptive, intelligent processes. Start with high-value, error-prone steps and expand from there.
8 min read
LLM-Drafted Tests for PySpark: The Model Writes, pytest Judges
Use Azure OpenAI structured outputs to draft edge-case test data for PySpark transforms, review it like code, and let pytest and assertDataFrameEqual decide.
7 min read
LLM-Drafted Data Quality Rules: Let the Model Propose, Not Enforce
Use Azure OpenAI structured outputs to draft Great Expectations rules from a data profile, then backtest and review them before they ever gate a pipeline.
7 min read
Generating dbt Models with Azure OpenAI: Let dbt Be the Judge
Use Azure OpenAI structured outputs to draft dbt models and YAML, then gate every draft with dbt build so nothing untested reaches review.
6 min read
Azure AI Foundry from Python: A Data Team's First Project
A practical first project in Azure AI Foundry for data teams: hubs and projects, the preview azure-ai-projects SDK, and an evaluation loop from day one.
7 min read
Reviewing Copilot Output in Fabric: A Workload-by-Workload Guide
Where Copilot in Microsoft Fabric stands in January 2025, what each workload's Copilot gets wrong, and how to review its output before it ships.
1 min read
Developer Experience for Data and AI Teams
Great developer experience is an investment that pays dividends in productivity, quality, and retention. Measure it and improve it continuously.
1 min read
DataOps Maturity: Building Reliable Data Operations
DataOps maturity directly impacts data reliability and team productivity. Invest in these practices to build a robust data operation.
1 min read
Microsoft Fabric at Ignite 2024: What's New for Data Professionals
The Fabric platform continues to mature into a comprehensive, AI-native analytics solution.
1 min read
Spark Optimization in Microsoft Fabric
1. Enable AQE : Adaptive Query Execution handles many optimizations automatically 2. Right size partitions : Target 128MB partitions 3. Broadcast small…
1 min read
Optimized Writes in Delta Lake
1. Enable by default : For most analytical workloads 2. Choose appropriate bin size : 128MB general, 256MB for Direct Lake 3. Combine with auto compact :…
1 min read
File Compaction Strategies in Delta Lake
1. Schedule regular compaction : Daily for high volume tables 2. Use partition aware compaction : Avoid rewriting entire large tables 3. Enable auto…
1 min read
Parquet File Optimization for Analytics
1. Target 128 256MB files : Optimal for most query engines 2. Use Snappy or ZSTD : Best compression/speed balance 3. Sort by filter columns : Improves…
1 min read
Building Data Products in Microsoft Fabric
Data products are the deliverables of a data mesh. Today I'm exploring how to build, manage, and consume data products in Microsoft Fabric.
1 min read
Copilot for Notebooks: AI-Assisted Data Engineering
Copilot for Notebooks transforms how data engineers write code. Today I'm exploring practical patterns for leveraging AI assistance in Fabric notebooks.
1 min read
SQL AI Functions in Databricks: LLMs Meet SQL
Databricks SQL AI functions bring large language model capabilities directly into SQL queries. Process text, generate insights, and enrich data without…
1 min read
Azure Databricks April 2024 Updates: AI and Analytics Convergence
Azure Databricks continues to evolve with powerful AI/BI features. April 2024 brings significant updates that blur the line between data engineering and…
1 min read
Automation in Fabric: Scheduling and Orchestration
Automation is key to maintaining reliable data pipelines. This guide covers scheduling, orchestration, and automation patterns in Microsoft Fabric.
1 min read
Fabric SDK: Python Development for Microsoft Fabric
The Microsoft Fabric SDK provides a Pythonic interface for working with Fabric services. This guide covers installation, configuration, and common…
2 min read
Microsoft Fabric Data Engineering: Best Practices After 6 Months in Production
Avoid over-partitioning. If partitions have < 1GB of data, consolidate. Use Data Pipelines, Not Just Notebooks
7 min read
Fabric's December 2023 Update: What I'd Act On in January
A practitioner's triage of the Microsoft Fabric December 2023 update: %%configure, Git conflict resolution, KQL in OneLake and what to leave alone.
1 min read
Fabric Architecture Patterns: Designing for Scale
Designing for scale in Fabric is about decomposition and clear contracts between layers. The medallion architecture works well: small, composable…
1 min read
Microsoft Fabric Best Practices for Production
Shipping Fabric into production exposes common anti-patterns — unbounded capacity use, duplicated data copies, and insufficient observability. The best…
4 min read
Microsoft Fabric is GA: What We Learned from Six Months of Preview
Microsoft Fabric reached General Availability at Ignite 2023 — November 15-17 in Seattle — and it's the outcome I genuinely thought might take another six…
1 min read
Microsoft Fabric Best Practices: Lessons Learned
Best practices and lessons learned from implementing Microsoft Fabric solutions for enterprise analytics.
1 min read
Microsoft Fabric Updates: Unified Analytics Platform
Exploring the latest Microsoft Fabric updates and how they transform enterprise data analytics.
1 min read
File Processing with AI: Automating Data Workflows
File processing — extracting structured data from PDFs, converting between formats, parsing inconsistent CSVs, pulling content from Excel sheets with…
1 min read
Data Wrangling in Microsoft Fabric: Practical Techniques
Data wrangling — getting messy, inconsistent, real-world data into a reliable clean form — is where most data engineering hours actually go. In Fabric…
1 min read
Fabric Notebooks: Features and Productivity Tips
Fabric Notebooks have some features I didn't notice until I'd been using them for a few weeks and wish someone had pointed out earlier. Inline data…
1 min read
Apache Spark in Microsoft Fabric: Getting Started
Fabric provides a managed Spark experience: Tomorrow we'll dive deeper into Fabric notebooks and their unique features. Spark in Fabric PySpark…
1 min read
Delta Tables in Microsoft Fabric: Deep Dive
The theoretical benefits of Delta Lake — ACID transactions, time travel, schema enforcement — are easy to describe, but the practical payoff becomes clear…
1 min read
Creating Your First Fabric Lakehouse: Step-by-Step Guide
A Lakehouse combines the best of data lakes and data warehouses: Once created, your Lakehouse has two main sections: The Files section is for unstructured…
1 min read
OneLake Explorer: Navigating Your Data Lake in Fabric
OneLake Explorer is a Windows desktop application that lets you browse the contents of your organisation's OneLake the same way you'd navigate files in…
2 min read
Getting Started with Microsoft Fabric: A Practical Guide
I've been onboarding several clients onto Fabric over the past few weeks and the first hour is always the same: getting oriented to an interface that's…
2 min read
Microsoft Fabric Public Preview: What You Need to Know
Six weeks into the Fabric public preview and I'm past the demo phase — I'm running actual data engineering workloads and discovering what holds up and what…
1 min read
Fabric Notebooks: Interactive Data Engineering and Science
Notebooks are the primary development environment in Fabric. Tomorrow, I will cover Fabric Pipelines for orchestration.
1 min read
Delta Lake in Microsoft Fabric: ACID Transactions for Data Lakes
Delta Lake isn't a bolt-on feature in Microsoft Fabric — it's the foundational table format for everything. Every table you write through a Spark notebook…
5 min read
Microsoft Fabric: The Biggest Data Launch Since SQL Server
At Microsoft Build 2023 on May 23, Satya Nadella announced Microsoft Fabric with the phrase "perhaps the biggest launch of a data product from Microsoft…
1 min read
Synapse Data Engineering in Microsoft Fabric: Spark at Scale
Synapse Data Engineering in Fabric provides a managed Spark experience that removes infrastructure concerns while delivering the full power of Apache Spark.…
1 min read
Building with Fabric Lakehouse: The Best of Data Lakes and Warehouses
The Lakehouse is where most of your Fabric data engineering work will happen. Tomorrow, I will cover Data Factory in Fabric for orchestrating data movement…
1 min read
Intelligent Document Processing Pipelines with AI
Intelligent document processing transforms unstructured documents into structured, actionable data. Combining OCR, layout analysis, and LLM understanding…
1 min read
Entity Extraction at Scale with LLMs
Entity extraction at scale transforms unstructured text into structured knowledge. Combining LLM intelligence with distributed processing enables insights…
1 min read
Anomaly Detection at Scale with AI
Anomaly detection at scale combines statistical rigor with AI understanding. Systems that both detect and explain anomalies enable faster, more confident…
1 min read
AI-Powered Data Quality: Intelligent Data Validation
AI-powered data quality goes beyond static rules. Intelligent systems understand context, detect subtle anomalies, and provide actionable recommendations…
1 min read
Spark ML Patterns for Production Systems
Spark ML provides battle-tested patterns for production machine learning. From feature engineering to model persistence, these patterns ensure reliable ML…
1 min read
Cognitive Services in Azure Synapse: Built-in AI Capabilities
Cognitive Services in Synapse brings AI capabilities directly into your data engineering workflows. Process millions of records with sentiment, entities…
1 min read
Azure Databricks + OpenAI: LLMs for Data Engineering
Azure Databricks with Azure OpenAI creates intelligent data platforms where natural language becomes the interface for data engineering tasks.
1 min read
Data Engineering Patterns That Defined 2022
2022 established these patterns as the foundation of modern data engineering. The lakehouse became the default architecture for new data platforms. Data…
1 min read
Data Platform Evolution in 2022: Trends and Technologies
The convergence of data lakes and data warehouses became mainstream: Streaming data processing became more accessible: Organizations adopted data mesh…
1 min read
Azure Synapse Analytics Updates: Ignite 2022 Highlights
Azure Synapse Analytics continues to mature as a comprehensive analytics platform. The improvements in serverless SQL, Spark integration, and data lake…
3 min read
Azure Databricks Unity Catalog: Unified Data Governance for the Lakehouse
Unity Catalog provides a unified governance layer across all your Databricks workspaces. Metastore: The top-level container for all metadata. You create one…
1 min read
Private Endpoints Everywhere: Securing Your Azure Data Platform
A private endpoint: Creates a network interface with a private IP in your VNet Maps to a specific Azure resource (or sub resource) Enables DNS resolution to…
2 min read
Azure Data Factory Managed Virtual Network: Secure Data Integration
Data Flows automatically use the managed VNet when connected to private endpoints.
1 min read
Materialized Views in Delta Live Tables
For large datasets, use streaming to incrementally update: Materialized views in Delta Live Tables provide: Precomputed results for fast query performance…
1 min read
Change Data Capture with Delta Live Tables
DLT's applychanges function processes these into clean, current-state tables.
1 min read
Data Quality with Delta Live Tables Expectations
The expression must evaluate to true for valid records.
1 min read
Delta Live Tables: Declarative Data Pipelines on Databricks
Traditional ETL: Delta Live Tables: DLT provides built in data quality enforcement: Handle CDC feeds with APPLY CHANGES: Delta Live Tables transforms data…
1 min read
Databricks Workflows: Orchestrating Data and ML Pipelines
Key features: Jobs : Multi task workflows with dependencies Triggers : Schedule, file arrival, or API based Compute : Job clusters or serverless Monitoring…
3 min read
Azure Data Factory Mapping Data Flows: ETL Patterns for the Modern Data Platform
Think of it as the best of both worlds: visual design for maintainability, Spark for scale.
1 min read
Azure Databricks Unity Catalog: Unified Data Governance at Scale
Unity Catalog solves these problems with a unified approach.
1 min read
Azure Synapse Analytics Updates: What's New in 2022
Azure Synapse Analytics in 2022 provides a more unified, performant, and feature-rich analytics platform.
1 min read
DataOps Practices: Agile for Data Teams
DataOps in 2021 matured from concept to standard practice. Teams that adopted these practices delivered faster and more reliably than those stuck in manual…
2 min read
Data Observability: Monitoring Your Data Pipelines
Data observability is the capability that answers "is my data healthy right now?" the same way application observability answers "is my application healthy…
1 min read
Data Quality Practices: Building Trust in Your Data
Data quality in 2021 became an engineering discipline. The tools matured, but success requires organizational commitment to treating data as a product.
1 min read
Data Lakehouse Patterns: Best of Both Worlds
The lakehouse architecture proved its value in 2021. Organizations are consolidating their data warehouses and data lakes into unified lakehouses, reducing…
1 min read
Data Mesh: From Concept to Practice
Data mesh isn't a technology choice - it's an organizational paradigm shift. 2021 was about understanding it; 2022 will be about implementing it.
1 min read
Data Engineering Trends That Defined 2021
Data engineering in 2021 matured from a support function to a strategic capability. The tools improved, patterns solidified, and the role gained the…
1 min read
Batch Inference at Scale with Azure ML Batch Endpoints
Batch endpoints enable cost-effective, scalable inference for large datasets without the complexity of managing infrastructure.
1 min read
Understanding Feature Stores for Machine Learning
Feature stores are becoming essential infrastructure for production ML systems. Understanding these concepts will help you build more reliable and…
1 min read
Managing Data with Azure ML Datasets
Proper data management with Azure ML Datasets is foundational to building reproducible, auditable machine learning pipelines.
3 min read
Microsoft Graph Data Connect: Bulk Access to Microsoft 365 Data
Graph Data Connect solves this by delivering bulk extracts of Microsoft 365 data directly to Azure Data Factory.
1 min read
PySpark Best Practices for Production Workloads
1. Define schemas explicitly Never rely on schema inference in production 2. Use built in functions Avoid UDFs when possible 3. Write pure transformation…
2 min read
Spark SQL Fundamentals and Best Practices
Spark SQL is where I've seen the highest productivity for data engineers who already know SQL but are new to distributed processing. You write familiar SQL…
2 min read
Synapse Pipelines vs Azure Data Factory - Making the Right Choice
The Synapse Pipelines vs. ADF question comes up in almost every Synapse project I start. The honest answer depends on where the rest of your data platform…
1 min read
Visual ETL with Azure Synapse Data Flows
Data flows in Synapse run on Spark clusters and offer: Visual drag and drop interface 80+ built in transformations Schema drift handling Data preview and…
2 min read
Big Data Processing with Azure Synapse Spark Pools
Synapse Spark pools are the feature that stops the "should I use Databricks or Synapse?" question from being purely a product choice and makes it an…
2 min read
Data Transformation with Azure Data Factory Mapping Data Flows
Mapping Data Flows are the feature that ended most of my "should I use PySpark or Spark SQL for this?" debates. Visual, yes—but reviewable in JSON and…
7 min read
Wiring a Medallion Pipeline on Azure Databricks with Auto Loader
How to connect Auto Loader, Structured Streaming and Delta MERGE into a bronze, silver and gold pipeline that restarts safely and doesn't run up the bill.
1 min read
Azure Databricks Workspace Management
Spinning up a Databricks workspace is the easy part. Making it production-ready is where I see the most avoidable rework. Cluster policies, AAD passthrough…
3 min read
Azure Synapse Analytics is Now Generally Available
All of this is unified through Synapse Studio - a single workspace where data engineers, data scientists, and analysts can collaborate.
1 min read
Azure Data Factory Data Flows: Visual ETL
I have a love-hate relationship with visual ETL tools. Drag-and-drop is wonderful right up until you need to diff two pipelines in source control. ADF Data…
2 min read
Getting Started with Delta Lake on Azure Databricks
Data lakes have a long-standing reputation problem: cheap to fill, painful to trust. Schema drift, half-written files from failed jobs, "is this row a…
1 min read
Databricks Delta Lake: Advanced Patterns
Delta Lake: the foundation of the modern lakehouse.
1 min read
Azure Databricks Jobs: Scheduled and Triggered Workflows
Notebooks are great until you need them to run on Tuesday at 2am, retry on failure, and chain into the next step. That's where Databricks Jobs come in.…
1 min read
Azure Data Factory Linked Services: Connect Everything
Best practice: use Managed Identity where supported. Linked Services are the foundation of Data Factory connectivity.
1 min read
Databricks Structured Streaming: Real-Time Data Processing
Incrementally process files as they arrive. Structured Streaming makes real-time processing accessible with familiar DataFrame semantics.
1 min read
Azure Synapse Spark Pools (Preview): Big Data Processing
Spark on Synapse is the same managed Spark story most cloud providers offer now, but it's done with the parts of Azure I already use — same workspace as the…
1 min read
Azure Data Lake Storage Gen2: Foundation of Modern Data Platforms
Gen2 is the standard for new data lakes on Azure. ADLS Gen2 is the foundation. Build your data platform on solid ground.
1 min read
Introduction to Delta Lake on Azure Databricks
Anyone who's run a "data lake" for any length of time has hit the same wall: parquet files everywhere, no transactional guarantees, partial-write disasters…
1 min read
Azure DevOps YAML Pipelines for Data Projects
The Azure DevOps Classic editor is comfortable, and I've watched many teams resist moving away from it for that exact reason. Until they need to branch a…
7 min read
Azure Data Factory: Designing a First Pipeline You Can Safely Rerun
How I build a first Azure Data Factory pipeline: parameterised by date, idempotent staging loads, no secrets in JSON, and UTC-only schedule triggers.