115 articles
Synapse serverless SQL infers long text columns as varchar(8000), and you can't fix it from PySpark. Here's why it happens — and how to fix it properly across Spark, SQL, and Power BI.
The data engineering profession evolved dramatically in 2025. As the year ends, here's my retrospective on the skills, tools, and mindsets that defined…
These Fabric updates demonstrate Microsoft's commitment to building a truly unified data platform that handles real-time and batch workloads while…
Batch processing introduces latency between data generation and insight availability. For fraud detection, operational monitoring, and customer engagement…
Without a feature store, teams often duplicate feature engineering work across projects. Features computed for training may differ from inference, causing…
Every write operation to a Delta table creates a new version. This transaction log enables point-in-time queries and rollback capabilities essential for…
Poor data quality leads to incorrect business decisions, failed ML models, and eroded trust in analytics platforms. A proactive approach to data validation…
Many organizations struggle with the gap between experimentation and production. Models that work in notebooks often fail in real-world scenarios due to…
Use Fabric notebooks to create interactive RAG applications that combine data exploration with AI-powered question answering over enterprise datasets.
Bronze holds raw ingested data. Silver contains cleansed and conformed data. Gold presents business-level aggregates ready for consumption.
Eventhouse is the storage layer optimized for time-series and streaming data. It automatically indexes data for lightning-fast queries.
Data Factory pipelines use a visual designer with activities that can be chained together.
Bronze tables store data exactly as received from source systems, preserving the original format for replayability and debugging.
Quality encompasses completeness, accuracy, consistency, timeliness, and validity. Each dimension requires specific checks.
Eventstreams support multiple source types including Azure Event Hubs, Azure IoT Hub, and custom applications. The visual designer makes it easy to…
Iceberg provides ACID transactions, schema evolution, time travel, and partition evolution for data lakes. Its format-agnostic design works with Spark…
VACUUM removes data files no longer referenced by the Delta log. Without regular vacuuming, your storage costs grow unbounded as old file versions accumulate.
A Lakehouse stores data in open formats like Delta Lake while providing SQL query capabilities, ACID transactions, and schema enforcement. In Microsoft…
Use Delta Lake constraints and expectations to ensure data quality at each layer. Fabric's Data Quality monitoring provides visibility into quality metrics…
Microsoft Fabric's Real-Time Intelligence workload has matured significantly since GA. Today I'm exploring how to build streaming analytics pipelines that…
Microsoft Fabric's Real-Time Intelligence workload has matured significantly since GA. We've implemented several production streaming scenarios, and I want…
AI-enhanced pipelines handle complex transformations and quality issues automatically.
This works on F2 and above now. For small businesses, this transforms how reports are built.
AI supercharges dbt workflows with intelligent generation and optimization.
AI-powered pipelines transform raw data into intelligent, enriched datasets. Design for both batch and streaming scenarios.
Streaming inference brings AI insights to real-time data. Design for throughput, latency, and reliability from the start.
AI transforms data pipelines from rigid rule-based systems to adaptive, intelligent processes. Start with high-value, error-prone steps and expand from there.
Documentation with AI: Automating Technical Writing for Data Projects
AI-assisted testing catches more bugs earlier. Combine generated tests with manual review to ensure comprehensive coverage.
AI transforms data quality from reactive checking to proactive management. Start with profiling and anomaly detection, then expand to automated rule…
AI accelerates dbt development but doesn't replace data engineering expertise. Use it to handle boilerplate and documentation while focusing your expertise…
Great developer experience is an investment that pays dividends in productivity, quality, and retention. Measure it and improve it continuously.
DataOps maturity directly impacts data reliability and team productivity. Invest in these practices to build a robust data operation.
The Fabric platform continues to mature into a comprehensive, AI-native analytics solution.
1. Enable AQE : Adaptive Query Execution handles many optimizations automatically 2. Right size partitions : Target 128MB partitions 3. Broadcast small…
1. Enable by default : For most analytical workloads 2. Choose appropriate bin size : 128MB general, 256MB for Direct Lake 3. Combine with auto compact :…
1. Schedule regular compaction : Daily for high volume tables 2. Use partition aware compaction : Avoid rewriting entire large tables 3. Enable auto…
1. Target 128 256MB files : Optimal for most query engines 2. Use Snappy or ZSTD : Best compression/speed balance 3. Sort by filter columns : Improves…
Data products are the deliverables of a data mesh. Today I'm exploring how to build, manage, and consume data products in Microsoft Fabric.
Copilot for Notebooks transforms how data engineers write code. Today I'm exploring practical patterns for leveraging AI assistance in Fabric notebooks.
Databricks SQL AI functions bring large language model capabilities directly into SQL queries. Process text, generate insights, and enrich data without…
Azure Databricks continues to evolve with powerful AI/BI features. April 2024 brings significant updates that blur the line between data engineering and…
Automation is key to maintaining reliable data pipelines. This guide covers scheduling, orchestration, and automation patterns in Microsoft Fabric.
The Microsoft Fabric SDK provides a Pythonic interface for working with Fabric services. This guide covers installation, configuration, and common…
Avoid over-partitioning. If partitions have < 1GB of data, consolidate. Use Data Pipelines, Not Just Notebooks
Designing for scale in Fabric is about decomposition and clear contracts between layers. The medallion architecture works well: small, composable…
Shipping Fabric into production exposes common anti-patterns — unbounded capacity use, duplicated data copies, and insufficient observability. The best…
Microsoft Fabric reached General Availability at Ignite 2023 — November 15-17 in Seattle — and it's the outcome I genuinely thought might take another six…
Best practices and lessons learned from implementing Microsoft Fabric solutions for enterprise analytics.
Exploring the latest Microsoft Fabric updates and how they transform enterprise data analytics.
File processing — extracting structured data from PDFs, converting between formats, parsing inconsistent CSVs, pulling content from Excel sheets with…
Data wrangling — getting messy, inconsistent, real-world data into a reliable clean form — is where most data engineering hours actually go. In Fabric…
Fabric Notebooks have some features I didn't notice until I'd been using them for a few weeks and wish someone had pointed out earlier. Inline data…
Fabric provides a managed Spark experience: Tomorrow we'll dive deeper into Fabric notebooks and their unique features. Spark in Fabric PySpark…
The theoretical benefits of Delta Lake — ACID transactions, time travel, schema enforcement — are easy to describe, but the practical payoff becomes clear…
A Lakehouse combines the best of data lakes and data warehouses: Once created, your Lakehouse has two main sections: The Files section is for unstructured…
OneLake Explorer is a Windows desktop application that lets you browse the contents of your organisation's OneLake the same way you'd navigate files in…
I've been onboarding several clients onto Fabric over the past few weeks and the first hour is always the same: getting oriented to an interface that's…
Six weeks into the Fabric public preview and I'm past the demo phase — I'm running actual data engineering workloads and discovering what holds up and what…
Notebooks are the primary development environment in Fabric. Tomorrow, I will cover Fabric Pipelines for orchestration.
Delta Lake isn't a bolt-on feature in Microsoft Fabric — it's the foundational table format for everything. Every table you write through a Spark notebook…
At Microsoft Build 2023 on May 23, Satya Nadella announced Microsoft Fabric with the phrase "perhaps the biggest launch of a data product from Microsoft…
Synapse Data Engineering in Fabric provides a managed Spark experience that removes infrastructure concerns while delivering the full power of Apache Spark.…
The Lakehouse is where most of your Fabric data engineering work will happen. Tomorrow, I will cover Data Factory in Fabric for orchestrating data movement…
Intelligent document processing transforms unstructured documents into structured, actionable data. Combining OCR, layout analysis, and LLM understanding…
Entity extraction at scale transforms unstructured text into structured knowledge. Combining LLM intelligence with distributed processing enables insights…
Anomaly detection at scale combines statistical rigor with AI understanding. Systems that both detect and explain anomalies enable faster, more confident…
AI-powered data quality goes beyond static rules. Intelligent systems understand context, detect subtle anomalies, and provide actionable recommendations…
Spark ML provides battle-tested patterns for production machine learning. From feature engineering to model persistence, these patterns ensure reliable ML…
Cognitive Services in Synapse brings AI capabilities directly into your data engineering workflows. Process millions of records with sentiment, entities…
Azure Databricks with Azure OpenAI creates intelligent data platforms where natural language becomes the interface for data engineering tasks.
2022 established these patterns as the foundation of modern data engineering. The lakehouse became the default architecture for new data platforms. Data…
The convergence of data lakes and data warehouses became mainstream: Streaming data processing became more accessible: Organizations adopted data mesh…
Azure Synapse Analytics continues to mature as a comprehensive analytics platform. The improvements in serverless SQL, Spark integration, and data lake…
Unity Catalog provides a unified governance layer across all your Databricks workspaces. Metastore: The top-level container for all metadata. You create one…
A private endpoint: Creates a network interface with a private IP in your VNet Maps to a specific Azure resource (or sub resource) Enables DNS resolution to…
Data Flows automatically use the managed VNet when connected to private endpoints.
For large datasets, use streaming to incrementally update: Materialized views in Delta Live Tables provide: Precomputed results for fast query performance…
DLT's applychanges function processes these into clean, current-state tables.
The expression must evaluate to true for valid records.
Traditional ETL: Delta Live Tables: DLT provides built in data quality enforcement: Handle CDC feeds with APPLY CHANGES: Delta Live Tables transforms data…
Key features: Jobs : Multi task workflows with dependencies Triggers : Schedule, file arrival, or API based Compute : Job clusters or serverless Monitoring…
Think of it as the best of both worlds: visual design for maintainability, Spark for scale.
Unity Catalog solves these problems with a unified approach.
Azure Synapse Analytics in 2022 provides a more unified, performant, and feature-rich analytics platform.
DataOps in 2021 matured from concept to standard practice. Teams that adopted these practices delivered faster and more reliably than those stuck in manual…
Data observability is the capability that answers "is my data healthy right now?" the same way application observability answers "is my application healthy…
Data quality in 2021 became an engineering discipline. The tools matured, but success requires organizational commitment to treating data as a product.
The lakehouse architecture proved its value in 2021. Organizations are consolidating their data warehouses and data lakes into unified lakehouses, reducing…
Data mesh isn't a technology choice - it's an organizational paradigm shift. 2021 was about understanding it; 2022 will be about implementing it.
Data engineering in 2021 matured from a support function to a strategic capability. The tools improved, patterns solidified, and the role gained the…
Batch endpoints enable cost-effective, scalable inference for large datasets without the complexity of managing infrastructure.
Feature stores are becoming essential infrastructure for production ML systems. Understanding these concepts will help you build more reliable and…
Proper data management with Azure ML Datasets is foundational to building reproducible, auditable machine learning pipelines.
Graph Data Connect solves this by delivering bulk extracts of Microsoft 365 data directly to Azure Data Factory.
1. Define schemas explicitly Never rely on schema inference in production 2. Use built in functions Avoid UDFs when possible 3. Write pure transformation…
Spark SQL is where I've seen the highest productivity for data engineers who already know SQL but are new to distributed processing. You write familiar SQL…
The Synapse Pipelines vs. ADF question comes up in almost every Synapse project I start. The honest answer depends on where the rest of your data platform…
Data flows in Synapse run on Spark clusters and offer: Visual drag and drop interface 80+ built in transformations Schema drift handling Data preview and…
Synapse Spark pools are the feature that stops the "should I use Databricks or Synapse?" question from being purely a product choice and makes it an…
Mapping Data Flows are the feature that ended most of my "should I use PySpark or Spark SQL for this?" debates. Visual, yes—but reviewable in JSON and…
Delta Lake has gone from "interesting open source project" to the default storage layer for data lakes inside about eighteen months. ACID, schema…
Spinning up a Databricks workspace is the easy part. Making it production-ready is where I see the most avoidable rework. Cluster policies, AAD passthrough…
All of this is unified through Synapse Studio - a single workspace where data engineers, data scientists, and analysts can collaborate.
I have a love-hate relationship with visual ETL tools. Drag-and-drop is wonderful right up until you need to diff two pipelines in source control. ADF Data…
Data lakes have a long-standing reputation problem: cheap to fill, painful to trust. Schema drift, half-written files from failed jobs, "is this row a…
Delta Lake: the foundation of the modern lakehouse.
Notebooks are great until you need them to run on Tuesday at 2am, retry on failure, and chain into the next step. That's where Databricks Jobs come in.…
Best practice: use Managed Identity where supported. Linked Services are the foundation of Data Factory connectivity.
Incrementally process files as they arrive. Structured Streaming makes real-time processing accessible with familiar DataFrame semantics.
Spark on Synapse is the same managed Spark story most cloud providers offer now, but it's done with the parts of Azure I already use — same workspace as the…
Gen2 is the standard for new data lakes on Azure. ADLS Gen2 is the foundation. Build your data platform on solid ground.
Anyone who's run a "data lake" for any length of time has hit the same wall: parquet files everywhere, no transactional guarantees, partial-write disasters…
The Azure DevOps Classic editor is comfortable, and I've watched many teams resist moving away from it for that exact reason. Until they need to branch a…
A surprising number of "modernise my reporting" engagements I see start the same way: data scattered across an on-prem SQL Server, a SaaS CRM, three Excel…