7 min read
Schema Evolution: Surviving the Day Upstream Renames a Column
Every source schema changes eventually. Which changes Delta accepts silently, which it rejects, a contract per layer, and the changes no schema check sees.
Data Quality
28 articles
7 min read
Late-Arriving Data: Deciding When a Day Is Actually Done
Late data is a distribution you can measure, not an edge case. How to set a waiting policy, handle what arrives after it, and say when a number is final.
7 min read
Event Time vs Processing Time: Know Which Clock Your Numbers Are On
Every record has three timestamps: when it happened, when it arrived, and when it was processed. Group by the wrong one and past days keep changing.
6 min read
The Grain of a Table: Declare It Before the First Join
Grain is what one row of a table means. Most wrong numbers come from a join or aggregation that quietly changed it. How to declare grain and test it.
9 min read
Materialized Lake Views as a Silver Flow: What to Hand Over
Now that Fabric materialized lake views are GA: which raw-to-silver steps I'd move onto them, how to keep rejected rows, and where notebooks still win.
7 min read
Write-Audit-Publish in Fabric: Stop Bad Data Before It Ships
Why many data quality root-cause fixes come down to where the check runs, and how to build a write-audit-publish boundary on a Fabric lakehouse.
9 min read
Publish Only Verified Loads with Fabric Warehouse Snapshots
A write-audit-publish pattern for Fabric Data Warehouse: load, run SQL checks, then roll a warehouse snapshot forward so analysts only see verified data.
8 min read
Changing a Shared Fabric Table Without Breaking Its Notebooks
How to version a Fabric lakehouse table contract once several notebooks depend on it: change classes, a writer guard, pinned readers and safe renames.
9 min read
Blameless Reviews for Data Incidents: Ask Which Control Failed
How to run blameless reviews for data quality incidents: record who detected it, which control was missing, and turn each review into a tracked change.
7 min read
Failure Budgets for Data Products: SLOs for Fresh, Correct Tables
Measure data product reliability in good hours, catch bad data at the publish boundary, and let a failure budget decide when pipeline changes stop.
7 min read
Pruning Data Quality Checks: Keep, Demote, or Delete
How to cut an inherited data quality rule set down to the checks that change decisions, using failure history, ownership and an explicit retirement path.
8 min read
Break Your Silver Layer on Purpose: Bad-Input Tests in Fabric
Why I keep a library of deliberately broken raw files for every Fabric bronze-to-silver transform, and how to run them as plain PySpark tests before release.
8 min read
Two Tickets per Data Incident: Contain First, Fix the Cause Second
Why every data quality incident should produce two pieces of work, a containment action and a root-cause fix, and how to run both on a Fabric lakehouse.
7 min read
Five Data Quality Signals for the Data-to-AI Handoff
Why I'd track five quality signals per feature table at the data engineering to AI handoff, and how to compute them in a Fabric notebook without a new tool.
8 min read
OneLake Table Contracts: Owners, Schemas and Failure Modes Up Front
Why OneLake governance has to be designed before scale: table contracts, schema-enabled lakehouses, explicit owners and deliberate failure modes in Fabric.
9 min read
Table Contracts in a Fabric Lakehouse: Where Raw Becomes a Product
Four decisions for a messy Fabric raw zone: contracts at silver, Delta constraints, quarantined rows with failed batches, and lakehouse schemas.
8 min read
Running Data Pipelines in Production: Reruns, Late Data and Drift
Eight years of data pipelines taught me to plan for reruns, late data and schema drift. The habits that matter, with Delta MERGE and Fabric examples.
1 min read
Implementing Data Quality Checks in Microsoft Fabric Lakehouses
Poor data quality leads to incorrect business decisions, failed ML models, and eroded trust in analytics platforms. A proactive approach to data validation…
1 min read
Data Mesh: Federated Computational Governance
Computational governance scales where manual review cannot. Embed policies in CI/CD pipelines to catch issues before they reach production.
1 min read
Data Quality for AI: Ensuring Clean Data for ML and LLM Applications
Quality encompasses completeness, accuracy, consistency, timeliness, and validity. Each dimension requires specific checks.
1 min read
Data Quality Automation with AI: Beyond Rule-Based Validation
AI-powered data quality catches issues that traditional rules miss.
7 min read
LLM-Drafted Data Quality Rules: Let the Model Propose, Not Enforce
Use Azure OpenAI structured outputs to draft Great Expectations rules from a data profile, then backtest and review them before they ever gate a pipeline.
1 min read
Data Contracts: Agreements Between Data Producers and Consumers
Data contracts formalize the agreement between data producers and consumers. Today I'm exploring how to implement effective data contracts in your data…
1 min read
Fabric Endorsement: Certifying Trusted Data Products
Fabric endorsement — the Promoted and Certified badge system for Fabric items — is a lightweight but important trust signal in a self-service analytics…
1 min read
AI-Powered Data Quality: Intelligent Data Validation
AI-powered data quality goes beyond static rules. Intelligent systems understand context, detect subtle anomalies, and provide actionable recommendations…
1 min read
Data Quality with Delta Live Tables Expectations
The expression must evaluate to true for valid records.
2 min read
Data Observability: Monitoring Your Data Pipelines
Data observability is the capability that answers "is my data healthy right now?" the same way application observability answers "is my application healthy…
1 min read
Data Quality Practices: Building Trust in Your Data
Data quality in 2021 became an engineering discipline. The tools matured, but success requires organizational commitment to treating data as a product.