7 min read
Schema Evolution: Surviving the Day Upstream Renames a Column
Every source schema changes eventually. Which changes Delta accepts silently, which it rejects, a contract per layer, and the changes no schema check sees.
Blog
Thoughts on data engineering, cloud architecture, and technology. 2205 articles and counting.
Showing 24 of 2205 posts — page 1 of 92
7 min read
Slowly Changing Dimensions Without Corrupting Your Facts
Should last year's revenue move when a customer changes segment? Pick an SCD type per column, write a replay-safe Type 2 MERGE, and join facts correctly.
8 min read
APPROXIMATEDISTINCTCOUNT in Direct Lake: When 1.6% Error Is Fine
APPROXIMATEDISTINCTCOUNT now runs on Import and Direct Lake models in preview. Decide per measure where about 1.6% error is fine, label it, and test the gain.
9 min read
Content Understanding vs Document Intelligence: Choose by Shape
Document Intelligence wins on fixed layouts, Content Understanding on variable or multimodal input: split the pipeline and leave stable extractors alone.
6 min read
The Small Files Problem: How Tables End Up With Two Million Files
Small files come from simple arithmetic, and every reader pays a cost per file. Where they come from, how to measure them, and how to compact them safely.
9 min read
Fabric's Data Engineering Agent: Set Guardrails Before Autonomy
Fabric's preview data engineering agent runs with the launching engineer's permissions. What to delegate first, which settings to pick, and what stays human.
8 min read
Apache Ossie and the Portable Semantic Layer: What It Fixes and What It Doesn't
Microsoft backed Apache Ossie at FabCon with a Power BI converter. It makes metric names portable, not DAX semantics, security rules or performance.
7 min read
BigQuery Mirroring Is GA: Zero-ETL Multicloud Has a Bill Too
Fabric mirroring for BigQuery is GA, but it moves cost to Google: CDC compute, egress, keyless reseeds and schema ownership decide when to mirror or copy.
9 min read
Fabric IQ Goes GA in Copilot: Your Semantic Model Is Now an Agent API
Fabric IQ is GA in Copilot Chat, Cowork and MCP. Treat each semantic model as an agent endpoint: RLS, OLS, metadata contracts and default-on access.
8 min read
DiskANN in Azure SQL Is GA: Do You Still Need a Separate Vector Store?
DiskANN vector indexes are now GA in Azure SQL and Fabric SQL. When embeddings belong next to your rows, and when Azure AI Search still wins.
7 min read
Partition Pruning: The Fastest Data Is the Data You Never Read
A query for one day should not read three years. How partition pruning works, the five ways it silently breaks, and how to pick partition columns.
9 min read
Foundry Routines Are GA: Scheduled Agents Without Building a Scheduler
Foundry routines run an agent on a timer, a schedule or a GitHub or Teams event. When to use them, when to use a workflow, and guardrails for unattended runs.
9 min read
Egress Policy Belongs on the Agent, Not in Its Code
Foundry's preview egress controls attach ordered destination rules to a hosted agent. Why that limits prompt-injection damage, and how to roll it out safely.
8 min read
GPT-6 Landed in Foundry: Upgrade Discipline Beats Upgrade Speed
GPT-6 arrived in Foundry ten weeks after GPT-5.6. Pin versions, gate on real queries, shadow traffic and plan retirements before you switch tiers on day one.
7 min read
Reading a Spark Query Plan: Ten Minutes That Explain a Two-Hour Job
The physical plan shows what Spark will actually do: shuffles, join strategy, pruning, Python UDFs. How to read one, and which plans should worry you.
8 min read
Workspace Identity for Eventstream Sources: Retire the Shared Access Key
Eventstream can now read Azure Event Hubs with workspace identity. How to move off SAS keys, the RBAC each side needs, and when a service principal still wins.
7 min read
Data Skew: Why 199 Tasks Finish in Seconds and One Takes an Hour
One hot key can make one task do a third of a job, and more nodes will not help. How to spot skew in the Spark UI, find the keys, and fix it properly.
8 min read
Release Plans Are Gone: Tracking Power Platform Change in 2026
Power Platform and Dynamics 365 release plans moved to the AI at Work roadmap. Why a CoE now needs continuous change intake, not two planning events a year.
8 min read
GPU-Accelerated Fabric Warehouse: How to Read the 7x Claim
Fabric Data Warehouse GPU query acceleration is in preview. Why the 7x benchmark won't predict your bill, and how to run a fair A/B test on your own queries.
7 min read
Late-Arriving Data: Deciding When a Day Is Actually Done
Late data is a distribution you can measure, not an edge case. How to set a waiting policy, handle what arrives after it, and say when a number is final.
8 min read
Fabric Data Agents Move to the Responses API: What Consumers Must Change
Fabric data agents now use GPT-5.1 and a Responses API client. What notebook, Foundry and app callers must change, and why you need a full eval re-run.
7 min read
PBIR Is Now the Default: What Changes for Report Source Control
Developer mode and PBIP are GA and PBIR becomes the default report format. Why that makes report review realistic, what breaks, and how to stage the move.
8 min read
Fabric Runtime 2.0 Is GA but Not Default: An Upgrade Plan
Fabric Runtime 2.0 is GA and set to become the default in late September. A plan to inventory, test libraries and Delta features, and pin runtimes first.
7 min read
Event Time vs Processing Time: Know Which Clock Your Numbers Are On
Every record has three timestamps: when it happened, when it arrived, and when it was processed. Group by the wrong one and past days keep changing.