9 min read
Publish Only Verified Loads with Fabric Warehouse Snapshots
A write-audit-publish pattern for Fabric Data Warehouse: load, run SQL checks, then roll a warehouse snapshot forward so analysts only see verified data.
Blog
Thoughts on data engineering, cloud architecture, and technology. 2205 articles and counting.
Showing 24 of 2205 posts — page 3 of 92
9 min read
Fabric Pipeline Deadlines: Alert on a Late Run Before Anyone Asks
How to catch a slipping Fabric pipeline before its deadline: alert on missing and late runs, not just failures, using job events, KQL and the jobs API.
8 min read
Sensitivity Labels in OneLake: Decide Item Boundaries Before Scale
Fabric sensitivity labels apply per item, not per table, so item boundaries, defaults, inheritance and protection policies need designing before OneLake grows.
8 min read
Changing a Shared Fabric Table Without Breaking Its Notebooks
How to version a Fabric lakehouse table contract once several notebooks depend on it: change classes, a writer guard, pinned readers and safe renames.
8 min read
Splitting an AI Journey's Latency Budget Across Its Steps
A journey-level latency budget only works when each step gets a cap, a timeout and a fallback; here is how to split one across a RAG request.
9 min read
Blameless Reviews for Data Incidents: Ask Which Control Failed
How to run blameless reviews for data quality incidents: record who detected it, which control was missing, and turn each review into a tracked change.
8 min read
Where Agent State Breaks: Resume, Retries and Duplicate Runs
Typed state and checkpoints don't stop an agent paying twice: five places agent workflow state breaks at its boundaries, and patterns that hold.
9 min read
RAG Citations People Can Check: Treat Them as Part of the Answer
Good retrieval is wasted if users cannot verify the answer, so design RAG citations to be checkable, validated in code and measured like relevance.
8 min read
Turning Production Queries into an LLM Eval Set You Can Trust
How I sample real user queries by stratum, redact PII before anyone reads them, split off a holdout and keep the eval set current as the product changes.
8 min read
Canary a Model Swap in Foundry: Sticky Splits and Version Stamps
An offline evaluation can't tell you how a new model handles real traffic. How I'd canary a Foundry model change with sticky splits and per-call version logs.
9 min read
Pricing a Threshold: Tying Fabric Activator Rules to Business Impact
Set Fabric Activator and KQL alert thresholds from the cost of a miss versus the cost of acting, then tier actions by the value at risk, not the raw reading.
8 min read
Fabric Warehouse Tables or Lakehouse Shortcuts: Where Each Table Lives
A placement rule for Fabric: when data belongs in a Warehouse table you own, and when reading a lakehouse shortcut through the SQL analytics endpoint is enough.
8 min read
Readable Fabric Pipelines: Names, Boundaries and Config Off the Canvas
How to keep Fabric pipelines readable as they grow: naming before the first Git commit, child pipelines with clear contracts, and variable libraries for config.
8 min read
Feeding AI Teams Through OneLake Shortcuts Without Over-Sharing
A passthrough OneLake shortcut adds speed but no boundary: how to give data science teams curated Fabric features without opening the whole lakehouse.
8 min read
Ingestion vs Curation in Fabric: Where I Draw the Workspace Line
Why I split Fabric ingestion and curation at the landing table, when that means separate workspaces joined by shortcuts, and when one lakehouse is enough.
8 min read
Four LLM Caches and How to Tell Which One Will Pay Off
Prompt, response, semantic and retrieval caches save different money and add different risks. Here's a break-even check to run before you build one.
7 min read
Failure Budgets for Data Products: SLOs for Fresh, Correct Tables
Measure data product reliability in good hours, catch bad data at the publish boundary, and let a failure budget decide when pipeline changes stop.
8 min read
Agent Tool Policy at the Gateway: MCP Through API Management
Tool rules inside agent code drift once several teams build agents. Put identity, rate limits and tracing for MCP tools in Azure API Management instead.
8 min read
Hybrid Search for RAG: Test What the BM25 Leg Rescues
Hybrid search is default RAG advice, but the BM25 leg has real costs; a three-way test on Azure AI Search shows which questions it rescues.
7 min read
Raising Groundedness Without Guesswork: Label the Failures First
Once retrieval is ruled out, ungrounded answers fall into a few repeatable patterns: label them, fix one at a time and track completeness too.
8 min read
Foundry Agent Versions as Releases: Git, Evals and Publishing
How I'd run Microsoft Foundry agent changes like releases in April 2026: definitions in Git, versions from CI, evaluation gates and Agent Application promotion.
9 min read
Store Changes, Not Readings: Shaping Noisy Streams in an Eventhouse
Use update policies, materialized views and a KQL state-change function in a Fabric Eventhouse to strip duplicates, chatter and flapping before anyone queries.
8 min read
When Two Fabric Reports Disagree: A Metric Drift Runbook
A triage runbook for disputed numbers in Fabric: check definition, timing, filters and logic in order, using time travel, snapshots and Semantic Link.
8 min read
Fabric Pipeline Latency: Find Where the Minutes Actually Go
How to split a slow Fabric pipeline run into start delay, queueing, Spark start-up, copy transfer and idle gaps, with KQL and the activity runs API.