8 min read
OneLake Shortcuts Are Dependencies: Inventory Them Before Scale
Treat every OneLake shortcut as a cross-team dependency: placement, identity, write paths, environment variables and a REST inventory of what points where.
Blog
Thoughts on data engineering, cloud architecture, and technology. 2205 articles and counting.
Showing 24 of 2205 posts — page 4 of 92
8 min read
When a Slow Fabric Lakehouse Is Really a Raw Zone Problem
Why slow Fabric silver loads often trace back to the raw zone: landing layout, header checks, small files and bronze table maintenance, with PySpark examples.
7 min read
Latency Budgets as a Cost Control: Pricing Each AI User Journey
Set a latency budget per user journey, then let it choose the Azure OpenAI deployment tier, model and reasoning effort you actually pay for.
7 min read
Pruning Data Quality Checks: Keep, Demote, or Delete
How to cut an inherited data quality rule set down to the checks that change decisions, using failure history, ownership and an explicit retirement path.
8 min read
Testing Agent Tool Policies: Prove the Limits Hold
A tool policy nobody tests is a comment. How to assert agent tool lists and approvals in CI, then gate tool use on a golden set of scenarios.
8 min read
Treat Retrieval Changes Like Schema Changes: A RAG Regression Gate
Make retrieval-first a team rule: an owned question set, a side-by-side index comparison on Azure AI Search, and a gate that fails on lost answers.
8 min read
Designing LLM Test Cases That Catch Hallucinations
Most eval sets only ask questions the documents can answer. Add unanswerable, false-premise and stale-context cases, each with an expected behaviour.
9 min read
Model Retirements in Foundry: Choose the Upgrade Policy on Purpose
Every Foundry model deployment has a version upgrade policy, and the default swaps your model at retirement. How to choose, audit and run model changes safely.
7 min read
Noise or a Real Change? Three KQL Checks Before Anyone Gets Paged
Use KQL baselines, week-over-week comparison and diffpatterns in an Eventhouse to tell a meaningful operational change from ordinary noise, and to explain it.
8 min read
One Metric, One Home: Ownership Rules That Stop Drift in Fabric
How to stop metric drift in Fabric by giving every metric one home layer and one named owner, then enforcing both rules with schemas, grants and versions.
8 min read
Fabric Pipeline Failure Paths: Catch the Error, Then Fail the Run
How dependency conditions decide a Fabric pipeline's final status, why try-catch hides failures, and how to log, alert and still fail the run honestly.
8 min read
Keeping OneLake Clean: Design the Workspace Lifecycle Before Scale
How to stop Fabric workspace and item sprawl before it starts: lifecycle states, domains, tags, endorsement, retention settings and a weekly hygiene audit.
8 min read
Break Your Silver Layer on Purpose: Bad-Input Tests in Fabric
Why I keep a library of deliberately broken raw files for every Fabric bronze-to-silver transform, and how to run them as plain PySpark tests before release.
8 min read
Where LLM Latency Goes, and Which Token Cuts Actually Help
Input and output tokens cost and delay answers differently, so split latency into first-token and generation time before trimming anything.
8 min read
Two Tickets per Data Incident: Contain First, Fix the Cause Second
Why every data quality incident should produce two pieces of work, a containment action and a root-cause fix, and how to run both on a Fabric lakehouse.
6 min read
Explicit State for Agent Workflows: Stop Making the Model Remember
Agents get confused when workflow state lives in the chat transcript. Four patterns for typed state, fresh context and code-owned transitions.
8 min read
Is Your Reranker Earning Its Place? Test It Before You Pay for It
The semantic ranker only reorders the top 50 results. A with-and-without test on a labelled set shows which RAG queries it rescues and which it just slows.
8 min read
Quality SLOs for LLM Features: An Error Budget for Bad Answers
Treat answer quality like availability: define a good answer, agree a target with the product owner, and let the error budget decide when changes stop.
9 min read
Foundry After the March GA: What's Governed and What's Still Preview
Foundry Agent Service went GA in mid-March 2026, but monitoring and tracing are still preview. What's governed now, and the controls I'd add first.
8 min read
Alerts People Don't Mute: Designing Activator Rules in Fabric
How to design Fabric Activator and KQL queryset alerts that fire on transitions, carry an owner and a next step, and stop teams muting the channel.
8 min read
SQL in Fabric: Give Analysts a View Layer, Not Your Tables
How I design an analyst-facing schema in Fabric Data Warehouse: views as the contract, permissions scoped to it, and SQL checks that catch breaks first.
8 min read
Fabric Data Movement: Design for the Half-Finished Copy
How to design Fabric Copy jobs and copy activities so a failed, partial or repeated copy never leaves duplicates, gaps or a watermark that moved too far.
9 min read
Fast Access Without Eroding OneLake Boundaries in Fabric
Why adding people as Fabric workspace Contributors is the access shortcut that erodes OneLake boundaries, and a faster pattern that keeps them intact.
9 min read
Who Gets Told? Owners, Error Codes and Runbooks in Fabric
A table contract is half the job: assigning owners per layer, collapsing load outcomes to three, and mapping Fabric pipeline failures to runbooks.