8 min read
Converting Direct Lake Tables to Import: Designing Hybrid Models
Power BI now converts single Direct Lake tables to Import in web modelling, making storage mode a per-table choice with real refresh and governance costs.
Blog
Thoughts on data engineering, cloud architecture, and technology. 2205 articles and counting.
Showing 24 of 2205 posts — page 2 of 92
8 min read
PostgreSQL Major Upgrades on Azure: Validate Early, Don't Pay to Wait
Pre-upgrade validation for Azure PostgreSQL Flexible Server is GA and Extended Support is now billed. An upgrade runbook and a version policy to avoid paying.
6 min read
The Grain of a Table: Declare It Before the First Join
Grain is what one row of a table means. Most wrong numbers come from a join or aggregation that quietly changed it. How to declare grain and test it.
9 min read
Claude's Server-Side Tools in Foundry: What to Let the Provider Run
Claude hosted on Azure in Foundry now runs web fetch, web search, tool search and an MCP connector for you. Decide which belong behind your own gateway.
8 min read
Idempotent Writes, and Why Retries Duplicate Data
Every pipeline gets retried eventually, and a non-idempotent write turns each retry into duplicates. Four patterns that make a write safe to repeat.
9 min read
Foundry Hosted Agents Are GA: When to Stop Running Your Own Containers
Hosted agents in Foundry Agent Service are GA. How they compare with Agent Framework on Container Apps or AKS on identity, networking, cold starts and tracing.
9 min read
Event-Driven Refresh for Materialized Lake Views: Schedules Still Win
Event-triggered refresh for Fabric materialized lake views trades predictable capacity and clean ordering for freshness. Where events fit and where they don't.
9 min read
Model Router Already Routes to Claude and Open Models: Govern the Pool
Foundry Model Router picks from OpenAI, Anthropic, DeepSeek, xAI and Meta models. Treat subsets, policy and model logging as governance, not tuning knobs.
9 min read
GPT-5.6 Sol, Terra or Luna: Pick the Tier per Job, Not per App
GPT-5.6 in Foundry has a 5x price spread across Sol, Terra and Luna. Choose a tier for each workflow step, back it with eval gates, then pick deployment types.
8 min read
Harness Agent or Plain Agent? What Agent Framework Now Decides for You
The Agent Framework Harness bundles planning, memory, compaction, approvals and telemetry. When to accept its opinions and when a plain Agent is better.
11 min read
Fabric Item CI/CD Is Mostly Solved. Your Tenant Isn't
Git integration and fabric-cicd ship Fabric items well. Workspaces, capacity, domains, roles and tenant settings still need their own source of truth.
9 min read
Auditing a Fabric Tenant on a Schedule with the Scanner and Admin APIs
Use a read-only service principal, the scanner APIs and the Fabric admin APIs to detect governance drift in a Fabric tenant, without handing it write access.
12 min read
Whose Identity Reaches the Target? OneLake Shortcut Security
How identity flows through OneLake shortcuts: passthrough vs delegated, SQL endpoint access modes, the Direct Lake over SQL exception, and lost policy.
8 min read
Synapse Serverless varchar(8000): Fixing Strings for Spark and Power BI
Why Synapse serverless SQL infers strings as varchar(8000), why Spark-created lake tables behave differently, and where Power BI string contracts belong.
8 min read
Rerun From Failed Activity in Fabric: Design for the Second Run
What a Fabric pipeline rerun actually runs again, why utcNow() and run IDs break recovery, and how to make rerun from failed activity safe.
8 min read
Findable by Design: Workspace Choices That Shape the OneLake Catalog
How workspace splits, item names, descriptions, endorsement and domains decide what people find in the OneLake catalog, plus a findability test you can script.
9 min read
Materialized Lake Views as a Silver Flow: What to Hand Over
Now that Fabric materialized lake views are GA: which raw-to-silver steps I'd move onto them, how to keep rejected rows, and where notebooks still win.
8 min read
Why an LLM Response Cache Moves Your p50 but Rarely Your p95
A response cache only speeds up the requests it hits, so the tail stays slow until hit rates get very high. Here's the arithmetic and what fixes p95 instead.
7 min read
Write-Audit-Publish in Fabric: Stop Bad Data Before It Ships
Why many data quality root-cause fixes come down to where the check runs, and how to build a write-audit-publish boundary on a Fabric lakehouse.
7 min read
Agent Fallbacks Are a Launch Requirement, Not an Ops Task
Decide what an agent does when a tool fails, a step runs out of budget or grounding is weak before launch, and make the release check fail without it.
8 min read
Hybrid Search in PostgreSQL: What It Costs to Own the Fusion
Azure AI Search and Cosmos DB fuse keyword and vector results for you; pgvector doesn't. What owning RRF in PostgreSQL involves, and when it pays.
7 min read
Does Your LLM Eval Score Predict Anything the Business Cares About?
Before an eval score becomes a product metric, join its verdicts to real outcomes like escalations and re-asks, and check whether failing answers do worse.
7 min read
Output Contracts: Catch a Model Swap Before Downstream Systems Do
Structured outputs fix the shape of a model's answer, not its meaning: a two-layer contract between a Foundry model and the systems that consume it.
8 min read
Real-Time Dashboards in Git: Reviewable Changes and a Way Back
Put Fabric Real-Time Dashboards under Git so tile and query changes are reviewed, validated before Update from Git, and reverted cleanly when they break.