21 min read
Fabric Data Agent Accuracy Is a Distribution, Not a Number
I asked a Fabric data agent 24 questions in three wordings, ten times each, on real AEMO data. It scored 82%, but only 39% of wordings were always right.
Evaluation
44 articles
7 min read
What xhigh Effort Bought Me on Short Tasks: Tokens, Not Answers
ultrathink now runs at xhigh effort in Claude Code. I ran six exact-answer tasks 96 times from low to max: every level got them right; higher just cost more.
6 min read
I Planted Five Bugs to Test Claude Code's Review Levels
I hid five bugs in a real change and ran /code-review at low, medium, high and xhigh. Medium found all five for 15 cents. Here's what each level adds.
7 min read
Haiku 5.5 vs Opus 5.5 on Real Repo Chores in Claude Code
I gave Haiku 5.5 and Opus 5.5 the same five chores in this blog's repo. Both passed every run. Haiku cost 25 times less, and the diffs show what Opus adds.
8 min read
GPT-6 Landed in Foundry: Upgrade Discipline Beats Upgrade Speed
GPT-6 arrived in Foundry ten weeks after GPT-5.6. Pin versions, gate on real queries, shadow traffic and plan retirements before you switch tiers on day one.
8 min read
Fabric Data Agents Move to the Responses API: What Consumers Must Change
Fabric data agents now use GPT-5.1 and a Responses API client. What notebook, Foundry and app callers must change, and why you need a full eval re-run.
9 min read
Model Router Already Routes to Claude and Open Models: Govern the Pool
Foundry Model Router picks from OpenAI, Anthropic, DeepSeek, xAI and Meta models. Treat subsets, policy and model logging as governance, not tuning knobs.
9 min read
GPT-5.6 Sol, Terra or Luna: Pick the Tier per Job, Not per App
GPT-5.6 in Foundry has a 5x price spread across Sol, Terra and Luna. Choose a tier for each workflow step, back it with eval gates, then pick deployment types.
7 min read
Does Your LLM Eval Score Predict Anything the Business Cares About?
Before an eval score becomes a product metric, join its verdicts to real outcomes like escalations and re-asks, and check whether failing answers do worse.
9 min read
RAG Citations People Can Check: Treat Them as Part of the Answer
Good retrieval is wasted if users cannot verify the answer, so design RAG citations to be checkable, validated in code and measured like relevance.
8 min read
Turning Production Queries into an LLM Eval Set You Can Trust
How I sample real user queries by stratum, redact PII before anyone reads them, split off a holdout and keep the eval set current as the product changes.
8 min read
Hybrid Search for RAG: Test What the BM25 Leg Rescues
Hybrid search is default RAG advice, but the BM25 leg has real costs; a three-way test on Azure AI Search shows which questions it rescues.
7 min read
Raising Groundedness Without Guesswork: Label the Failures First
Once retrieval is ruled out, ungrounded answers fall into a few repeatable patterns: label them, fix one at a time and track completeness too.
8 min read
Testing Agent Tool Policies: Prove the Limits Hold
A tool policy nobody tests is a comment. How to assert agent tool lists and approvals in CI, then gate tool use on a golden set of scenarios.
8 min read
Treat Retrieval Changes Like Schema Changes: A RAG Regression Gate
Make retrieval-first a team rule: an owned question set, a side-by-side index comparison on Azure AI Search, and a gate that fails on lost answers.
8 min read
Designing LLM Test Cases That Catch Hallucinations
Most eval sets only ask questions the documents can answer. Add unanswerable, false-premise and stale-context cases, each with an expected behaviour.
8 min read
Is Your Reranker Earning Its Place? Test It Before You Pay for It
The semantic ranker only reorders the top 50 results. A with-and-without test on a labelled set shows which RAG queries it rescues and which it just slows.
8 min read
Quality SLOs for LLM Features: An Error Budget for Bad Answers
Treat answer quality like availability: define a good answer, agree a target with the product owner, and let the error budget decide when changes stop.
7 min read
A Token Waste Audit: Cut What the Answer Doesn't Use
Profile each LLM request by segment, rank the cuts by quality risk, and keep only the token savings that survive a paired evaluation against the baseline.
7 min read
Wrong RAG Answer? Rule Out Retrieval Before You Edit the Prompt
Most wrong RAG answers start with the passages, not the prompt. A labelled question set and a retrieval-only triage on Azure AI Search show which one to fix.
8 min read
Gating LLM Changes on Groundedness Flips, Not Fluency Averages
How I compare a baseline and a candidate prompt or model on frozen context, count groundedness regressions per row, and treat fluency as a tie-breaker.
8 min read
LLM-as-a-Judge: Check the Judge Before You Trust the Score
An LLM judge has its own biases: how to write pass/fail criteria, measure its kappa and false-pass rate against human labels, and know when to skip it.
8 min read
Testing AI Systems: Which Tests Run Where, and Why
A layered test strategy for LLM apps: stubbed unit tests per commit, pass-rate gates on PRs, nightly evaluator runs and red teaming before release.
7 min read
Eight Prompt Engineering Myths to Drop in 2026
Eight prompt engineering myths worth dropping in 2026, from verbose prompts to reflexive chain-of-thought and temperature rules, and what to measure instead.
1 min read
Continuous Evaluation: Monitoring AI Quality in Production
Continuous evaluation maintains AI quality standards throughout the system lifecycle.
1 min read
Evaluation Metrics for AI: Measuring What Matters
Regular evaluation with consistent metrics drives continuous improvement in AI quality.
6 min read
Azure AI Foundry from Python: A Data Team's First Project
A practical first project in Azure AI Foundry for data teams: hubs and projects, the preview azure-ai-projects SDK, and an evaluation loop from day one.
8 min read
AI Safety Research in 2025: What Builders Should Act On Now
Late-2024 safety research on scheming, alignment faking and prompt injection, translated into concrete controls for Azure teams shipping AI in 2025.
1 min read
Weights and Biases for LLM Applications: Complete Guide
Weights & Biases provides a comprehensive platform for LLM observability, evaluation, and collaboration. Its strength lies in combining experiment tracking…
4 min read
Quality-Based LLM Routing: Maximizing Output Quality
Quality isn't one dimensional. Consider: Accuracy : Factual correctness Completeness : Covering all aspects Coherence : Logical flow and consistency…
1 min read
End-to-End RAG Evaluation: Complete System Assessment
Individual component metrics tell part of the story, but end-to-end evaluation measures how well your entire RAG pipeline performs as a system.
1 min read
Context Recall in RAG: Are You Finding All Relevant Documents?
While context precision measures noise in retrieved results, context recall measures completeness. Are you retrieving all the documents needed to fully…
1 min read
Context Precision in RAG: Evaluating Retrieval Quality
Context precision measures whether the retrieved documents are actually relevant to answering the question. High precision means less noise for the…
1 min read
Answer Relevancy in RAG: Does the Response Address the Question?
An answer can be factually correct and grounded in context but still fail to address what was actually asked. Answer relevancy measures how well the…
1 min read
Faithfulness Scoring in RAG: Detecting Hallucinations
Faithfulness is perhaps the most critical metric for RAG systems. An unfaithful answer that hallucinates information not in the source documents can be…
1 min read
Generation Metrics for RAG: Measuring Answer Quality
While retrieval metrics measure what documents are found, generation metrics evaluate the quality of the synthesized answer. This guide covers metrics…
1 min read
Retrieval Metrics for RAG: A Deep Dive
The retrieval component of RAG systems directly impacts generation quality. This guide provides a comprehensive overview of retrieval metrics and how to…
1 min read
RAGAS Framework: Automated RAG Evaluation
RAGAS (Retrieval Augmented Generation Assessment) is an open-source framework for evaluating RAG pipelines. This guide covers how to implement and use RAGAS…
1 min read
RAG Evaluation: Measuring Retrieval-Augmented Generation Quality
Retrieval-Augmented Generation (RAG) systems combine retrieval and generation components, each requiring specific evaluation strategies. This guide covers…
1 min read
Task-Specific Evaluation for LLMs: Beyond Generic Benchmarks
Generic benchmarks like MMLU and HumanEval don't predict performance on your specific use cases. This guide covers how to design and implement task-specific…
1 min read
MMLU Scores Explained: What They Really Mean
MMLU (Massive Multitask Language Understanding) is one of the most cited LLM benchmarks. Understanding what it measures and its limitations is crucial for…
1 min read
LLM Benchmarking: Understanding Model Performance
Understanding LLM benchmarks is essential for making informed model selection decisions. This guide covers the major benchmarks and how to interpret their…
1 min read
Model Evaluation Frameworks: A Comprehensive Guide
Rigorous model evaluation is critical for production AI systems. This guide covers the major evaluation frameworks and how to implement comprehensive…
1 min read
LLM Benchmarking: Evaluating Models for Your Use Case
LLM benchmarking methodology matters more than the benchmark scores themselves, because the standard public benchmarks (MMLU for general knowledge…