7 min read
Does Your LLM Eval Score Predict Anything the Business Cares About?
Before an eval score becomes a product metric, join its verdicts to real outcomes like escalations and re-asks, and check whether failing answers do worse.
LLMOps
16 articles
7 min read
Output Contracts: Catch a Model Swap Before Downstream Systems Do
Structured outputs fix the shape of a model's answer, not its meaning: a two-layer contract between a Foundry model and the systems that consume it.
8 min read
Turning Production Queries into an LLM Eval Set You Can Trust
How I sample real user queries by stratum, redact PII before anyone reads them, split off a holdout and keep the eval set current as the product changes.
8 min read
Canary a Model Swap in Foundry: Sticky Splits and Version Stamps
An offline evaluation can't tell you how a new model handles real traffic. How I'd canary a Foundry model change with sticky splits and per-call version logs.
8 min read
Foundry Agent Versions as Releases: Git, Evals and Publishing
How I'd run Microsoft Foundry agent changes like releases in April 2026: definitions in Git, versions from CI, evaluation gates and Agent Application promotion.
9 min read
Model Retirements in Foundry: Choose the Upgrade Policy on Purpose
Every Foundry model deployment has a version upgrade policy, and the default swaps your model at retirement. How to choose, audit and run model changes safely.
8 min read
Quality SLOs for LLM Features: An Error Budget for Bad Answers
Treat answer quality like availability: define a good answer, agree a target with the product owner, and let the error budget decide when changes stop.
8 min read
Monitoring AI in Production: Layers, Owners and a Review Rhythm
Run AI monitoring as an operating rhythm: four layers with owners, what pages versus what waits, and tagging every change so regressions are traceable.
8 min read
LLM Observability: The Few Signals That Actually Matter
Which LLM signals deserve a dashboard or a page: tokens by feature, latency split by phase, throttling, and sampled quality, using OpenTelemetry on Azure.
1 min read
LLMOps: Operationalizing Large Language Models in Production
LLMs present distinct operational challenges: non-deterministic outputs, prompt sensitivity, context window management, and the difficulty of defining…
1 min read
AI Observability with Azure Monitor and Application Insights
Monitor these essential AI metrics: latency (p50, p95, p99), token consumption per request, error rates by model and endpoint, cache hit rates for…
1 min read
LLMOps: Operationalizing Large Language Models
LLMOps ensures reliable, cost-effective LLM operations at scale.
1 min read
LLMOps Emergence: Operating Large Language Models in Production
LLMOps is essential for reliable LLM applications. Start with prompt management and evaluation, then add observability and cost tracking as you scale.
1 min read
Evaluation Improvements in Azure AI Studio
Without proper evaluation: Models may hallucinate without detection Quality degrades silently over time Compliance violations go unnoticed User experience…
1 min read
Prompt Flow Improvements: Building Production AI Pipelines
Prompt Flow has evolved significantly since its introduction. Today I'm exploring the latest improvements for building production-ready AI pipelines.
1 min read
Introduction to LLMOps: MLOps for Large Language Models
LLMOps brings discipline to LLM application development. Start with these foundations and iterate as your applications mature.