7 min read
Claude Code's Monitor Tool vs Polling: What Actually Happened
I gave Opus 5.5 a build to watch and a dev server to start, with and without the Monitor tool. It used Monitor inconsistently, and headless runs hid the answer.
Testing
34 articles
6 min read
I Planted Five Bugs to Test Claude Code's Review Levels
I hid five bugs in a real change and ran /code-review at low, medium, high and xhigh. Medium found all five for 15 cents. Here's what each level adds.
7 min read
I Put a Guard on az in a Claude Code Mod, Then Tried to Get Past It
A Claude Code mod to hold destructive Azure CLI commands passed 12 unit tests and stopped nothing live. Here's why, the fix, and the gap no regex closes.
7 min read
Agent Fallbacks Are a Launch Requirement, Not an Ops Task
Decide what an agent does when a tool fails, a step runs out of budget or grounding is weak before launch, and make the release check fail without it.
7 min read
Output Contracts: Catch a Model Swap Before Downstream Systems Do
Structured outputs fix the shape of a model's answer, not its meaning: a two-layer contract between a Foundry model and the systems that consume it.
8 min read
Testing Agent Tool Policies: Prove the Limits Hold
A tool policy nobody tests is a comment. How to assert agent tool lists and approvals in CI, then gate tool use on a golden set of scenarios.
8 min read
Designing LLM Test Cases That Catch Hallucinations
Most eval sets only ask questions the documents can answer. Add unanswerable, false-premise and stale-context cases, each with an expected behaviour.
8 min read
Break Your Silver Layer on Purpose: Bad-Input Tests in Fabric
Why I keep a library of deliberately broken raw files for every Fabric bronze-to-silver transform, and how to run them as plain PySpark tests before release.
8 min read
LLM-as-a-Judge: Check the Judge Before You Trust the Score
An LLM judge has its own biases: how to write pass/fail criteria, measure its kappa and false-pass rate against human labels, and know when to skip it.
8 min read
Testing AI Systems: Which Tests Run Where, and Why
A layered test strategy for LLM apps: stubbed unit tests per commit, pass-rate gates on PRs, nightly evaluator runs and red teaming before release.
1 min read
Testing LLM Applications: Strategies Beyond Traditional Unit Tests
LLM outputs vary between runs and model versions. Testing must focus on behavioral properties rather than exact string matching, while still catching…
1 min read
Testing AI Applications: Strategies for LLM-Based Systems
Test AI components within the full application context, including error handling, timeout behavior, and integration with downstream systems.
1 min read
Evaluating RAG Systems: Metrics and Automated Testing with Azure AI
Fail builds when quality metrics drop below thresholds, catching regressions before they reach production.
1 min read
AI Testing Frameworks: Comprehensive Testing for AI Systems
Comprehensive AI testing ensures reliable behavior across diverse scenarios.
1 min read
Testing AI Applications: Strategies for Non-Deterministic Systems
Robust AI testing combines deterministic checks with AI-powered evaluation.
8 min read
LLM-Drafted Tests for PySpark: The Model Writes, pytest Judges
Use Azure OpenAI structured outputs to draft edge-case test data for PySpark transforms, review it like code, and let pytest and assertDataFrameEqual decide.
1 min read
UI Automation with AI: Beyond Traditional Scripting
AI-powered UI automation adapts to changes, understands context, and can recover from errors - capabilities that traditional automation simply cannot match.
1 min read
End-to-End RAG Evaluation: Complete System Assessment
Individual component metrics tell part of the story, but end-to-end evaluation measures how well your entire RAG pipeline performs as a system.
1 min read
Task-Specific Evaluation for LLMs: Beyond Generic Benchmarks
Generic benchmarks like MMLU and HumanEval don't predict performance on your specific use cases. This guide covers how to design and implement task-specific…
1 min read
Model Evaluation Frameworks: A Comprehensive Guide
Rigorous model evaluation is critical for production AI systems. This guide covers the major evaluation frameworks and how to implement comprehensive…
2 min read
Seed Parameter: Achieving Reproducible LLM Outputs
The seed parameter introduced in GPT-4 Turbo at DevDay 2023 is a useful addition for testing and debugging, but it's important to understand what…
5 min read
LLM Application Testing: Strategies and Frameworks
Testing LLM applications is a problem that doesn't have a satisfying general-purpose solution yet, and that's worth acknowledging before diving into…
1 min read
LLM Evaluation Metrics: Measuring What Matters
Understanding and implementing key metrics for evaluating LLM application performance and quality.
1 min read
Evaluation Frameworks for LLM Applications
Building comprehensive evaluation frameworks to measure and improve LLM application quality.
2 min read
TDD (Test-Driven Development) Overview with NextJS Example
This will create a new NextJS project with TypeScript in a directory named my-app. Change into the directory and run npm run dev to start the development…
1 min read
TDD (Test-Driven Development) Overview with Python Example
This cycle is repeated for each new piece of functionality. The result is a suite of tests that provide confidence in the code and make it easier to…
8 min read
Debugging an Azure Functions Blob Trigger Locally with Azurite
Run and debug an isolated .NET Azure Functions blob trigger against Azurite, and understand the receipts, retries and limits of local testing.
4 min read
Test-Driven Development in TypeScript: A Practical Guide
To get started with TDD in TypeScript, you will need to set up a TypeScript project and install the necessary dependencies.
2 min read
Test Driven Development in Rust
To get started with TDD in Rust, you'll need to have a few things set up. First, you'll need to have the Rust programming language installed on your system.…
1 min read
Synthetic Data Generation: When Real Data Isn't Available
Synthetic data in 2021 became practical for production use. The key is validating that synthetic data maintains the statistical properties needed for your…
1 min read
Azure Load Testing: Cloud-Native Performance Testing
Azure Load Testing makes performance validation accessible and integrated into modern DevOps workflows. By catching performance regressions early, you can…
1 min read
Azure Chaos Studio: Chaos Engineering for Resilient Applications
Chaos engineering is the practice of experimenting on a system to build confidence in its ability to withstand turbulent conditions. Netflix pioneered this…
1 min read
Azure DevOps Test Plans: Quality Assurance at Scale
Automated tests are the bulk of my testing pyramid, but there's a tier you can't replace: the human running through a workflow looking for the thing nobody…
1 min read
Load Testing Azure Applications with Apache JMeter
Deploy JMeter on Azure VMs for distributed testing: Run JMeter in containers for quick, scalable tests: Metric Description Target Response Time (avg)…