2 min read
Evaluating LLM Outputs: Beyond Vibes
Production AI needs real evaluation. Here's how I approach it. You test with 5 prompts. They look good. Ship it.
Testing
25 articles
1 min read
Testing AI Systems: What Actually Works
Traditional testing assumes deterministic behavior. AI systems are probabilistic. Same input, different output.
1 min read
Testing LLM Applications: Strategies Beyond Traditional Unit Tests
LLM outputs vary between runs and model versions. Testing must focus on behavioral properties rather than exact string matching, while still catching…
1 min read
Testing AI Applications: Strategies for LLM-Based Systems
Test AI components within the full application context, including error handling, timeout behavior, and integration with downstream systems.
1 min read
Evaluating RAG Systems: Metrics and Automated Testing with Azure AI
Fail builds when quality metrics drop below thresholds, catching regressions before they reach production.
1 min read
AI Testing Frameworks: Comprehensive Testing for AI Systems
Comprehensive AI testing ensures reliable behavior across diverse scenarios.
1 min read
Testing AI Applications: Strategies for Non-Deterministic Systems
Robust AI testing combines deterministic checks with AI-powered evaluation.
1 min read
Testing Data Pipelines with AI: From Unit Tests to Integration Testing
AI-assisted testing catches more bugs earlier. Combine generated tests with manual review to ensure comprehensive coverage.
1 min read
UI Automation with AI: Beyond Traditional Scripting
AI-powered UI automation adapts to changes, understands context, and can recover from errors - capabilities that traditional automation simply cannot match.
1 min read
End-to-End RAG Evaluation: Complete System Assessment
Individual component metrics tell part of the story, but end-to-end evaluation measures how well your entire RAG pipeline performs as a system.
1 min read
Task-Specific Evaluation for LLMs: Beyond Generic Benchmarks
Generic benchmarks like MMLU and HumanEval don't predict performance on your specific use cases. This guide covers how to design and implement task-specific…
1 min read
Model Evaluation Frameworks: A Comprehensive Guide
Rigorous model evaluation is critical for production AI systems. This guide covers the major evaluation frameworks and how to implement comprehensive…
2 min read
Seed Parameter: Achieving Reproducible LLM Outputs
The seed parameter introduced in GPT-4 Turbo at DevDay 2023 is a useful addition for testing and debugging, but it's important to understand what…
5 min read
LLM Application Testing: Strategies and Frameworks
Testing LLM applications is a problem that doesn't have a satisfying general-purpose solution yet, and that's worth acknowledging before diving into…
1 min read
LLM Evaluation Metrics: Measuring What Matters
Understanding and implementing key metrics for evaluating LLM application performance and quality.
1 min read
Evaluation Frameworks for LLM Applications
Building comprehensive evaluation frameworks to measure and improve LLM application quality.
2 min read
TDD (Test-Driven Development) Overview with NextJS Example
This will create a new NextJS project with TypeScript in a directory named my-app. Change into the directory and run npm run dev to start the development…
1 min read
TDD (Test-Driven Development) Overview with Python Example
This cycle is repeated for each new piece of functionality. The result is a suite of tests that provide confidence in the code and make it easier to…
4 min read
Test-Driven Development in TypeScript: A Practical Guide
To get started with TDD in TypeScript, you will need to set up a TypeScript project and install the necessary dependencies.
2 min read
Test Driven Development in Rust
To get started with TDD in Rust, you'll need to have a few things set up. First, you'll need to have the Rust programming language installed on your system.…
1 min read
Synthetic Data Generation: When Real Data Isn't Available
Synthetic data in 2021 became practical for production use. The key is validating that synthetic data maintains the statistical properties needed for your…
1 min read
Azure Load Testing: Cloud-Native Performance Testing
Azure Load Testing makes performance validation accessible and integrated into modern DevOps workflows. By catching performance regressions early, you can…
1 min read
Azure Chaos Studio: Chaos Engineering for Resilient Applications
Chaos engineering is the practice of experimenting on a system to build confidence in its ability to withstand turbulent conditions. Netflix pioneered this…
1 min read
Azure DevOps Test Plans: Quality Assurance at Scale
Automated tests are the bulk of my testing pyramid, but there's a tier you can't replace: the human running through a workflow looking for the thing nobody…
1 min read
Load Testing Azure Applications with Apache JMeter
Deploy JMeter on Azure VMs for distributed testing: Run JMeter in containers for quick, scalable tests: Metric Description Target Response Time (avg)…