8 min read
Why an LLM Response Cache Moves Your p50 but Rarely Your p95
A response cache only speeds up the requests it hits, so the tail stays slow until hit rates get very high. Here's the arithmetic and what fixes p95 instead.
Caching
20 articles
8 min read
Four LLM Caches and How to Tell Which One Will Pay Off
Prompt, response, semantic and retrieval caches save different money and add different risks. Here's a break-even check to run before you build one.
1 min read
Semantic Caching for LLM Applications: Reducing Costs and Latency
Traditional caching requires exact matches. Users asking "What is Azure?" and "Can you explain Azure?" would generate two separate API calls. Semantic…
1 min read
Semantic Caching for LLM Applications: Reducing Costs and Latency
Unlike exact-match caching, semantic caching uses embeddings to find similar queries even when worded differently.
1 min read
Implementing Semantic Caching for LLM Applications
Use Redis Vector Search for efficient similarity matching at scale. Tune the similarity threshold based on your use case - higher values ensure more precise…
1 min read
Caching Strategies: Building Modern AI Systems
Understanding caching strategies is essential for production AI systems. Here's what you need to know.
1 min read
Prompt Caching Strategies: Reducing Latency and Cost
Smart caching can reduce AI costs by 30-50% for applications with repetitive queries.
1 min read
Caching Strategies for LLM Cost Reduction
1. Start with exact match Simple and effective 2. Add semantic caching For variable phrasing 3. Set appropriate TTL Balance freshness and savings 4. Monitor…
1 min read
Context Caching for LLM Applications
Tomorrow we'll explore conversation summarization techniques. Redis Caching Semantic Search LLM Caching Patterns
1 min read
Caching Strategies for LLM Applications
Effective caching significantly reduces LLM costs and latency. Tomorrow, I will cover response streaming patterns.
1 min read
LLM Caching Strategies: Reducing Costs and Latency
LLM caching strategies are essential for production systems. By combining exact matching, semantic similarity, and intelligent invalidation, you can…
8 min read
Redis Caching Patterns in .NET 7 Isolated Azure Functions
Cache-aside, write-through, write-behind and refresh-ahead with Azure Cache for Redis in .NET 7 isolated Azure Functions, and when each one is the wrong choice.
1 min read
Output Caching in ASP.NET Core 7: A Complete Guide
Output caching stores the complete HTTP response and serves it directly for subsequent identical requests, bypassing the entire request pipeline. Unlike…
1 min read
GitHub Actions Caching Strategies
Proper caching can reduce build times by 50-80% for dependency-heavy projects.
2 min read
Result Set Caching for Query Acceleration in Synapse
Result set caching in Synapse Dedicated SQL Pool stores the output of a query in the dedicated pool's storage and returns the cached result for subsequent…
1 min read
Mastering Azure CDN Caching Rules and Optimization
CDN caching rules are the configuration that determines how long cached content stays at the edge before the CDN checks with the origin for updates—and…
1 min read
Azure Cache for Redis: High-Performance Caching Strategies
Azure Cache for Redis is the caching layer I add to almost every production Azure application that has read-heavy patterns with data that doesn't change on…
2 min read
Cosmos DB Integrated Cache: Reducing RU Costs at Scale
The Cosmos DB integrated cache is the feature I've been waiting for since the first time I watched a read-heavy application burn through its RU budget…
1 min read
Azure Cache for Redis: Caching Patterns
Redis caching transforms application performance at scale.
1 min read
Azure Cache for Redis Enterprise: High Performance Caching
Redis Enterprise: when standard caching isn't enough.