8 min read
Why an LLM Response Cache Moves Your p50 but Rarely Your p95
A response cache only speeds up the requests it hits, so the tail stays slow until hit rates get very high. Here's the arithmetic and what fixes p95 instead.
API Management
9 articles
8 min read
Four LLM Caches and How to Tell Which One Will Pay Off
Prompt, response, semantic and retrieval caches save different money and add different risks. Here's a break-even check to run before you build one.
8 min read
Agent Tool Policy at the Gateway: MCP Through API Management
Tool rules inside agent code drift once several teams build agents. Put identity, rate limits and tracing for MCP tools in Azure API Management instead.
9 min read
Azure OpenAI in Production, January 2025: Let the Platform Do the Work
Deployment types, keyless auth, SDK retries, APIM token limits and Global Batch: what to build on Azure OpenAI in January 2025, and what to stop building.
1 min read
Rate Limit Management for AI APIs
Tomorrow we'll explore token estimation techniques. Rate Limiting Patterns Circuit Breaker Pattern Azure OpenAI Best Practices
1 min read
Azure OpenAI Quotas: Understanding and Managing Limits
Tomorrow we'll explore rate limit management strategies. Azure OpenAI Quotas Quota Management Rate Limit Best Practices
7 min read
APIM Policies in Production: Limits, Fan-out and Fallbacks
Four Azure API Management policy patterns that hold up in production: identity-aware limits, Key Vault named values, parallel fan-out and stale fallbacks.
1 min read
Azure API Management: Policy Deep Dive
API Management is one of those services I underestimated for years. "It's a proxy, right?" The penny drops when you realise policies are the product. Rate…
1 min read
Azure API Management: API Gateway Patterns
Three years ago a client asked me to "put a gateway in front of my APIs." It turned into a six-month conversation about rate limiting, OAuth, partner…