Latency

Prompt Caching: Reducing Latency and Cost for Repetitive Contexts

Many applications send the same system prompt and context repeatedly: 1. Identify common prefixes System prompts, few shot examples 2. Batch similar…