7 min read
Late-Arriving Data: Deciding When a Day Is Actually Done
Late data is a distribution you can measure, not an edge case. How to set a waiting policy, handle what arrives after it, and say when a number is final.
Streaming
31 articles
7 min read
Event Time vs Processing Time: Know Which Clock Your Numbers Are On
Every record has three timestamps: when it happened, when it arrived, and when it was processed. Group by the wrong one and past days keep changing.
1 min read
Real-Time Streaming Analytics with Azure Stream Analytics and Event Hubs
Enrich streaming data with reference data lookups for context-aware processing and alerting in real-time scenarios.
1 min read
Building TypeScript AI Applications with Vercel AI SDK
The SDK supports function calling with automatic TypeScript type inference, making it easy to extend your AI with custom capabilities.
1 min read
Real-Time Data Processing with Microsoft Fabric Eventstreams
Eventstreams support multiple source types including Azure Event Hubs, Azure IoT Hub, and custom applications. The visual designer makes it easy to…
1 min read
Streaming LLM Responses: Patterns for Real-Time User Experience
Add a blinking cursor effect while streaming to indicate ongoing generation. Users perceive streaming responses as 3-5x faster than equivalent non-streaming…
3 min read
Microsoft Fabric Real-Time Intelligence: Patterns for Streaming Analytics
Microsoft Fabric's Real-Time Intelligence workload has matured significantly since GA. We've implemented several production streaming scenarios, and I want…
1 min read
Streaming Inference: Building Modern AI Systems
Understanding streaming inference is essential for production AI systems. Here's what you need to know.
1 min read
Streaming Inference Patterns: Processing Data as It Flows
Streaming inference brings AI insights to real-time data. Design for throughput, latency, and reliability from the start.
1 min read
Real-Time AI Patterns: Building Low-Latency Intelligent Systems
Real-time AI requires discipline around latency. Design for the worst case and optimize for the common case.
8 min read
LLMs on an Event Hubs Stream: Tiered Triage with Azure Functions
How to put Azure OpenAI inside an Event Hubs pipeline without stalling partitions: batch triggers, a cheap triage tier, structured outputs, fail-closed errors.
7 min read
Where Streaming Logic Belongs in Fabric: Eventstream, KQL or Spark
A decision guide for placing filters, enrichment, aggregation and alerts across Eventstream, Eventhouse and Spark in Fabric Real-Time Intelligence.
1 min read
Streaming Machine Learning: Training and Inference on Real-Time Data
Streaming ML enables intelligent real-time systems that continuously learn and adapt. Start with robust feature engineering and monitoring before enabling…
1 min read
Fabric Eventstream Enhancements: Real-Time Data at Scale
Eventstream enhancements make real-time analytics more powerful and accessible. Start with simple streaming scenarios and progressively add complexity.
8 min read
Fabric Eventstream Patterns: Deciding Where Stream Logic Lives
Practical Fabric Eventstream patterns: what belongs in the no-code event processor, what belongs in KQL, and what to fix before events arrive.
6 min read
Fabric Real-Time Analytics After GA: An End-to-End Streaming Design
How Eventstream, KQL databases, Power BI and the Data Activator preview fit together in Fabric as of January 2024, and when to choose something else.
1 min read
Eventstreams in Fabric: Streaming Data Ingestion
Tomorrow we'll explore Real Time Dashboards for visualizing streaming data. Eventstream Documentation Event Hub Integration Streaming Best Practices
1 min read
Fabric Real-Time Analytics: Stream Processing at Scale
'{' ' "SoftDeletePeriod": "30.00:00:00",' ' "Recoverability": "Enabled"' '}'
1 min read
Response Streaming: Real-Time LLM Output
Streaming improves the user experience significantly. Tomorrow, I will cover error handling for AI applications.
1 min read
Synapse Real-Time Analytics in Microsoft Fabric: Streaming at Scale
Real-Time Analytics in Fabric provides powerful streaming capabilities for time-sensitive workloads. Tomorrow, I will cover Power BI integration in Fabric.
1 min read
Streaming Responses with Azure OpenAI
Streaming transforms AI applications from feeling sluggish to feeling responsive. The implementation adds complexity, but the UX improvement is substantial.…
7 min read
Streaming Azure OpenAI Completions from the API to the Browser
How stream=True works on the Azure OpenAI 2022-12-01 Completions API, what you give up by streaming, and how to relay tokens to a browser without losing any.
1 min read
Working with IoT Hub Built-in Endpoints
The built-in endpoint provides a reliable, scalable way to consume IoT device telemetry for real-time processing.
1 min read
Real-Time Analytics on Azure: Building Live Dashboards
Azure's real time analytics stack enables instant insights from millions of events per second, powering live dashboards and alerts.
1 min read
Streaming Tables in Delta Live Tables
DLT streaming tables use append mode by default. For aggregations: DLT handles checkpointing automatically: Streaming tables in Delta Live Tables provide:…
1 min read
Using Azure Event Hubs with Kafka Protocol
Event Hubs with Kafka protocol offers: No Kafka cluster management Automatic scaling 99.99% SLA Native Azure integration Pay per throughput pricing Event…
1 min read
Real-Time Data Processing with Spark Structured Streaming
The appeal of Spark Structured Streaming is that you write it almost identically to a batch Spark job. Same DataFrame API, same transformations, same Spark…
7 min read
Wiring a Medallion Pipeline on Azure Databricks with Auto Loader
How to connect Auto Loader, Structured Streaming and Delta MERGE into a bronze, silver and gold pipeline that restarts safely and doesn't run up the bill.
1 min read
Azure Media Services: Video Streaming Platform
Azure Media Services: broadcast-quality video for everyone.
1 min read
Databricks Structured Streaming: Real-Time Data Processing
Incrementally process files as they arrive. Structured Streaming makes real-time processing accessible with familiar DataFrame semantics.
1 min read
Azure Event Hubs with Kafka API
The Kafka API compatibility makes Event Hubs a drop-in replacement for many streaming scenarios.