Microsoft Fabric Real-Time Analytics: Streaming Data Pipelines

I wrote “Microsoft Fabric Real-Time Analytics: Streaming Data Pipelines” to share practical, production-minded guidance on this topic.

Creating an Eventhouse

Eventhouse is the storage layer optimized for time-series and streaming data. It automatically indexes data for lightning-fast queries.

Ingesting Streaming Data

Use Eventstream to capture data from various sources including Azure Event Hubs, Kafka, and IoT Hub.

// Create a table for IoT sensor data
.create table SensorReadings (
    DeviceId: string,
    Temperature: real,
    Humidity: real,
    Pressure: real,
    Timestamp: datetime,
    Location: string
)

// Create mapping for JSON ingestion
.create table SensorReadings ingestion json mapping 'SensorMapping'
'['
'   {"column":"DeviceId", "path":"$.deviceId"},'
'   {"column":"Temperature", "path":"$.temperature"},'
'   {"column":"Humidity", "path":"$.humidity"},'
'   {"column":"Pressure", "path":"$.pressure"},'
'   {"column":"Timestamp", "path":"$.timestamp"},'
'   {"column":"Location", "path":"$.location"}'
']'

Real-Time Aggregations

KQL excels at time-windowed aggregations for monitoring dashboards.

// Calculate 5-minute averages per device
SensorReadings
| where Timestamp > ago(1h)
| summarize
    AvgTemperature = avg(Temperature),
    MaxTemperature = max(Temperature),
    MinTemperature = min(Temperature),
    ReadingCount = count()
    by DeviceId, bin(Timestamp, 5m)
| order by Timestamp desc

// Detect anomalies using statistical methods
SensorReadings
| where Timestamp > ago(24h)
| summarize
    AvgTemp = avg(Temperature),
    StdTemp = stdev(Temperature)
    by DeviceId
| join kind=inner (
    SensorReadings
    | where Timestamp > ago(1h)
) on DeviceId
| where Temperature > AvgTemp + (3 * StdTemp)
| project DeviceId, Temperature, AvgTemp, StdTemp, Timestamp

Materialized Views for Performance

Pre-compute expensive aggregations for dashboard queries.

.create materialized-view HourlyDeviceStats on table SensorReadings
{
    SensorReadings
    | summarize
        AvgTemperature = avg(Temperature),
        AvgHumidity = avg(Humidity),
        EventCount = count()
        by DeviceId, bin(Timestamp, 1h)
}

Real-Time Analytics in Fabric eliminates the complexity of managing separate streaming and batch systems, providing a unified platform for all your time-sensitive data needs.\n\n## Takeaways\n\nAdd a concise, personal takeaway and recommended next steps here.\n