Microsoft Fabric Real-Time Analytics: Streaming Data Pipelines

Microsoft Fabric Real-Time Analytics provides sub-second query performance on streaming data. By combining Eventhouse with KQL (Kusto Query Language), you can build powerful analytics pipelines that process millions of events per second.

Creating an Eventhouse

Eventhouse is the storage layer optimized for time-series and streaming data. It automatically indexes data for lightning-fast queries.

Ingesting Streaming Data

Use Eventstream to capture data from various sources including Azure Event Hubs, Kafka, and IoT Hub.

// Create a table for IoT sensor data
.create table SensorReadings (
    DeviceId: string,
    Temperature: real,
    Humidity: real,
    Pressure: real,
    Timestamp: datetime,
    Location: string
)

// Create mapping for JSON ingestion
.create table SensorReadings ingestion json mapping 'SensorMapping'
'['
'   {"column":"DeviceId", "path":"$.deviceId"},'
'   {"column":"Temperature", "path":"$.temperature"},'
'   {"column":"Humidity", "path":"$.humidity"},'
'   {"column":"Pressure", "path":"$.pressure"},'
'   {"column":"Timestamp", "path":"$.timestamp"},'
'   {"column":"Location", "path":"$.location"}'
']'

Real-Time Aggregations

KQL excels at time-windowed aggregations for monitoring dashboards.

// Calculate 5-minute averages per device
SensorReadings
| where Timestamp > ago(1h)
| summarize
    AvgTemperature = avg(Temperature),
    MaxTemperature = max(Temperature),
    MinTemperature = min(Temperature),
    ReadingCount = count()
    by DeviceId, bin(Timestamp, 5m)
| order by Timestamp desc

// Detect anomalies using statistical methods
SensorReadings
| where Timestamp > ago(24h)
| summarize
    AvgTemp = avg(Temperature),
    StdTemp = stdev(Temperature)
    by DeviceId
| join kind=inner (
    SensorReadings
    | where Timestamp > ago(1h)
) on DeviceId
| where Temperature > AvgTemp + (3 * StdTemp)
| project DeviceId, Temperature, AvgTemp, StdTemp, Timestamp

Materialized Views for Performance

Pre-compute expensive aggregations for dashboard queries.

.create materialized-view HourlyDeviceStats on table SensorReadings
{
    SensorReadings
    | summarize
        AvgTemperature = avg(Temperature),
        AvgHumidity = avg(Humidity),
        EventCount = count()
        by DeviceId, bin(Timestamp, 1h)
}

Real-Time Analytics in Fabric eliminates the complexity of managing separate streaming and batch systems, providing a unified platform for all your time-sensitive data needs.