Azure Data Explorer: Fast Analytics on Log Data

Azure Data Explorer (ADX) is Microsoft’s secret weapon for log and telemetry analytics. It’s what powers Azure Monitor, Application Insights, and Microsoft Sentinel.

Why ADX?

• Sub-second queries on billions of rows
• Native time-series optimizations
• KQL query language (surprisingly intuitive)
• Built-in streaming ingestion

KQL Basics

// Find slow requests in the last hour
requests
| where timestamp > ago(1h)
| where duration > 5000  // milliseconds
| project timestamp, name, duration, resultCode
| order by duration desc
| take 100

// Aggregate by time buckets
requests
| where timestamp > ago(24h)
| summarize count(), avg(duration) by bin(timestamp, 1h)
| render timechart

// Join with exceptions
requests
| where timestamp > ago(1h)
| join kind=inner (
    exceptions
    | where timestamp > ago(1h)
) on operation_Id
| project timestamp, requestName=name, exceptionType=type, exceptionMessage=message

Ingestion Options

// Streaming ingestion via SDK
var kustoUri = "https://myadx.australiaeast.kusto.windows.net";
var ingestUri = "https://ingest-myadx.australiaeast.kusto.windows.net";

using var client = KustoIngestFactory.CreateStreamingIngestClient(kustoUri);
using var stream = new MemoryStream(Encoding.UTF8.GetBytes(jsonData));

await client.IngestFromStreamAsync(
    stream,
    new KustoIngestionProperties("database", "table")
    {
        Format = DataSourceFormat.json
    });

For log analytics at scale, ADX is hard to beat.