2 min read
Job Clusters vs All-Purpose Clusters: Choosing the Right Approach
The job cluster versus all-purpose cluster decision in Databricks is primarily a cost decision: all-purpose clusters stay running between tasks (you pay for…
Blog
Thoughts on data engineering, cloud architecture, and technology. 2205 articles and counting.
Showing 24 of 2205 posts — page 73 of 92
2 min read
Databricks Cluster Policies for Governance and Cost Control
Databricks cluster policies are the governance layer that prevents the classic enterprise data platform problem: data scientists spinning up 64-node GPU…
2 min read
Azure Databricks Cluster Configuration Best Practices
Azure Databricks cluster configuration is where cost and performance trade-offs become very concrete: the wrong cluster type for a workload is either money…
2 min read
Workload Management in Azure Synapse
Workload management in Synapse Dedicated SQL Pool is the resource governance system that prevents low-priority ad-hoc queries from consuming all compute and…
2 min read
Result Set Caching for Query Acceleration in Synapse
Result set caching in Synapse Dedicated SQL Pool stores the output of a query in the dedicated pool's storage and returns the cached result for subsequent…
2 min read
High-Performance Data Loading with COPY Command in Synapse
The COPY command replaced PolyBase as the recommended data loading mechanism for Synapse Dedicated SQL Pool because it's simpler to use and performs at the…
2 min read
PolyBase for Data Virtualization in Azure Synapse
PolyBase is the T-SQL feature that makes Synapse Dedicated SQL Pool a data virtualisation engine, not just a data warehouse—it allows you to define external…
2 min read
Azure Synapse Dedicated SQL Pool Best Practices
Synapse Dedicated SQL Pool (formerly Azure SQL Data Warehouse) is the massively parallel processing (MPP) engine for structured, petabyte-scale analytical…
2 min read
Azure Synapse Serverless SQL Pools for Big Data Analytics
Synapse Serverless SQL Pool is the query engine that turned Azure Data Lake into an ad-hoc analytics platform—you write T-SQL, point it at files in ADLS…
3 min read
Microsoft Ignite 2021: Data, AI, and the Intelligent Cloud
Microsoft Ignite Fall 2021 brought a wave of announcements across Azure. While hybrid work and Microsoft Teams dominated the headlines, the data and AI…
2 min read
External Tables in Azure Data Explorer
External tables in ADX expose data in Azure Blob Storage or ADLS Gen2 as queryable tables without ingesting the data into ADX—useful when the data is too…
2 min read
Materialized Views in Azure Data Explorer
Materialized views in ADX pre-aggregate data according to a defined query, updating continuously as new data is ingested—so when you query the view, you're…
2 min read
Kusto Functions for Reusable Query Patterns
KQL functions in Azure Data Explorer are the reusability mechanism that prevents the same complex query logic from being copy-pasted across dashboards…
2 min read
ADX Continuous Export for Data Archival and Compliance
ADX Continuous Export is the managed pipeline that continuously moves data from ADX tables to Azure Blob Storage or ADLS Gen2—useful for archiving cold data…
2 min read
Azure Data Explorer Integration for Advanced Analytics
Azure Data Explorer is the analytics service that addresses a specific limitation of Log Analytics: KQL query performance on large historical datasets…
2 min read
Cross-Workspace Queries in Azure Monitor
Cross-workspace queries in KQL let you write a single query that spans multiple Log Analytics workspaces—essential when your monitoring architecture splits…
2 min read
Log Analytics Workspace Design for Enterprise Kubernetes
Log Analytics workspace design is one of those infrastructure decisions that feels low-stakes until your bill arrives or you hit a query that crosses a…
2 min read
Deep Dive into Azure Monitor for Containers
Azure Monitor for Containers (the Container Insights feature) is the native Azure observability solution for AKS that doesn't require running your own…
2 min read
Building Effective Grafana Dashboards for AKS
Grafana is the visualisation layer that makes Prometheus metrics interpretable at a glance—and for AKS, the starting point is the community dashboards that…
2 min read
Prometheus Metrics Collection in AKS
Prometheus became the observability standard for Kubernetes because its data model—time-series metrics with labels—maps naturally to the dynamic…
2 min read
Comprehensive AKS Monitoring with Container Insights
Container Insights is the Azure Monitor feature that closes the observability gap for AKS clusters—without it, you have Kubernetes metrics available in the…
2 min read
Azure AD Workload Identity for AKS: The Future of Pod Identity
Azure AD Workload Identity is the architecture-level improvement over AAD Pod Identity that removes the NMI DaemonSet and uses Kubernetes native service…
1 min read
Secure Azure Access with AKS Pod Identity
AAD Pod Identity was the original mechanism for giving Kubernetes pods an Azure AD identity so they could access Azure resources—Key Vault secrets, Storage…
2 min read
Serverless Kubernetes with AKS Virtual Nodes
Virtual nodes in AKS enable pods to run on Azure Container Instances rather than on VM-based node pool nodes—useful for burst workloads that spike…