7 min read
Reading a Spark Query Plan: Ten Minutes That Explain a Two-Hour Job
The physical plan shows what Spark will actually do: shuffles, join strategy, pruning, Python UDFs. How to read one, and which plans should worry you.
Databricks
64 articles
9 min read
Leaving ADF and Databricks for Fabric: What Held Up
What consolidating Azure Data Factory, Databricks, Azure SQL and Power BI onto Microsoft Fabric delivered in production, and where the seams still show.
9 min read
Databricks or Fabric in 2026: Choosing Your Primary Platform
A practical decision guide for picking Azure Databricks or Microsoft Fabric as your primary data platform in 2026, and when running both is worth the cost.
1 min read
Databricks AI: Unified Data and AI Platform
Databricks unifies data, ML, and AI on a single lakehouse platform.
1 min read
Microsoft Fabric and Databricks: Coexistence Strategies
1. Use shared storage : ADLS as the common data layer 2. Standardize on Delta Lake : Compatible format for both 3. Optimize for readers : Both platforms…
1 min read
Foundation Model APIs in Databricks: Enterprise LLM Access
Databricks Foundation Model APIs provide enterprise-ready access to state-of-the-art LLMs. This guide covers using these APIs for building AI applications.
1 min read
Serverless Model Serving: Pay-Per-Request ML Inference
Serverless model serving eliminates infrastructure management while providing cost-effective, scalable ML inference. This guide covers implementing…
1 min read
Databricks Model Serving Updates: April 2024
Databricks Model Serving continues to evolve with new features for deploying and scaling ML models. This guide covers the latest updates and best practices.
1 min read
Online Feature Serving in Databricks: Real-Time ML Features
Online feature serving enables real-time ML inference by providing low-latency access to features. This guide covers setting up and using Databricks online…
1 min read
Feature Engineering in Databricks: From Raw Data to ML Features
Feature engineering transforms raw data into meaningful inputs for machine learning models. Databricks provides powerful tools for building and managing…
1 min read
Unity Catalog for ML: Governed Machine Learning
Unity Catalog extends governance to machine learning assets. Manage models, features, and experiments with the same rigor as your data.
1 min read
Databricks Vector Search Deep Dive: Production Patterns
Taking Vector Search to production requires careful consideration of performance, reliability, and maintenance. This guide covers production-ready patterns.
1 min read
Vector Search in Databricks: Semantic Search at Scale
Databricks Vector Search enables semantic similarity search over your lakehouse data. Build RAG applications, recommendation systems, and intelligent search…
1 min read
The ai_query Function: Custom LLM Calls in Databricks SQL
The aiquery() function enables custom LLM interactions directly in SQL. Unlike specialized functions, it allows you to craft any prompt and get intelligent…
1 min read
SQL AI Functions in Databricks: LLMs Meet SQL
Databricks SQL AI functions bring large language model capabilities directly into SQL queries. Process text, generate insights, and enrich data without…
1 min read
Dashboards in Databricks: Building Data Applications
Databricks dashboards (including Lakeview) provide powerful visualization capabilities integrated with the lakehouse. This guide covers building effective…
1 min read
Databricks Genie Spaces: Natural Language Data Exploration
Genie Spaces enable non-technical users to explore data using natural language. This guide covers setting up and optimizing Genie Spaces for your organization.
1 min read
Databricks AI/BI: Intelligent Analytics Platform
Databricks AI/BI combines the power of the lakehouse with AI-driven analytics. This guide explores how to leverage AI/BI for intelligent data analysis.
1 min read
Azure Databricks April 2024 Updates: AI and Analytics Convergence
Azure Databricks continues to evolve with powerful AI/BI features. April 2024 brings significant updates that blur the line between data engineering and…
3 min read
AI Assistants in Jupyter and Databricks Notebooks
Error: {type(error).name}: {str(error)} response = openai.ChatCompletion.create( engine="gpt-4", messages=[{"role": "user", "content": prompt}] ) return…
1 min read
Azure Databricks + OpenAI: LLMs for Data Engineering
Azure Databricks with Azure OpenAI creates intelligent data platforms where natural language becomes the interface for data engineering tasks.
3 min read
Azure Databricks Unity Catalog: Unified Data Governance for the Lakehouse
Unity Catalog provides a unified governance layer across all your Databricks workspaces. Metastore: The top-level container for all metadata. You create one…
1 min read
Materialized Views in Delta Live Tables
For large datasets, use streaming to incrementally update: Materialized views in Delta Live Tables provide: Precomputed results for fast query performance…
1 min read
Streaming Tables in Delta Live Tables
DLT streaming tables use append mode by default. For aggregations: DLT handles checkpointing automatically: Streaming tables in Delta Live Tables provide:…
1 min read
Change Data Capture with Delta Live Tables
DLT's applychanges function processes these into clean, current-state tables.
1 min read
Data Quality with Delta Live Tables Expectations
The expression must evaluate to true for valid records.
1 min read
Delta Live Tables: Declarative Data Pipelines on Databricks
Traditional ETL: Delta Live Tables: DLT provides built in data quality enforcement: Handle CDC feeds with APPLY CHANGES: Delta Live Tables transforms data…
1 min read
Databricks Workflows: Orchestrating Data and ML Pipelines
Key features: Jobs : Multi task workflows with dependencies Triggers : Schedule, file arrival, or API based Compute : Job clusters or serverless Monitoring…
1 min read
AutoML in Databricks: Automated Machine Learning Made Simple
Databricks AutoML automatically: Prepares and preprocesses data Engineers features Selects algorithms Tunes hyperparameters Evaluates and compares models…
1 min read
MLflow Model Registry: Versioning and Deploying ML Models
The registry organizes models with: Registered Models : Named model artifacts Model Versions : Specific iterations of a model Stages : Lifecycle states…
1 min read
Databricks Model Serving: Real-Time ML Inference at Scale
Key features: Serverless : No infrastructure management Auto scaling : Handles variable traffic automatically Low latency : Sub second response times…
1 min read
Databricks Feature Store: Centralized Feature Management for ML
The Feature Store solves these problems.
1 min read
Databricks ML: End-to-End Machine Learning on the Lakehouse
The ML platform includes: Feature Store : Centralized feature management AutoML : Automated model training and selection MLflow : Experiment tracking and…
2 min read
Photon Engine: Accelerating Spark Workloads on Databricks
Photon is Databricks' native vectorised query execution engine written in C++—a replacement for the JVM-based Apache Spark execution engine for SQL and…
2 min read
Databricks SQL: Analytics Warehousing on the Lakehouse
Databricks SQL offers: SQL Endpoints : Serverless compute for SQL queries Query Editor : Web based SQL IDE Dashboards : Built in visualization and reporting…
1 min read
Delta Sharing: Secure Data Exchange Across Organizations
The protocol is open source, meaning recipients don't need Databricks to access shared data.
1 min read
Data Governance in Azure Databricks: Policies, Practices, and Implementation
A complete governance framework addresses: Access Control : Who can access what data Data Quality : Ensuring data accuracy and completeness Data Lineage :…
1 min read
Azure Databricks Unity Catalog: Unified Data Governance at Scale
Unity Catalog solves these problems with a unified approach.
1 min read
Data Lakehouse Patterns: Best of Both Worlds
The lakehouse architecture proved its value in 2021. Organizations are consolidating their data warehouses and data lakes into unified lakehouses, reducing…
2 min read
Repos in Databricks: Managing Production Deployments
Databricks Repos in production use means the code running in your prod workspace is explicitly linked to a specific Git commit—not "whatever notebooks…
2 min read
Git Integration in Databricks for Version Control
Databricks Git integration (Repos) connects the workspace directly to GitHub, GitLab, Azure DevOps, or Bitbucket, replacing the manual pattern of…
2 min read
Databricks Notebook Workflows for Data Pipelines
Databricks notebook workflows—chaining notebooks together using dbutils.notebook.run()—are the stepping stone between "notebooks as scripts" and proper…
1 min read
Databricks REST API for Advanced Automation
The Databricks REST API exposes every workspace capability that the UI and CLI provide—and then some—making it the integration point for external…
1 min read
Mastering the Databricks CLI for Automation
The Databricks CLI is the command-line interface for Databricks workspace operations—running jobs, managing clusters, deploying libraries, uploading…
2 min read
Job Clusters vs All-Purpose Clusters: Choosing the Right Approach
The job cluster versus all-purpose cluster decision in Databricks is primarily a cost decision: all-purpose clusters stay running between tasks (you pay for…
2 min read
Databricks Cluster Policies for Governance and Cost Control
Databricks cluster policies are the governance layer that prevents the classic enterprise data platform problem: data scientists spinning up 64-node GPU…
2 min read
Azure Databricks Cluster Configuration Best Practices
Azure Databricks cluster configuration is where cost and performance trade-offs become very concrete: the wrong cluster type for a workload is either money…
1 min read
Databricks SQL Analytics is Now Generally Available
Databricks SQL Analytics provides: Native SQL interface to query Delta Lake tables SQL Endpoints with auto scaling compute Built in visualization and…
1 min read
Mastering Delta Lake MERGE Operations for Upserts and SCD
The MERGE statement combines INSERT, UPDATE, and DELETE operations in a single atomic transaction: Handle complex business logic with conditional updates:…
2 min read
MLflow Experiments - Tracking and Managing Machine Learning Models
MLflow became the experiment tracking tool I recommend to every ML team regardless of their cloud platform choice. The core value proposition is simple…
1 min read
Databricks Notebooks - Collaborative Data Science and Engineering
Notebooks support multiple languages in a single document: Python PySpark and pandas Scala Native Spark SQL Spark SQL R SparkR and local R Markdown…
1 min read
Real-Time Data Processing with Spark Structured Streaming
The appeal of Spark Structured Streaming is that you write it almost identically to a batch Spark job. Same DataFrame API, same transformations, same Spark…
2 min read
Delta Lake Time Travel - Versioned Data Lake Queries
Delta Lake time travel is the feature that makes "oops" survivable. An engineer runs a DELETE with a typo in the WHERE clause. A batch job writes corrupt…
2 min read
Azure Databricks Workspace Management and Data Governance Best Practices
Databricks workspace governance was the problem nobody thought about until there were thirty workspaces, fifteen clusters running overnight, and six teams…
2 min read
MLflow Integration with Azure Databricks for MLOps
MLflow is the closest thing to a standard the ML tooling space has right now—experiment tracking, run metadata, model registry, and a deployment abstraction…
7 min read
Wiring a Medallion Pipeline on Azure Databricks with Auto Loader
How to connect Auto Loader, Structured Streaming and Delta MERGE into a bronze, silver and gold pipeline that restarts safely and doesn't run up the bill.
7 min read
Databricks SQL Analytics in Preview: What to Test Before You Commit
SQL Analytics on Azure Databricks is in public preview. Here is what it is, how to pilot it properly, and when Synapse or Power BI alone is the better fit.
1 min read
Azure Databricks Workspace Management
Spinning up a Databricks workspace is the easy part. Making it production-ready is where I see the most avoidable rework. Cluster policies, AAD passthrough…
2 min read
Getting Started with Delta Lake on Azure Databricks
Data lakes have a long-standing reputation problem: cheap to fill, painful to trust. Schema drift, half-written files from failed jobs, "is this row a…
1 min read
Databricks Delta Lake: Advanced Patterns
Delta Lake: the foundation of the modern lakehouse.
1 min read
Azure Databricks Jobs: Scheduled and Triggered Workflows
Notebooks are great until you need them to run on Tuesday at 2am, retry on failure, and chain into the next step. That's where Databricks Jobs come in.…
1 min read
Databricks Structured Streaming: Real-Time Data Processing
Incrementally process files as they arrive. Structured Streaming makes real-time processing accessible with familiar DataFrame semantics.
1 min read
MLflow on Azure Databricks: Experiment Tracking
MLflow makes ML experiments reproducible and models traceable.
1 min read
Introduction to Delta Lake on Azure Databricks
Anyone who's run a "data lake" for any length of time has hit the same wall: parquet files everywhere, no transactional guarantees, partial-write disasters…