1 min read
Building Data Pipelines with Microsoft Fabric Data Factory
Data Factory pipelines use a visual designer with activities that can be chained together.
ETL
37 articles
1 min read
AI in Data Pipelines: Intelligent ETL and ELT
AI-enhanced pipelines handle complex transformations and quality issues automatically.
1 min read
AI Data Pipeline Patterns: Intelligent ETL and Processing
AI-powered pipelines transform raw data into intelligent, enriched datasets. Design for both batch and streaming scenarios.
1 min read
AI in Data Pipelines: Intelligent ETL and Data Processing
AI transforms data pipelines from rigid rule-based systems to adaptive, intelligent processes. Start with high-value, error-prone steps and expand from there.
1 min read
Change Data Capture in Microsoft Fabric
CDC captures row level changes (inserts, updates, deletes) from source systems: The simplest option for supported sources: For more control over the CDC…
1 min read
Automation in Fabric: Scheduling and Orchestration
Automation is key to maintaining reliable data pipelines. This guide covers scheduling, orchestration, and automation patterns in Microsoft Fabric.
1 min read
Fabric Data Pipelines: Orchestration Patterns
A Fabric pipeline isn't just a sequence of activities — it's an orchestration layer with conditional branching, parameter passing, loops, and error handling…
1 min read
Dataflow Gen2: Power Query Transformations in Fabric
A month into production-style Fabric testing and Dataflow Gen2 has become my go-to recommendation for teams without Spark expertise who need repeatable data…
1 min read
Copy Activity in Fabric: Data Movement Patterns
The Copy Activity in Fabric Data Factory is the same copy engine as Azure Data Factory — the underlying data movement service optimised for throughput with…
1 min read
Fabric Data Factory: Data Integration Reimagined
Tomorrow we'll dive deeper into Copy Activity patterns. Data Factory in Fabric Data Pipeline Documentation Pipeline Expressions
1 min read
Dataflows Gen2: Low-Code Data Transformation in Fabric
Dataflows Gen2 provide an accessible way to build data transformations. Tomorrow, I will cover Mirroring in Fabric.
1 min read
Fabric Pipelines: Orchestrating Data Workflows
Fabric Pipelines enable robust data orchestration with full control flow capabilities. Tomorrow, I will cover Dataflows Gen2.
1 min read
Data Factory in Microsoft Fabric: Modern Data Integration
Data Pipelines in Fabric are similar to Azure Data Factory pipelines but with deeper Fabric integration.
1 min read
SSIS to Azure Migration: Strategies and Tools
Lift and shift existing SSIS packages with minimal changes: Convert SSIS logic to native Azure Data Factory: SSIS Component Data Flow Equivalent OLE DB…
1 min read
Delta Live Tables: Declarative Data Pipelines on Databricks
Traditional ETL: Delta Live Tables: DLT provides built in data quality enforcement: Handle CDC feeds with APPLY CHANGES: Delta Live Tables transforms data…
3 min read
Azure Data Factory Mapping Data Flows: ETL Patterns for the Modern Data Platform
Think of it as the best of both worlds: visual design for maintainability, Spark for scale.
1 min read
Data Flows Gen2: Visual Data Transformation at Scale
Data Flows Gen2 democratize data transformation, enabling both developers and data analysts to build scalable ETL solutions.
2 min read
High-Performance Data Loading with COPY Command in Synapse
The COPY command replaced PolyBase as the recommended data loading mechanism for Synapse Dedicated SQL Pool because it's simpler to use and performs at the…
2 min read
PolyBase for Data Virtualization in Azure Synapse
PolyBase is the T-SQL feature that makes Synapse Dedicated SQL Pool a data virtualisation engine, not just a data warehouse—it allows you to define external…
1 min read
Azure Data Factory Stored Procedure Activity: Database Logic in Pipelines
The Stored Procedure Activity bridges the gap between data movement and data transformation, enabling complex database logic to be orchestrated as part of…
1 min read
Azure Data Factory ForEach Activity: Parallel Processing Patterns
The ForEach Activity is the key to building scalable, efficient data pipelines that can process thousands of items in parallel while maintaining control and…
1 min read
Azure Data Factory Lookup Activity: Retrieving Metadata for Dynamic Pipelines
The Lookup Activity is the foundation for building intelligent, data-driven pipelines that adapt their behavior based on configuration and runtime data.
1 min read
Dynamic Content in Azure Data Factory: Expressions and Functions
Dynamic content in ADF is the expression system that makes pipelines adaptive rather than static—generating file paths from date parameters, constructing…
1 min read
Parameterized Pipelines in Azure Data Factory: Building Reusable Data Workflows
Parameterized pipelines dramatically reduce maintenance overhead and enable rapid deployment of new data integration scenarios without code changes.
1 min read
Azure Data Factory Copy Activity: Moving Data at Scale
The Copy Activity in Azure Data Factory is the activity I configure more than any other—it's the data movement primitive that connects over 90 source and…
1 min read
Mastering Delta Lake MERGE Operations for Upserts and SCD
The MERGE statement combines INSERT, UPDATE, and DELETE operations in a single atomic transaction: Handle complex business logic with conditional updates:…
2 min read
Synapse Pipelines vs Azure Data Factory - Making the Right Choice
The Synapse Pipelines vs. ADF question comes up in almost every Synapse project I start. The honest answer depends on where the rest of your data platform…
2 min read
Azure Data Factory Triggers - Orchestrating Data Pipelines
ADF trigger design is where data pipeline reliability is won or lost. Schedule triggers are obvious—run at midnight, run every hour—but the interesting…
1 min read
Azure Search Indexers - Automated Data Ingestion Strategies
Indexers handle the ETL process for search: Requires a column that increases monotonically: Enable change tracking on your database: Source JSON: Source…
1 min read
Visual ETL with Azure Synapse Data Flows
Data flows in Synapse run on Spark clusters and offer: Visual drag and drop interface 80+ built in transformations Schema drift handling Data preview and…
2 min read
Data Transformation with Azure Data Factory Mapping Data Flows
Mapping Data Flows are the feature that ended most of my "should I use PySpark or Spark SQL for this?" debates. Visual, yes—but reviewable in JSON and…
2 min read
Power BI Dataflows - Best Practices for Enterprise Data Preparation
Dataflows are collections of entities (tables) that are created and managed in workspaces in the Power BI service. They allow you to: Connect to various…
2 min read
Building Data Pipelines with Delta Lake on Azure Databricks
Delta Lake has gone from "interesting open source project" to the default storage layer for data lakes inside about eighteen months. ACID, schema…
1 min read
Azure Data Factory Data Flows: Visual ETL
I have a love-hate relationship with visual ETL tools. Drag-and-drop is wonderful right up until you need to diff two pipelines in source control. ADF Data…
1 min read
Azure Data Factory Triggers: Scheduling and Event-Driven Pipelines
A Data Factory pipeline without a trigger is a stored procedure that nobody calls. Triggers are the boring half of orchestration that decides whether your…
1 min read
Power BI Dataflows: Self-Service ETL
Dataflows bridge the gap between self-service BI and governed enterprise data.
1 min read
Building Data Pipelines with Azure Data Factory
A surprising number of "modernise my reporting" engagements I see start the same way: data scattered across an on-prem SQL Server, a SaaS CRM, three Excel…