8 min read
Rerun From Failed Activity in Fabric: Design for the Second Run
What a Fabric pipeline rerun actually runs again, why utcNow() and run IDs break recovery, and how to make rerun from failed activity safe.
Data Factory
34 articles
9 min read
Fabric Pipeline Deadlines: Alert on a Late Run Before Anyone Asks
How to catch a slipping Fabric pipeline before its deadline: alert on missing and late runs, not just failures, using job events, KQL and the jobs API.
8 min read
Readable Fabric Pipelines: Names, Boundaries and Config Off the Canvas
How to keep Fabric pipelines readable as they grow: naming before the first Git commit, child pipelines with clear contracts, and variable libraries for config.
8 min read
Fabric Pipeline Latency: Find Where the Minutes Actually Go
How to split a slow Fabric pipeline run into start delay, queueing, Spark start-up, copy transfer and idle gaps, with KQL and the activity runs API.
8 min read
Fabric Pipeline Failure Paths: Catch the Error, Then Fail the Run
How dependency conditions decide a Fabric pipeline's final status, why try-catch hides failures, and how to log, alert and still fail the run honestly.
8 min read
Fabric Data Movement: Design for the Half-Finished Copy
How to design Fabric Copy jobs and copy activities so a failed, partial or repeated copy never leaves duplicates, gaps or a watermark that moved too far.
9 min read
Who Gets Told? Owners, Error Codes and Runbooks in Fabric
A table contract is half the job: assigning owners per layer, collapsing load outcomes to three, and mapping Fabric pipeline failures to runbooks.
9 min read
Fabric Daily Loads: Gate on Data Readiness, Not the Clock
How to remove brittle dependencies from daily Fabric loads with one orchestrating pipeline, readiness checks, idempotent writes and deliberate retries.
9 min read
Leaving ADF and Databricks for Fabric: What Held Up
What consolidating Azure Data Factory, Databricks, Azure SQL and Power BI onto Microsoft Fabric delivered in production, and where the seams still show.
1 min read
Building Data Pipelines with Microsoft Fabric Data Factory
Data Factory pipelines use a visual designer with activities that can be chained together.
1 min read
Copy Activity in Fabric: Data Movement Patterns
The Copy Activity in Fabric Data Factory is the same copy engine as Azure Data Factory — the underlying data movement service optimised for throughput with…
1 min read
Fabric Data Factory: Data Integration Reimagined
Tomorrow we'll dive deeper into Copy Activity patterns. Data Factory in Fabric Data Pipeline Documentation Pipeline Expressions
1 min read
Fabric Pipelines: Orchestrating Data Workflows
Fabric Pipelines enable robust data orchestration with full control flow capabilities. Tomorrow, I will cover Dataflows Gen2.
1 min read
Data Factory in Microsoft Fabric: Modern Data Integration
Data Pipelines in Fabric are similar to Azure Data Factory pipelines but with deeper Fabric integration.
1 min read
SSIS to Azure Migration: Strategies and Tools
Lift and shift existing SSIS packages with minimal changes: Convert SSIS logic to native Azure Data Factory: SSIS Component Data Flow Equivalent OLE DB…
2 min read
Azure Data Factory Managed Virtual Network: Secure Data Integration
Data Flows automatically use the managed VNet when connected to private endpoints.
3 min read
Azure Data Factory Mapping Data Flows: ETL Patterns for the Modern Data Platform
Think of it as the best of both worlds: visual design for maintainability, Spark for scale.
1 min read
Data Flows Gen2: Visual Data Transformation at Scale
Data Flows Gen2 democratize data transformation, enabling both developers and data analysts to build scalable ETL solutions.
1 min read
Azure Data Factory Stored Procedure Activity: Database Logic in Pipelines
The Stored Procedure Activity bridges the gap between data movement and data transformation, enabling complex database logic to be orchestrated as part of…
1 min read
Azure Data Factory Web Activity: Integrating REST APIs in Data Pipelines
The Web activity in ADF is the HTTP client that lets your pipeline call external REST APIs—triggering a Logic App, posting a Teams notification, calling a…
1 min read
Azure Data Factory Until Activity: Implementing Loops and Polling Patterns
The Until activity in ADF is the polling loop that keeps running until a condition is true—useful for pipelines that trigger an external process and need to…
1 min read
Azure Data Factory ForEach Activity: Parallel Processing Patterns
The ForEach Activity is the key to building scalable, efficient data pipelines that can process thousands of items in parallel while maintaining control and…
1 min read
Azure Data Factory Lookup Activity: Retrieving Metadata for Dynamic Pipelines
The Lookup Activity is the foundation for building intelligent, data-driven pipelines that adapt their behavior based on configuration and runtime data.
1 min read
Dynamic Content in Azure Data Factory: Expressions and Functions
Dynamic content in ADF is the expression system that makes pipelines adaptive rather than static—generating file paths from date parameters, constructing…
1 min read
Parameterized Pipelines in Azure Data Factory: Building Reusable Data Workflows
Parameterized pipelines dramatically reduce maintenance overhead and enable rapid deployment of new data integration scenarios without code changes.
1 min read
Azure Data Factory Copy Activity: Moving Data at Scale
The Copy Activity in Azure Data Factory is the activity I configure more than any other—it's the data movement primitive that connects over 90 source and…
2 min read
Synapse Pipelines vs Azure Data Factory - Making the Right Choice
The Synapse Pipelines vs. ADF question comes up in almost every Synapse project I start. The honest answer depends on where the rest of your data platform…
2 min read
ADF Self-Hosted Integration Runtime - Connecting On-Premises Data
The Self-Hosted Integration Runtime is the component that makes ADF viable for the majority of real enterprise projects—the ones where the source database…
2 min read
Azure Data Factory Triggers - Orchestrating Data Pipelines
ADF trigger design is where data pipeline reliability is won or lost. Schedule triggers are obvious—run at midnight, run every hour—but the interesting…
2 min read
Data Transformation with Azure Data Factory Mapping Data Flows
Mapping Data Flows are the feature that ended most of my "should I use PySpark or Spark SQL for this?" debates. Visual, yes—but reviewable in JSON and…
1 min read
Azure Data Factory Data Flows: Visual ETL
I have a love-hate relationship with visual ETL tools. Drag-and-drop is wonderful right up until you need to diff two pipelines in source control. ADF Data…
1 min read
Azure Data Factory Linked Services: Connect Everything
Best practice: use Managed Identity where supported. Linked Services are the foundation of Data Factory connectivity.
1 min read
Azure Data Factory Triggers: Scheduling and Event-Driven Pipelines
A Data Factory pipeline without a trigger is a stored procedure that nobody calls. Triggers are the boring half of orchestration that decides whether your…
7 min read
Azure Data Factory: Designing a First Pipeline You Can Safely Rerun
How I build a first Azure Data Factory pipeline: parameterised by date, idempotent staging loads, no secrets in JSON, and UTC-only schedule triggers.