1 min read
Fabric Data Factory Notes: designing pipelines for failure, not the happy path
I focused on making delivery decisions auditable and repeatable—documenting intent, success criteria, and rollback paths to reduce tribal knowledge.
Data Factory
32 articles
1 min read
Orchestration Lessons in Fabric: debugging pipeline latency before it becomes a fire drill
I worked on smoothing the handoff between data engineering and AI teams—standardizing feature contracts, embedding validation, and adding lightweight…
1 min read
Orchestration Lessons in Fabric: how I keep orchestration readable over time
I turned implicit processes into explicit operating rules—defining owners, acceptance tests, and lightweight runbooks so teams can move confidently and…
1 min read
Orchestration Lessons in Fabric: debugging pipeline latency before it becomes a fire drill
I focused on making delivery decisions auditable and repeatable—documenting intent, success criteria, and rollback paths to reduce tribal knowledge.
1 min read
Orchestration Lessons in Fabric: designing pipelines for failure, not the happy path
I spent the day reducing cognitive overhead for engineers and analysts—introducing clearer table contracts, simpler failure modes, and concise runbooks that…
1 min read
Data Movement Without Drama: designing pipelines for failure, not the happy path
I spent the day reducing cognitive overhead for engineers and analysts—introducing clearer table contracts, simpler failure modes, and concise runbooks that…
1 min read
Pipelines I Trust in Fabric: reducing brittle dependencies in daily loads
I worked on smoothing the handoff between data engineering and AI teams—standardizing feature contracts, embedding validation, and adding lightweight…
1 min read
Building Data Pipelines with Microsoft Fabric Data Factory
Data Factory pipelines use a visual designer with activities that can be chained together.
1 min read
Copy Activity in Fabric: Data Movement Patterns
The Copy Activity in Fabric Data Factory is the same copy engine as Azure Data Factory — the underlying data movement service optimised for throughput with…
1 min read
Fabric Data Factory: Data Integration Reimagined
Tomorrow we'll dive deeper into Copy Activity patterns. Data Factory in Fabric Data Pipeline Documentation Pipeline Expressions
1 min read
Fabric Pipelines: Orchestrating Data Workflows
Fabric Pipelines enable robust data orchestration with full control flow capabilities. Tomorrow, I will cover Dataflows Gen2.
1 min read
Data Factory in Microsoft Fabric: Modern Data Integration
Data Pipelines in Fabric are similar to Azure Data Factory pipelines but with deeper Fabric integration.
1 min read
SSIS to Azure Migration: Strategies and Tools
Lift and shift existing SSIS packages with minimal changes: Convert SSIS logic to native Azure Data Factory: SSIS Component Data Flow Equivalent OLE DB…
2 min read
Azure Data Factory Managed Virtual Network: Secure Data Integration
Data Flows automatically use the managed VNet when connected to private endpoints.
3 min read
Azure Data Factory Mapping Data Flows: ETL Patterns for the Modern Data Platform
Think of it as the best of both worlds: visual design for maintainability, Spark for scale.
1 min read
Data Flows Gen2: Visual Data Transformation at Scale
Data Flows Gen2 democratize data transformation, enabling both developers and data analysts to build scalable ETL solutions.
1 min read
Azure Data Factory Stored Procedure Activity: Database Logic in Pipelines
The Stored Procedure Activity bridges the gap between data movement and data transformation, enabling complex database logic to be orchestrated as part of…
1 min read
Azure Data Factory Web Activity: Integrating REST APIs in Data Pipelines
The Web activity in ADF is the HTTP client that lets your pipeline call external REST APIs—triggering a Logic App, posting a Teams notification, calling a…
1 min read
Azure Data Factory Until Activity: Implementing Loops and Polling Patterns
The Until activity in ADF is the polling loop that keeps running until a condition is true—useful for pipelines that trigger an external process and need to…
1 min read
Azure Data Factory ForEach Activity: Parallel Processing Patterns
The ForEach Activity is the key to building scalable, efficient data pipelines that can process thousands of items in parallel while maintaining control and…
1 min read
Azure Data Factory Lookup Activity: Retrieving Metadata for Dynamic Pipelines
The Lookup Activity is the foundation for building intelligent, data-driven pipelines that adapt their behavior based on configuration and runtime data.
1 min read
Dynamic Content in Azure Data Factory: Expressions and Functions
Dynamic content in ADF is the expression system that makes pipelines adaptive rather than static—generating file paths from date parameters, constructing…
1 min read
Parameterized Pipelines in Azure Data Factory: Building Reusable Data Workflows
Parameterized pipelines dramatically reduce maintenance overhead and enable rapid deployment of new data integration scenarios without code changes.
1 min read
Azure Data Factory Copy Activity: Moving Data at Scale
The Copy Activity in Azure Data Factory is the activity I configure more than any other—it's the data movement primitive that connects over 90 source and…
2 min read
Synapse Pipelines vs Azure Data Factory - Making the Right Choice
The Synapse Pipelines vs. ADF question comes up in almost every Synapse project I start. The honest answer depends on where the rest of your data platform…
2 min read
ADF Self-Hosted Integration Runtime - Connecting On-Premises Data
The Self-Hosted Integration Runtime is the component that makes ADF viable for the majority of real enterprise projects—the ones where the source database…
2 min read
Azure Data Factory Triggers - Orchestrating Data Pipelines
ADF trigger design is where data pipeline reliability is won or lost. Schedule triggers are obvious—run at midnight, run every hour—but the interesting…
2 min read
Data Transformation with Azure Data Factory Mapping Data Flows
Mapping Data Flows are the feature that ended most of my "should I use PySpark or Spark SQL for this?" debates. Visual, yes—but reviewable in JSON and…
1 min read
Azure Data Factory Data Flows: Visual ETL
I have a love-hate relationship with visual ETL tools. Drag-and-drop is wonderful right up until you need to diff two pipelines in source control. ADF Data…
1 min read
Azure Data Factory Linked Services: Connect Everything
Best practice: use Managed Identity where supported. Linked Services are the foundation of Data Factory connectivity.
1 min read
Azure Data Factory Triggers: Scheduling and Event-Driven Pipelines
A Data Factory pipeline without a trigger is a stored procedure that nobody calls. Triggers are the boring half of orchestration that decides whether your…
1 min read
Building Data Pipelines with Azure Data Factory
A surprising number of "modernise my reporting" engagements I see start the same way: data scattered across an on-prem SQL Server, a SaaS CRM, three Excel…