1 min read
Fabric Data Pipelines: Orchestration Patterns
A Fabric pipeline isn't just a sequence of activities — it's an orchestration layer with conditional branching, parameter passing, loops, and error handling…
Blog
Thoughts on data engineering, cloud architecture, and technology. 2205 articles and counting.
Showing 24 of 2205 posts — page 46 of 92
1 min read
Dataflow Gen2: Power Query Transformations in Fabric
A month into production-style Fabric testing and Dataflow Gen2 has become my go-to recommendation for teams without Spark expertise who need repeatable data…
1 min read
Copy Activity in Fabric: Data Movement Patterns
The Copy Activity in Fabric Data Factory is the same copy engine as Azure Data Factory — the underlying data movement service optimised for throughput with…
1 min read
Fabric Data Factory: Data Integration Reimagined
Tomorrow we'll dive deeper into Copy Activity patterns. Data Factory in Fabric Data Pipeline Documentation Pipeline Expressions
1 min read
Data Wrangling in Microsoft Fabric: Practical Techniques
Data wrangling — getting messy, inconsistent, real-world data into a reliable clean form — is where most data engineering hours actually go. In Fabric…
1 min read
Fabric Notebooks: Features and Productivity Tips
Fabric Notebooks have some features I didn't notice until I'd been using them for a few weeks and wish someone had pointed out earlier. Inline data…
1 min read
Apache Spark in Microsoft Fabric: Getting Started
Fabric provides a managed Spark experience: Tomorrow we'll dive deeper into Fabric notebooks and their unique features. Spark in Fabric PySpark…
1 min read
Delta Tables in Microsoft Fabric: Deep Dive
The theoretical benefits of Delta Lake — ACID transactions, time travel, schema enforcement — are easy to describe, but the practical payoff becomes clear…
1 min read
Creating Your First Fabric Lakehouse: Step-by-Step Guide
A Lakehouse combines the best of data lakes and data warehouses: Once created, your Lakehouse has two main sections: The Files section is for unstructured…
1 min read
OneLake Explorer: Navigating Your Data Lake in Fabric
OneLake Explorer is a Windows desktop application that lets you browse the contents of your organisation's OneLake the same way you'd navigate files in…
1 min read
Fabric Workspace Setup: Organization and Best Practices
How you structure workspaces in Fabric matters more than most getting-started guides acknowledge. A workspace is the unit of security, capacity binding, and…
2 min read
Microsoft Fabric Trial: What You Get and How to Maximize It
The Fabric trial is one of the more generous evaluation paths Microsoft has offered for a data platform — a 60-day, F64-equivalent capacity for free is a…
2 min read
Getting Started with Microsoft Fabric: A Practical Guide
I've been onboarding several clients onto Fabric over the past few weeks and the first hour is always the same: getting oriented to an interface that's…
2 min read
Microsoft Fabric Public Preview: What You Need to Know
Six weeks into the Fabric public preview and I'm past the demo phase — I'm running actual data engineering workloads and discovering what holds up and what…
1 min read
Circuit Breakers for AI Applications: Preventing Cascade Failures
This concludes our June 2023 series on Azure, Data, and AI topics. We covered: Microsoft Fabric deep dives (Lakehouse, Delta Lake, OneLake) Power BI Direct…
1 min read
Fallback Patterns for AI Applications
Fallback patterns ensure continuous service availability. Tomorrow, I will cover circuit breakers for AI applications.
1 min read
Retry Strategies for AI Applications
Azure OpenAI rate limits in mid-2023 are set per deployment per region, and hitting them is a normal operating condition rather than an exceptional one for…
1 min read
Error Handling for AI Applications: Building Resilient Systems
Robust error handling ensures AI applications remain useful even during failures. Tomorrow, I will cover retry strategies in more depth.
1 min read
Response Streaming: Real-Time LLM Output
Streaming improves the user experience significantly. Tomorrow, I will cover error handling for AI applications.
1 min read
Caching Strategies for LLM Applications
Effective caching significantly reduces LLM costs and latency. Tomorrow, I will cover response streaming patterns.
1 min read
Token Budgeting: Cost-Effective LLM Applications
Token costs for GPT-4 in mid-2023 are real enough to design around: roughly $0.03 per 1K input tokens and $0.06 per 1K output tokens for the 8K context…
1 min read
Context Pruning: Managing Token Limits in LLM Applications
Effective context pruning ensures your LLM applications work reliably. Tomorrow, I will cover token budgeting strategies.
1 min read
Conversation Management: Building Production Chat Systems
Robust conversation management enables reliable production chat systems. Tomorrow, I will cover context pruning strategies.
1 min read
Multi-Turn Conversations: Managing State in AI Applications
Multi-turn conversation management is essential for production chatbots. Tomorrow, I will cover conversation management strategies.