1 min read
Optimized Writes in Delta Lake
1. Enable by default : For most analytical workloads 2. Choose appropriate bin size : 128MB general, 256MB for Direct Lake 3. Combine with auto compact :…
Blog
Thoughts on data engineering, cloud architecture, and technology. 2205 articles and counting.
Showing 24 of 2205 posts — page 29 of 92
1 min read
File Compaction Strategies in Delta Lake
1. Schedule regular compaction : Daily for high volume tables 2. Use partition aware compaction : Avoid rewriting entire large tables 3. Enable auto…
1 min read
Parquet File Optimization for Analytics
1. Target 128 256MB files : Optimal for most query engines 2. Use Snappy or ZSTD : Best compression/speed balance 3. Sort by filter columns : Improves…
1 min read
Delta Lake Everywhere: The Universal Table Format
Delta Lake provides a reliable, performant foundation for modern data platforms. Its ACID transactions, time travel, and schema evolution capabilities make…
1 min read
OneLake Interoperability: Connecting Your Data Ecosystem
1. Use managed identity : Preferred for Azure services 2. Leverage shortcuts : Don't copy data unnecessarily 3. Optimize file sizes : 128MB 1GB for best…
1 min read
Microsoft Fabric and Databricks: Coexistence Strategies
1. Use shared storage : ADLS as the common data layer 2. Standardize on Delta Lake : Compatible format for both 3. Optimize for readers : Both platforms…
1 min read
Cross-Cloud Analytics with Microsoft Fabric
Most enterprises use multiple clouds: 1. Minimize cross cloud joins : Materialize frequently joined data 2. Use shortcuts wisely : Only for data that…
1 min read
Lakehouse Federation in Microsoft Fabric
1. Define clear domains : Each lakehouse should own specific data 2. Minimize cross domain joins : Materialize frequently joined data 3. Implement…
1 min read
Data Virtualization in Microsoft Fabric
Traditional ETL moves data physically: Virtualization queries in place: Shortcuts create virtual pointers to external data: 1. Use for read heavy workloads…
1 min read
Change Data Capture in Microsoft Fabric
CDC captures row level changes (inserts, updates, deletes) from source systems: The simplest option for supported sources: For more control over the CDC…
1 min read
Real-Time Sync Patterns in Microsoft Fabric
For sub second latency: For minute level latency with relational sources: Combine streaming and batch for different data types: Distribute data to multiple…
1 min read
Open Mirroring in Fabric: Build Custom Data Sources
Open Mirroring provides a standardized way to: Write Delta Lake tables directly to a mirrored database location Maintain the mirrored database experience…
1 min read
Snowflake Mirroring to Fabric: Cross-Cloud Data Integration
Mirroring eliminates the need for complex ETL between these systems.
1 min read
Cosmos DB Mirroring to Fabric: Analytical Queries at Scale
Unlike SQL mirroring which uses change tracking, Cosmos DB mirroring leverages the built in analytical store: 1. Go to your Cosmos DB account 2. Navigate to…
1 min read
Azure SQL Mirroring to Fabric: Complete Setup Guide
1. Navigate to your workspace 2. Click New Mirrored Database 3. Select Azure SQL Database 4. Configure connection: Server name Database name Authentication…
1 min read
Fabric Mirroring GA: Real-Time Database Replication
Mirroring creates a continuously synchronized copy of your operational database in OneLake as Delta Lake tables: As of GA: Azure SQL Database : Full support…
1 min read
Filtered Vector Search in Azure AI Search: Best Practices
Pre-filtering: Filter before vector search Post-filtering: Filter after vector search Azure AI Search uses pre-filtering with automatic optimization.
1 min read
HNSW Tuning for Azure AI Search: A Practical Guide
HNSW builds a multi layer graph where: Higher layers : Sparse, for fast long range navigation Lower layers : Dense, for precise local search Layer 0 :…
1 min read
Product Quantization for Extreme-Scale Vector Search
PQ divides each vector into subvectors and quantizes each independently: For even faster search, combine with inverted file index: Dataset Size n subvectors…
1 min read
Scalar Quantization for Vector Search: A Practical Guide
Scalar quantization maps floating point values to integers by dividing the value range into buckets: Understanding quantization error helps set…
2 min read
Binary Vectors and Quantization in Azure AI Search
With quantization, this can be reduced to 1-2 GB.
1 min read
Improved Vector Search in Azure AI Search: Deep Dive
Vector search finds similar items by comparing their mathematical representations (embeddings). The key components: Embeddings : Dense vectors representing…
1 min read
Azure AI Search Updates: What's New in Summer 2024
Expanded Vector Dimensions Semantic Ranking Improvements Customer-Managed Keys for Vectors