1 min read
Spark Optimization in Microsoft Fabric
1. Enable AQE : Adaptive Query Execution handles many optimizations automatically 2. Right size partitions : Target 128MB partitions 3. Broadcast small…
Apache Spark
6 articles
1 min read
Apache Spark in Microsoft Fabric: Getting Started
Fabric provides a managed Spark experience: Tomorrow we'll dive deeper into Fabric notebooks and their unique features. Spark in Fabric PySpark…
1 min read
Synapse Data Engineering in Microsoft Fabric: Spark at Scale
Synapse Data Engineering in Fabric provides a managed Spark experience that removes infrastructure concerns while delivering the full power of Apache Spark.…
1 min read
Databricks Notebooks - Collaborative Data Science and Engineering
Notebooks support multiple languages in a single document: Python PySpark and pandas Scala Native Spark SQL Spark SQL R SparkR and local R Markdown…
1 min read
PySpark Best Practices for Production Workloads
1. Define schemas explicitly Never rely on schema inference in production 2. Use built in functions Avoid UDFs when possible 3. Write pure transformation…
2 min read
Spark SQL Fundamentals and Best Practices
Spark SQL is where I've seen the highest productivity for data engineers who already know SQL but are new to distributed processing. You write familiar SQL…