1 min read
Azure Synapse Analytics: Build 2022 Updates
Connect operational data to analytics: Enhanced query capabilities: Use Spark for complex analytics: Time series and log analytics: Build data pipelines:…
Big Data
14 articles
1 min read
Synapse Spark Pools: Optimization and Best Practices
Properly optimized Spark pools can process petabytes of data efficiently and cost-effectively.
2 min read
Azure Databricks Cluster Configuration Best Practices
Azure Databricks cluster configuration is where cost and performance trade-offs become very concrete: the wrong cluster type for a workload is either money…
2 min read
Azure Synapse Serverless SQL Pools for Big Data Analytics
Synapse Serverless SQL Pool is the query engine that turned Azure Data Lake into an ad-hoc analytics platform—you write T-SQL, point it at files in ADLS…
2 min read
Azure Data Explorer Integration for Advanced Analytics
Azure Data Explorer is the analytics service that addresses a specific limitation of Log Analytics: KQL query performance on large historical datasets…
1 min read
Azure Data Lake Storage Gen2: Hierarchical Namespace for Big Data Analytics
Azure Data Lake Storage Gen2 provides the foundation for modern data analytics, combining the best of blob storage scalability with the directory semantics…
1 min read
Getting Started with Apache Spark 3.0 in Azure Synapse Analytics
First, let us create a new Spark pool with version 3.0 in Azure Synapse: One of the most significant improvements in Spark 3.0 is Adaptive Query Execution.…
1 min read
PySpark Best Practices for Production Workloads
1. Define schemas explicitly Never rely on schema inference in production 2. Use built in functions Avoid UDFs when possible 3. Write pure transformation…
2 min read
Spark SQL Fundamentals and Best Practices
Spark SQL is where I've seen the highest productivity for data engineers who already know SQL but are new to distributed processing. You write familiar SQL…
2 min read
Azure Data Lake Storage Gen2 - Building Modern Data Lakes
ADLS Gen2 is the storage layer that most enterprise data platforms in Australia are converging on. The combination of blob-scale storage (petabytes, low…
1 min read
Enterprise Big Data Analytics with Azure HDInsight
HDInsight supports multiple cluster types for different workloads: Apache Spark Fast, general purpose cluster computing Apache Hadoop Batch processing with…
2 min read
Big Data Processing with Azure Synapse Spark Pools
Synapse Spark pools are the feature that stops the "should I use Databricks or Synapse?" question from being purely a product choice and makes it an…
3 min read
Azure Synapse Analytics is Now Generally Available
All of this is unified through Synapse Studio - a single workspace where data engineers, data scientists, and analysts can collaborate.
2 min read
Introduction to Azure Synapse Analytics
"We have a data warehouse and a Hadoop cluster and they don't talk to each other" was a sentence I heard for years. Synapse is Microsoft's answer: a single…