35 articles
Synapse serverless SQL infers long text columns as varchar(8000), and you can't fix it from PySpark. Here's why it happens — and how to fix it properly across Spark, SQL, and Power BI.
After a year of implementing data solutions on Azure, I've compiled the most important lessons that can save you time, money, and headaches. These come from…
Synapse enables AI at data warehouse scale with native ML integration.
Synapse Data Engineering in Fabric provides a managed Spark experience that removes infrastructure concerns while delivering the full power of Apache Spark.…
Cognitive Services in Synapse brings AI capabilities directly into your data engineering workflows. Process millions of records with sentiment, entities…
Azure Synapse with OpenAI transforms data warehousing from technical SQL expertise to conversational data access. Business users can explore enterprise data…
Configure diagnostic settings to export to Data Lake: Use serverless SQL to query exported logs: Complex log analysis: Process logs with Azure Functions:…
// Enable streaming ingestion .alter database SensorAnalytics policy streamingingestion enable
Connect operational data to analytics: Enhanced query capabilities: Use Spark for complex analytics: Time series and log analytics: Build data pipelines:…
Data Flows Gen2 democratize data transformation, enabling both developers and data analysts to build scalable ETL solutions.
Properly optimized Spark pools can process petabytes of data efficiently and cost-effectively.
These optimizations can dramatically improve dedicated SQL pool performance while reducing resource consumption.
Serverless SQL charges based on data processed. Optimization reduces both cost and query time.
Azure Synapse Analytics in 2022 provides a more unified, performant, and feature-rich analytics platform.
Azure Synapse Link for Dataverse eliminates the traditional barriers between operational and analytical systems. Business users get insights from live data…
Workload management in Synapse Dedicated SQL Pool is the resource governance system that prevents low-priority ad-hoc queries from consuming all compute and…
Result set caching in Synapse Dedicated SQL Pool stores the output of a query in the dedicated pool's storage and returns the cached result for subsequent…
The COPY command replaced PolyBase as the recommended data loading mechanism for Synapse Dedicated SQL Pool because it's simpler to use and performs at the…
PolyBase is the T-SQL feature that makes Synapse Dedicated SQL Pool a data virtualisation engine, not just a data warehouse—it allows you to define external…
Synapse Dedicated SQL Pool (formerly Azure SQL Data Warehouse) is the massively parallel processing (MPP) engine for structured, petabyte-scale analytical…
Synapse Serverless SQL Pool is the query engine that turned Azure Data Lake into an ad-hoc analytics platform—you write T-SQL, point it at files in ADLS…
Microsoft Ignite Fall 2021 brought a wave of announcements across Azure. While hybrid work and Microsoft Teams dominated the headlines, the data and AI…
First, let us create a new Spark pool with version 3.0 in Azure Synapse: One of the most significant improvements in Spark 3.0 is Adaptive Query Execution.…
Data flows in Synapse run on Spark clusters and offer: Visual drag and drop interface 80+ built in transformations Schema drift handling Data preview and…
Dedicated SQL pools distribute data across 60 distributions for parallel processing: Control node : Orchestrates queries and manages metadata Compute nodes…
The question that sparked this post came from a data engineer on a client project: "can we run Power BI reports against Cosmos DB without destroying the…
Synapse Spark pools are the feature that stops the "should I use Databricks or Synapse?" question from being purely a product choice and makes it an…
Artificial Intelligence (AI) and Machine Learning (ML) are trending topics right now. In 2021, there are countless of ways to have a form of "AI" in your…
Serverless SQL: query anything, pay for what you scan.
All of this is unified through Synapse Studio - a single workspace where data engineers, data scientists, and analysts can collaborate.
Note: While dedicated SQL pools are mature (evolved from Azure SQL DW), the Synapse unified experience is still in preview.
A second Synapse-preview note, on what I think is the genuinely new idea in the platform: serverless SQL. Point T-SQL at parquet, CSV, or JSON in your data…
Spark on Synapse is the same managed Spark story most cloud providers offer now, but it's done with the parts of Azure I already use — same workspace as the…
Synapse Link: analytics without the pipeline complexity.
"We have a data warehouse and a Hadoop cluster and they don't talk to each other" was a sentence I heard for years. Synapse is Microsoft's answer: a single…