8 min read
Synapse Serverless varchar(8000): Fixing Strings for Spark and Power BI
Why Synapse serverless SQL infers strings as varchar(8000), why Spark-created lake tables behave differently, and where Power BI string contracts belong.
PySpark
13 articles
8 min read
Break Your Silver Layer on Purpose: Bad-Input Tests in Fabric
Why I keep a library of deliberately broken raw files for every Fabric bronze-to-silver transform, and how to run them as plain PySpark tests before release.
1 min read
Implementing Data Quality Checks in Microsoft Fabric Lakehouses
Poor data quality leads to incorrect business decisions, failed ML models, and eroded trust in analytics platforms. A proactive approach to data validation…
1 min read
Microsoft Fabric Notebooks: Collaborative Data Science at Scale
Fabric notebooks provide a seamless path from exploration to production, with built-in governance, collaboration, and scalable compute.
1 min read
Fabric Data Pipelines: Orchestrating ML Feature Engineering at Scale
For real-time inference, query the feature store directly from your model serving endpoint. Fabric's query optimization ensures millisecond-level feature…
8 min read
LLM-Drafted Tests for PySpark: The Model Writes, pytest Judges
Use Azure OpenAI structured outputs to draft edge-case test data for PySpark transforms, review it like code, and let pytest and assertDataFrameEqual decide.
1 min read
Copilot for Notebooks: AI-Assisted Data Engineering
Copilot for Notebooks transforms how data engineers write code. Today I'm exploring practical patterns for leveraging AI assistance in Fabric notebooks.
1 min read
Data Wrangling in Microsoft Fabric: Practical Techniques
Data wrangling — getting messy, inconsistent, real-world data into a reliable clean form — is where most data engineering hours actually go. In Fabric…
1 min read
Apache Spark in Microsoft Fabric: Getting Started
Fabric provides a managed Spark experience: Tomorrow we'll dive deeper into Fabric notebooks and their unique features. Spark in Fabric PySpark…
1 min read
Fabric Notebooks: Interactive Data Engineering and Science
Notebooks are the primary development environment in Fabric. Tomorrow, I will cover Fabric Pipelines for orchestration.
1 min read
Synapse Data Science in Microsoft Fabric: ML Made Simple
Fabric's Data Science experience integrates seamlessly with the rest of the platform, allowing you to go from raw data in Lakehouse to deployed models in a…
1 min read
Synapse Data Engineering in Microsoft Fabric: Spark at Scale
Synapse Data Engineering in Fabric provides a managed Spark experience that removes infrastructure concerns while delivering the full power of Apache Spark.…
1 min read
PySpark Best Practices for Production Workloads
1. Define schemas explicitly Never rely on schema inference in production 2. Use built in functions Avoid UDFs when possible 3. Write pure transformation…