11 articles
Synapse serverless SQL infers long text columns as varchar(8000), and you can't fix it from PySpark. Here's why it happens — and how to fix it properly across Spark, SQL, and Power BI.
Poor data quality leads to incorrect business decisions, failed ML models, and eroded trust in analytics platforms. A proactive approach to data validation…
Fabric notebooks provide a seamless path from exploration to production, with built-in governance, collaboration, and scalable compute.
For real-time inference, query the feature store directly from your model serving endpoint. Fabric's query optimization ensures millisecond-level feature…
Copilot for Notebooks transforms how data engineers write code. Today I'm exploring practical patterns for leveraging AI assistance in Fabric notebooks.
Data wrangling — getting messy, inconsistent, real-world data into a reliable clean form — is where most data engineering hours actually go. In Fabric…
Fabric provides a managed Spark experience: Tomorrow we'll dive deeper into Fabric notebooks and their unique features. Spark in Fabric PySpark…
Notebooks are the primary development environment in Fabric. Tomorrow, I will cover Fabric Pipelines for orchestration.
Fabric's Data Science experience integrates seamlessly with the rest of the platform, allowing you to go from raw data in Lakehouse to deployed models in a…
Synapse Data Engineering in Fabric provides a managed Spark experience that removes infrastructure concerns while delivering the full power of Apache Spark.…
1. Define schemas explicitly Never rely on schema inference in production 2. Use built in functions Avoid UDFs when possible 3. Write pure transformation…