8 min read
LLMs on an Event Hubs Stream: Tiered Triage with Azure Functions
How to put Azure OpenAI inside an Event Hubs pipeline without stalling partitions: batch triggers, a cheap triage tier, structured outputs, fail-closed errors.
Blog
Thoughts on data engineering, cloud architecture, and technology. 2205 articles and counting.
Showing 24 of 2205 posts — page 22 of 92
7 min read
Turning Call Recordings into Data with Azure AI Speech and GPT-4o
A batch pipeline for call and meeting audio: Azure AI Speech transcription with diarization, then GPT-4o structured outputs into rows you can query.
6 min read
Reading Charts and Dashboards with GPT-4o: What to Trust
Where GPT-4o vision helps data teams read charts and dashboard screenshots, what it gets wrong, what each image costs, and how to check its numbers in code.
7 min read
Multimodal RAG on Azure: Caption PDF Figures Before You Embed Them
Make diagrams and charts in PDFs retrievable with Document Intelligence 4.0 figure extraction, GPT-4o descriptions and Azure AI Search hybrid queries.
7 min read
Hybrid Search in PostgreSQL: pgvector, Full-Text and RRF in One Query
How to build hybrid search inside PostgreSQL with pgvector, full-text search and weighted reciprocal rank fusion, and when a dedicated search service is better.
8 min read
Where Should Your Vectors Live on Azure? Search Index vs Database
Azure AI Search, Cosmos DB for NoSQL and PostgreSQL with pgvector can all hold embeddings now. Here is how I decide between a search index and your database.
9 min read
Azure OpenAI in Production, January 2025: Let the Platform Do the Work
Deployment types, keyless auth, SDK retries, APIM token limits and Global Batch: what to build on Azure OpenAI in January 2025, and what to stop building.
7 min read
LLM-Drafted Data Dictionaries: Profile First, Describe Second
Draft column descriptions for an Azure SQL database with Azure OpenAI, grounded in profiling stats, reviewed by a data owner, then stored as MS_Description.
8 min read
LLM-Drafted Tests for PySpark: The Model Writes, pytest Judges
Use Azure OpenAI structured outputs to draft edge-case test data for PySpark transforms, review it like code, and let pytest and assertDataFrameEqual decide.
7 min read
LLM-Drafted Data Quality Rules: Let the Model Propose, Not Enforce
Use Azure OpenAI structured outputs to draft Great Expectations rules from a data profile, then backtest and review them before they ever gate a pipeline.
7 min read
Generating dbt Models with Azure OpenAI: Let dbt Be the Judge
Use Azure OpenAI structured outputs to draft dbt models and YAML, then gate every draft with dbt build so nothing untested reaches review.
6 min read
Drafting a Star Schema with GPT-4o and Structured Outputs
Use GPT-4o structured outputs to draft a star schema from requirements, check it with plain code, and generate Fabric Warehouse DDL that respects its limits.
7 min read
Natural Language over a Power BI Model: Let the LLM Pick, Not Write
Answer plain-English questions from a Power BI semantic model by having Azure OpenAI choose measures and filters, then run them with semantic link.
7 min read
GraphRAG 1.x: When Global Search Justifies the Indexing Bill
How Microsoft's GraphRAG 1.x builds a knowledge graph, what local, global and DRIFT search do, and when its indexing cost is worth paying over vector RAG.
8 min read
RAG Beyond Top-K: Retrieval Upgrades Worth Making in January 2025
Which RAG retrieval upgrades Azure AI Search gives you as GA or preview in January 2025, which you build yourself, and the order I'd add them in.
7 min read
Prompting in 2025: What Changes with o1 and Structured Outputs
Which prompt techniques still earn their place on Azure OpenAI in January 2025, which moved into the API, and which now work against reasoning models like o1.
8 min read
Five LLM Application Patterns, From Least to Most Autonomy
A practical ladder of LLM patterns on Azure in January 2025: structured outputs, grounded RAG, tool workflows, self-review and agents.
6 min read
Azure AI Foundry from Python: A Data Team's First Project
A practical first project in Azure AI Foundry for data teams: hubs and projects, the preview azure-ai-projects SDK, and an evaluation loop from day one.
7 min read
Rolling Out a Data Mesh on Fabric: A Phased Plan for 2025
A phased plan for rolling out data mesh on Microsoft Fabric and Purview in early 2025: what to build first, what's still preview, and when to stop.
7 min read
Where Streaming Logic Belongs in Fabric: Eventstream, KQL or Spark
A decision guide for placing filters, enrichment, aggregation and alerts across Eventstream, Eventhouse and Spark in Fabric Real-Time Intelligence.
7 min read
Reviewing Copilot Output in Fabric: A Workload-by-Workload Guide
Where Copilot in Microsoft Fabric stands in January 2025, what each workload's Copilot gets wrong, and how to review its output before it ships.
8 min read
Planning Fabric Work for 2025: What's GA, Preview, and Worth Waiting On
A January 2025 status check on Microsoft Fabric after Ignite 2024: which features are GA, which are preview, and how to sequence this year's work.
7 min read
o1 Is GA, o3 Is Announced: Reasoning Models in January 2025
What changed from o1-preview to o1 (2024-12-17), what o3 actually is today, and how I'd use reasoning_effort and developer messages right now.
8 min read
AI Safety Research in 2025: What Builders Should Act On Now
Late-2024 safety research on scheming, alignment faking and prompt injection, translated into concrete controls for Azure teams shipping AI in 2025.