Fabric Data Agent Accuracy Is a Distribution, Not a Number

I asked a Fabric data agent “What was Queensland’s highest demand over the last quarter?” ten times, each in a fresh conversation. Once it said 8,187.54 MW, which is correct. Nine times it said 5,993.13 MW, a single five-minute interval from 1 October. Across all three wordings it was right 14 times in 30, so for this question a one-off evaluation run is close to a coin toss.

Marc’s post on data-marc.com makes the case for automated tests that store the query producing the expected answer rather than the answer itself. He names determinism as the goal but doesn’t measure it. So I did: 24 questions, three wordings each, ten runs per wording, on real Australian electricity market data, with every stage of the agent’s work captured. Accuracy is a distribution, and the number your evaluation prints is one draw from it.

The rig

The data is AEMO’s aggregated price and demand data: five-minute regional prices and demand for all five NEM regions from January 2025 to September 2026, 918,720 intervals, used with attribution under AEMO’s copyright permissions. It lands in a lakehouse as a small star schema, with a Direct Lake semantic model on top carrying six measures. One data agent has both sources attached.

The questions include real traps. AEMO timestamps mark the end of each interval, so September’s last row is stamped 00:00 on 1 October. My ground truth follows the date dimension and assigns each row to the date on its stamp. “Average price” can mean the simple mean of intervals or the demand-weighted price, and for South Australia in 2025 those are $86.73/MWh and $117.95/MWh. The financial year runs July to June. Every expected value comes from a stored T-SQL query run against the SQL analytics endpoint at evaluation time (Marc’s dynamic ground truth), and all 24 matched an independent pandas recomputation from the raw CSVs exactly.

The harness calls the published agent through FabricOpenAIResponses (SDK 0.1.33a0, which added headless credentials; see how the Responses API client changed) in a new conversation for every ask. It records the rewritten question, data source, generated query and answer, and grades each answer twice: with the SDK’s judge prompt as a second turn, as evaluate_data_agent does, and with a deterministic numeric comparator. Throttled attempts were re-asked. evaluate_data_agent can repeat questions itself (num_query_repeats), and its _steps table records each tool call’s natural-language query, but it doesn’t report pass^k.

The agent ran on the Standard runtime, not the preview runtime; the API runs used F16 and the 29 October notebook and portal checks used F4. The SDK and evaluate_data_agent are in preview (0.1.33a0 is an alpha on PyPI), and these results cover 24 questions on one dataset.

82% accurate, 39% dependable

Across 720 runs the agent was right 82.4% of the time. Scored one pass at a time, the same 72 wordings came in anywhere from 73.6% to 86.1%. And only 28 of the 72 wordings were right on all ten runs. Forty were flaky: right on some runs, wrong on others, with nothing changed in between.

The metric that captures this is pass^k from the τ-bench paper: the chance that all k independent attempts at a task succeed, averaged across tasks. Pass^10 here is 38.9%.

Only 3 of the 24 questions were right every time under all three wordings.

The variance enters before the query is written

The Responses API returns every tool call. In the trace, the orchestrator rewrites the question into a fuller instruction, picks a source, generates and runs a query, then writes the answer.

The rewrite and the DAX text differed on almost every run, yet most of that variation is harmless: aliases, column choices and formatting that return the same number. Routing barely moved. Only 3 of 720 runs touched the lakehouse, even though it was attached.

In the two questions I traced end to end, what decided the answer was the rewrite. For the Queensland question, runs whose rewrite said “the most recent calendar quarter available in the data” returned the 1 October interval; runs that said “the last completed calendar quarter relative to today” returned the right peak. That one phrase predicted the outcome in 26 of 29 answered runs. For “the average price across the NEM in September 2026”, the agent used the demand-weighted measure in all 11 runs where the rewrite contained the word “weighted”, and the simple-average measure in 18 of the 19 where it didn’t.

Four ways it went wrong

Definitional drift. In v0, 27% of definitional runs returned a defensible number under a different definition, usually the simple average where the business meant demand-weighted. Every South Australia 2025 run that produced a query (28 of 30) used the measure literally named [Average Price], and six of those answers described the result as “volume-weighted”. The label matched the business definition and the number didn’t, which is exactly how a wrong answer gets past a reviewer.

The portal’s test chat found a third reading. With the v0 settings in the draft, “What was the average price across the NEM in September 2026?” came back as $53.63/MWh, labelled volume-weighted, five times in a row. That is the unweighted mean of the five regional demand-weighted prices, which counts Tasmania the same as New South Wales. It never appeared in any of the 45 API runs of that question. (The test chat keeps one thread, so those answers could see each other.)

Data-edge artefacts. The single 00:00 interval on 1 October exists because AEMO stamps interval ends. Half the Queensland runs treated it as a quarter of data, and the portal chat went further, reporting the period covered as 1 October to 31 December 2026.

Prior knowledge leaking in. The NEM moved to five-minute settlement on 1 October 2021. Eight rewrites still described half-hourly or 30-minute intervals, and two answers to “how many hours of negative prices” divided the interval count by two, returning 15,940 hours instead of 2,656.7.

Declines. In 42 runs (5.8%) the semantic model tool produced no query, and the agent said so rather than guessing. That’s the right failure, but it hit 32 of the 72 wordings, so it shows up as flakiness on otherwise reliable questions.

What instructions fixed, and what they broke

For v1 I did what the guidance suggests: definitions in the agent instructions, data source instructions naming the right measure for each term, and three example queries on the lakehouse. Every wording ran five times, compared with v0’s first five runs.

- "Average price" means the demand-weighted (volume-weighted) average:
  SUM(rrp * demand) / SUM(demand). Say that you used the demand-weighted average.
- "Peak demand" means the highest 5-minute total demand in MW.
- Financial year means the Australian financial year, 1 July to 30 June.
- settlement_date is the END of each 5-minute interval in AEST (UTC+10, no daylight saving).
- Report prices to 2 decimal places and demand to 1 decimal place. Give one final number per question.

Five runs per wording	v0	v1
Accuracy (pass^1)	81.7%	85.0%
Wordings right on all five runs	37 of 72	44 of 72
Definitional questions correct	66.7%	88.9%
Aggregation questions correct	85.6%	75.6%
Wrong answers (not alternative definitions)	5.0%	8.3%
Wordings flipping right/wrong among answered runs	10	15
Runs routed to the lakehouse	2 of 360	45 of 360

The definitions worked where they were aimed. South Australia’s 2025 average price went from 0 correct in 15 runs to 12 of 15. They also leaked into questions about average demand: 16 of 45 rewrites asked for a “demand-weighted average demand”, against none in v0, and the agent dutifully computed one. Tasmania’s 2025 average demand came back as 1,069.0 MW instead of 1,049.7 MW in seven of 15 runs. Aggregation accuracy fell ten points.

The routing hint sent all 15 timestamp lookups to the lakehouse, and the 45 lakehouse runs were right 91% of the time. The Queensland “last quarter” question got no instruction, and two of its three wordings were wrong on all five v1 runs.

Microsoft’s own tooling reproduces the label problem. I put three of the shakiest v1 wordings through evaluate_data_agent in a Fabric notebook, with num_query_repeats=5 against the published v1 agent. It marked 12 of 15 answers as not matching, one right in five for each question. Three South Australia answers read “the demand-weighted average price … was $86.73/MWh”, and four NEM answers gave $47.16 the same way: the instruction’s label on the simple average. The SDK’s own _steps table shows how, with a rewrite asking for “the NEM-wide demand-weighted average wholesale price … using the predefined Average Price measure”.

The judge held up, with two sharp edges

I expected the SDK’s judge to be a source of noise. It wasn’t. Across all 1,080 runs in both configurations it replied with a bare “Yes”, “No” or “Unclear”. Its verdict matched my deterministic comparator on 1,079 runs; the other was a single “Unclear” on a wrong answer. To test the judge on its own, I held 60 stored v1 answers fixed (30 correct, 30 not) and re-ran the judge five times on each by branching off the stored answer with previous_response_id. All 300 replays returned the original verdict.

Two details in 0.1.33a0 are still worth knowing. First, the judge is the data agent itself: evaluate_data_agent sends the grading prompt as a second message in the same conversation, and the default prompt asks about “the most recent answer”. The verdict is then parsed with "no" in message.lower() before checking for “yes”, so a judge that ever says “Yes, but note…” scores a fail. Second, the critic_prompt example on Microsoft Learn uses an {actual_answer} placeholder the SDK never fills. It raises KeyError after the agent has answered, and because results are written only at the end, the run is lost. Here it is in a Fabric notebook, 31 seconds in:

Until that changes, write a custom critic prompt the way the default one works: refer to the most recent answer and use only {query} and {expected_answer}.

The harness

import json
from azure.identity import AzureCliCredential
from fabric.dataagent.client import FabricOpenAIResponses

client = FabricOpenAIResponses(
    artifact_name="<your-data-agent-name>",
    workspace_name="<your-workspace-id>",  # name or ID; an ID avoids lookup ambiguity
    ai_skill_stage="production",
    credential=AzureCliCredential(),  # fine for a few hundred calls; use a caching credential for more
)

def ask_fresh(question: str) -> dict:
    """One question, one brand-new conversation, every step kept."""
    conversation = client.conversations.create()
    response = client.responses.create(input=question, conversation=conversation.id)
    steps = [
        {"tool": item.name, "args": json.loads(item.arguments)}
        for item in response.output
        if item.type == "function_call"
    ]
    rewrite = next((s["args"].get("query") for s in steps if s["tool"].startswith("trace.")), None)
    code = [s["args"]["code"] for s in steps if "code" in s["args"]]
    return {"answer": response.output_text, "rewrite": rewrite, "code": code}

runs = [ask_fresh("What was Queensland's highest demand over the last quarter?") for _ in range(10)]
client.close()

Tool names like trace.analyze_semantic_model aren’t a documented contract; print your own agent’s item types first.

The pass^k estimator is one line per question wording:

from math import comb

def pass_hat_k(n: int, correct: int, k: int) -> float:
    """Unbiased estimate that k independent asks are all correct (tau-bench)."""
    if k > n:
        raise ValueError("k must be <= n")
    return comb(correct, k) / comb(n, k)

What I’d do on Monday

• Run every evaluation question at least five times in fresh conversations, and report pass^5 next to accuracy. evaluate_data_agent takes num_query_repeats and logs the rewrite in the function_queries column of its _steps table, so pass^k is a group-by over its output, not a custom harness.
• Ask each important question in two or three wordings. Wording moved the Queensland question from 8 in 10 right to 1 in 10.
• Name measures for their definition. Without instructions, 101 of 120 average-price runs picked the measure called Average Price.
• Write definitions into the agent instructions and back them with example queries, then rerun the same distribution, not a single pass.
• Grade numbers with a deterministic comparator, and keep the LLM judge for wording.
• Treat a decline as a failure in your metric but a safe one in your risk review.

Your data agent is still only as good as your data, as Marc says. It is also only as dependable as its least stable rewrite, and you won’t see that until you ask the same question more than once.