Structural detection (pillar 2) catches loops, retries, and cost spikes with zero LLM calls. But some failures aren't structural — the agent's tool-call pattern looks completely fine, and it still hallucinated a fact, didn't actually finish the task, or left the user visibly frustrated across a conversation. That's what semantic evaluation is for.
How a run gets evaluated
- A background worker polls for completed runs on a 5-second interval.
- Adaptive sampling decides whether this specific run gets evaluated — see below.
- If sampled in, the configured evaluators run against the run's own stored events.
- Any finding is grouped into its recurring pattern, checked against accumulated false-positive feedback, optionally re-checked by a second model for high-stakes findings, and written alongside your structural signals — same dashboard, same alert channel.
Adaptive sampling
Evaluating every single run through an LLM doesn't scale, so most runs are sampled rather than always evaluated:
| Rule | Default rate | Why |
|---|---|---|
| A structural detector already fired | 100% | Augment a known problem with semantic context — highest value per evaluation |
Agent flagged semantic_critical | 100% | You said this agent's correctness matters enough to always check |
| Run had a retrieval event | 20% | RAG is the highest hallucination-risk pattern |
| Everything else | 5% | Baseline coverage without evaluating every run |
Sampling is deterministic — the same run always gets the same decision, no flapping on retries. Per-agent overrides let you set a custom sample rate, a monthly evaluation budget, and which evaluators run.
Evaluators
Two evaluators ship today, both backed by DeepEval:
- Hallucination — did the agent state something as fact that its own context doesn't support
- Task Completion — did the agent actually do what it was asked, not just respond plausibly
A third, User Frustration, operates on entire conversations rather than single runs — it reads the last several runs in a conversation and evaluates cross-turn frustration signals no single-run evaluator can see.
False positive management
A semantic-evaluation feature that erodes trust with noisy alerts is worse than no feature at all. Four mitigations, all built in:
- Confidence scoring — every finding has a 0.0–1.0 confidence score. Below-floor findings are stored and visible but never alerted.
- Grouping and dedup — findings from the same agent, evaluator, and root-cause pattern are grouped into one recurring issue, not N separate alerts.
- Feedback capture — mark a finding "not a real issue" from the dashboard. Enough false-positive marks on a pattern suppress or downweight future matches.
- Second-opinion evaluation — a high-severity Hallucination finding gets independently re-checked by a different model before it stays HIGH. Disagreement downgrades it to MEDIUM.
Billing and quotas
Two independent monthly quotas, both enforced before an evaluation runs — a skipped evaluation costs nothing: an org-wide cap across every agent, and an optional per-agent budget. Usage and a month-end cost projection are available from the dashboard.
What this is not
- Not a replacement for structural detection — it augments it. The 23 structural detectors remain the always-on, zero-cost, zero-LLM first line.
- Not a runtime guardrail — semantic evaluation can never trigger a policy. By the time a semantic finding exists, the run it describes is already over.
- Not required — disabled by default. Everything else in Dunetrace works identically without it.
Already running Langfuse, LangSmith, or Braintrust?
Semantic evaluation is Dunetrace's own native option — you don't have to use it. Pull your existing tracer's evaluation results in instead, or alongside it, on the same Integrations page.