Architecture

System overview

Dunetrace is a pipeline of independent services communicating through a shared Postgres database — no message broker. Each service does one job.

Agent Code
  └─▶ Dunetrace SDK       (raw event data → ingest events + OTel spans)
        └─▶ Ingest API    (POST /v1/ingest → Postgres, returns 202)
                ├─▶ Detector     (poll → RunState → 29 detectors → signals)
                ├─▶ Semantic     (optional, off by default → semantic signals)
                ├─▶ Integrations (optional, off by default → pulled evaluations)
                ├─▶ Alerts       (poll → explain → Slack / webhook)
                └─▶ Customer API (runs, signals, explanations → dashboard)
        ├─▶ stdout NDJSON (emit_as_json=True → Loki / Grafana Alloy)
        └─▶ OTel exporter (otel_exporter=… → Tempo / Honeycomb / Datadog)

Services

Ingest API · port 8001

The entry point for all SDK traffic. Its only job is to accept events as fast as possible and not lose them. Validates the schema, authenticates via the api_keys table, returns 202 Accepted before touching the database, then writes in a background task.

Why the 202 before writing? Your agent is waiting. Round-trip latency should be as short as possible. Validation is synchronous; persistence is async.

Detector worker

Background polling loop, every 5 seconds. The only process that runs detection logic.

1. Fetches runs completed since last poll plus runs stalled longer than 90s
2. Skips runs already in processed_runs
3. Reconstructs RunState by replaying events
4. Runs the Tier 1 detectors against the RunState. PROMPT_INJECTION_SIGNAL is handled by the SDK on raw input; the worker extracts the evidence from the run.started payload.
5. Writes any FailureSignal rows
6. UPSERTs the issues table for each fired signal and advances the clean-run counter; auto-resolves after 5 consecutive clean runs
7. Marks the run processed

Why polling instead of streaming? A polling worker needs no message broker, survives restarts gracefully, and is trivial to reason about. At sub-100 runs/sec, 5-second latency is acceptable.

Poll watermark. Run discovery is bounded by a persisted per-shard watermark, so each poll scans only recent events rather than the entire retention window. This also keeps the monthly events partitions prunable — the partition key is received_at, and a query that never mentions it cannot prune. The window is expressed as "runs touched by a recent event" rather than "recent terminal events", which is what keeps late-arriving events triggering a re-detection. The watermark only advances after a poll that fully drains its backlog, so a busy or restarted worker can never step over unprocessed runs. Tune the re-scan overlap with WATERMARK_GRACE_SECS (default 3600).

Horizontal scaling. Set SHARD_COUNT=N and run N replicas with distinct SHARD_INDEX values; each polls only the runs whose agent_id hashes to its bucket. A misconfigured replica fails at startup rather than silently claiming no work.

Explain layer

A library — not a service. Imported by both the alerts worker and the customer API. Takes a FailureSignal, returns an Explanation in under 1 ms. Uses deterministic string templates, not LLM calls.

Three reasons for no LLM: latency (templates are instant), cost (zero per-signal API cost), consistency (same signal → same explanation, makes testing predictable).

Alerts worker

Background polling loop, every 10 seconds. The only process that sends external notifications. Fetches unalerted signals, computes rate context concurrently, calls explain(), formats for Slack Block Kit or webhook JSON, POSTs with exponential backoff up to 3 attempts. Marks alerted=TRUE only after at least one destination succeeds.

Running more than one replica. Signals are claimed before delivery: the row is stamped in the same statement that selects it, using FOR UPDATE SKIP LOCKED. alerted can't serve that purpose on its own, because it is only set after a successful send — two workers scanning the same window would both see the signal outstanding and both deliver it. Claims expire after CLAIM_TIMEOUT_SECS (default 300) so a worker that dies mid-delivery doesn't strand its rows, and a poll that ends without delivering hands its claims back immediately.

To scale throughput, shard as well: SHARD_COUNT / SHARD_INDEX, or ALERTS_SHARD_COUNT / ALERTS_SHARD_INDEX to scale the alerts worker independently of the detector. Sharding on agent_id matters beyond throughput — one alert is sent per (org_id, agent_id, failure_type) group, so a group has to stay whole on a single worker.

Customer API · port 8002

Read-only FastAPI service. Powers the dashboard and any customer integrations. All endpoints require Authorization: Bearer <api_key>, except in AUTH_MODE=dev. Signal responses include the full explanation inline.

See Semantic Evaluation and Integrations for the full endpoint reference on each of those surfaces.

Dashboard · port 3000

A single-page HTML app served by nginx. No build step — plain HTML/CSS/JS fetching from the Customer API. Auto-refreshes every 15 seconds. All data is computed client-side.

SDK framework integrations

Two first-class SDKs send events to the same ingest API — runs from either appear together in the dashboard under the same agent_id.

The Python SDK ships framework integrations for LangChain, CrewAI, AutoGen, and Langfuse. The TypeScript SDK supports HTTP ingest and Loki NDJSON; OTel spans are Python-only.

SDK output modes

Three independent output paths that can be combined:

All three can be active at once. OTel and NDJSON are zero-cost when disabled. Pass endpoint=None for OTel-only or Loki-only deployments.

Database schema

Seven tables. Event payloads land in the payload JSONB column; when you self-host, that data never leaves your own PostgreSQL.

CREATE TABLE events (
    id             BIGSERIAL PRIMARY KEY,
    batch_id       TEXT             NOT NULL,
    event_type     TEXT             NOT NULL,
    run_id         TEXT             NOT NULL,
    agent_id       TEXT             NOT NULL,
    agent_version  TEXT             NOT NULL,
    step_index     INTEGER          NOT NULL,
    timestamp      DOUBLE PRECISION NOT NULL,
    payload        JSONB            NOT NULL,
    parent_run_id  TEXT,
    received_at    TIMESTAMPTZ      NOT NULL DEFAULT NOW()
);

CREATE TABLE failure_signals (
    id             BIGSERIAL PRIMARY KEY,
    failure_type   TEXT        NOT NULL,
    severity       TEXT        NOT NULL,
    run_id         TEXT        NOT NULL,
    agent_id       TEXT        NOT NULL,
    agent_version  TEXT        NOT NULL,
    step_index     INTEGER     NOT NULL,
    confidence     REAL        NOT NULL,
    evidence       JSONB       NOT NULL,
    detected_at    TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    shadow         BOOLEAN     NOT NULL DEFAULT TRUE,
    alerted        BOOLEAN     NOT NULL DEFAULT FALSE
);

CREATE TABLE processed_runs (…);
CREATE TABLE api_keys (…);
CREATE TABLE issues (…);
CREATE TABLE digest_log (…);

CREATE TABLE fixes (
    id                    BIGSERIAL    PRIMARY KEY,
    run_id                TEXT         NOT NULL,
    signal_id             BIGINT       NOT NULL,
    fix_content           TEXT         NOT NULL,
    fix_type              TEXT         NOT NULL DEFAULT 'prompt_addition',
    applied_via           TEXT         NOT NULL,   -- 'github_pr' or 'clipboard'
    langfuse_prompt_name  TEXT,                    -- historical column, from the removed Langfuse apply-fix flow — always NULL now
    langfuse_version      INTEGER,                 -- historical column, repurposed to store the GitHub PR number when applied_via='github_pr'
    applied_at            TIMESTAMPTZ  NOT NULL DEFAULT NOW()
);

Performance

Agent overhead: under 500 μs per run with default HTTP ingest. The drain thread is entirely background. Even under backpressure (ingest API down), the ring buffer drops the oldest events rather than blocking the agent.

Failure modes

• Ingest API down — drain thread drops unshippable events. Agent never blocks. Events during outage are lost.
• Detector worker down — runs queue up. When the worker restarts, it catches up. Signals delayed but not lost.
• Postgres down — ingest returns 503. SDK buffers up to 10,000 events, then rolls. Observability data loss is acceptable during DB outages.
• Alerts worker down — signals accumulate as alerted=FALSE. On restart, delivery resumes. At-least-once.