Architecture

Conduit is three Go services (gateway, collector and UI) and two SDKs (Go and Python), built on NATS JetStream, ClickHouse, VictoriaMetrics and Grafana Tempo. This document describes how data moves through the system, what each component guarantees, and how the data is stored.

Components

Component	Source	Role
SDKs	sdk/go, sdk/python	Create spans in the agent process and send them to the gateway over gRPC with mTLS
Gateway	cmd/gateway	Authenticates agents, validates spans, rate-limits each agent and publishes to NATS
Collector	cmd/collector	Consumes from NATS and writes to ClickHouse (required), VictoriaMetrics and Tempo (both best-effort)
UI	cmd/ui	Server-rendered trace list and trace detail, read from ClickHouse, with sign-in
NATS JetStream		A durable buffer between ingestion and storage. Spans, logs and metrics use separate work-queue streams (SPANS, LOGS, METRICS), so a backlog in one never blocks the others.
ClickHouse		The system of record for spans, LLM data and trace rollups
VictoriaMetrics		Agent metrics (the system of record for them) and span duration metrics
Tempo		A copy of each trace, in OTLP format, for Grafana

Data flow

sequenceDiagram
    autonumber
    participant SDK as Agent SDK
    participant GW as Gateway
    participant N as NATS (SPANS)
    participant C as Collector
    participant CH as ClickHouse
    participant VM as VictoriaMetrics
    participant T as Tempo
    SDK->>GW: IngestSpan (gRPC, mTLS client cert)
    Note over GW: auth → rate limit → validate
    GW->>N: publish span (JSON)
    GW-->>SDK: accepted / rejection reason
    N->>C: deliver (durable consumer)
    C->>CH: insert (batch API, nanosecond timestamps)
    alt ClickHouse insert fails
        C-->>N: Nak, redeliver in 5s
    else stored
        C->>VM: span duration (best effort)
        C->>T: OTLP/HTTP span (best effort, OTLP-shaped IDs only)
        C-->>N: Ack (deleted from the work queue)
    end

1. SDK. Spans are created in the agent process. Trace and span IDs are sized to the W3C and OTLP standards: 16-byte trace IDs and 8-byte span IDs. Context propagates through context.Context in Go and through contextvars in Python, so nesting is correct across goroutines, threads and asyncio tasks. Sending is synchronous, one RPC per span.
2. Gateway. Handles each request in this order:
   • Authenticate the TLS 1.3 client certificate against the configured CA. The agent’s identity is the certificate’s CommonName.
   • Rate-limit with a token bucket per identity, charged per telemetry item.
   • Validate the span, including ID charset and size, name, status, timestamps, attribute limits and LLM field bounds.
   • Publish to the SPANS stream.

   The wire API is Conduit’s own protobuf IngestionService (proto/ingestion.proto), not OTLP.

3. NATS JetStream. A work queue: each span is deleted once the collector acknowledges it, and the stream has a 7-day maximum age as a backstop.
4. Collector. At-least-once delivery into ClickHouse. A failed insert is retried after a 5-second delay, and a message that can’t be decoded is discarded for good. Writes to VictoriaMetrics and Tempo are best-effort: their failures are logged, not retried, so they can never duplicate rows in ClickHouse.
5. UI. Reads only from ClickHouse:
   • Trace list: the queries are always bounded by a time window, and paging uses a keyset cursor.
   • Trace detail: capped at 2000 spans per trace.
   • Sign-in: a login form with signed session cookies, or Google Cloud IAP.

Delivery and failure behavior

Failure	Effect
The gateway rejects a span (invalid, over the rate limit, unauthenticated)	The SDK gets an error. Span.End / span.end() raises or returns it, with the reason.
NATS is unavailable	The gateway fails the publish and returns an error to the SDK. Its /readyz reports nats: unavailable.
ClickHouse is unavailable	The collector naks, NATS keeps the span, and it’s redelivered every 5 seconds until ClickHouse returns, for up to 7 days.
VictoriaMetrics or Tempo is unavailable	Logged. The span is still stored in ClickHouse.
A collector instance restarts	The durable consumer resumes, and unacknowledged spans are redelivered.

The SDKs don’t retry. A failed send is reported to the caller, who decides what to do.

Storage schema

Schema management

The schema is defined by numbered migrations in internal/storage/migrations/, which are embedded in the binaries.

• Who applies them: the collector, at startup, before it reconciles retention. collector migrate does the same and exits, for running as a Job before a rollout. The UI never changes the schema.
• Bookkeeping: applied versions are recorded in schema_migrations with a SHA-256 checksum. If a migration is edited after it was applied, or the database has a version the binary doesn’t know, the collector refuses to start.
• Idempotency: ClickHouse has no DDL transactions or locks, so every migration must be idempotent (IF NOT EXISTS / IF EXISTS), and a test enforces it. That makes it safe for two collectors to start at once.
• Existing installs: these adopt migrations automatically. Migration 0001 is the original DDL, so it applies as a no-op and is recorded.

ClickHouse spans

Column	Type	Notes
trace_id, span_id, parent_id	String	parent_id is empty for root spans
agent_name	String	Always the mTLS CommonName, never a client-supplied value
span_name	String
start_time, end_time	DateTime64(9, 'UTC')	Nanosecond precision, written with the native batch protocol
duration_ms	UInt64
status	Enum8('ok', 'error', 'unset')
attributes	Map(String, String)	Includes error.message for failed spans
llm_model, llm_prompt, llm_completion	Nullable(String)	Set only on LLM spans
llm_input_tokens, llm_output_tokens	Nullable(UInt32)
llm_cost_usd	Nullable(Float64)
ingested_at	DateTime	Defaults to now()

The table is a MergeTree, partitioned by day and ordered by (agent_name, trace_id, start_time). Rows expire after SPAN_RETENTION_DAYS (default 90, range 1–3650).

• Who sets it: the collector owns retention, as part of owning the schema. On startup it compares the table’s TTL with the setting and runs ALTER TABLE spans MODIFY TTL … only if they differ. Shortening retention lets ClickHouse delete rows that are now past it.
• The UI only creates the tables if they’re missing, and never alters them.

ClickHouse logs

Log records sent in IngestBatch. Each is validated at the gateway: the severity enum, a body of 1 byte to 32 KiB, a timestamp no more than an hour in the future, well-formed optional trace_id and span_id (a span needs a trace), and the shared attribute limits. Agent identity comes from the certificate, as for spans.

Column	Type	Notes
timestamp	DateTime64(9, 'UTC')	Defaults to the receive time if the client sends none
agent_name	String	The mTLS CommonName
severity	Enum8('unspecified', 'debug', 'info', 'warn', 'error')
body	String
attributes	Map(String, String)
trace_id, span_id	String	Empty unless the log belongs to a trace or span

The table is partitioned by day and ordered by (agent_name, timestamp). SPAN_RETENTION_DAYS applies to it as well.

VictoriaMetrics: agent metrics

Gauges and counters sent in IngestBatch are written to VictoriaMetrics. The gateway validates them:

• Names must be valid Prometheus names. conduit_* is reserved for Conduit’s own series, so agents can’t forge them, and __* is reserved too.
• Labels: at most 32, with Prometheus-style keys and values of at most 1 KiB.
• Values must be finite, and counters must be ≥ 0.
• The agent label is always set from the certificate.

For metrics, VictoriaMetrics is the system of record, so a failed write is retried, unlike the best-effort span-duration export. Label values are escaped for the Prometheus text format (backslash, quote and newline), so a value can’t inject extra samples.

ClickHouse trace_summary

A materialized view (AggregatingMergeTree) with per-trace rollups: start, end, span count, error count, tokens and cost. It uses SimpleAggregateFunction columns, so rows from separate inserts merge correctly. Query it with the matching aggregate functions and GROUP BY trace_id. Nothing in the UI reads it yet.

What the UI reads

Query	Bounded by
Trace list	A time window (1h, 24h or 7d), up to 100 traces per page, keyset paging
Trace detail	Up to 2000 spans, ordered by start time
Agent suggestions	A time window, up to 200 names
Logs page	A time window, up to 200 records, newest first. Filters by agent, minimum severity, case-insensitive text and trace.

The UI never reads prompts or completions. See Roadmap: open decisions.

Sensitive data

By default, prompts and completions are stored in ClickHouse whenever an SDK records them with RecordLLMCall / record_llm_call.

To never store them, set STORE_LLM_PAYLOADS=false on the gateway.

• What’s dropped: prompts and completions are removed at the gateway, before anything is published to NATS, logged or stored.
• What’s kept: the model, token counts and cost are always kept.
• Validation: a span whose prompt would have been too large is still accepted, because the prompt is dropped before validation.
• Typos: an unrecognized value stops the gateway at startup, rather than silently storing payloads.

With storage on, treat ClickHouse access as access to prompt data. Payloads already stored aren’t removed when you turn the switch off.

Operational endpoints

Every service exposes /livez and /readyz. The gateway and collector serve them on HEALTH_ADDR (default :8081); the UI serves them on its main port. See Deployment: health checks.

Design decisions

• Separate health port for the gateway. Its gRPC port requires mTLS client certificates, which Kubernetes and compose probes can’t present.
• Generated code is committed. The Go and Python protobuf stubs are in the repository, so a checkout builds with nothing but Go. CI fails if they drift from the .proto files.
• No JavaScript framework. The UI uses html/template, htmx and two small scripts, all embedded in the binary. Its strict CSP rules out inline scripts and styles.
• ClickHouse as the system of record. Only ClickHouse failures trigger redelivery. Treating the other sinks as best-effort keeps duplicate rows out of ClickHouse.