# Conduit documentation (full)

> Secure, self-hosted observability for AI agents: traces, logs and metrics over mTLS, stored in ClickHouse and VictoriaMetrics, with a server-rendered UI. Go services, Go and Python SDKs.

---

<!-- Source: https://tjscanlan.github.io/conduit/index.md -->

# Conduit

> Secure, self-hosted observability for AI agents: traces, logs and metrics over mTLS, stored in ClickHouse and VictoriaMetrics, with a server-rendered UI. Go services, Go and Python SDKs.

Conduit collects traces from AI agents. Each trace is a run broken into spans for planning, tool calls and LLM calls, with status, errors, token counts and cost. Conduit stores the traces and shows them in a fast, server-rendered UI. It's designed for teams that want agent telemetry inside their own infrastructure, with a small and auditable attack surface.

## Highlights

- **Identity you can trust.** Agents authenticate with mTLS client certificates, and the certificate's CommonName is the agent's identity. An agent can't spoof it with an attribute.
- **Hardened ingestion.** Every span is validated before it reaches storage. Rate limits apply per agent and per telemetry item, so batching can't get around them.
- **LLM-aware traces.** Model, tokens, cost and errors are recorded per span and rolled up per trace. Prompts and completions are stored separately from span metadata.
- **A UI that's quick to read.** One row per trace, filtering by time, agent and errors, stable paging, and a span tree with a waterfall timeline.
- **Small and auditable.** Each service is a static Go binary in a `FROM scratch` image. The UI uses no JavaScript framework, has a strict Content-Security-Policy, and embeds its own assets.
- **Production basics included.** UI sign-in with a login form or Google Cloud IAP, health and readiness endpoints, Kubernetes manifests, and a CI pipeline that runs the whole stack end to end.

## How it works

```mermaid
flowchart LR
    A["Agent<br/>Go or Python SDK"] -- "gRPC + mTLS :4317" --> G["Gateway<br/>auth · validation · rate limits"]
    G -- "JetStream (work queue)" --> N[("NATS")]
    N --> C["Collector"]
    C --> CH[("ClickHouse<br/>spans, LLM data")]
    C -. "best effort" .-> VM[("VictoriaMetrics<br/>span durations")]
    C -. "OTLP/HTTP, best effort" .-> T[("Tempo<br/>traces")]
    CH --> U["UI<br/>login or GCP IAP"]
```

The gateway accepts spans only from agents holding a certificate signed by your CA. The collector writes each span to ClickHouse, the system of record, and redelivers it if that write fails. It also copies spans to VictoriaMetrics and Tempo on a best-effort basis. The UI reads from ClickHouse. See [Architecture](architecture.md) for details.

## Quick start

You need Docker (Compose v2, 2 GiB of memory). One command runs the whole
platform from the signed release images:

```bash
git clone https://github.com/tjscanlan/conduit && cd conduit
make quickstart          # or: ./deployments/quickstart/quickstart.sh
```

It generates a dev CA, agent certificates, and a UI login, waits until
everything is healthy, and prints your login. Open <http://localhost:8080>:
a simulated `demo-agent` is already sending traces, and a Get started page
walks you through sending your own agent's first trace. The banner also shows how
to send a trace from your own code as `my-agent`; CI checks that a cold start
reaches the first trace in under five minutes.

Releases also ship [`conduitctl`](cli.md), the same
quickstart as a single binary with no checkout needed: `conduitctl
quickstart`, plus `agent-cert` and `doctor`.

`quickstart.sh status` shows the login again, `down` stops the stack (keeping
data), and `reset` deletes everything. Next, run one of the
[examples](examples.md) (hello-trace, tool calling, asyncio, Claude,
OpenAI) or instrument your own agent with the [Go SDK](sdk-go.md) or
[Python SDK](sdk-python.md). To work
on Conduit itself (building images from source), see
[Deployment](deployment.md#local-docker-compose-from-source).

## Documentation

**Read the docs at <https://tjscanlan.github.io/conduit/>**, including the [visual roadmap](https://tjscanlan.github.io/conduit/roadmap.html). The same pages are in [`docs/`]().

**For AI agents:**
- [`llms.txt`](https://tjscanlan.github.io/conduit/llms.txt) indexes the docs and links each page's raw Markdown.
- [`llms-full.txt`](https://tjscanlan.github.io/conduit/llms-full.txt) has all the docs in one file.
- [`AGENTS.md`](AGENTS.md) tells coding agents how to build, test and change this repo.

| Guide | What's in it |
|-------|--------------|
| [Architecture](architecture.md) | Components, data flow, delivery guarantees and storage schema |
| [Security](security.md) | The threat model, mTLS identity, rate limiting, UI authentication and browser hardening |
| [Deployment](deployment.md) | Docker Compose, Kubernetes, health checks, upgrades and configuration reference |
| [Go SDK](sdk-go.md) · [Python SDK](sdk-python.md) | Instrumenting agents |
| [Roadmap](roadmap.md) | What's shipped, what's next and the open decisions |
| [Releasing](releasing.md) | Versioning policy, verifying signed images, upgrading and cutting a release |
| [Changelog](CHANGELOG.md) | What changed in each release |
| [Contributing](contributing.md) | Development setup, tests, code generation and pull requests |
| [Brand](brand.md) | Logo, palette and accessibility rules for the UI |

## Project status

Conduit is **pre-1.0**. The core pipeline is tested end to end in CI on every push: SDK → gateway → NATS → collector → ClickHouse, VictoriaMetrics, Tempo → UI. APIs and the storage schema may still change between releases. See the [roadmap](roadmap.md).

## License

[Apache License 2.0](LICENSE)

---

<!-- Source: https://tjscanlan.github.io/conduit/architecture.md -->

# Architecture

Conduit is three Go services (gateway, collector and UI) and two SDKs (Go
and Python), built on NATS JetStream, ClickHouse, VictoriaMetrics and Grafana
Tempo. This document describes how data moves through the system, what each
component guarantees, and how the data is stored.

## Components

| Component | Source | Role |
|-----------|--------|------|
| **SDKs** | `sdk/go`, `sdk/python` | Create spans in the agent process and send them to the gateway over gRPC with mTLS |
| **Gateway** | `cmd/gateway` | Authenticates agents, validates spans, rate-limits each agent and publishes to NATS |
| **Collector** | `cmd/collector` | Consumes from NATS and writes to ClickHouse (required), VictoriaMetrics and Tempo (both best-effort) |
| **UI** | `cmd/ui` | Server-rendered trace list and trace detail, read from ClickHouse, with sign-in |
| NATS JetStream | | A durable buffer between ingestion and storage. Spans, logs and metrics use separate work-queue streams (`SPANS`, `LOGS`, `METRICS`), so a backlog in one never blocks the others. |
| ClickHouse | | The system of record for spans, LLM data and trace rollups |
| VictoriaMetrics | | Agent metrics (the system of record for them) and span duration metrics |
| Tempo | | A copy of each trace, in OTLP format, for Grafana |

## Data flow

```mermaid
sequenceDiagram
    autonumber
    participant SDK as Agent SDK
    participant GW as Gateway
    participant N as NATS (SPANS)
    participant C as Collector
    participant CH as ClickHouse
    participant VM as VictoriaMetrics
    participant T as Tempo
    SDK->>GW: IngestSpan (gRPC, mTLS client cert)
    Note over GW: auth → rate limit → validate
    GW->>N: publish span (JSON)
    GW-->>SDK: accepted / rejection reason
    N->>C: deliver (durable consumer)
    C->>CH: insert (batch API, nanosecond timestamps)
    alt ClickHouse insert fails
        C-->>N: Nak, redeliver in 5s
    else stored
        C->>VM: span duration (best effort)
        C->>T: OTLP/HTTP span (best effort, OTLP-shaped IDs only)
        C-->>N: Ack (deleted from the work queue)
    end
```

1. **SDK.** Spans are created in the agent process. Trace and span IDs are
   sized to the W3C and OTLP standards: 16-byte trace IDs and 8-byte span IDs.
   Context propagates through `context.Context` in Go and through
   `contextvars` in Python, so nesting is correct across goroutines,
   threads and asyncio tasks. Sending is synchronous, one RPC per span.
2. **Gateway.** Handles each request in this order:
   - **Authenticate** the TLS 1.3 client certificate against the configured
     CA. The agent's identity is the certificate's CommonName.
   - **Rate-limit** with a token bucket per identity, charged per telemetry
     item.
   - **Validate** the span, including ID charset and size, name, status,
     timestamps, attribute limits and LLM field bounds.
   - **Publish** to the `SPANS` stream.

   The wire API is Conduit's own protobuf `IngestionService`
   (`proto/ingestion.proto`), not OTLP.
3. **NATS JetStream.** A work queue: each span is deleted once the
   collector acknowledges it, and the stream has a 7-day maximum age as a
   backstop.
4. **Collector.** At-least-once delivery into ClickHouse. A failed insert is
   retried after a 5-second delay, and a message that can't be decoded is
   discarded for good. Writes to VictoriaMetrics and Tempo are best-effort:
   their failures are logged, not retried, so they can never duplicate rows
   in ClickHouse.
5. **UI.** Reads only from ClickHouse:
   - **Trace list:** the queries are always bounded by a time window, and
     paging uses a keyset cursor.
   - **Trace detail:** capped at 2000 spans per trace.
   - **Sign-in:** a login form with signed session cookies, or Google Cloud
     IAP.

## Delivery and failure behavior

| Failure | Effect |
|---------|--------|
| The gateway rejects a span (invalid, over the rate limit, unauthenticated) | The SDK gets an error. `Span.End` / `span.end()` raises or returns it, with the reason. |
| NATS is unavailable | The gateway fails the publish and returns an error to the SDK. Its `/readyz` reports `nats: unavailable`. |
| ClickHouse is unavailable | The collector naks, NATS keeps the span, and it's redelivered every 5 seconds until ClickHouse returns, for up to 7 days. |
| VictoriaMetrics or Tempo is unavailable | Logged. The span is still stored in ClickHouse. |
| A collector instance restarts | The durable consumer resumes, and unacknowledged spans are redelivered. |

The SDKs don't retry. A failed send is reported to the caller, who decides
what to do.

## Storage schema

### Schema management

The schema is defined by numbered migrations in
`internal/storage/migrations/`, which are embedded in the binaries.
- **Who applies them:** the collector, at startup, before it reconciles
  retention. `collector migrate` does the same and exits, for running as a
  Job before a rollout. The UI never changes the schema.
- **Bookkeeping:** applied versions are recorded in `schema_migrations` with
  a SHA-256 checksum. If a migration is edited after it was applied, or the
  database has a version the binary doesn't know, the collector refuses to
  start.
- **Idempotency:** ClickHouse has no DDL transactions or locks, so every
  migration must be idempotent (`IF NOT EXISTS` / `IF EXISTS`), and a test
  enforces it. That makes it safe for two collectors to start at once.
- **Existing installs:** these adopt migrations automatically. Migration
  0001 is the original DDL, so it applies as a no-op and is recorded.

### ClickHouse `spans`

| Column | Type | Notes |
|--------|------|-------|
| `trace_id`, `span_id`, `parent_id` | `String` | `parent_id` is empty for root spans |
| `agent_name` | `String` | Always the mTLS CommonName, never a client-supplied value |
| `span_name` | `String` | |
| `start_time`, `end_time` | `DateTime64(9, 'UTC')` | Nanosecond precision, written with the native batch protocol |
| `duration_ms` | `UInt64` | |
| `status` | `Enum8('ok', 'error', 'unset')` | |
| `attributes` | `Map(String, String)` | Includes `error.message` for failed spans |
| `llm_model`, `llm_prompt`, `llm_completion` | `Nullable(String)` | Set only on LLM spans |
| `llm_input_tokens`, `llm_output_tokens` | `Nullable(UInt32)` | |
| `llm_cost_usd` | `Nullable(Float64)` | |
| `ingested_at` | `DateTime` | Defaults to `now()` |

The table is a `MergeTree`, partitioned by day and ordered by
`(agent_name, trace_id, start_time)`. Rows expire after
**`SPAN_RETENTION_DAYS`** (default 90, range 1–3650).
- **Who sets it:** the collector owns retention, as part of owning the schema. On startup it compares the
  table's TTL with the setting and runs `ALTER TABLE spans MODIFY TTL …`
  only if they differ. Shortening retention lets ClickHouse delete rows
  that are now past it.
- **The UI** only creates the tables if they're missing, and never alters
  them.

### ClickHouse `logs`

Log records sent in `IngestBatch`. Each is validated at the gateway: the
severity enum, a body of 1 byte to 32 KiB, a timestamp no more than an hour
in the future, well-formed optional `trace_id` and `span_id` (a span needs
a trace), and the shared attribute limits. Agent identity comes from the
certificate, as for spans.

| Column | Type | Notes |
|--------|------|-------|
| `timestamp` | `DateTime64(9, 'UTC')` | Defaults to the receive time if the client sends none |
| `agent_name` | `String` | The mTLS CommonName |
| `severity` | `Enum8('unspecified', 'debug', 'info', 'warn', 'error')` | |
| `body` | `String` | |
| `attributes` | `Map(String, String)` | |
| `trace_id`, `span_id` | `String` | Empty unless the log belongs to a trace or span |

The table is partitioned by day and ordered by `(agent_name, timestamp)`.
`SPAN_RETENTION_DAYS` applies to it as well.

### VictoriaMetrics: agent metrics

Gauges and counters sent in `IngestBatch` are written to VictoriaMetrics.
The gateway validates them:
- **Names** must be valid Prometheus names. `conduit_*` is reserved for
  Conduit's own series, so agents can't forge them, and `__*` is reserved
  too.
- **Labels:** at most 32, with Prometheus-style keys and values of at most
  1 KiB.
- **Values** must be finite, and counters must be ≥ 0.
- **The `agent` label** is always set from the certificate.

For metrics, VictoriaMetrics is the system of record, so a failed write is
retried, unlike the best-effort span-duration export. Label values are
escaped for the Prometheus text format (backslash, quote and newline), so a
value can't inject extra samples.

### ClickHouse `trace_summary`

A materialized view (`AggregatingMergeTree`) with per-trace rollups: start,
end, span count, error count, tokens and cost. It uses `SimpleAggregateFunction`
columns, so rows from separate inserts merge correctly. Query it with the
matching aggregate functions and `GROUP BY trace_id`. Nothing in the UI
reads it yet.

### What the UI reads

| Query | Bounded by |
|-------|-----------|
| Trace list | A time window (1h, 24h or 7d), up to 100 traces per page, keyset paging |
| Trace detail | Up to 2000 spans, ordered by start time |
| Agent suggestions | A time window, up to 200 names |
| Logs page | A time window, up to 200 records, newest first. Filters by agent, minimum severity, case-insensitive text and trace. |

The UI never reads prompts or completions. See [Roadmap: open decisions](roadmap.md#open-decisions).

## Sensitive data

By default, prompts and completions are stored in ClickHouse whenever an SDK
records them with `RecordLLMCall` / `record_llm_call`.

**To never store them, set `STORE_LLM_PAYLOADS=false` on the gateway.**
- **What's dropped:** prompts and completions are removed at the gateway,
  before anything is published to NATS, logged or stored.
- **What's kept:** the model, token counts and cost are always kept.
- **Validation:** a span whose prompt would have been too large is still
  accepted, because the prompt is dropped before validation.
- **Typos:** an unrecognized value stops the gateway at startup, rather
  than silently storing payloads.

With storage on, treat ClickHouse access as access to prompt data.
Payloads already stored aren't removed when you turn the switch off.

## Operational endpoints

Every service exposes `/livez` and `/readyz`. The gateway and collector serve
them on `HEALTH_ADDR` (default `:8081`); the UI serves them on its main port.
See [Deployment: health checks](deployment.md#health-checks).

## Design decisions

- **Separate health port for the gateway.** Its gRPC port requires mTLS
  client certificates, which Kubernetes and compose probes can't present.
- **Generated code is committed.** The Go and Python protobuf stubs are in
  the repository, so a checkout builds with nothing but Go. CI fails if
  they drift from the `.proto` files.
- **No JavaScript framework.** The UI uses `html/template`, htmx and two
  small scripts, all embedded in the binary. Its strict CSP rules out
  inline scripts and styles.
- **ClickHouse as the system of record.** Only ClickHouse failures trigger
  redelivery. Treating the other sinks as best-effort keeps duplicate rows
  out of ClickHouse.

---

<!-- Source: https://tjscanlan.github.io/conduit/deployment.md -->

# Deployment

Conduit runs as three binaries, `gateway`, `collector` and `ui`. Each is
shipped as a `FROM scratch` image built from `deployments/docker/Dockerfile`
with `--build-arg BINARY=<name>`. The binaries depend on NATS JetStream,
ClickHouse, VictoriaMetrics and Tempo.

| Component | Port | Talks to |
|-----------|------|----------|
| gateway | 4317 (gRPC, mTLS); 8081 health | NATS |
| collector | 8081 health | NATS, ClickHouse, VictoriaMetrics, Tempo (OTLP/HTTP :4318) |
| ui | 8080 (HTTP; `/healthz` and `/readyz` unauthenticated) | ClickHouse |

## Health checks

Every service exposes unauthenticated health endpoints over plain HTTP:

| Endpoint | Meaning | Gateway / collector | UI |
|----------|---------|---------------------|----|
| `/livez` (UI: `/healthz`) | The process is up | `HEALTH_ADDR`, default `:8081` | main port, 8080 |
| `/readyz` | Dependencies are reachable: NATS for the gateway; NATS and ClickHouse for the collector; ClickHouse for the UI | `HEALTH_ADDR` | main port |

- **Why a separate port for the gateway:** its gRPC port requires mTLS client
  certificates, which probes can't present.
- **What `/readyz` returns:** a 200 or a 503, with each check reported only
  as `ok` or `unavailable`. The underlying errors go to the service log, so
  the endpoint never exposes hostnames or credentials.
- **Health checks in scratch images:** the images have no shell or curl, so
  each binary includes `/app healthcheck`, which calls its own `/readyz` and
  exits 0 or 1. Compose health checks use it.
- **Keep `HEALTH_ADDR` internal:** don't expose it outside the cluster.

## Local: quickstart

`make quickstart` (or `deployments/quickstart/quickstart.sh`) runs the
released images with nothing but Docker:

- **Credentials:** the one-shot `conduit-init` container writes a dev CA, the
  gateway certificate, client certificates for `demo-agent` and `my-agent`,
  and a random UI login into `deployments/quickstart/.conduit/` (gitignored).
  It runs as your user, so the agent keys are readable by your own code, and
  it keeps existing files on later runs.
- **Secrets:** the script writes a random ClickHouse password and session key
  to `deployments/quickstart/.env` (gitignored, owner-only).
- **Network:** only the UI (`127.0.0.1:8080`) and the gateway
  (`127.0.0.1:4317`) listen, on loopback. Change the ports with
  `CONDUIT_UI_PORT` and `CONDUIT_GATEWAY_PORT`.
- **Small machines:** ClickHouse uses a low-memory profile and system log
  tables are off, so the stack fits a 2 GiB Docker VM. Span retention is
  7 days.
- **Demo data:** a simulated `demo-agent` (no model calls; every span has
  `demo=true`) sends a run every 8 seconds. Start without it using
  `CONDUIT_DEMO_AGENT=0`, or stop it with
  `docker compose -p conduit-quickstart stop demo-agent`.
- **Version:** images default to `v0.2.0`; set `CONDUIT_VERSION` to pick
  another release.

**Get started:** until your own agent sends a trace, opening the UI lands
on a Get started page (`/start`). It checks that the stack and the sample
data are working, and shows the commands that make an agent certificate
(on the command line: the UI never serves keys). It then gives SDK
snippets and waits, live, for the first trace from an agent other than
`demo-agent`, linking to it when it arrives. You can hide it from the page.

`CONDUIT_FROM_SOURCE=1 quickstart.sh` builds the gateway, collector and UI
from your checkout instead of pulling the release, one image at a time so
it fits a 2 GiB Docker VM.

`quickstart.sh status` reprints the login, `down` stops the stack, and
`reset` deletes its data and credentials. `scripts/quickstart-check.sh` is
the CI check: from a cold start, the first trace must be visible in the UI
within 300 seconds.

The quickstart is for local evaluation only. Its CA key sits next to the
certificates it signs; use real PKI (below) for anything shared.

## Local: docker compose (from source)

For working on Conduit itself, this stack builds the images from your
checkout.

```bash
./scripts/gen-certs.sh                         # dev CA, gateway cert, example agent cert
./scripts/gen-certs.sh agent my-agent          # more agents as needed

# UI login (see security.md for other auth modes)
printf '%s' 'a-long-dev-password' | go run ./cmd/ui hash-password \
  | sed 's/^/admin:/' > deployments/docker/ui-users

export CLICKHOUSE_PASSWORD=$(openssl rand -hex 16)
docker compose -f deployments/docker/docker-compose.yml up --build
```

Then open http://localhost:8080 and sign in as `admin`. Agents connect to
`localhost:4317` using `certs/ca.crt` and their own client cert (see
[sdk-go.md](sdk-go.md)).

To check the whole pipeline end to end, run
`UI_USER=admin UI_PASSWORD=… ./scripts/smoke-test.sh` with the same
`CLICKHOUSE_PASSWORD` exported. It sends a trace with the Go SDK and checks
that it reaches ClickHouse, VictoriaMetrics, Tempo and the UI. CI runs it on
every push.

Compose sets `UI_COOKIE_INSECURE=true` because it serves plain HTTP on
localhost. Never set it anywhere else. Without `UI_SESSION_KEY`, sessions
end whenever the UI restarts.

## Kubernetes

The manifests in `deployments/k8s/` assume a `conduit` namespace,
[cert-manager](https://cert-manager.io), and NATS, ClickHouse,
VictoriaMetrics and Tempo reachable at the hostnames in `configmap.yaml`.

### 1. Certificates (`mtls-cert.yaml`)

The file creates a root CA, `conduit-ca-issuer`, a gateway server cert
(secret `conduit-mtls-certs`) and an example agent cert. The gateway and
the agents all chain to that one CA. The gateway Deployment maps the
secret's `tls.crt`, `tls.key` and `ca.crt` onto `server.crt`, `server.key`
and `ca.crt`.

For production, swap the bootstrap self-signed root CA for your PKI but
keep the issuer name. Issue one `Certificate` per agent with
`usages: [client auth]`. Its `commonName` becomes the agent's identity.

### 2. Secrets

Create `conduit-secrets` in the `conduit` namespace:

| Key | Used by | Value |
|-----|---------|-------|
| `clickhouse-dsn` | collector, ui | `clickhouse://user:pass@host:9000/observability` |
| `ui-users` | ui (login mode) | lines of `username:<output of ui hash-password>` |
| `ui-session-key` | ui (login mode) | 32 or more random bytes, for example `openssl rand -hex 32` |

```bash
kubectl -n conduit create secret generic conduit-secrets \
  --from-literal=clickhouse-dsn="$CLICKHOUSE_DSN" \
  --from-file=ui-users=./ui-users \
  --from-literal=ui-session-key="$(openssl rand -hex 32)"
```

### 3. Apply

```bash
kubectl apply -f deployments/k8s/configmap.yaml \
              -f deployments/k8s/mtls-cert.yaml \
              -f deployments/k8s/deployment.yaml \
              -f deployments/k8s/service.yaml
```

The gateway Service is a ClusterIP. Expose it through a LoadBalancer or a
TCP/gRPC passthrough if agents run outside the cluster. TLS must terminate
at the gateway itself, or mTLS identity is lost.

On GKE, use `deployments/k8s/gke-iap.yaml` instead of the `ui` Service to
put the UI behind Identity-Aware Proxy (see [security.md](security.md#iap-mode-google-cloud)).

### Upgrading an existing deployment

- **Schema migrations:** the collector applies pending migrations at
  startup. To migrate before rolling out new collectors, for example in a
  Kubernetes Job, run `/app migrate` with the collector image and its
  environment. Never edit a shipped migration file; the collector refuses
  to start if a checksum changes.

- **NATS stream:** the SPANS stream now uses work-queue retention, so
  acked spans are deleted. JetStream can't change the retention of an
  existing stream. Once, with the gateway and collector stopped, run
  `nats stream rm SPANS` so it's recreated with the new settings.
- **UI auth:** `UI_API_TOKEN` is gone. Configure `UI_AUTH_MODE` and its
  settings (see [security.md](security.md#ui-authentication)).
- **Tempo endpoint:** `TEMPO_ENDPOINT` must point at Tempo's **OTLP/HTTP**
  port, `:4318`. The collector posts to `/v1/traces`. Port `:4317` is
  OTLP/gRPC and won't work.

## Configuration reference

| Variable | Component | Default |
|----------|-----------|---------|
| `GATEWAY_ADDR` | gateway | `:4317` |
| `TLS_CERT`, `TLS_KEY`, `TLS_CA` | gateway | `/etc/certs/server.crt`, `server.key`, `ca.crt` |
| `RATE_LIMIT_RPS` | gateway | `100` telemetry items per second per agent |
| `HEALTH_ADDR` | gateway, collector | `:8081` (liveness and readiness endpoints) |
| `SPAN_RETENTION_DAYS` | collector | `90` (1–3650). The collector reconciles the spans table's TTL on startup; shortening it deletes data that's now past retention. |
| `STORE_LLM_PAYLOADS` | gateway | `true`. Set `false` to drop prompts and completions at ingestion; model, tokens and cost are kept. An unrecognized value stops startup. |
| `NATS_URL` | gateway, collector | `nats://localhost:4222` |
| `CLICKHOUSE_DSN` | collector, ui | local dev DSN. Always set it explicitly. Add `?secure=true` for TLS to ClickHouse. |
| `VICTORIAMETRICS_URL` | collector | `http://localhost:8428` |
| `TEMPO_ENDPOINT` | collector | `http://localhost:4318` |
| `LISTEN_ADDR` | ui | `:8080` |
| `UI_AUTH_MODE` and related | ui | see [security.md](security.md#ui-authentication) |
| `UI_GATEWAY_ADDR` | ui | `localhost:4317`. The gateway address shown in the Get started page's snippets (display only). |
| `UI_DEMO_AGENT` | ui | `demo-agent`. The sample agent whose traces don't count as a user's first trace on Get started. |

---

<!-- Source: https://tjscanlan.github.io/conduit/cli.md -->

# conduitctl

`conduitctl` runs a local Conduit with one command and helps connect agents
to it. It needs only Docker: the quickstart's compose files are built into
the binary, and the released, signed images are pulled.

## Install

Download the archive for your platform from the
[latest release](https://github.com/tjscanlan/conduit/releases/latest), check
it, and put `conduitctl` on your `PATH`:

```bash
# Linux and macOS; pick linux|darwin and amd64|arm64
V=v0.2.0 OS=darwin ARCH=arm64
curl -LO https://github.com/tjscanlan/conduit/releases/download/$V/conduitctl_${V}_${OS}_${ARCH}.tar.gz
curl -LO https://github.com/tjscanlan/conduit/releases/download/$V/SHA256SUMS
curl -LO https://github.com/tjscanlan/conduit/releases/download/$V/SHA256SUMS.sigstore.json
cosign verify-blob SHA256SUMS --bundle SHA256SUMS.sigstore.json \
  --certificate-identity "https://github.com/tjscanlan/conduit/.github/workflows/release.yml@refs/tags/$V" \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com   # optional, recommended
shasum -a 256 -c SHA256SUMS --ignore-missing
tar -xzf conduitctl_${V}_${OS}_${ARCH}.tar.gz && sudo mv conduitctl_${V}_${OS}_${ARCH}/conduitctl /usr/local/bin/
```

Or, with Go: `go install github.com/tjscanlan/conduit/cmd/conduitctl@latest`.

`conduitctl` ships with v0.2.0 and later (v0.1.0 has no binaries, and its
images lack the quickstart's `conduit-init` and `conduit-demo-agent`).
From a checkout, `make quickstart` runs the same stack.

## Commands

| Command | What it does |
|---|---|
| `conduitctl quickstart` | Writes the compose bundle and generated secrets to `~/.conduit/quickstart`, starts the stack, waits for health and the first demo trace, and prints the UI login and the `CONDUIT_*` settings for your agent. Running it again restarts with the same data and credentials. |
| `conduitctl status` | Shows the services, the login, and the settings again. |
| `conduitctl agent-cert NAME` | Issues a client certificate for agent `NAME`, signed by the quickstart's CA, and prints how to use it. Valid existing certificates are kept. |
| `conduitctl doctor` | Checks Docker (running, Compose v2, at least 2 GiB of memory), free ports, every service's health, the UI, the gateway (an mTLS call as `my-agent`), and that demo data reached ClickHouse. Each failure says what to do. |
| `conduitctl down` | Stops the stack, keeping data and credentials. |
| `conduitctl reset` | Stops the stack and deletes its data and credentials. |
| `conduitctl version` | Prints the version; images default to the same tag. |

Flags (all commands): `-dir` (default `~/.conduit/quickstart`), `-ui-port`
(default 8080), `-gateway-port` (default 4317), `-images` (image tag; default
is the CLI's own version), and `-no-demo` (skip the simulated `demo-agent`).

## Things to know

- **Local evaluation only:** the CA key sits next to the certificates it
  signs, and only `127.0.0.1` ports are opened. For anything shared, see
  [Deployment](deployment.md).
- **One local stack at a time:** `conduitctl` and `make quickstart` run the
  same compose project (`conduit-quickstart`) and share its data volumes. So
  `docker compose -p conduit-quickstart logs gateway` works with either.
- **Keep `-dir` under your home directory:** Docker Desktop and Colima share
  only some host folders with containers, usually your home.
- **Development builds:** a `conduitctl` built from a checkout reports
  version `dev` and needs `-images`. `scripts/build-quickstart-images.sh
  ci-local` builds every image locally for `-images ci-local`; CI tests the
  CLI this way.

---

<!-- Source: https://tjscanlan.github.io/conduit/security.md -->

# Security

Conduit handles agent telemetry that can include sensitive data: prompts,
completions, tool arguments and error messages. This document describes the
threat model and the controls that address it.

## Threat model

| Threat | Control |
|--------|---------|
| An unauthenticated client sends spans | The gateway requires a TLS 1.3 client certificate signed by your CA (`RequireAndVerifyClientCert`). |
| An agent impersonates another agent | Identity is the certificate's CommonName, set by the gateway. A client-supplied `agent_name` attribute is overwritten. |
| An agent floods ingestion, or a buggy one does | Per-identity token buckets charged per telemetry item, with a 1000-item batch cap ([below](#ingestion-rate-limiting)). |
| Malformed or hostile payloads reach storage | Strict validation at the gateway: ID charset and size, name and attribute limits, status enum, timestamp sanity, LLM field bounds, finite non-negative cost. ClickHouse queries use bound parameters only. |
| An SDK is tricked into trusting a fake gateway | The SDKs trust only the configured CA (`TLSCACert` / `tls_ca`), never the system roots. |
| Unauthorized UI access | The UI fails closed: a login form with signed cookies, or GCP IAP JWT verification. `none` must be set explicitly ([below](#ui-authentication)). |
| XSS, clickjacking and CSRF in the UI | `html/template` escaping, a strict CSP with no inline script or style, `frame-ancestors 'none'`, SameSite cookies, and same-origin checks on POST requests. |
| Prompts or completions are retained against policy | `STORE_LLM_PAYLOADS=false` drops them at the gateway, before NATS, logs or storage ([architecture](architecture.md#sensitive-data)). |
| Health endpoints leak internals | `/readyz` reports only `ok` or `unavailable`. Errors go to logs. |
| Supply-chain compromise | Few dependencies, govulncheck in CI, GitHub Actions pinned by commit SHA, `FROM scratch` images, generated code pinned and drift-checked. |

**Out of scope, or the operator's job:**
- **Network isolation:** keep NATS, ClickHouse, Tempo, VictoriaMetrics and
  the health ports internal.
- **Encryption at rest** for ClickHouse.
- **Access to ClickHouse itself.** Anyone with ClickHouse access can read
  prompt data (see [Architecture: sensitive data](architecture.md#sensitive-data)).

## Ingestion rate limiting

The gateway rate-limits each agent separately, using a token bucket keyed
on the CommonName of its mTLS client certificate. Authentication runs
first, so every request already has a verified identity when it reaches
the limiter.

`RATE_LIMIT_RPS` (default 100) counts telemetry items per second, not
RPCs. An `IngestBatch` call costs one token for each span, metric and log
it carries, so batching can't multiply an agent's throughput. Bursts can
use up to `RATE_LIMIT_RPS` tokens at once.

Some batches are rejected with `InvalidArgument` because they could never
succeed:
- the batch has more items than `RATE_LIMIT_RPS`
- the batch has more than 1000 items

A request rejected only because the bucket is currently empty gets
`ResourceExhausted` and can be retried.

## UI authentication

The UI (`cmd/ui`) always requires authentication unless you explicitly
disable it. Pick a mode with `UI_AUTH_MODE`:

| Mode | Use when | Identity comes from |
|------|----------|---------------------|
| `login` (default) | Self-hosted, or anywhere without an identity-aware proxy | Login form, then a signed session cookie |
| `iap` | Running on Google Cloud behind Identity-Aware Proxy | IAP's signed JWT on each request |
| `none` | Throwaway local development only | Nobody (every request is `anonymous`) |

The UI refuses to start if the selected mode is misconfigured, for example
`login` without a users file or `iap` without an audience. A misconfigured
mode never quietly falls back to `none`.

`GET /healthz` and `/static/` are always unauthenticated so load balancer
health checks work. Neither serves any telemetry data.

### Login mode

| Variable | Required | Description |
|----------|----------|-------------|
| `UI_USERS_FILE` | yes | One `username:bcrypt-hash` per line. `#` comments are allowed. |
| `UI_SESSION_KEY` | recommended | HMAC key for session cookies, at least 32 bytes. Every replica must share it. If it's unset, a random key is generated at startup, so sessions are lost on restart and don't work across replicas. |
| `UI_SESSION_TTL` | no | Session lifetime, `8h` by default. |
| `UI_COOKIE_INSECURE` | no | `true` lets the cookie travel over plain HTTP. Use it only for local compose. |

Create a users file entry with the UI binary itself. The runtime image has no
shell, so this works without extra tools:

```bash
printf '%s' "$PASSWORD" | go run ./cmd/ui hash-password    # from source
printf '%s' "$PASSWORD" | docker run --rm -i conduit-ui hash-password
echo "alice:<hash>" >> deployments/docker/ui-users          # gitignored
```

`htpasswd -nB alice` output works too.

How login is protected:
- Passwords are checked with bcrypt. When a username doesn't exist, the UI
  still compares against a dummy hash, so response timing doesn't reveal
  which usernames exist.
- Sessions are stateless and signed with HMAC-SHA256. Nothing is stored
  server-side, so a session can't be revoked before it expires. To end
  every session at once, rotate `UI_SESSION_KEY`.
- The session cookie is `__Host-conduit_session`, with `HttpOnly`, `Secure`,
  `SameSite=Lax` and `Path=/` set.
- `POST /login` and `POST /logout` reject cross-site requests, checking
  `Sec-Fetch-Site` and falling back to `Origin`. This blocks login CSRF and
  forced logout.
- Login attempts are rate limited to 5 per minute for each username and
  each client IP.
- The `next` redirect parameter accepts only local paths, which prevents
  open redirects.

### IAP mode (Google Cloud)

| Variable | Required | Description |
|----------|----------|-------------|
| `UI_IAP_AUDIENCE` | yes | `/projects/PROJECT_NUMBER/global/backendServices/SERVICE_ID` for a load balancer backend, or `/projects/PROJECT_NUMBER/apps/PROJECT_ID` for App Engine. |

On every request, the UI checks the `x-goog-iap-jwt-assertion` header:
- **Signature:** must be ES256 and verify against Google's published keys at
  `https://www.gstatic.com/iap/verify/public_key-jwk`. The keys are cached
  for an hour and refetched when a new key ID appears, at most once per
  minute.
- **Claims:** the issuer must be `https://cloud.google.com/iap`, the
  audience must match `UI_IAP_AUDIENCE`, and the token must not be expired.

It never trusts the unsigned `x-goog-authenticated-user-email` header. A
request that reaches the pod without passing through IAP, such as one from
inside the cluster or through a misconfigured load balancer, is rejected.

Clicking **Sign out** redirects to `/_gcp_iap/clear_login_cookie`. That's
IAP's own logout endpoint.

Use `deployments/k8s/gke-iap.yaml` to set this up on GKE: it has the
BackendConfig with IAP enabled, the health check on `/healthz`, a ClusterIP
Service, and an Ingress. Access is granted with
`roles/iap.httpsResourceAccessor`.

### Browser hardening

Every response carries a strict Content-Security-Policy that allows only
same-origin scripts and styles, with no inline script and
`frame-ancestors 'none'`. Responses also set `X-Content-Type-Options: nosniff`,
`X-Frame-Options: DENY` and `Referrer-Policy: no-referrer`.

Templates must not contain inline event handlers, `<script>` blocks or
`style` attributes. `TestTemplatesHaveNoInlineScript` enforces this.

---

<!-- Source: https://tjscanlan.github.io/conduit/examples.md -->

# Examples

Runnable agents in [`examples/`](https://github.com/tjscanlan/conduit/tree/main/examples),
in Go and Python. Each one sends a trace to a local Conduit and prints a link
to it in the UI. CI runs every example on every push, so they stay working.

## Run one in two commands

```bash
make quickstart                        # once: Conduit on localhost, needs only Docker
cd examples/go && go run ./tool-agent  # or: python examples/python/tool_agent.py
```

The examples find the certificates `make quickstart` generated and send as
agent `my-agent`. To point them somewhere else, set `CONDUIT_GATEWAY_ADDR`,
`CONDUIT_CERT_DIR` (a directory with `ca.crt` and `<agent>.crt`/`.key`), and
`CONDUIT_AGENT`.

For Python, install the SDK first: `pip install conduit-observability`, or
`pip install "conduit-observability[anthropic,openai]"` to include the Claude
and OpenAI SDKs. To try SDK changes from a checkout, use `pip install -e sdk/python`.

## The examples

| Example | Go (`examples/go`) | Python (`examples/python`) | Needs |
|---|---|---|---|
| **Hello trace.** The smallest useful trace: a run with one LLM call. Start here. | `go run ./hello-trace` | `hello_trace.py` | nothing |
| **Tool-calling agent.** A model/tool loop with a simulated model. | `go run ./tool-agent` | `tool_agent.py` | nothing |
| **Async agent.** Concurrent tool calls with `asyncio.gather`; spans still nest correctly. | | `async_agent.py` | nothing |
| **Claude agent.** The same loop with the official Anthropic SDK and real token usage. | `go run ./claude-agent` | `claude_agent.py` | `ANTHROPIC_API_KEY` |
| **OpenAI agent.** The same loop with the official OpenAI SDK (Chat Completions). | `go run ./openai-agent` | `openai_agent.py` | `OPENAI_API_KEY` |

The Claude examples default to `claude-sonnet-5` (override with
`ANTHROPIC_MODEL`). The OpenAI examples default to `gpt-5-mini` (override with
`OPENAI_MODEL`). Without a key, they say so and exit.

## What a tool-calling trace looks like

Every tool-calling example produces the same span tree. It's a good shape
to copy into your own agent:

```text
agent.run                 one per task; attributes: question, example
├── llm.call              one per model call: model, prompt, completion, tokens, cost
├── tool.get_weather      one per tool call: tool.input; error status if the tool failed
├── tool.get_weather
└── llm.call              the final answer
```

- **Tools run outside the `llm.call` span.** Each `llm.call` span ends before
  its tools run, so its duration is the model's latency alone.
- **Tool spans are children of the run, not of the model call.** On the
  waterfall, concurrent tool calls then show up side by side (see the async
  example).
- **Record the provider's real usage.** Take the token counts and model name
  from the API response (`usage.input_tokens` for Anthropic,
  `usage.prompt_tokens` for OpenAI). `EstimateCostUSD` / `estimate_cost_usd`
  turns them into an approximate cost for models it knows.

## How CI checks them without API keys

`scripts/examples-check.sh` runs every example against a running quickstart
and confirms each trace appears in the UI with the expected spans. The
Claude and OpenAI examples run against `scripts/fake-llm-server.py`, a
stdlib fake of the Messages and Chat Completions APIs. Both official SDKs
honor `ANTHROPIC_BASE_URL` and `OPENAI_BASE_URL`, so the examples' real
request building and response parsing are exercised. The fake rejects a
second turn that lacks correct tool results.

---

<!-- Source: https://tjscanlan.github.io/conduit/sdk-go.md -->

# Go SDK

`sdk/go` sends spans from a Go agent to the Conduit gateway over mTLS and
gRPC. It's a separate Go module, `github.com/tjscanlan/conduit/sdk/go`, and
it requires Go 1.26 or newer.

For runnable Go agents (hello-trace, tool calling, Claude, OpenAI) that
work against `make quickstart` with no setup, see [Examples](examples.md).

## Quick start (2 minutes)

With a local stack from `make quickstart` (needs only Docker):

```bash
export CONDUIT_CERT_DIR=$PWD/deployments/quickstart/.conduit/certs CONDUIT_AGENT=my-agent
go get github.com/tjscanlan/conduit/sdk/go@latest
```

```go
client, err := sdk.NewFromEnv() // CONDUIT_CERT_DIR, CONDUIT_AGENT, CONDUIT_GATEWAY_ADDR
if err != nil {
	log.Fatal(err) // says what to set or fix
}
defer client.Close()
if err := client.Check(ctx); err != nil { // optional: fail fast if the gateway won't accept us
	log.Fatal(err)
}

run, ctx := client.StartSpan(ctx, "agent.run")
// ... your agent's work; child spans from ctx nest under run ...
run.SetOK()
if err := run.End(ctx); err != nil {
	log.Print(err)
}
```

Open <http://localhost:8080>: the trace appears under agent `my-agent`.
The [examples](examples.md) build on this: tool calls, Claude, and OpenAI.

## Configuration

`sdk.NewFromEnv()` reads:

| Variable | Meaning |
|---|---|
| `CONDUIT_CERT_DIR` | Directory with `ca.crt` and `<agent>.crt`/`<agent>.key` (the layout `make quickstart` and `scripts/gen-certs.sh` produce). |
| `CONDUIT_AGENT` | Which agent's certificate to use from that directory. Optional when it holds exactly one client certificate. |
| `CONDUIT_TLS_CERT`, `CONDUIT_TLS_KEY`, `CONDUIT_TLS_CA` | Explicit file paths; they override `CONDUIT_CERT_DIR`. |
| `CONDUIT_GATEWAY_ADDR` | Gateway address, default `localhost:4317` (`OTEL_GATEWAY_ADDR` is accepted too). The host must be in the gateway certificate. |

`sdk.ConfigFromEnv()` returns the resolved `sdk.Config` without connecting;
`sdk.New(sdk.Config{...})` takes the same settings in code.

## When it doesn't connect

The SDK checks for the common first-run mistakes and says what to do.
`New` and `NewFromEnv` catch file problems right away:

- **Nothing configured:** names the variables to set.
- **Several agents in the directory:** lists them and asks for `CONDUIT_AGENT`.
- **A missing file:** points at `make quickstart` / `scripts/gen-certs.sh`.
- **A key that belongs to another certificate.**
- **The gateway's `server.crt` used as an agent certificate.**
- **An expired, not-yet-valid, or CommonName-less certificate.**

`Check` and every send add a hint to gRPC errors:

- the gateway isn't running or reachable;
- the gateway's certificate isn't signed by your `ca.crt`;
- the host isn't in the gateway's certificate (use a name it lists, such as `localhost`);
- the gateway rejected the agent's certificate (signed by a CA it doesn't trust);
- the address isn't the gateway's gRPC port;
- rate limiting.

The original gRPC error stays wrapped, so `status.FromError` still works.

## Production: agent certificates

Every agent authenticates with its own client certificate. The
certificate's CommonName is the agent's identity: the gateway rate-limits
by it and records it on every span as `agent_name`. Setting an attribute
can't change it.

For local development:

```bash
./scripts/gen-certs.sh                    # once: CA, gateway cert, example agent cert
./scripts/gen-certs.sh agent research-bot # one per agent
```

This writes `certs/research-bot.crt` and `certs/research-bot.key`. In
production, issue client certificates from your own PKI, signed by the CA
the gateway trusts (`TLS_CA`).

## Add the module

```bash
go get github.com/tjscanlan/conduit/sdk/go@v0.2.0
```

The SDK lives in a subdirectory of the repo, so its tags are prefixed:
`sdk/go/v0.2.0`. It is released together with the main module, which provides
the generated proto package. To develop against a checkout instead:

```bash
go work init ./your-agent ./path/to/conduit/sdk/go ./path/to/conduit
```

## Instrument

```go
import sdk "github.com/tjscanlan/conduit/sdk/go"

client, err := sdk.New(sdk.Config{
	GatewayAddr: "localhost:4317",          // host must match a SAN on the gateway cert
	TLSCert:     "certs/research-bot.crt",
	TLSKey:      "certs/research-bot.key",
	TLSCACert:   "certs/ca.crt",            // the only CA trusted for the gateway
})
if err != nil {
	log.Fatal(err)
}
defer client.Close()

span, ctx := client.StartSpan(ctx, "agent.run")
span.SetAttr("task", "summarize")

llm, _ := client.StartSpan(ctx, "llm.call") // child of agent.run: same trace, parent set
resp, err := callModel(ctx, prompt)
if err != nil {
	llm.SetError(err)
} else {
	llm.SetOK().RecordLLMCall(sdk.LLMCall{
		Model:        resp.Model,
		Prompt:       prompt,
		Completion:   resp.Text,
		InputTokens:  resp.InputTokens,
		OutputTokens: resp.OutputTokens,
		CostUSD:      resp.CostUSD,
	})
}
if err := llm.End(ctx); err != nil {
	log.Printf("span not delivered: %v", err)
}

span.SetOK()
_ = span.End(ctx)
```

### API

| Call | What it does |
|------|--------------|
| `sdk.NewFromEnv()` / `sdk.ConfigFromEnv()` | Builds the client (or just its `Config`) from the environment; see [Configuration](#configuration). |
| `client.Check(ctx)` | Sends an empty batch to confirm the gateway is reachable and accepts this agent's certificate. Returns a diagnosed error otherwise. |
| `sdk.New(cfg)` | Connects to the gateway over TLS 1.3. The gateway's certificate is verified against `TLSCACert` only, never the system roots. Returns an error if any certificate file is missing or invalid. |
| `client.StartSpan(ctx, name)` | Starts a span. If `ctx` carries a span from an earlier `StartSpan`, the new span joins its trace as a child; otherwise it starts a new trace. Returns the span and a context carrying it. |
| `span.SetAttr(k, v)` | Adds a string attribute. The gateway allows up to 128 attributes, keys up to 128 bytes and values up to 4096 bytes. A span over those limits is rejected. |
| `span.SetOK()` / `span.SetError(err)` | Sets the span status. `SetError` also stores `err.Error()` in `error.message`, cut to 4096 bytes. `SetError(nil)` does nothing. Spans you don't mark have status `unset`. |
| `span.RecordLLMCall(call)` | Attaches model, prompt, completion, token counts and cost. Prompts and completions are capped at 128 KiB each. |
| `span.End(ctx)` | Sends the span synchronously with one RPC. Returns an error if delivery fails or the gateway rejects the span, with the gateway's reason. |
| `client.Log(ctx, severity, body, attrs)` | Sends one log record (`sdk.SeverityDebug`, `Info`, `Warn` or `Error`). If `ctx` carries a span from `StartSpan`, the record is linked to that span and its trace. Returns an error if the gateway rejects it. Bodies are limited to 32 KiB. |
| `client.Gauge(ctx, name, value, labels)` / `client.Counter(...)` | Sends one metric sample to VictoriaMetrics. Names follow Prometheus rules, and names starting with `conduit_` or `__` are reserved. The `agent` label is always set from your certificate. Counters must be ≥ 0, and values must be finite. |
| `client.Close()` | Closes the connection. |

`sdk.EstimateCostUSD(model, in, out)` uses a small built-in price table
that will go out of date. Prefer the cost your provider reports.

### Rate limits

Each agent identity may send `RATE_LIMIT_RPS` spans per second (default
100), with bursts up to that number. Over the limit, `End` returns a
`ResourceExhausted` error. See [security.md](security.md#ingestion-rate-limiting).

### Things to know

- **Synchronous delivery:** `End` makes one RPC per span and blocks until
  the gateway answers. Call it off your hot path, or pass a context with a
  deadline.
- **No retries:** the SDK doesn't retry failed sends, so handle the error
  from `End` if delivery matters.
- **Agent name comes from the certificate:** `Config.AgentName` isn't sent.
  Identity always comes from the client certificate.

---

<!-- Source: https://tjscanlan.github.io/conduit/sdk-python.md -->

# Python SDK

`sdk/python` (the `conduit-observability` distribution, imported as `conduit_observability`)
sends spans from a Python agent to the Conduit gateway over mTLS and gRPC.
It requires Python 3.10 or newer.

For runnable Python agents (hello-trace, tool calling, Claude, OpenAI) that
work against `make quickstart` with no setup, see [Examples](examples.md).

## Quick start (2 minutes)

With a local stack from `make quickstart` (needs only Docker):

```bash
export CONDUIT_CERT_DIR=$PWD/deployments/quickstart/.conduit/certs CONDUIT_AGENT=my-agent
pip install conduit-observability     # 0.2.0 or newer
```

```python
import conduit_observability as conduit

conduit.configure()            # reads CONDUIT_CERT_DIR, CONDUIT_AGENT, CONDUIT_GATEWAY_ADDR
conduit.get_client().check()   # optional: fail fast if the gateway won't accept us

@conduit.traced()              # one span per call, named after the function
def search(query):
    ...

with conduit.span("agent.run") as run:
    search("conduit")          # its span is a child of agent.run
    run.set_ok()
conduit.shutdown()
```

Open <http://localhost:8080>: the trace appears under agent `my-agent`.
The [examples](examples.md) build on this: tool calls, asyncio, Claude, and OpenAI.

### Record real LLM usage

`llm_call_from_anthropic` and `llm_call_from_openai` turn a provider
response into an `LLMCall`. Each one captures the model, the text and tool
calls, the real token counts and an estimated cost. They read the response
by attribute, so they never import the provider SDKs. The `anthropic` and
`openai` extras install those SDKs for you if you want them.

```python
with conduit.span("llm.call") as llm:
    msg = anthropic_client.messages.create(model=..., max_tokens=1024, messages=messages)
    llm.record_llm_call(conduit.llm_call_from_anthropic(msg, prompt=user_text))
    llm.set_ok()
# OpenAI: conduit.llm_call_from_openai(resp) for Chat Completions or Responses API results
```

## Configuration

`conduit.configure()` and `Client.from_env()` read:

| Variable | Meaning |
|---|---|
| `CONDUIT_CERT_DIR` | Directory with `ca.crt` and `<agent>.crt`/`<agent>.key` (the layout `make quickstart` and `scripts/gen-certs.sh` produce). |
| `CONDUIT_AGENT` | Which agent's certificate to use from that directory. Optional when it holds exactly one agent (`ca` and `server` files are skipped). |
| `CONDUIT_TLS_CERT`, `CONDUIT_TLS_KEY`, `CONDUIT_TLS_CA` | Explicit file paths; they override `CONDUIT_CERT_DIR`. The 0.1.0 names `TLS_CERT`, `TLS_KEY`, `TLS_CA` still work. |
| `CONDUIT_GATEWAY_ADDR` | Gateway address, default `localhost:4317` (`OTEL_GATEWAY_ADDR` is accepted too). The host must be in the gateway certificate. |

To be explicit, call `conduit.configure(gateway_addr=..., tls_cert=..., tls_key=..., tls_ca=...)`
or pass a `Client`.

**Module API vs `Client`:** the module-level `span()` and `@traced` never
break your agent. If nothing can be configured, they become no-ops and log
one warning. Use a `Client` directly for explicit control, for example
`raise_errors=True`.

## When it doesn't connect

The SDK raises `ConfigError` (a `ConduitError`) with what to do when the
client is created:

- **Nothing configured.**
- **Several agents in the directory:** it lists them.
- **A missing file:** points at `make quickstart` / `scripts/gen-certs.sh`.
- **A key that belongs to another certificate.**
- **A CA file that isn't PEM.**

`check()` and every send add a hint to gRPC errors:

- the gateway isn't running or reachable;
- the host isn't in the gateway's certificate;
- a certificate verification or TLS handshake failure, which usually means
  the agent's certificate and the gateway's come from different CAs;
- rate limiting, timeouts.

## Production: agent certificates

The process is the same as for the Go SDK. The certificate's CommonName is
the agent's identity: the gateway rate-limits by it and records it on every
span as `agent_name`. See [sdk-go.md](sdk-go.md#production-agent-certificates).

```bash
./scripts/gen-certs.sh agent research-bot
```

## Install

```bash
pip install conduit-observability
```

Import it as `conduit_observability`.

## Instrument with a Client

```python
from conduit_observability import Client, LLMCall

client = Client(
    "localhost:4317",                 # host must match a SAN on the gateway cert
    tls_cert="certs/research-bot.crt",
    tls_key="certs/research-bot.key",
    tls_ca="certs/ca.crt",            # the only CA trusted for the gateway
)
# or: Client.from_env(); see Configuration

with client.span("agent.run") as run:
    run.set_attr("task", "summarize")

    with client.span("llm.call") as llm:        # child of agent.run
        resp = call_model(prompt)                # an exception here marks llm as an error
        llm.record_llm_call(LLMCall(
            model=resp.model, prompt=prompt, completion=resp.text,
            input_tokens=resp.input_tokens, output_tokens=resp.output_tokens,
            cost_usd=resp.cost_usd,
        ))
        llm.set_ok()

client.close()                                   # or use `with Client(...) as client:`
```

Spans opened with `client.span(...)` nest automatically. The current span
is tracked with `contextvars`, so nesting is correct per thread and per
asyncio task, and a single `Client` can be shared by the whole process.

### API

| Call | What it does |
|------|--------------|
| `conduit.configure(client=None, **kwargs)` | Sets the module-level client (from the environment by default). `get_client()` returns it; `shutdown()` closes it. |
| `conduit.span(name)` / `@conduit.traced(name=None, **attrs)` | A span on the module-level client: a context manager, or a decorator for sync and async functions (OK on return, error if it raises). `current_span()` is the innermost open span. |
| `llm_call_from_anthropic(msg, prompt="")` / `llm_call_from_openai(resp, prompt="")` | An `LLMCall` from a provider response, with real usage and an estimated cost. |
| `Client.from_env(timeout=10.0)` / `client.check()` | Builds a client from the environment; checks the gateway accepts it (raises a diagnosed `ConduitError` if not). |
| `Client(addr, tls_cert, tls_key, tls_ca, timeout=10.0)` | Opens a TLS channel to the gateway. The gateway's certificate is verified against `tls_ca` only. |
| `client.span(name, *, raise_errors=False)` | A context manager. It opens a child of the current span, makes it current, and ends and sends it on exit. If the block raises, the span is marked as an error with `error.message`, and the exception propagates. |
| `client.start_span(name)` | Starts a child of the current span without making it current. You call `end()` yourself. |
| `span.set_attr(k, v)` | Adds a string attribute. The gateway allows up to 128 attributes, keys up to 128 bytes and values up to 4096 bytes. A span over those limits is rejected. |
| `span.set_ok()` / `span.set_error(err)` | Sets the span status. `set_error` stores the message in `error.message`, cut to 4096 bytes. `set_error(None)` does nothing. |
| `span.record_llm_call(LLMCall(...))` | Attaches model, prompt, completion, token counts and cost. Prompts and completions are capped at 128 KiB each. |
| `span.end()` | Sends the span once; calling it again does nothing. Raises `ConduitError` if delivery fails or the gateway rejects the span. |
| `client.log(severity, body, attributes=None)` | Sends one log record (`SEVERITY_DEBUG`, `SEVERITY_INFO`, `SEVERITY_WARN` or `SEVERITY_ERROR`). Inside `with client.span(...)` it's linked to that span and trace. Raises `ConduitError` if it's rejected. Bodies are limited to 32 KiB. |
| `client.gauge(name, value, labels=None)` / `client.counter(...)` | Sends one metric sample. The same naming rules apply as in the Go SDK: Prometheus names, `conduit_` and `__` reserved, and the `agent` label set from your certificate. |
| `span.trace_id`, `span.span_id` | 32-hex and 16-hex IDs, for log correlation. |

### Delivery errors

- **`span.end()`** raises `ConduitError` when a span isn't accepted, for
  example if the gateway is unreachable, the agent is over its rate limit,
  or validation fails.
- **`with client.span(...)`** logs delivery failures to the `conduit_observability`
  logger by default, so instrumentation can never crash your agent. Pass
  `raise_errors=True` to get `ConduitError` instead. Even then, an exception
  raised by your own code inside the block always wins and is never masked.

### Things to know

- **Synchronous delivery:** each span is one blocking RPC, sent when the
  span ends. Set `timeout` if the gateway might be slow.
- **No retries:** the SDK doesn't retry failed sends.
- **Identity comes from the certificate:** setting an `agent_name`
  attribute, or passing the legacy `agent_name` argument, doesn't change
  the agent's identity.
- **Price table:** `estimate_cost_usd(model, in, out)` uses a small
  built-in table that will go out of date. Prefer the cost your provider
  reports.

## Development

```bash
cd sdk/python
uv venv && uv pip install -e '.[dev]'    # or python -m venv / pip
.venv/bin/python -m pytest
```

The tests run against an in-process gRPC server that requires client
certificates, using certificates generated on the fly. The generated stubs
in `src/conduit_observability/_proto` are committed. After editing `proto/*.proto`,
regenerate them with `make proto`, which also regenerates the Go code.

---

<!-- Source: https://tjscanlan.github.io/conduit/api.md -->

# API reference

Agents talk to the gateway over gRPC with mutual TLS on port 4317, using the `IngestionService` in package `observability.v1`. The SDKs wrap this API. Use it directly only if you're writing a client in another language. This page is generated from the `.proto` files and the gateway's validation constants, so it always matches the code.

## Authentication and identity

Every RPC requires a TLS 1.3 client certificate signed by the CA the gateway trusts. The certificate's CommonName is the agent's identity. The gateway stamps it on every span (`agent_name`), log (`agent_name`) and metric (`agent` label), overwriting any value the client supplies.

## Errors and partial acceptance

- **`IngestSpan`** returns `accepted: false` with a `rejection_reason` when validation fails.
- **`IngestBatch`** validates each item independently. It returns per-kind accepted counts, plus an `errors` list with entries like `log[2]: validation: log body is required`, where the index is the item's position in its kind's list.
- **gRPC status `RESOURCE_EXHAUSTED`:** the agent is over its rate limit (charged per item). Retry later.
- **gRPC status `INVALID_ARGUMENT`:** the batch is larger than 1000 items or the per-agent limit; split it.
- **gRPC status `UNAUTHENTICATED`:** no valid client certificate.

## Ingestion limits

| Applies to | Field | Rule |
|---|---|---|
| Span | `trace_id`, `span_id`, `parent_id` | 1–64 characters of `[A-Za-z0-9_-]` (`parent_id` may be empty). SDKs send W3C sizes: 32 and 16 hex characters. |
| Span | `name` | Required, at most 256 bytes |
| Span | duration | `end_time` ≥ `start_time`, at most 24 hours |
| Span, log | `attributes` | At most 128 entries; keys 1–128 bytes; values at most 4 KiB |
| Span | `llm_call.prompt` / `completion` | At most 128 KiB each (dropped at the gateway when `STORE_LLM_PAYLOADS=false`) |
| Span | `llm_call.model` | At most 256 bytes |
| Span | `llm_call` tokens and cost | Token counts ≥ 0; `cost_usd` finite and ≥ 0 |
| Log | `body` | Required, at most 32 KiB |
| Log, metric | `timestamp` | No more than 1 hour in the future; the receive time if unset |
| Log | `trace_id` / `span_id` | Optional, same format as spans; `span_id` requires `trace_id` |
| Metric | `name` | Prometheus name `[a-zA-Z_:][a-zA-Z0-9_:]*`, at most 200 characters; `conduit_*` and `__*` are reserved |
| Metric | `labels` | At most 32; keys `[a-zA-Z_][a-zA-Z0-9_]*` (at most 128, not `__*`); values at most 1 KiB. The `agent` label is always set by the gateway. |
| Metric | `value` | Finite; counters ≥ 0 |
| Batch | items per `IngestBatch` | At most 1000 (spans + metrics + logs), and no more than the per-agent rate limit (`RATE_LIMIT_RPS`) |

## `ingestion.proto`

### service `IngestionService`

IngestionService is the gateway's inbound API. All RPCs require mTLS — agent identity comes from the client certificate.

| RPC | Request | Response | Description |
|---|---|---|---|
| `IngestSpan` | `IngestSpanRequest` | `IngestSpanResponse` | IngestSpan submits a single span. Used for real-time streaming from agents. |
| `IngestBatch` | `IngestBatchRequest` | `IngestBatchResponse` | IngestBatch submits spans, metrics, and logs. Each item is validated and accepted independently (not atomically); rejected items are reported in IngestBatchResponse.errors. |

### message `IngestSpanRequest`

| # | Field | Type | Description |
|---|---|---|---|
| 1 | `span` | `Span` |  |

### message `IngestSpanResponse`

| # | Field | Type | Description |
|---|---|---|---|
| 1 | `accepted` | `bool` |  |
| 2 | `rejection_reason` | `string` | set if accepted=false |

### message `IngestBatchRequest`

| # | Field | Type | Description |
|---|---|---|---|
| 1 | `spans` | `repeated Span` |  |
| 2 | `metrics` | `repeated Metric` |  |
| 3 | `logs` | `repeated LogRecord` |  |

### message `IngestBatchResponse`

| # | Field | Type | Description |
|---|---|---|---|
| 1 | `spans_accepted` | `int32` |  |
| 2 | `metrics_accepted` | `int32` |  |
| 3 | `logs_accepted` | `int32` |  |
| 4 | `errors` | `repeated string` | one per rejected item, e.g. "log[2]: validation: ..." |

## `span.proto`

### message `Span`

Span represents a single unit of work in an agent execution trace.

| # | Field | Type | Description |
|---|---|---|---|
| 1 | `trace_id` | `string` |  |
| 2 | `span_id` | `string` |  |
| 3 | `parent_id` | `string` | empty if root span |
| 4 | `name` | `string` |  |
| 5 | `start_time` | `google.protobuf.Timestamp` |  |
| 6 | `end_time` | `google.protobuf.Timestamp` |  |
| 7 | `status` | `SpanStatus` |  |
| 8 | `attributes` | `map<string, string>` |  |
| 9 | `llm_call` | `LLMCallData` | Optional — set only for LLM call spans |

### enum `SpanStatus`

| Value | Number | Description |
|---|---|---|
| `SPAN_STATUS_UNSET` | 0 |  |
| `SPAN_STATUS_OK` | 1 |  |
| `SPAN_STATUS_ERROR` | 2 |  |

### message `LLMCallData`

LLMCallData captures the full context of a single inference call. Stored separately from span metadata to allow payload-level access control.

| # | Field | Type | Description |
|---|---|---|---|
| 1 | `model` | `string` |  |
| 2 | `prompt` | `string` |  |
| 3 | `completion` | `string` |  |
| 4 | `input_tokens` | `int32` |  |
| 5 | `output_tokens` | `int32` |  |
| 6 | `cost_usd` | `double` |  |

## `log.proto`

### message `LogRecord`

| # | Field | Type | Description |
|---|---|---|---|
| 1 | `timestamp` | `google.protobuf.Timestamp` |  |
| 2 | `severity` | `LogSeverity` |  |
| 3 | `body` | `string` |  |
| 4 | `attributes` | `map<string, string>` |  |
| 5 | `trace_id` | `string` |  |
| 6 | `span_id` | `string` |  |

### enum `LogSeverity`

| Value | Number | Description |
|---|---|---|
| `LOG_SEVERITY_UNSPECIFIED` | 0 |  |
| `LOG_SEVERITY_DEBUG` | 1 |  |
| `LOG_SEVERITY_INFO` | 2 |  |
| `LOG_SEVERITY_WARN` | 3 |  |
| `LOG_SEVERITY_ERROR` | 4 |  |

## `metric.proto`

### message `Metric`

| # | Field | Type | Description |
|---|---|---|---|
| 1 | `name` | `string` |  |
| 2 | `labels` | `map<string, string>` |  |
| 3 | `value` | `double` |  |
| 4 | `timestamp` | `google.protobuf.Timestamp` |  |
| 5 | `type` | `MetricType` |  |

### enum `MetricType`

| Value | Number | Description |
|---|---|---|
| `METRIC_TYPE_UNSPECIFIED` | 0 |  |
| `METRIC_TYPE_GAUGE` | 1 |  |
| `METRIC_TYPE_COUNTER` | 2 |  |

---

<!-- Source: https://tjscanlan.github.io/conduit/roadmap.md -->

# Roadmap

Conduit's goal is to be the observability layer a security-conscious team can run next to its AI agents. It should be self-hosted, small enough to audit, and focused on what matters for agents: steps, tools, models, errors and cost.

This roadmap is ordered by priority, not by date. Items move as we learn. To suggest a change, open an issue or a PR against `docs/_data/roadmap.json`.

## Shipped

**Foundation**
- mTLS agent identity, with the certificate CommonName as `agent_name`
- Span validation at ingestion
- Per-agent rate limiting, charged per telemetry item
- NATS JetStream work queue with at-least-once delivery into ClickHouse
- Best-effort export to VictoriaMetrics and Tempo
- Span status (ok, error, unset) from end to end, and nanosecond timestamps
- Logs (ClickHouse) and metrics (VictoriaMetrics) ingestion, each with its own NATS stream, plus SDK `Log`, `Gauge` and `Counter` APIs

**SDKs**
- Go and Python SDKs: mTLS that trusts only your CA, W3C-sized IDs, status and error recording
- Python context that is safe across threads and asyncio tasks

**UI**
- Sign-in: a login form with signed session cookies, or Google Cloud IAP
- Strict CSP and browser hardening
- Trace list: one row per trace, filters by time, agent and errors, keyset paging, agent suggestions
- Trace detail: span tree, waterfall timeline, error and LLM details, deep links
- Logs page: filters by time, severity, agent, text and trace, with deep links to spans

**Data governance**
- `STORE_LLM_PAYLOADS=false` drops prompts and completions at the gateway
- `SPAN_RETENTION_DAYS`, with the TTL reconciled on existing tables
- Versioned, checksummed ClickHouse schema migrations (`collector migrate`)

**Operations**
- `/livez` and `/readyz` on every service, and a `healthcheck` subcommand for scratch images
- Compose restart policies and Kubernetes probes, plus GKE IAP manifests

**Quality**
- CI: race tests, a generated-code drift check, govulncheck, Docker builds, and a full-stack smoke test with both SDKs
- Brand identity, documentation set, and this docs site

## Now: hardening for first production users

| Item | Area | Why |
|------|------|-----|
| **Prompt and completion access control** | UI | Payloads are stored but never shown. Showing them needs a decision on who may see them (see [open decisions](#open-decisions)). |

## Next: scale and integration

| Item | Area | Why |
|------|------|-----|
| **Batched, asynchronous SDK delivery** | SDKs | One blocking RPC per span is simple but slow for chatty agents. Add a buffered exporter with backpressure. |
| **Read the trace list from `trace_summary`** | Storage | Avoid aggregating raw spans for every list query at high volume. Needs root names and agents in the view. |
| **OpenTelemetry interoperability** | Pipeline | Accept OTLP from existing OpenTelemetry agents, and use OTel semantic conventions for LLM attributes. |
| **Search** | UI | Search attributes, error messages and span names across traces. |
| **Published SDKs** | SDKs | A PyPI package and a tagged Go module with stable import paths (see open decisions). |
| **Helm chart** | Operations | Package the Kubernetes manifests with sensible defaults. |

## Later: insight

| Item | Area | Why |
|------|------|-----|
| **Cost and token dashboards** | UI | Spend by agent, model and time, built on the existing rollups. |
| **Alerts** | Operations | Error-rate and latency thresholds per agent, with webhook delivery. |
| **Trace comparison** | UI | Compare two runs of the same task side by side. |
| **Multi-tenancy** | Security | Separate projects or teams with scoped UI access. |

## Open decisions

These need a maintainer's call before the related work can start.

| Decision | Options | Raised in |
|----------|---------|-----------|
| **Who can see prompts and completions** | Every signed-in user; A separate role or permission; Reveal-on-click with an audit log | [PR #3](https://github.com/tjscanlan/conduit/pull/3) |
| **`nats-server` as a test dependency** | Add an in-process NATS for integration tests; Rely on the compose smoke test, which already covers the path. Leaning toward the smoke test. | [PR #2](https://github.com/tjscanlan/conduit/pull/2) |

---

<!-- Source: https://tjscanlan.github.io/conduit/releasing.md -->

# Releasing

How Conduit is versioned, how to verify what you download, how to upgrade, and
how a maintainer cuts a release.

## Versioning policy

Conduit uses [Semantic Versioning](https://semver.org/). It is **pre-1.0**: a
minor version (`0.x.0`) may contain breaking changes, and a patch version
(`0.x.y`) will not. Breaking changes are called out in the
[changelog](https://github.com/tjscanlan/conduit/blob/main/CHANGELOG.md).

The policy covers this public API:

- The gRPC ingestion API in `proto/` ([reference](api.md))
- The exported APIs of the Go and Python SDKs
- Environment variables and subcommands ([configuration reference](deployment.md#configuration-reference))

It does not cover packages under `internal/`, the root Go module's other packages
(that module runs the services and provides generated types to the SDK), the UI's
HTML, or the exact ClickHouse schema. Schema changes reach existing data through
migrations, which the collector applies at startup.

All artifacts of a release share one version number.

| Artifact | Where | Version form |
|----------|-------|--------------|
| Images | `ghcr.io/tjscanlan/conduit-{gateway,collector,ui,init,demo-agent}` | `v0.2.0` |
| Go SDK | `github.com/tjscanlan/conduit/sdk/go` | tag `sdk/go/v0.2.0`, used as `@v0.2.0` |
| Python SDK | [`conduit-observability`](https://pypi.org/project/conduit-observability/) | `0.1.0` |

## Verifying images

Images are built by the `release` workflow and signed with keyless
[cosign](https://docs.sigstore.dev/), so a valid signature shows the image came
from this repository's workflow at a version tag:

```bash
cosign verify ghcr.io/tjscanlan/conduit-gateway:v0.2.0 \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --certificate-identity-regexp '^https://github\.com/tjscanlan/conduit/\.github/workflows/release\.yml@refs/tags/v'
```

Each image also carries an SBOM and build provenance as attestations in its
manifest list. To see them:

```bash
docker buildx imagetools inspect ghcr.io/tjscanlan/conduit-gateway:v0.2.0 --raw
```

In production, deploy by digest rather than by tag.

## Upgrading

1. Read the changelog entries between your version and the new one.
2. Follow [Upgrading an existing deployment](deployment.md#upgrading-an-existing-deployment).

## Cutting a release

Only a maintainer pushes tags.

### One-time setup

On PyPI, add a trusted publisher for the project `conduit-observability`: owner
`tjscanlan`, repository `conduit`, workflow `release.yml`, environment `pypi`. If
the project doesn't exist yet, add it as a *pending* publisher in your account
settings. If it does, add it in the project's settings. Consider also requiring a
reviewer on the `pypi` environment in the repository settings.

### Each release

1. On a branch, set the new version in all of these, then merge:
   - `sdk/python/pyproject.toml`
   - the `require github.com/tjscanlan/conduit` line in `sdk/go/go.mod`
   - the image tags in `deployments/k8s/deployment.yaml`
   - the `${CONDUIT_VERSION:-vX.Y.Z}` image defaults in `deployments/quickstart/compose.yml`
     (embedded in `conduitctl`)
   - `CHANGELOG.md`: move `[Unreleased]` into an `[X.Y.Z]` section and update the
     links at the bottom

   `scripts/release-check.sh vX.Y.Z` checks that these agree, and CI runs it.
2. Tag the merge commit and push both tags at once. The SDK requires the root
   module at the same version, and the workflow refuses a mismatch:

   ```bash
   git switch main && git pull
   git tag v0.2.0 && git tag sdk/go/v0.2.0
   git push --atomic origin v0.2.0 sdk/go/v0.2.0
   ```

3. Watch the `release` workflow. It tests the commit, builds, pushes and signs the
   images (gateway, collector, ui, and the quickstart's init and demo-agent),
   builds the `conduitctl` binaries with a cosign-signed `SHA256SUMS`, publishes to PyPI, checks that the Go SDK resolves the way a consumer
   would fetch it, and creates the GitHub Release from the changelog section.

A failed run can be re-run: images are rebuilt and re-signed, and PyPI files that
already exist are skipped. PyPI versions and Go module tags are permanent, so fix a
bad release with a new patch version rather than by moving a tag.

---

<!-- Source: https://tjscanlan.github.io/conduit/contributing.md -->

# Contributing

Thanks for helping improve Conduit. This guide covers local setup, the
checks your change has to pass, and how changes get reviewed and merged.

## Prerequisites

| Tool | Version | Used for |
|------|---------|----------|
| Go | 1.26+ (the `toolchain` line in `go.mod` pins the exact patch release) | Everything |
| Docker with Compose v2 | recent | The local stack and the smoke test |
| Python | 3.10+ | The Python SDK |
| [uv](https://docs.astral.sh/uv/) (optional) | recent | Python environments and codegen |
| OpenSSL | any | `scripts/gen-certs.sh` |

## Repository layout

```
cmd/{gateway,collector,ui}   service entry points
internal/{auth,health,schema,storage,telemetry}
proto/                        .proto sources and the generated Go code (committed)
sdk/go                        Go SDK, a separate module
sdk/python                    Python SDK (src/conduit_observability, with generated stubs committed)
ui/                           templates and static assets, embedded into cmd/ui
deployments/{docker,k8s}      compose stack and Kubernetes manifests
scripts/                      certs, codegen, smoke test
docs/                         this documentation
```

## Checks your change must pass

CI runs all of these on every push and pull request. Run the relevant ones
locally first.

```bash
gofmt -l .                                  # must print nothing
go build ./... && go vet ./...
go test -race ./...                         # root module
(cd sdk/go && go test -race ./...)          # Go SDK (a separate module)
(cd sdk/python && python -m pytest)         # Python SDK
make lint                                   # go vet + govulncheck
```

**End-to-end smoke test.** Start the stack (see [Deployment](deployment.md)), then run:

```bash
CLICKHOUSE_PASSWORD=... UI_USER=admin UI_PASSWORD=... ./scripts/smoke-test.sh
```

It sends traces with both SDKs and checks ClickHouse, VictoriaMetrics, Tempo,
the UI and every service's health endpoint. CI runs it against a fresh stack.

## Generated code

The Go and Python protobuf stubs are committed. After editing anything in
`proto/`, run:

```bash
make proto
```

and commit the regenerated files in the same commit. `make proto` also
regenerates `docs/api.md`, the API reference, which is built from the
`.proto` files and the validation constants in `internal/schema`. Run
`make api-docs` on its own after changing only a limit. Both generators pin
their tool versions, and CI fails if the committed code differs from what
they produce.

## Schema changes

Add a new file to `internal/storage/migrations/` named
`NNNN_description.sql`, using the next number.
- **One statement per file,** and it must be idempotent: `CREATE … IF NOT
  EXISTS`, `ALTER TABLE … ADD COLUMN IF NOT EXISTS`, and so on.
  `TestMigrationsAreIdempotent` enforces this.
- **Never edit or renumber a migration that has shipped.** Applied
  migrations are checksummed, and the collector refuses to start if one
  changes.

## Conventions

- **Go:** use the standard library where you can. A new module needs a
  short justification in the PR, because the project's pitch includes "no
  supply-chain surprises". Match the style of the surrounding code.
- **Tests:** a behavior change comes with a test. Security-relevant code,
  such as auth, validation and rate limiting, needs tests for the
  rejection paths as well as the happy path.
- **Storage:** ClickHouse queries use bound parameters, never string
  interpolation of user input. Pass times as integers. Binding a
  `time.Time` positionally truncates it to whole seconds.
- **UI:**
  - **No inline scripts or styles:** the CSP forbids them, and
    `TestTemplatesHaveNoInlineScript` enforces it.
  - **Links:** build links with query strings as `template.URL` in Go.
    `html/template` re-encodes query strings interpolated into an `href`.
  - **Colors:** follow the contrast rules in [brand.md](brand.md).
- **Docs:** if you change behavior, configuration or a guarantee, update
  the matching document in `docs/` in the same PR. If you add a doc, also
  add it to `docs/_data/docs.json`, with a one-line description. That file
  drives the site nav and the agent-readable `llms.txt`, and CI fails if a
  doc is missing from it.
- **Commits:** use the form `area: what and why`, for example
  `collector: retry failed ClickHouse writes`, and keep each commit to one
  change.

## Pull requests

1. Branch from `main`.
2. Keep each PR to one coherent theme. The description should explain
   what changed and why, and call out:
   - breaking changes and upgrade steps
   - security-relevant changes
   - what you verified, and what you didn't
3. CI must be green, with all jobs passing: `test`, `generated`,
   `python-sdk`, `vuln`, `docker` and `smoke`.
4. A maintainer merges with a merge commit.

### Automated contributions

Some changes come from an automated work loop driven by `.claude/loop.md`.
It works on `loop/<feature-name>` branches and opens a PR at each stopping
point. It never merges and never pushes to `main`. Its working notes are in
`.claude/loop-log.md`. These PRs go through the same review as any other.

## Reporting security issues

Please **don't** open a public issue for a vulnerability. Report it
privately to the maintainers, through GitHub's private vulnerability
reporting for this repository. See [Security](security.md) for the threat
model.

---

<!-- Source: https://tjscanlan.github.io/conduit/brand.md -->

# Brand

## Logo

<img src="assets/img/logo.svg" alt="Conduit logo" width="96" height="96">

The mark is five concentric arcs forming a "C" that opens toward the
direction of flow. The colors run from red on the outside to yellow at the
center, a nod to data moving through a conduit.

| File | Use |
|------|-----|
| `ui/static/img/logo.svg` | The mark on light or dark backgrounds, at 24 px and up |
| `ui/static/img/favicon.svg` | The mark on a white rounded tile, for browser tabs and app icons |

Pair the mark with the word **Conduit** set in the UI's sans-serif at 700
weight. Leave clear space of at least a quarter of the mark's width around
it. Don't recolor, rotate, stretch or outline the mark.

## Palette

| | Hex | RGB | Role |
|---|-----|-----|------|
| ![](https://img.shields.io/badge/-%20%20%20%20-ec0000) | `#ec0000` | 236, 0, 0 | Primary: links, primary buttons, errors |
| ![](https://img.shields.io/badge/-%20%20%20%20-ec2400) | `#ec2400` | 236, 36, 0 | Gradient and accents |
| ![](https://img.shields.io/badge/-%20%20%20%20-ec5300) | `#ec5300` | 236, 83, 0 | Accents, chart series |
| ![](https://img.shields.io/badge/-%20%20%20%20-ec9b00) | `#ec9b00` | 236, 155, 0 | Waterfall bars, warnings, highlights |
| ![](https://img.shields.io/badge/-%20%20%20%20-ecca00) | `#ecca00` | 236, 202, 0 | Tags and highlights |
| ![](https://img.shields.io/badge/-%20%20%20%20-ffffff) | `#ffffff` | 255, 255, 255 | Surfaces |

**Supporting neutrals,** needed for legible text since the palette has no dark color:
- `#1c1917` ink, for body text and the nav bar
- `#57534e` secondary text
- `#e7e5e4` borders
- `#fafaf9` page background

The brand gradient runs through all five colors from left to right. Use it
only for the nav rule, card accents and the logo, never behind text.

## Accessibility rules

These come from measured contrast ratios (WCAG 2.1), and the UI follows them.

| Combination | Ratio | Rule |
|-------------|-------|------|
| `#ec0000` on white | 4.6:1 | **The only palette color allowed as body text on white.** |
| White on `#ec0000` | 4.6:1 | OK for button labels |
| `#ec2400` / `#ec5300` on white | 4.4 / 3.6:1 | Large text (18 px and up) and graphics only |
| `#ec9b00` / `#ecca00` on white | 2.3 / 1.6:1 | **Never** for text. Use them as fills with ink text on top (7.7 / 10.8:1). |
| Error text on the red tint `#fef2f2` | | Use `#b80000` (6.3:1). `#ec0000` drops to 4.2:1 there. |
| `#57534e` on white | 7.6:1 | Secondary text |

Never rely on color alone for status. Status badges always include a word
(`OK`, `ERROR`), and error rows also get a red edge.