/// HIGH-PERFORMANCE LOG INGESTION SERVICE

8,000,000 ROWS. 6.5 SEC.
ZERO DROPS.

Buffers log lines in memory, flushes in large batches through DuckDB’s appender into a DuckLake catalog — a local file for metadata, Parquet data files on disk or S3. One Go binary. No JVM. No agent mesh. No cluster.

$ ./duckgraph -addr :8080 -data data [ README.MD → ]

2.9M ROWS/S — PEAK ACCEPT
GOCCY JSON PARSE

1.05M ROWS/S — SUSTAINED
R2 DATA PATH

0.69 BYTES/ROW — ZSTD
REPETITIVE LOGS

200ms REPLICA LAG — SHARE
CATALOG MIRROR

[ POST /v1/logs — NDJSON STREAM ] LIVE 1,048,576 ROWS/S

[ 01 / BENCHMARKS ]

M4 · GRAIN RUNTIME · MEASURED, NOT EXTRAPOLATED ///

INGEST → R2 DATA PATH

BATCH	SUSTAINED
8,192	~24K r/s
65,536	~129K r/s
131,072	~258K r/s
262,144	~440K r/s
524,288	~600K r/s
1,048,576	~1M+ r/s

SMALL BATCHES ARE COMMIT-LATENCY-BOUND — 8 WRITE THREADS PIPELINE THE PUTS

ACCEPT PATH — PARSER

PARSER	NS/LINE	THROUGHPUT	ALLOCS
encoding/json	1324	192 MB/s	12
goccy/go-json	251	1012 MB/s	6

LARGE-SCALE RUNS

8M REPETITIVE	7.7S ACCEPT	49 FILES · 5.3MB
4M HIGH-ENTROPY	6.2S ACCEPT	WIRE-BOUND 90MBPS

ZSTD → 0.69 B/ROW MAKES BANDWIDTH IRRELEVANT; RANDOM PAYLOAD SATURATES THE PIPE

BOTTLENECK MAP

REGIME	LIMITER
SMALL BATCH	COMMIT LATENCY
R2 + REPETITIVE	JSON PARSE → FIXED
R2 + RANDOM	UPLOAD BANDWIDTH
LOCAL DISK	ENCODE CPU — 2M+ r/s
PG CATALOG	PG COMMIT — ~370K r/s

NOTHING BEATS PHYSICS — THE UPLINK IS THE CEILING, NOT THE SOFTWARE

[ 02 / PIPELINE ]

ONE WRITER SERIALIZES FLUSHES · MVCC SNAPSHOT READS ///

CLIENT>>> HTTP :8080
POST /v1/logs>>> CHANNEL BUFFER
QUEUE 262144>>> BATCH FLUSHER
500MS TICK>>> DUCKDB APPENDER>>> DUCKLAKE CATALOG>>> PARQUET FILES
LOCAL | S3://

TOPOLOGY / SINGLE

./duckgraph -mode serve \
  -data data \
  -admin-password '…'

ONE BINARY — INGEST + QUERY + DASHBOARD

TOPOLOGY / REPLICAS

writer: CHECKPOINT →
  catalog-snapshot.duckdb
reader: -mode query \
  -catalog …snapshot.duckdb
lag ≤ publish + refresh

FILE CATALOG STAYS FAST — SNAPSHOTS FAN OUT

TOPOLOGY / PG MIRRORS

-replicate-catalog \
  'postgres:dbname=mirror…'
mode share | copy
share: same files, ~5s
copy:  physical, ~17s

READERS ATTACH POSTGRES DIRECTLY — N OF THEM

[ 03 / CAPABILITY MATRIX ]

SHIPPED FEATURES — ALL IN ONE BINARY ///

[ AUTH.RBAC ]

API keys + sessions. PBKDF2-SHA256. admin / writer / reader — guards on last-admin and self-demotion.

[ INGEST ]

JSON object, array, NDJSON, gzip. 202 accepted, 503 on full queue. ts in RFC3339 or epoch.

[ QUERY ]

Severity floor, multi-service/host, substring, attr.* JSON filters, ordering, pagination.

[ SSE.TAIL ]

Live tail over Server-Sent Events. Same filters as query. 15s heartbeats.

[ SDK.GO ]

Dependency-free client. Batching, gzip, retries w/ backoff, slog handler, drop counter.

[ UI.HTMX ]

Server-rendered dashboard, htmx vendored — zero CDN. Built-in admin panel.

[ REPLICATE ]

Native in-writer supervisor. share or copy mode, two destinations, 30s watchdog resumes.

[ COMPACT ]

DuckLake maintenance on a schedule — merge small files, expire snapshots, delete orphans.

[ S3.R2 ]

Object-storage data path. Metadata/connection caching, http_retries. Auto 8 write threads.

[ 04 / WIRE FORMAT ]

REAL REQUESTS — COPY, PASTE, SHIP ///

+INGEST.SH+

curl -X POST :8080/v1/logs \
  -H 'Authorization: Bearer dgk_…' \
  -H 'Content-Type: application/x-ndjson' \
  --data-binary @batch.ndjson

# 202 {"accepted": 65536}

+QUERY.JSON+

POST /v1/query
{
  "min_level": "warn",
  "services": ["api"],
  "attrs": {"env": "prod"},
  "q": "timeout",
  "order": "desc",
  "limit": 100
}

+SHIP.GO+

c, _ := sdk.New(base,
  sdk.WithAPIKey("dgk_…"),
  sdk.WithGzip())

slog.SetDefault(slog.New(
  sdk.NewSlogHandler(c,
    &sdk.SlogOptions{
      Service: "myapp"})))

slog.Info("started")

[ 05 / FAILURE MODES — EXERCISED ]

NOT THEORETICAL ///

SCENARIO	RESULT	STATUS
SHARE + INLINED ROWS	HARD ERROR ON REPLICATE	FIXED — INLINE=0 + RESUME-ALL
JOB ERRORS MID-RUN	SILENT STALENESS	FIXED — 30S WATCHDOG
WRITER RESTART	—	RESUMES FROM DEST STATE
SOURCE FILES DELETED (COPY)	MIRROR UNAFFECTED	VERIFIED — OWNS COPIES
WRITER CATALOG LOST	PG MIRROR = FULL METADATA REPLICA	RE-REGISTERABLE
SUPERVISOR DEATH	STOPS WITH WRITER	RESUMES ON RESTART

8,000,000 ROWS. 6.5 SEC.ZERO DROPS.