Architecture

This page maps the services and objects behind a lakestream, from the catalog down to WAL and compacted objects.

System components

A lakestream is built from three services, each responsible for one part of the path from an appended record to a queryable table.

Stream Catalog Service

The Stream Catalog Service is the centralized metadata authority. It assigns offsets, maintains the Stream Offset Index, and holds log and stream metadata — including the Stream ID assigned to each log at creation and, where the pattern supports transactions, transaction state. Centralizing offset assignment here, rather than having each partition's leader assign its own, is what makes the pattern leaderless: no node has to coordinate with peers to agree on the next offset, and no single node's failure blocks that partition's ordering. See diskless & leaderless for what this changes about failover.

That metadata needs a durable, low-latency store underneath it. Ursa, the reference implementation, uses Oxia, a distributed key-value store, to hold offset assignments and log metadata; Ursa for Kafka also persists idempotent-producer state there, so a reassigned broker can recover producer sequence state without replaying the entire log.

Stream Storage Service

The Stream Storage Service is the read and write path for log data. Writes land first as row-oriented WAL objects — the write-ahead log tier, which can be backed by a replicated log system such as Apache BookKeeper, an object store, or a distributed file system. Reads come from whichever objects currently hold the requested range: WAL objects for the hot window, compacted objects in lakehouse storage for anything already folded into long-term retention.

This split — a write-optimized hot tier and a scan-optimized long-term tier, both addressed through the same offset space — is what lets one service answer both a low-latency tail read and a range read over older data.

Compaction Service

The Compaction Service is the background process that turns many small, multiplexed WAL objects into fewer, log-specific compacted objects. Per log, it:

• converts row-oriented WAL data into columnar Parquet
• sizes the resulting files for query performance rather than write latency
• registers each compacted file with the destination table format's catalog
• updates the Stream Offset Index so offsets resolve to the new physical locations
• enforces the log's retention and deletion policy

That row-to-columnar conversion uses the log's schema, managed through an external schema registry, to map records onto Parquet's columnar structure. Compaction is the step that turns a log written once into a table — see zero-ETL.

The object lifecycle

A record's path through the pattern: it is appended, recorded in a WAL object, later folded into a compacted object, and — for internal tables — committed to a table. At every step, the Stream Offset Index is updated to map that record's logical offset to wherever it currently lives physically, so a consumer resuming from an offset, or a query engine reading a table snapshot, resolves to the right bytes regardless of which stage the record is in.

Most compacted objects are columnar Parquet, the form that backs table reads; a log that does not need table access can stay on a row-based compacted object instead, since compaction's job is consolidating WAL objects, not forcing a format change. The Stream Offset Index can also carry secondary indexes beyond the logical offset — for example, mapping Pulsar message IDs to offsets when the pattern serves as tiered storage behind a classic Pulsar cluster.

append
  │
  ▼
WAL object ────────────────► tail reads (hot window)
  │
  │  compaction
  ▼
compacted object (row / Parquet)
  │
  │  catalog commit (Parquet only)
  ▼
table (Iceberg / Delta Lake) ─────► table scans

Stream Offset Index: logical offset → physical location — updated at every step

See the specification for how these services and objects are defined as a concrete contract, and the glossary for short definitions of each term on this page.