Architecture

Where concepts: architecture describes the Lakestream pattern's services in the abstract, this page maps them onto Ursa's actual code — the modules you'd build, embed, or read to find the same three services at work.

The module stack

Reading top to bottom, from the interfaces a caller programs against down to the engine that moves bytes:

lakestream-api — the normative interfaces

lakestream-api is the Lakestream API itself: StreamCatalog, Log, LogCursor, LogStorage, StreamReader, and StreamWriter, defined as interfaces and records with no implementation and no internal dependencies of its own. It's the same module the specification documents level by level — Ursa's copy of the API is that module, not a separate mirror of it.

ursa-storage-lakestream — the implementation and integration surface

ursa-storage-lakestream implements the API: IndexedStreamCatalog for catalog operations, LogImpl and LogCursorImpl for per-log operations, and the stream layouts — IndexedLayout, SingleLogLayout — that map a stream onto one or more logs. This is the module external consumers embed directly; Ursa for Kafka links against it, and it's what Java clients covers for embedding it yourself.

ursa-storage-core — the internal engine

ursa-storage-core is where WAL objects are actually written and read: WalStorage over the object store or replicated log system underneath, FileStorage for the cloud calls themselves, and the read and write caches in front of both. As WAL storage backends, AWS S3, GCS, and Azure Blob Storage are supported today; Apache BookKeeper support is in progress — see feature matrix. Core's own StorageApi predates the Lakestream API and is being superseded by it: new code targets lakestream-api and ursa-storage-lakestream instead of calling StorageApi directly.

ursa-storage-lakehouse — table formats

ursa-storage-lakehouse depends on core and implements the table formats compacted objects can be committed into — Apache Iceberg and Delta Lake — along with the multi-catalog support covered in the feature matrix.

ursa-storage-materialization — schema and format conversion

ursa-storage-materialization applies a log's schema to compacted data on the way into a table format, converting row-oriented entries into the columnar records ursa-storage-lakehouse writes — the step that lets a schema evolve without breaking the tables it materializes into.

ursa-storage-compact — the compaction service

ursa-storage-compact depends on core, ursa-storage-ml, and ursa-storage-lakehouse, and orchestrates the WAL-to-compacted-object pipeline described below.

ursa-storage-ml and the Pulsar modules — protocol adapters

ursa-storage-ml bridges Pulsar's ManagedLedger interface to Log and LogCursor — a Pulsar-only adapter; stream and cursor logic itself lives in ursa-storage-lakestream, not here. ursa-storage-pulsar and ursa-storage-pulsar-ml extend that bridge: tiered storage offload, and the ManagedLedger implementation that replaces BookKeeper, respectively. Together, this is the layer that lets a Pulsar- or Kafka-compatible broker run on Ursa storage without touching the Lakestream API's Java types directly.

lakestream-api                interfaces: StreamCatalog, Log, LogCursor, LogStorage
      │
      ▼
ursa-storage-lakestream       implementation + integration surface
      │                       (IndexedStreamCatalog, LogImpl, layouts)
      ▼
ursa-storage-core             internal engine: WalStorage, FileStorage, caches
      │                       WAL objects on S3 / GCS / Azure Blob (BookKeeper in progress)
      ▼
ursa-storage-compact           folds WAL objects into compacted objects
      │                       via ursa-storage-materialization (schema)
      ▼                       into ursa-storage-lakehouse (Iceberg / Delta, multi-catalog)

ursa-storage-ml, ursa-storage-pulsar, ursa-storage-pulsar-ml
      wrap Log / LogCursor as Pulsar's ManagedLedger / ManagedCursor, so
      Pulsar- and Kafka-compatible brokers run on this same stack.

The runtime view

The modules above implement three services — described at the pattern level in concepts: architecture, and here as what Ursa's code actually runs:

The Stream Catalog Service assigns offsets and owns the Stream Offset Index, the mapping from logical offset to physical location every read resolves through. Centralizing that assignment in one service — backed by Oxia, a distributed key-value store — is what removes the need for leader election: no broker node owns a partition, so no single node's failure blocks it.
The Stream Storage Service is the read and write path ursa-storage-core implements: writes land as WAL objects; reads come from whichever object, WAL or compacted, currently holds the requested range.
The Compaction Service, implemented by ursa-storage-compact, periodically folds a log's WAL objects into a compacted object — converting row-oriented data to Parquet with ursa-storage-materialization's schema handling, registering the result with the destination table format's catalog via ursa-storage-lakehouse, and updating the Stream Offset Index to point at the new location.

Oxia sits beneath all three as the metadata store: stream and log metadata, offset assignments, and cursor state.

See concepts: architecture for this same system described independent of any one implementation's module boundaries.

On this page