Specification overview
The Lakestream API: three levels, one storage contract.
The Lakestream API is organized into three levels that share one underlying storage contract.
What the spec is today
The normative surface is code: the lakestream-api Java module, in the ursa-storage repo, under the io.streamnative.lakestream.api package. It defines interfaces and records only — no implementations — and every operation a conforming storage layer exposes is described there first. This section works through that module, level by level.
Three companion pages describe the storage-level formats these interfaces assume, rather than define directly: data format covers the WAL object and compacted object binary layouts, metadata covers how stream and offset metadata is laid out, and operations covers the append, read, acknowledge, retain, and compact contract each storage layer implements underneath the interfaces below.
The three levels
Reading top-down, from what a protocol layer touches first to what it ultimately runs on:
Level 2 — stream catalog
StreamCatalog, Stream, StreamLayout, StreamReader, and StreamWriter — streams, namespaces, and the routing that maps a logical stream onto one or more logs. This is what a protocol layer builds on: topic and partition semantics, at the Kafka or Pulsar level, are implemented in terms of these types. See Stream Catalog.
Level 1 — log storage
LogStorage and LogId — durable, addressable logs as primitives, with no notion of streams or catalogs above them. A caller holding a LogId can append to and read from a log without going through a catalog at all. See log storage.
Level 0 — log and cursor
Log and LogCursor — per-log operations obtained through a stream rather than the catalog directly: append, read, acknowledge, fence. This is the layer Ursa's Pulsar-compatible ManagedLedger and ManagedCursor wrap. See log storage.
Why layered
An implementation, or an embedder linking the library directly, only has to reach as high as the level it needs. A Kafka- or Pulsar-compatible broker needs Level 2 for topic and partition semantics. A component that only needs a durable, appendable log — no catalog, no stream metadata — can address Level 0 or Level 1 directly; nothing above Level 1 requires it.
Implementations
Ursa is the reference implementation and targets all three levels. Clients & libraries covers the two ways to consume a lakestream: embedding the Java library and calling this API directly, or speaking a protocol to a broker built on it.
None of this means much without a definition of "conforming" — see conformance for the properties a system has to provide before it can call itself a lakestream.