Lakestream

PROJECTS

Projects

Lakestream is an open API and specification for stream storage on object storage, with a stream materialization framework that defines how a stream becomes a lakehouse table. Ursa implements it as a storage engine; Ursa for Apache Kafka (UFK) is a Kafka distribution built on it. Both are open source under Apache 2.0. Alongside them, streaming-proof checks the delivery and ordering guarantees of a running system, and leaderless-log-protocol specifies and model-checks the coordination protocols the storage layer rests on. All four live in the openlakestream GitHub organization.

Runtimes

Ursa

STORAGE ENGINE · 1.0

StreamNative's implementation of the spec: WAL objects on object storage, compaction to Parquet, materialization to Iceberg and Delta. Java, embeddable, and the subject of the VLDB 2025 Best Industry Paper.

DOCS →openlakestream/ursaVLDB paper

Ursa for Apache Kafka (UFK)

KAFKA DISTRIBUTION · 4.3.1

A Kafka distribution built on the Lakestream API and specification, whose diskless topics store records through Ursa: brokers hold no partition data, any broker is eligible to serve a partition, and compaction writes Parquet that can be queried as a lakehouse table. One config flag per topic.

DOCS →openlakestream/kafka

Protocols and verification

leaderless-log-protocol

PROTOCOL SPECS · TLA+ · FIZZBEE

Markdown specifications of the leaderless log protocol and task claiming, model-checked in TLA+ and Fizzbee over bounded configurations, and the coordination-delegated pattern they build on.

DOCS →openlakestream/leaderless-log-protocol

streaming-proof

VERIFICATION HARNESS

A black-box harness that drives a running deployment through its ordinary client protocol and reports missed, duplicated, and out-of-order messages per key. Results describe one run, not a certification.

DOCS →openlakestream/streaming-proof

Which one do I want?