Why Lakestream

This page lays out the cost and complexity drivers that make lakehouse-native streaming worth standardizing.

The cost of duplication

A broker-replicated streaming system pays for durability with replication: every message a producer sends gets copied to follower replicas so the system can survive a node failure, and when those replicas span availability zones, that copy is also cross-AZ network traffic. The nodes serving this traffic are also the nodes holding the data on local or attached disk, so compute and storage scale together — adding capacity means moving partition data, not just adding a process.

That is the cost of keeping one system durable. Getting the same events into a lakehouse adds a second: a connector or job reads the stream and writes it again, as Parquet, into Iceberg or Delta Lake tables. The bytes a broker replicated for durability and the bytes a lakehouse holds for analytics are, by then, two unrelated copies of the same events — each with its own storage cost, its own access controls, and no guarantee the two agree at any given moment.

The complexity of ETL pipelines

That second copy does not appear on its own; a pipeline has to produce it, continuously. The pipeline reads the stream, transforms records into the destination table's schema, and writes them on its own schedule, which means every event has a gap between when it's produced and when it's queryable, sized to that schedule rather than to how fast the stream actually moves.

The pipeline is also a maintained system in its own right. Schema changes on the source have to be propagated to it, backfills and reprocessing have to be re-run through it, and if two teams want table access to the same stream, they either share one pipeline as a dependency or build two that can drift from each other.

The case for a standard

A single product could remove one team's pipeline. It would not remove the underlying problem, because it would tie the result to that product's storage format and that product's supported engines — a different kind of lock-in in place of the pipeline. Standardizing the pattern instead — open formats, on object storage, under a storage contract any implementation can read and write — removes that trade-off. A Kafka-compatible broker, a Pulsar-compatible broker, or a native client can all write the same kind of log; any engine that reads Iceberg or Delta Lake can query it, without agreeing on anything beyond the format.

The next three pages work through how: stream–table duality removes the second copy, diskless & leaderless removes the replication cost, and zero-ETL removes the pipeline. Ursa for Kafka is one concrete example of a broker built this way.