Stream–table duality

Stream–table duality describes how one copy of data serves both a tailing stream and a queryable table.

One copy, two views

Records append to a log in arrival order. The newest data — the hot window — lives in row-oriented WAL objects, which multiplex records from many logs and are written for fast, durable appends, not for scanning. On a schedule, the compaction service folds the WAL objects belonging to one log into a compacted object, typically stored as columnar Parquet, and commits it to a table catalog (Parquet only) — the same mechanism that lets tables appear without a separate ingestion pipeline (see zero-ETL).

From the moment that commit lands, the same compacted object backs both a stream read and a table read. There is no export step and no second write; see architecture for how the catalog, storage, and compaction services divide this work.

The tailing stream

A consumer that wants offset-ordered, low-latency reads tails the stream: it reads WAL objects for the hot window and, once data compacts, reads the equivalent range from compacted objects instead. The Stream Offset Index maps each logical offset to whichever physical object currently holds it, so a consumer resuming from an offset does not need to know whether that offset's data is still in a WAL object or has already been compacted.

The queryable table

The same compacted objects are Parquet files committed to an Iceberg or Delta Lake table, so any engine that reads those formats can scan the log as a table: filter it, aggregate it, join it against other tables, with whichever query engine the reader already runs. No separate copy is maintained for analytics — compaction is the only step between a record landing and that record being queryable.

Internal and external tables

A compacted log can be exposed as a table in two ways. An internal table — also called SBT, for stream-backed table — is one the storage layer manages end-to-end: because its compacted objects are indexed by both the Stream Offset Index and the table's metadata, the storage layer owns the whole lifecycle, and the data stays readable as a stream. It is typically still registered into an external data catalog so query engines can discover it, even though the storage layer continues to own it underneath. Internal tables suit raw data — the bronze layer, in medallion-architecture terms.

An external table — also called SDT, for stream-delivered-to table — is delivered into a catalog you run instead, where both data and metadata belong to that external system. Once data lands there it is no longer indexed by the Stream Offset Index, so it cannot be read back as a stream; getting a stream view of the same events means keeping a second, indexed copy. In exchange, external tables can be repartitioned or updated in ways an internal table cannot, which suits curated, silver- or gold-layer data. See lakehouse tables for how Ursa configures both modes.