Stream–table duality

Stream–table duality is producing two access patterns from one retained log inside the storage lifecycle: offset-ordered stream consumption and analytical table scans. The stream side is served from the log's own compacted storage; the table side is materialized from the same log by the stream materialization framework, not exported by a separate connector.

One log, two outputs

Entries first land in raw WAL objects. Background compaction reads a WAL range once and fans that single pass out to every configured output: internal compacted objects for the stream and, when a materialization policy resolves, an external destination table. The two outputs are separate files with separate lifecycles. "Duality" describes their common source and shared production step, not shared bytes.

It also does not mean there is no physical rewrite, no storage-level redundancy, or never a period in which WAL and compacted files coexist.

The stream view

A stream reader uses a log ID and record offset. Ursa's unified reader resolves the index and routes to raw storage or a compacted-object reader. A reader resuming at an older offset does not need to choose a tier itself, but the requested range must still be retained and the appropriate compacted reader must be configured.

Entries can contain multiple records. An entry at offset o with n records covers [o, o + n); the next entry begins after that range. Ordering is per log. Combining a stream's logs does not create a global order.

The table view

A query engine reads a committed table snapshot through its table format and catalog. It does not read uncommitted Parquet files or automatically include the WAL tail.

Consequently, the stream and table can expose different frontiers: a successful append may already be consumable while its table materialization is still queued, writing files, or waiting for a catalog commit. Query-time ordering is also different: a table scan is not an offset-ordered subscription.

Internal compacted objects and delivered tables

Ursa keeps two kinds of compaction output:

Earlier Ursa code and design documents described a third arrangement, an "internal" or "stream-backed table" (SBT) whose compacted files were registered in a lakehouse catalog as well as the offset index. Ursa 1.0.0 removed that concept: internal COs are not catalog tables, and TableCatalogType.NONE selects storage-only compaction with no table sink at all. The TableMode enum (MANAGED / EXTERNAL / CUSTOM) that expressed table ownership was removed with it.

External delivery can support destination-specific partitioning or upserts. Such transformations must not be assumed to preserve the source log's append-only ordering or byte representation. The internal COs and the delivered table are separate outputs, not a zero-copy pair.

What the duality does and does not promise

• A record acknowledged to the log is readable as a stream before it is visible in any table.
• One compaction pass produces both outputs, so there is no second read of the source and no separately operated export job. There are still two sets of files.
• Trimming internal COs through stream retention does not touch the delivered table, and dropping or rewriting the delivered table does not change the stream's offset index.

Choose the destination and verify its implemented data path. See architecture for index publication and zero-ETL for materialization policy.