Glossary

This glossary distinguishes the public stream model from older storage terminology. Implementation details refer to Ursa and Ursa for Apache Kafka (UFK); see architecture for how the components fit together.

Catalog

An ambiguous term unless qualified. The stream catalog manages namespaces, streams, layouts, and policies. A table catalog manages or locates table metadata for analytical readers. Registering a table in an externally operated catalog does not make it an external/delivered table.

Compacted object (CO)

An object produced by consolidating data for a log. It can retain raw entries or use a columnar representation. Ursa's internal compacted objects are referenced by the offset index and serve stream reads; they are storage files, not table files, and are not registered in a table catalog.

Compaction

Consolidating log data into retained files. Lakehouse materialization can additionally decode records and write Parquet. This use of "compaction" is not inherently Kafka key-based log compaction, which keeps the latest value per key.

Compaction Service

Background scheduling and processing for compaction tasks. File generation, table commits, index replacement, and cleanup are separate steps; retention is not exclusively owned by one service. See architecture.

Cursor

A per-log read and acknowledgment handle (LogCursor). The read offset is the next position to read; the mark-delete offset acknowledges entries through that position. Durable cursors persist acknowledgment state; ephemeral cursors hold in-memory state. Cursor acknowledgment is not the same as physically deleting source data.

Delta Lake

An open table format supported by Ursa's lakehouse materialization. It is an external destination: the delivered table does not provide stream read-back.

Diskless

Serving without broker-local durable partition replicas. Brokers may still have buffers, caches, and recoverable in-memory state. Durability depends on the configured shared storage and metadata services. See diskless & leaderless.

Entry

The unit submitted to a log append: a payload with a record count. One entry can contain one or many records and occupies a contiguous range of record offsets. A batch append is not a transaction across multiple logs.

Entry format

The encoding of an entry's payload and metadata. The storage API accepts bytes and a record count; decoding those bytes into table columns requires the matching serialization support.

External table (SDT)

In the lakehouse data path, a stream-delivered-to table: destination data is written from the same WAL read as the internal compacted objects but into separate files owned by the destination system. It does not itself provide source-log replay. Ursa 1.0.0 has no table-ownership mode. See stream–table duality.

Fencing

Preventing operations against state that should no longer accept them. A log can be marked FENCED; stream lifecycle fencing, per-holder storage leases, and Kafka producer-snapshot ownership checks are separate mechanisms. Do not interpret a local log fence as a complete distributed single-writer protocol.

Hot window

The recent, uncompacted portion of a log served from raw WAL storage or caches. Its extent depends on processing progress, not just elapsed time; table queries do not automatically include it.

Iceberg

Apache Iceberg, an open table format for files and snapshots. Ursa materializes into Iceberg as an external destination. Query engines read committed table metadata, not the live WAL or the stream offset index.

Implementation

A system that implements the standard's model and API. Ursa implements Lakestream as a storage engine; its backend choices and plugin coverage should not be treated as universal requirements.

Internal compacted objects

Ursa's per-log compacted objects, indexed by the stream offset index for replay and never registered in a table catalog. Earlier Ursa releases could also register them in a lakehouse catalog as an "internal" or "stream-backed table" (SBT); Ursa 1.0.0 removed that concept, so they are storage files only.

Lakehouse storage

The retained-file tier used by lakehouse processing and analytical readers, commonly Parquet files in object storage. Stream read-back additionally requires an offset index and a compatible compacted-object reader.

Lakestream

Lakestream is an open API and specification for stream storage on object storage, with a stream materialization framework that defines how a stream becomes a lakehouse table. Ursa implements it as a storage engine; Ursa for Apache Kafka is a Kafka distribution built on it. Both are open source under Apache 2.0. Capitalized, the word names the standard; lowercase, "a lakestream" is a conforming system. See What is Lakestream.

Leaderless

In the serving data path, offset allocation and durability do not require leader/follower replication among brokers. Shared metadata still coordinates ordering, and Kafka still advertises a selected serving broker in its leader field.

Log

An independently ordered, append-only sequence of entries identified by LogId. A stream can contain multiple logs. Offsets are local to a log, and retained data can be trimmed or deleted through lifecycle operations.

Log format

The storage representation of entries, WAL objects, compacted objects, and their indexes. It is distinct from the stream layout, which determines which logs belong to a stream.

Log ID

The numeric LogId identifying one log. Older Ursa storage code often names this value streamId; it is not the public StreamIdentifier of a multi-log stream.

Materialization

Decoding log entries and producing a destination representation according to schema and policy. In the lakehouse path, flushing materialized files and committing them to table metadata are distinct stages.

Materialization policy

TableMaterializationPolicy: namespace defaults and stream overrides for destination selection and materialization behavior. Resolution requires an available referenced catalog and allows stream-level opt-out. Supported behavior depends on the selected writer.

Medallion architecture

A data-modeling convention with bronze (raw), silver (cleaned), and gold (curated) layers. These are application choices, not Lakestream API modes or guarantees about table ownership.

Metadata store

The durable store for shared metadata and storage indexes. Ursa uses Oxia; the public API does not require callers to operate on Oxia keys directly.

Object storage

Blob storage such as S3, GCS, or Azure Blob Storage, used by Ursa's object-backed WAL and retained-file paths. Its redundancy and availability are separate from Kafka's replication factor.

Offset

A record position within one log. An entry at offset o containing n records covers [o, o + n). Offsets are not global across logs, timestamps, or physical file locations.

Oxia

The distributed key-value store Ursa uses for metadata and indexed offset sequencing. The Kafka integration also stores producer-state snapshots there; recovery can replay entries after a snapshot.

Parquet

A columnar file format used by lakehouse materialization. A Parquet file alone is neither a committed table snapshot nor a replayable log: each access path needs its corresponding metadata and reader.

Partitioning

The composition of a stream into logs. Ursa currently implements INDEXED layouts with ordinal log selection. Table partitioning is a separate destination layout choice and need not match source-log partitioning.

Reassignment

Changing which broker serves a diskless partition. Shared files avoid local partition-data migration, but routing updates, state recovery, and cache warming can still be needed.

Record

An individual event or message within a log entry. A record offset identifies its position within that log's ordered sequence.

RecordBatch

Kafka's encoded batch of records. It is protocol-specific, not a universal alias for every Lakestream entry; entries carry opaque payloads and record counts at the API boundary.

Retention

Policy governing how long or how much data remains available. Logical trimming, physical file cleanup, and table lifecycle are distinct operations. The Kafka integration also schedules retention based on topic metadata.

RoutingKey

The context passed to select a log for a stream write. The current type carries an optional integer index: ofIndex selects an ordinal log, and roundRobin requests round-robin routing. It has no record-key field or implemented hash/range routing.

Schema

The record structure used for decoding and column materialization. Schema sources and supported evolution vary by serialization and destination; not all schemas require an external registry.

Schema registry

A service that can supply versioned record schemas for supported encodings. It is one schema source, not an unconditional prerequisite for every log append.

Stream

A named catalog entity with configuration, schema, lifecycle state, and a layout of one or more logs. It is not merely a second name for a single log or one globally ordered sequence.

Stream Catalog Service

The catalog layer managing stream metadata and opening explicit data-plane handles. Ursa's storage index and Oxia sequencing support the log data path; loading catalog metadata itself does not append records or allocate their offsets.

Stream ID

Qualify this term: the public API uses StreamIdentifier for a namespace-qualified stream name, while older storage internals use numeric streamId for a single log. See Log ID above.

Stream layout

StreamLayout: the mapping from a stream to its logs, including log selection for writes. Ursa's current implementation is IndexedLayout; a future strategy declared in the API is not automatically supported by the factory.

Stream materialization framework

The part of the standard that defines how a stream becomes a table. Its policy model lives in lakestream-api, in the io.lakestream.api.materialization package. Ursa supplies the service-provider interface and the materializers that execute a resolved policy. See stream materialization framework.

Stream Offset Index

Metadata mapping log-offset ranges to physical file locations and formats. It supports read routing and changes as data compacts. Consumer acknowledgment state and table snapshot metadata are separate from this index.

Stream position

A layout-aware position containing a log ID and offset (StreamPosition). Identifying only the numeric offset is insufficient for a stream with multiple logs.

Stream Storage Service

The data path for durable appends, reads, index access, and trimming. Ursa separates the raw storage API from unified read routing into raw and columnar representations.

Streamhouse

An open, vendor-neutral category for data architectures that keep the current state of a business continuously available to applications, analytics, and agents. The term is defined by the Streamhouse Working Group at streamhouse.com. Lakestream is one way to build such an architecture's stream-storage layer; it is neither the category nor a requirement of it. See Lakestream and the Streamhouse.

Stream–table duality

Producing stream consumption and table scans from one retained log inside the storage lifecycle: internal compacted objects serve the stream, and they are Parquet that can be registered as a lakehouse table and queried. A table can also be materialized from the same compaction pass into an external destination. Where it is, duality does not imply shared files with that destination, identical visibility frontiers, immediate table freshness, or read-back from it.

Table format

A format such as Iceberg or Delta Lake that organizes table files, metadata, and snapshots. The format is distinct from a particular catalog service or query engine.

Table metadata

The metadata through which a query engine discovers committed files and snapshots. Publishing it is distinct from writing files and updating the stream offset index.

Table mode

Removed. Earlier API revisions had a TableMode enum (MANAGED, EXTERNAL, CUSTOM) on TableConf; Ursa 1.0.0 dropped it, and a destination table's lifecycle belongs to the destination system.

Table scan

Reading a committed table through an analytical engine. Unlike tailing a log, a scan does not imply offset order or access to the latest WAL entries.

Tailing

Continuing to read a log in offset order as entries become available. Readers can traverse raw and compacted ranges when those ranges remain indexed and supported readers are configured.

Trim

Logical or physical removal of a log prefix. softTrim takes an inclusive boundary and can leave bytes in storage; hardTrim takes an exclusive boundary for physical removal. Neither is the same operation as cursor acknowledgment.

WAL object (WO)

A raw object that can multiplex entries from multiple logs. In the object-backed implementation, it is flushed before its index results complete the corresponding appends. It may coexist with compacted output until cleanup.

Write-ahead log (WAL)

The write-optimized first durable representation of appended entries. It is not itself an Iceberg or Delta table snapshot.

Zero-copy

A claim this documentation does not make about external materialization. Internal compacted objects and delivered tables are separate files; what they share is the WAL read pass that produces them. Nothing here means zero buffer copies, no WAL-to-Parquet rewrite, or no storage redundancy.

This is about delivery into an external table. The internal compacted objects themselves are Parquet files that serve stream reads and can also be registered as a lakehouse table and queried.

Zero-ETL

Storage-managed materialization instead of a separately operated export connector. Decoding, file generation, table commits, configuration, and processing latency remain. See zero-ETL.