Glossary

This glossary defines terms used across the Lakestream standard and its implementations.

Catalog

The external system — such as an Iceberg or Delta Lake catalog — that tracks a table's metadata and lets query engines discover and read it. An internal table is typically still registered into a catalog you don't manage, even though the storage layer owns its lifecycle; an external table's catalog is one you run yourself.

Compacted object (CO)

A per-log object created by folding together the WAL objects belonging to one log — typically stored as columnar Parquet, though it can stay row-based when a log doesn't need table access. Compacted objects are designed for long-term retention and fast scans, and once committed to a table catalog, back both stream reads and table reads.

Compaction

The process of folding row-oriented WAL objects into a log's compacted object, usually converting to columnar Parquet, though the result can stay row-based. Compaction is what turns a log written once into a table — see zero-ETL.

Compaction Service

The background service that performs compaction: it merges WAL objects into compacted objects, sizes files for query performance, registers them with the destination table catalog, updates the Stream Offset Index, and enforces retention. See architecture.

Cursor

A consumer's tracked read position within a stream, used to resume consumption after a disconnect or restart. The Stream Offset Index underpins cursor tracking and recovery.

Delta Lake

An open table format Lakestream can commit compacted objects into, alongside Apache Iceberg. Both formats provide ACID transactions, schema evolution, time travel, and partition pruning over data stored as Parquet.

Diskless

Describing serving nodes that keep no local partition state: durability lives in object storage instead of on the node's disk. See diskless & leaderless.

Entry

A batch of multiple records grouped together for efficient processing and storage. An entry is the unit of storage atomicity: every record in a batch is committed and stored together, typically after client-side batching and compression.

Entry format

The serialization format for entries within a WAL object, including their metadata and offsets.

External table (SDT)

A table delivered into a catalog you run, where both data and metadata belong to that external system. Once written, external-table data is no longer indexed by the Stream Offset Index, so it can't be read back as a stream. External tables support repartitioning and upserts, which suits curated data.

Fencing

Transitioning a log to a fenced state, so further appends to it fail until it's reactivated. It's the storage layer's single-writer safety mechanism: a log's previous writer can't keep appending after fencing, even if it hasn't yet noticed a new writer took over.

Hot window

The configurable, recent span of a log still held in WAL objects rather than folded into compacted objects. Reads against the hot window come from WAL objects; older reads come from compacted objects.

Iceberg

Apache Iceberg, an open table format Lakestream can commit compacted objects into, alongside Delta Lake. Both formats provide ACID transactions, schema evolution, time travel, and partition pruning over data stored as Parquet.

Internal table (SBT)

A table the storage layer manages end-to-end, also called a stream-backed table. Because its compacted objects are indexed by both the Stream Offset Index and the table's metadata, the data stays readable as both a stream and a table — see stream–table duality.

Lakehouse storage

The long-term storage tier holding compacted objects, optimized for retention and analytical scans rather than low-latency appends.

Lakestream

Capitalized, the open standard itself: the pattern plus its API. Lowercase, "a lakestream" is any system that conforms to the standard. See What is Lakestream.

Leaderless

Describing an architecture where no node needs leader-based coordination with peers to serve a partition, because offset assignment is centralized in the Stream Catalog Service rather than negotiated among replicas. See diskless & leaderless.

Log

The ordered, append-only sequence of entries that records everything written to a stream. The log is the authoritative source of truth for a stream's data; each record in it is indexed by an offset.

Log format

The storage layout of a log: how its WAL objects and compacted objects are structured. Also called stream format.

Materialization

Applying a stream's schema to compacted data and committing the result as columns in the destination table format, governed by policies for how and when schema changes take effect.

Medallion architecture

A layering convention for lakehouse data — bronze (raw), silver (cleaned), and gold (curated) — that internal and external tables map onto naturally: internal tables suit bronze data, external tables suit silver and gold.

Metadata store

The durable, low-latency store backing the Stream Catalog Service's offset assignments and log metadata. The pattern requires one; it doesn't mandate a specific implementation. See architecture.

Object storage

Cloud or on-premises blob storage — such as S3, GCS, or Azure Blob Storage — used as the durability layer for WAL objects and compacted objects, in place of attached disks.

Offset

The atomically increasing sequence number that indexes each record within a log.

Oxia

A distributed key-value store. Ursa, the reference implementation, uses it as the metadata store behind the Stream Catalog Service, and Ursa for Kafka additionally uses it to persist idempotent-producer state for stateless broker failover.

Parquet

The columnar file format compacted objects are stored in, and the format Lakestream tables are built from.

Partitioning

Dividing a topic or dataset into parallel logs, each an independently ordered append-only sequence. External tables can also be repartitioned on write, independent of how the source log was partitioned.

Reassignment

Handing ownership of a partition or log to a different serving node after a failure. Because durability lives in object storage rather than on the previous node's disk, reassignment requires no data copy.

Record

A single unit of data in a stream — also called an event or message — produced to or consumed from a stream.

RecordBatch

Another name for an entry: a batch of records grouped for efficient, atomic storage.

Reference implementation

A conforming system that implements the Lakestream pattern and its API, serving as proof the standard is implementable. Ursa is the reference implementation; other conforming systems aren't required to share its code.

Retention

The configured duration WAL objects and compacted objects are kept before deletion, enforced by the Compaction Service.

RoutingKey

The argument a stream write passes to select which log within the stream receives the entry. It currently carries either an explicit log index or a request for round-robin placement across the stream's logs; key-based (hash) routing is designed into the type but not yet implemented.

Schema

The structural definition of a log's records, typically managed by an external schema registry. Compatibility modes govern how a schema can evolve without breaking the tables it materializes into.

Schema registry

The external system a log's schema is managed by, used to transform records from row-based serialization into columnar storage and to enforce compatibility as a schema evolves.

Stream

The logical, real-time view of a log: an ordered sequence of records a consumer can tail. See stream–table duality.

Stream Catalog Service

The centralized metadata service responsible for offset assignment, the Stream Offset Index, and log and stream metadata. It's what makes the pattern leaderless. See architecture.

Stream ID

The unique identifier assigned to a log at creation by the metadata service.

Stream Offset Index

A multi-level index mapping logical offsets to their physical locations within WAL objects and compacted objects. It underpins offset commits, cursor tracking, consumer recovery, and consumer resume, and updates incrementally as data is appended and compacted.

Stream Storage Service

The service combining write-ahead log storage for newly ingested records with lakehouse storage for long-term, per-log retention. See architecture.

Stream–table duality

The property that one copy of data is simultaneously readable as a stream and queryable as a table, with no pipeline between the two. See stream–table duality.

Table format

An open format — Apache Iceberg or Delta Lake — for organizing compacted data with ACID transactions, schema evolution, and time travel.

Table metadata

The catalog-level metadata that lets compacted objects be committed to a table without duplicating or re-copying the underlying data.

Table scan

Reading a table's compacted objects through a query engine — filtering, aggregating, or joining them — rather than tailing them in offset order.

Tailing

Reading a stream in offset order as new records arrive, backed by WAL objects for the hot window and compacted objects for anything already folded into long-term storage.

WAL object (WO)

A row-based object aggregating records from one or more logs, written sequentially for efficient, immediately durable appends. WAL objects are retained for a configurable hot window before compaction.

Write-ahead log (WAL)

The append-only storage tier newly ingested records land in first, ensuring immediate durability before compaction folds them into a compacted object.

Zero-copy

Reading a log as a table without duplicating or re-copying its data — the same compacted objects serve both access paths. This is the mechanism behind stream–table duality.

Zero-ETL

Ingestion writing directly to open table formats, so tables appear without a separate ETL pipeline, connectors, or staging copy. See zero-ETL.