Stream Catalog (Level 2)

The Stream Catalog is the top level of the Lakestream API: streams, layouts, readers, and writers. It also covers how a stream declares its table side, through an optional materialization policy.

The catalog

StreamCatalog is the entry point: namespace and stream CRUD, plus factory methods for the layout, writer, and reader behind a given stream. Namespaces group streams; every stream lives in exactly one. Creating a stream takes an identifier, a StreamConfig (a property bag — retention and cleanup policy travel as string properties rather than typed fields), a Partitioning, and a SchemaConfig:

public interface StreamCatalog extends AutoCloseable {
    CompletableFuture<Stream> createStream(StreamIdentifier id, StreamConfig config,
                                            Partitioning partitioning, SchemaConfig schema,
                                            Map<String, String> properties);

    CompletableFuture<Stream> loadStream(StreamIdentifier identifier);

    CompletableFuture<StreamWriter> openWriter(StreamIdentifier identifier);

    CompletableFuture<StreamReader> openReader(StreamIdentifier identifier);
}

Every operation is asynchronous. Beyond creation and load, the catalog also handles lifecycle — sealStream stops writes while leaving reads open, truncateStream clears data — and property and materialization-policy updates on both namespaces and streams.

Streams and layout

Loading or creating a stream returns a Stream — a handle combining metadata (identifier, config, partitioning, schema, state) with data-plane access (layout, writer, reader, and getLog for a specific log):

public interface Stream extends AutoCloseable {
    StreamIdentifier identifier();
    StreamLayout layout();
    StreamWriter writer();
    StreamReader reader();
    Log getLog(LogId logId);

    CompletableFuture<StreamPosition> softTrim(StreamPosition position);
    CompletableFuture<Void> hardTrim(StreamPosition position);
}

StreamLayout is how a stream maps onto one or more logs — there is no intermediate partition type. Its Partitioning names a strategy and carries strategy-specific config (for example, numPartitions under INDEXED). Two strategies exist today: INDEXED, a fixed number of logs addressed by integer index, and RANGE, key-range segments with split and merge, reserved for a future release. The layout resolves a write to a target log (resolveForWrite, given a RoutingKey that carries either an explicit index or a request for round-robin placement) and enumerates a stream's logs in order (logIds) — both are asynchronous calls, consistent with the rest of this API. StreamPosition is an opaque handle the layout creates and interprets; it's what softTrim and hardTrim take, so a range layout can resolve a single trim position to whichever logs it actually spans.

Reading and writing

StreamWriter resolves the target log from a routing key and appends to it in one call:

public interface StreamWriter extends AutoCloseable {
    CompletableFuture<WriteResult> write(RoutingKey key, int numberOfRecords, ByteBuf data);

    StreamLayout layout();

    record WriteResult(LogId logId, long offset) {}
}

StreamReader is the mirror on the read side — read(logId, startOffset, maxMessageCount, maxSizeBytes) — and handles routing between row-oriented and Parquet storage transparently, so a caller reading across the WAL-to-compacted-object boundary doesn't have to know which tier an offset currently lives in. See architecture for that boundary.

Declaring a table

A stream can also declare its table side. StreamCatalog registers named TableCatalog handles to a backing store (registerTableCatalog, getTableCatalog) — Iceberg, Delta Lake, Delta Lake under Unity Catalog, ClickHouse, or none. A TableMaterializationPolicy, set at the namespace level as a baseline and optionally overridden per stream, decides whether and how a stream materializes: which catalog it targets, WriteMode (APPEND, UPSERT, or CDC), TableMode for lifecycle ownership of the destination table (MANAGED, where the implementation creates and owns the table and may drop it when the stream is deleted; EXTERNAL, a pre-existing table the implementation writes to but never drops; or CUSTOM), partitioning (PartitionSpec, with transforms like IDENTITY and BUCKET mirroring Iceberg's vocabulary), and an EvolutionPolicy governing which schema changes — adding a column, widening a type — are allowed to reach the table automatically. Stream.effectiveMaterialization() resolves the merged, stream-over-namespace policy into a catalog, table identifier, and effective settings a materializer can act on directly.

One TableCatalogType value, NONE, marks the case where there's no external catalog at all: the stream's compacted objects stay internal — the managed, SBT path — rather than committing to Iceberg or Delta. See stream–table duality for what that internal/external split means for how the data can be read back.