Diskless architecture

The broker splits produce, fetch, and ListOffsets requests by storage mode. Classic partitions use Kafka's local log; diskless partitions route to the Ursa provider. A request can contain both kinds of partition. Upstream Kafka has no pluggable replica manager, so Ursa for Apache Kafka (UFK) carries the diskless path in its own ReplicaManager.

Kafka clients
      |
      v
Broker routing and owner checks
      |                              |
      v                              v
Ursa provider                    classic local log
      |
Lakestream StreamCatalog -> partition Log / LogCursor
      |                              |
      v                              v
Oxia metadata                    object-storage WAL
                                     |
                                Ursa compactor
                                  /       \
                                 v         v
                       compacted objects external Iceberg table
                       (Kafka reads)     (when enabled)

The provider and its dependencies load through an isolated runtime classloader. This integration uses the Lakestream API, not Pulsar ManagedLedger.

Topics, streams, and offsets

The Ursa integration creates a stream on first open and grows its indexed partition layout when Kafka adds partitions. Each Kafka partition opens the corresponding Lakestream log. The stream name includes the topic UUID (<topic>-topic-id-<uuid>), so deleting and recreating a topic does not reopen the old log. Source metadata also preserves the logical Kafka topic name.

The write path validates the Kafka record batches, rejects transactional batches, and copies the records into an append payload. It calls Log.append(recordCount, payload) asynchronously. The returned LogEntryHeader.offset() supplies the produce response's base offset. One append can contain multiple records: these are record-offset ranges, not ManagedLedger entry IDs used as Kafka offsets.

For idempotent writes, the broker validates producer epochs and sequences and detects duplicates before append. State is held in memory and snapshotted to Oxia by interval or record threshold. Recovery claims the zone-scoped snapshot with version checks and replays the log tail before accepting writes. This supports non-transactional idempotence, not Kafka transactions.

Fetch and ListOffsets

The fetch path reads through a Lakestream cursor and reconstructs Kafka record batches, preserving the assigned record offsets. Ursa's cursor can read WAL entries or compacted objects; compacting the WAL does not make retained records unavailable to Kafka consumers.

The high watermark is the offset just past the last durable record. Earliest ListOffsets reads the current log start; latest returns the high watermark. Timestamp lookup searches entry headers and then scans record timestamps, with a bounded scan. See limitations for the timestamp and visibility boundaries.

The diskless fetch path implements request-level long polling. It registers append waiters before reading. If a request has no records or errors and has caught-up partitions, it waits for an append or fetch.max.wait.ms, then re-reads those partitions once. Any available records cause an immediate response; it does not accumulate data to satisfy fetch.min.bytes as classic Kafka does. Notifications come from the local writer; writes through another owner become visible through refreshed storage reads.

Serving owners, not classic data-leader elections

Owner selection sorts eligible live brokers by ID and hashes the topic UUID and partition index to select an owner. Kafka metadata exposes that owner as the protocol leader and sole replica/ISR member. Produce, fetch, and ListOffsets use the same owner selection; requests to another broker receive a retryable routing error rather than permission to write anywhere.

When the eligible broker set changes, clients refresh metadata and reach the newly selected owner. No local message-log transfer or classic per-partition data-leader election is needed. KRaft is still responsible for cluster membership and topic metadata, and idempotent-producer recovery can add delay.

A comma-separated zone_id token in client.id restricts selection to brokers whose broker.rack matches that zone. If no live broker matches, or no zone is supplied, selection falls back to all eligible live brokers. Different zones can therefore have different owners for the same shared log. See configuration for an example.

RF=1 means there is no Kafka follower-fetch replication for diskless records. An acknowledged diskless produce waits for Ursa rather than waiting for Kafka ISR replication. This removes Kafka's inter-broker replication traffic for those records, not all cross-zone traffic from the deployment.

Flush batching and request pipelining

Ursa flushes the write buffer on the interval (250 ms by default) or size threshold (256 MiB), whichever triggers first. Produce latency includes waiting for that flush and completing the remote write; the interval alone is not a latency SLA.

socket.server.enable.request.pipelining=true allows overlapping produce requests on one connection while responses remain ordered. The broker caps this with socket.server.request.pipelining.max.in.flight.requests.per.connection (default 5). This avoids serializing every produce behind an earlier remote flush, but increases in-flight memory. It is a produce optimization, not unrestricted pipelining for every request type.

Retention, compaction, and external tables

Kafka's retention.ms and retention.bytes are translated into Log.softTrim calls. A soft trim advances the readable log start; it is not immediate physical object deletion.

The standalone compactor publishes tasks and writes compacted objects. Ursa's WAL cleaner uses the oldest first-uncompacted position across streams to determine safe deletion prefixes. A stalled compaction stream can hold back shared WAL reclamation. Run the compactor even when you do not need an external Iceberg table.

External materialization is a separate output. In the Docker stack, the lakehouse profile starts Polaris, while URSA_MATERIALIZATION_ENABLED=true enables the compactor's external Iceberg sink. Starting Polaris alone does not create tables. Kafka retention must not be treated as an external-table retention policy. See Ursa lakehouse tables for internal and external output ownership.

Topic deletion is handled by the active controller's storage lifecycle integration, with periodic sweeps for orphaned state. Deletion and cleanup are asynchronous; recreating the Kafka topic uses a new stream identity. In the demo's default external naming, the same logical topic name targets the same Iceberg table, so a recreated topic can append to that table.