Diskless architecture

Diskless storage is a bypass inside ReplicaManager, Kafka's broker-side component for reading and writing partition data. For a classic topic, ReplicaManager behaves exactly as it always has. For a diskless topic, it routes the same requests into the diskless storage layer instead — the collection of handlers that read and write through Ursa rather than a local log.

Where the bypass happens

Kafka clients (producers, consumers, admin)
            │
            ▼
       ReplicaManager
            │
   is this partition's topic diskless?
      ┌─────┴─────┐
     yes           no
      │             │
      ▼             ▼
the diskless    the local log
storage layer     (.log files)
      │
      ▼
     Ursa  ──────▶  object storage
      │              (LOCAL / S3 / GCS / Azure Blob)
      ▼
    Oxia  (producer state, stream metadata)

The routing decision is made per partition, not per broker or per request type — a single broker serves classic and diskless partitions side by side, and the same produce, fetch, and ListOffsets APIs work against both.

The write path

A produce request for a diskless partition still enters through ReplicaManager the same way a classic one does; what changes is what happens next. Instead of appending to a local log, ReplicaManager hands the partition's records to the diskless storage layer, which validates producer sequence state in Oxia — and, for idempotent producers, updates it — then appends the records to an Ursa-backed ManagedLedger for that partition. The append is asynchronous: the request handler thread that started it is free to serve other requests while Ursa's write buffer batches the append toward its next flush. Ursa acknowledges the append with a position, and the entry id in that position becomes the record batch's base offset — so the offset a producer sees comes from Ursa's assignment, not a local log's next-offset counter. Classic topics never enter this path; they append to the local log exactly as Kafka always has.

The read path

Fetch requests for diskless partitions run the same routing in reverse: the diskless storage layer reads from the partition's ManagedLedger instead of a local log, converting what it reads back into the record batches the Kafka protocol expects, and patching each batch's base offset to match the entry id it was written under — so a consumer sees consistent offsets regardless of which path served the read. ListOffsets requests resolve against Ursa too: earliest and latest map directly to positions Ursa already tracks (the first available position, and the high watermark derived from the last confirmed entry), and a timestamp-based lookup narrows to a candidate entry by publish time before scanning records for an exact match. None of this is visible at the protocol level — a client can't tell whether a given fetch was served from a local log or from Ursa.

Producer state moves to Oxia

Idempotent producers rely on Kafka tracking a sequence number per producer per partition — normally in local .snapshot files a broker rebuilds from its log on restart. Diskless topics keep that state in Oxia instead: an in-memory cache backs every validation on the write path, and it's snapshotted to Oxia on a periodic interval, or after enough records have been appended, whichever comes first. A broker that loses that cache — because it restarted, or because a different broker now owns the partition — recovers by loading the last snapshot from Oxia and replaying whatever entries Ursa recorded after it, rather than replaying its own local log from the start.

Replication factor 1, and no ISR wait

Diskless topics are created at replication factor 1, and the controller enforces it — there's no second or third replica for followers to keep in sync, because Ursa's own durability already covers what ISR replication exists to provide. That removes a wait from the write path: a classic produce request with acks=all sits in Kafka's DelayedProduce purgatory until followers catch up, but a diskless produce request only waits on Ursa's acknowledgment, so it never enters that wait at all. The metadata a diskless partition reports reflects the same fact — its in-sync-replica set is the leader alone, and the broker-side threads that would normally fetch a partition's data from its leader to stay in sync skip diskless partitions entirely, since there's nothing for them to replicate. This is also what failover without elections depends on: because Ursa, not a specific broker's disk, holds the data, any broker can pick up a diskless partition and start serving it as soon as it's elected leader, with no re-replication step first.

Request pipelining and the flush interval

Ursa's write buffer flushes on an interval — ursa.storage.write.buffer.flush.interval.ms, 250ms by default — and a diskless produce request only completes once its data has cleared that flush. On its own, that adds up to roughly half a flush interval of extra latency on average. But Kafka's network layer has historically allowed only one in-flight request per connection: once a broker reads a request, it stops reading that connection until the response is fully sent. For diskless topics, where the response is gated on a remote flush rather than a fast local write, that turns the connection itself into a queue — a producer with several in-flight requests on one connection can end up waiting multiples of the flush interval, not a fraction of it. With a 250ms flush interval and five in-flight requests, for example, the resulting average lands closer to three flush cycles (~750ms) and the tail closer to five (~1250ms) — which is how a 250ms flush setting turns into produce latency well over a second. socket.server.enable.request.pipelining, off by default, is the fix: it lets the broker keep reading new requests from a connection while earlier ones are still waiting on a flush, while still sending responses back in order. It's recommended on for any deployment running diskless topics.

See configuration for the write-buffer and pipelining settings referenced above.