Ursa for Kafka

Ursa for Kafka is a fork of Apache Kafka in which designated topics run diskless: the broker's local log is bypassed entirely, and records are stored through Ursa on object storage instead. Idempotent-producer state — the sequence bookkeeping Kafka needs for exactly-once semantics — moves out of local .snapshot files and into Oxia, the distributed key-value store Ursa also uses for its own metadata. Classic topics are untouched by any of this; diskless is an opt-in mode, not a replacement for how Kafka already works.

What diskless buys

Because a diskless topic's data isn't pinned to whichever broker happens to hold its local log, instant failover becomes possible: any broker can serve a partition immediately after leader election, with no re-replication step first. Brokers themselves become stateless with respect to message data — scaling one up or down doesn't depend on migrating local disk contents. And because Kafka-level ISR replication is bypassed for diskless topics, with Ursa handling durability instead, the cross-AZ network traffic that comes from followers fetching every record from a leader disappears for that data. Storage and compute are decoupled: what a broker does is serve requests, not own bytes.

How this differs from tiered storage

Apache Kafka's own KIP-405 introduced tiered storage, which moves older log segments to remote storage while recent data stays local — the local log remains the primary store for anything actively being written or read. Diskless storage is a different shape of change: for a diskless topic, there is no local log to tier from. Ursa is the primary and only store from the first byte written, not a destination cold segments eventually migrate to.

Turning it on

Diskless mode is a per-topic setting, ursa.storage.enable=true, applied at topic creation through the standard kafka-topics.sh --create --config flag — see configuration for the full settings table. Nothing about the client protocol changes: existing producers and consumers work against a diskless topic without modification.

Status

Ursa for Kafka is a fork of Apache Kafka developed on branch 4.2-ursa, currently versioned 4.2.1-SNAPSHOT. As a Kafka fork, it carries the same Apache-2.0 license.

Where next

• Quickstart — run a three-broker diskless cluster with Docker.
• Diskless architecture — how produce, fetch, and ListOffsets route around the local log.
• Ursa — the storage engine diskless topics write through.
• Diskless & leaderless — the pattern this implements, described independent of Kafka.