Kafka compatibility

Ursa for Apache Kafka (UFK) changes where a diskless topic's records live, not the protocol clients speak. This page describes diskless topics. Classic topics in the same cluster keep Kafka's local-log path and every behavior that comes with it.

✔ marks supported and ✖ marks rejected. A rejection is an error code the broker returns, not an undefined behavior.

Clients and tools

UFK adds no request types, no request versions and no field changes: the message definitions under clients/src/main/resources/common/message/ are upstream Kafka's. A client negotiates API versions and reads metadata exactly as it does against stock Kafka, so producers, consumers, admin clients and the stock bin/ tools work against a diskless topic without recompiling or reconfiguring.

What a client sees differs in one respect. The serving owner for a diskless partition is reported as the partition leader and as its only replica and ISR member. That owner is selected by hashing the topic UUID and partition index across the live brokers, not elected the way a classic partition's data leader is. A request sent to a broker that is not the current owner comes back as a retryable routing error, which clients handle with their normal metadata refresh. See diskless architecture.

The Compose stack exercises these stock tools against diskless topics:

kafka-topics.sh, kafka-configs.sh, kafka-console-producer.sh, kafka-console-consumer.sh, kafka-producer-perf-test.sh, kafka-consumer-perf-test.sh, kafka-share-consumer-perf-test.sh, kafka-share-groups.sh, kafka-broker-api-versions.sh and kafka-metadata-quorum.sh.

Consumer groups and share groups

Group coordination is unchanged, because it does not run through the diskless path at all: __consumer_offsets is always a classic topic, so offset commits and group state keep Kafka's own storage and replication. See limitations.

Share groups

Share groups consume diskless topics through a dedicated path rather than the classic one. A share fetch that spans both storage modes is split: the classic partitions are served from local logs, the diskless partitions through the Ursa provider, and the two results are combined into one response. Share-partition initialization resolves its start offset through the diskless offset lookup instead of reading local partition state, which a diskless partition does not have.

The share-demo Compose profile runs the whole path end to end:

make share-demo

It creates a single-partition diskless topic, sets share.auto.offset.reset=earliest on the group with kafka-configs.sh --entity-type groups, produces 40,000 records, consumes them with two share consumers in one group, and finishes with kafka-share-groups.sh --describe --offsets. The target tears the stack down and removes volumes when it exits, including on Ctrl+C.

Share-group state and acknowledgement bookkeeping are Kafka's own and are not diskless. What changes is only where the records are read from.

Producers

Idempotent producers are supported. The broker validates producer epochs and sequence numbers and detects duplicates before appending. Producer state is held in memory and snapshotted to Oxia rather than to local .snapshot files, so it survives an owner change without a local log to recover from; recovery claims the zone-scoped snapshot and replays the log tail. Sequence violations surface as the usual INVALID_PRODUCER_EPOCH and OUT_OF_ORDER_SEQUENCE_NUMBER errors. Read the recovery boundary on limitations before relying on deduplication across a failure.

Transactions are rejected on every path that could involve a diskless topic, each with a logged warning on the broker:

A failed partition check fails the whole AddPartitionsToTxn request, so a transaction that touches one diskless topic does not partially succeed. Idempotence is not a substitute: an application built on Kafka transactions cannot use diskless topics by turning idempotence on. Transactions on diskless topics are tracked as a roadmap item under LIP-162.

Topic settings

The controller rejects these at the request:

One case is rewritten rather than rejected. When ursa.storage.topic.default.enable makes a new topic diskless and the request also asked for a replication factor above 1, the controller sets the factor to -1 and the topic is created with the default of 1. An explicit ursa.storage.enable=true with a replication factor above 1 is rejected instead.

Adopting diskless

Diskless is a topic-level config, so adoption is per topic and existing topics are untouched by enabling the cluster-level system. Classic and diskless topics run side by side in one cluster, under one set of ACLs and one broker deployment.

ursa.storage.enable is immutable after creation. Changing a topic's storage mode in either direction means creating a new topic with the setting you want and moving the data with your own tooling; UFK ships no converter and no in-place migration.

Brokers still need local storage after adoption: KRaft metadata, internal topics and any classic topics keep their own disks.

Next steps

• Diskless architecture — how produce, fetch and owner selection work.
• Limitations — the behavioral boundaries behind this table.
• Implementation status — how UFK maps Kafka topics onto Lakestream streams.