lakestream
Ursa for Kafka

Limitations

What diskless topics don't support yet.

This page states plainly what diskless topics don't do yet — not a roadmap with dates, since none of these carry a committed timeline, but the current state of the preview. None of it applies to classic topics, which are unaffected by any of this.

Current state of the preview

Diskless storage is implemented but not yet released. The gaps below are what to plan around today, not permanent design limits.

  1. No transactional producers. Diskless topics reject transactional produce requests outright — a producer configured with a transactional.id can't write to one. Non-transactional producers, including idempotent ones, work today.
  2. Replication factor 1, and it's permanent. Diskless topics are created at RF=1, and the setting can't be changed afterward. There's no in-place conversion between diskless and classic storage; moving a topic from one to the other means creating a new topic and migrating the data into it.
  3. Internal topics stay on local storage. __consumer_offsets and __transaction_state always use classic, per-broker storage, even on a broker with diskless storage enabled. Consumer offset tracking doesn't move to Ursa just because your data topics do.
  4. No Kafka-style key/value log compaction. A diskless topic can't run compacting cleanup semantics that collapse a key's history down to its latest value. This is a different thing from Ursa's own WAL-to-Parquet compaction, which periodically materializes a topic's raw write-ahead-log data into columnar storage for analytics — that process changes storage layout, not what a consumer reads, and doesn't substitute for key-based log compaction.
  5. Fetch requests don't long-poll against Ursa yet. Classic Kafka can hold a fetch open until enough data arrives (fetch.min.bytes) or a timeout elapses. For diskless topics, DelayedFetch currently returns immediately regardless of those settings, so a light-traffic diskless consumer makes more, smaller round-trips than a classic one would under the same configuration.

None of these are permanent design boundaries — transaction support and Ursa-aware long-poll fetch are both listed as intended future work — but as of this preview, they're what to plan around. Track progress at github.com/ursaio/ursa-for-kafka.