Configuration

Ursa for Apache Kafka (UFK) is configured through broker settings, topic defaults and zone-aware routing properties.

Configure the diskless storage system in the cluster's broker/controller configuration, then select storage mode when creating each topic. Configure participating nodes consistently; setting a property on one broker does not propagate it to the others.

The tables below list broker defaults, not the camelCase properties used by the Ursa compactor. The Docker stack overrides some of these values, as noted below.

Storage and metadata

Property	Default	Description
`ursa.storage.enable`	`false`	Enable the diskless storage system. Required before creating diskless topics.
`ursa.storage.topic.default.enable`	`false`	Make new non-internal topics diskless when no explicit topic override is supplied.
`ursa.storage.class.path`	`""`	Override the isolated Ursa provider classpath. When unset, the loader resolves the installation's `ursa-storage/*` runtime.
`ursa.catalog.oxia.service.url`	`oxia://localhost:6648/default`	Lakestream stream catalog endpoint.
`ursa.oxia.service.url`	`oxia://localhost:6648/default`	Ursa storage metadata and Kafka producer-state snapshots.
`ursa.storage.backend.type`	`LOCAL`	`LOCAL`, `S3`, `GCS`, or `AZURE_BLOB`; `AZUREBLOB` is also accepted.
`ursa.storage.path`	`/tmp/ursa-data`	Local path for `LOCAL`; WAL object prefix for remote backends.
`ursa.storage.compaction.bucket`	`kafka-ursa-storage`	Bucket/container holding compacted objects. Set explicitly alongside the WAL bucket.
`ursa.storage.compaction.prefix`	`/tmp/compaction-data`	Prefix for compacted objects.
`ursa.storage.lifecycle.sweep.interval.ms`	`600000`	Active-controller sweep interval for orphaned diskless storage state.

The two Oxia URLs have different roles even when they point to the same service and namespace. The current integration does not use the old ursa.storage.oxia.service.url, pulsar.oxia.service.url, ursa.storage.namespace, or ursa.storage.wal.directory settings. Do not carry those keys forward as substitutes for the current catalog and storage configuration.

LOCAL is useful for local testing; independent broker-local directories do not provide the shared storage needed for multi-broker failover.

Object storage connection

Property	Default	Description
`ursa.storage.s3.endpoint`	`""`	Endpoint override; also mapped to the cloud endpoint for GCS/Azure backends.
`ursa.storage.s3.bucket`	`kafka-ursa-storage`	WAL bucket/container; also used for GCS/Azure backends.
`ursa.storage.s3.region`	`us-east-1`	Region passed to remote backends.
`ursa.storage.s3.access.key`	`""`	Explicit S3 access key.
`ursa.storage.s3.secret.key`	`""`	Explicit S3 secret key.
`ursa.storage.s3.session.token`	`""`	Optional token for temporary S3 credentials.
`ursa.storage.s3.path.style.access`	unset (`null`)	Optional S3 path-style override; the MinIO demo sets `true`.

Despite the shared bucket/endpoint property names, S3 credentials and path-style options are forwarded only for the S3 backend. They are not GCS or Azure credentials. Keep broker and compactor storage locations and metadata endpoints aligned so the compactor processes the same WAL and brokers can read its output.

Buffering, producer state, and networking

Property	Default	Description
`ursa.storage.write.buffer.flush.interval.ms`	`250`	WAL flush interval in milliseconds.
`ursa.storage.write.buffer.size`	`4194304` (4 MiB)	Size of each WAL write-buffer segment.
`ursa.storage.write.buffer.flush.size`	`268435456` (256 MiB)	Byte threshold that triggers an early flush.
`ursa.storage.producer.state.snapshot.interval.ms`	`30000`	Producer-state snapshot interval; `<= 0` disables the time trigger.
`ursa.storage.producer.state.snapshot.record.threshold`	`10000`	Appended-record threshold for snapshots; `<= 0` disables this trigger.
`socket.server.enable.request.pipelining`	`false`	Allow overlapping produce requests per connection with ordered responses.
`socket.server.request.pipelining.max.in.flight.requests.per.connection`	`5`	Per-connection broker cap when pipelining is enabled; minimum `1`.

The Docker stack enables pipelining and uses a 16 MiB buffer segment, but those are demo overrides, not broker defaults. Lower flush intervals trade smaller batches and more object-store writes for lower waiting time. Pipelining trades additional in-flight memory for less connection-level serialization.

Topic settings

Property	Default	Description
`ursa.storage.enable`	`false`, unless the cluster's topic default enables it	Select diskless storage at topic creation.

Create diskless topics with --config ursa.storage.enable=true and --replication-factor 1. When RF is omitted in the creation request, the controller chooses 1 for a diskless topic; an explicit non-1 RF is rejected. An explicit topic ursa.storage.enable=false overrides the diskless topic default.

The storage-mode flag cannot be changed after creation. Migrate into a new topic to change storage modes. Internal topics remain classic, and explicitly requesting diskless storage for an internal topic is rejected.

retention.ms and retention.bytes drive logical soft trims, not immediate S3 deletion. cleanup.policy=compact neither provides Kafka key-based compaction on diskless topics nor exempts them from retention. See limitations.

S3 settings to add to a configured cluster

This is a storage fragment, not a complete Kafka server configuration. Keep the cluster's existing KRaft, listener, security, and local log settings. Replace endpoints, bucket names, and credential placeholders for your deployment.

ursa.storage.enable=true
ursa.catalog.oxia.service.url=oxia://oxia:6648/default
ursa.oxia.service.url=oxia://oxia:6648/default

ursa.storage.backend.type=S3
ursa.storage.path=ursa/wal
ursa.storage.s3.bucket=my-kafka-bucket
ursa.storage.s3.region=us-east-1
ursa.storage.s3.endpoint=https://s3.us-east-1.amazonaws.com
ursa.storage.s3.access.key=<your-access-key>
ursa.storage.s3.secret.key=<your-secret-key>

ursa.storage.compaction.bucket=my-kafka-bucket
ursa.storage.compaction.prefix=ursa/compacted
socket.server.enable.request.pipelining=true

Supply credentials through your deployment's secret mechanism rather than committing them to a properties file.

The distribution built by ./gradlew releaseTarGz packages the provider and dependencies under ursa-storage/. A broker alone is not the complete deployment: supply reachable Oxia and object storage and run the standalone compactor. The quickstart wires those services together.

Zone-aware routing

Set broker.rack to the broker's zone. On clients, put a zone_id token in the comma-separated client.id, for example:

# Broker configuration
broker.rack=us-east-1a

# Producer or consumer configuration
client.id=my-app,zone_id=us-east-1a

Owner selection prefers live brokers with that rack. Missing, empty, or unmatched zones fall back to the full eligible broker set. Use a stable zone for an idempotent producer: producer-state snapshots are zone-scoped, so changing zones is not equivalent to retrying on another owner in the same zone. This routing uses client.id, not client.rack.