Quickstart

This quickstart runs the demo cluster from the ursa-for-kafka repository's Docker setup: three brokers, Oxia for metadata, and MinIO standing in for S3. Commands and ports below are taken from that setup's own README, in docker/examples/docker-compose-files/cluster/ursa/.

Prerequisites

• Docker >= 20.10.4
• Docker Compose v2+
• Python >= 3.7 (for building the image)
• Java >= 17 (for building Kafka)

Build the image

git clone https://github.com/ursaio/ursa-for-kafka.git
cd ursa-for-kafka/docker/examples/docker-compose-files/cluster/ursa
./build-image.sh

This builds Kafka with diskless storage support and produces a kafka-diskless:latest image. Build a linux/amd64 (x86_64) image instead if you need one — useful on Apple Silicon:

./build-image.sh --amd64

Run the demo

make demo

This starts Oxia, MinIO, and three Kafka brokers, creates a diskless topic test-diskless with 64 partitions, runs two producer instances at 10k messages/second each, runs one consumer instance, and cleans up when it finishes — including on Ctrl+C.

Service ports

The MinIO console is at http://localhost:19001, with username and password both minioadmin; browse the kafka-ursa bucket there to see the stored log segments. The demo sets all of this through Docker Compose environment variables rather than a server.properties file — Kafka's environment-variable convention prefixes a property with KAFKA_, uppercases it, and replaces each . with _, so ursa.storage.enable becomes KAFKA_URSA_STORAGE_ENABLE. See configuration for what each underlying property does.

Creating your own diskless topic

make demo cleans up after itself, so to keep a cluster running and create topics by hand, start it without the demo workload:

make up

make ps shows what's running, and make list-topics lists topics once you've created one. Create a diskless topic the same way you would any other Kafka topic, with one addition — --config ursa.storage.enable=true — and one requirement: replication factor 1.

./bin/kafka-topics.sh --bootstrap-server localhost:29092 --create \
  --topic my-diskless-topic \
  --partitions 10 \
  --replication-factor 1 \
  --config ursa.storage.enable=true

The Makefile wraps the same idea: make create-topic creates one with its built-in defaults, and make create-topic TOPIC=my-topic PARTITIONS=6 overrides the name and partition count.

Testing failover

Because a diskless partition's data lives in Ursa rather than on any one broker's disk, stopping a broker doesn't take its partitions down with it:

docker compose stop kafka-1

./bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:39092 \
  --topic test-diskless \
  --from-beginning

docker compose start kafka-1

The consumer keeps reading against kafka-2 while kafka-1 is down. See diskless architecture for why no re-replication step is needed before the remaining brokers can serve it.

What's next

• Diskless architecture for how the produce and fetch paths work under this cluster.
• Configuration for every broker and topic setting.
• Limitations for what this preview doesn't support yet.