Operations

Compaction is a separate process from whatever writes the streams. It rewrites log ranges into compacted objects and, when materialization is enabled, writes table files from the same pass. A deployment that appends but never compacts keeps working: it accumulates write-ahead log (WAL) objects and produces no tables.

This page covers running that process. For the settings it reads, see compaction settings. For a Kubernetes deployment with manifests, see Ursa for Apache Kafka (UFK) deployment; for how it scales next to UFK and what a stalled stream does to reclamation, see operating the compactor.

Entry point

The service is io.lakestream.ursa.compact.CompactionMain, which takes a properties file and nothing else:

CompactionMain -c <configuration-file>
CompactionMain --conf <configuration-file>
CompactionMain --conf=<configuration-file>
CompactionMain -h | --help

In the packaged distribution, bin/compact start runs it with $COMPACT_CONF. The lakestream/kafka image ships the same entry point as /opt/kafka/bin/ursa-compactor.sh --conf <file>.

The launcher script in a source checkout is not a ready-to-run installation on its own: it looks for the distribution JAR and lib/ beside it.

Exit codes

Environment variables

bin/compact reads these. Each has a default, so none is required.

Set COMPACT_LOG_CONF to your own Log4j 2 configuration file. The script adds that file's directory to the classpath and passes its base name to Log4j, so the file can live anywhere.

JVM options the launcher sets

Worth knowing because they are not obvious from the configuration file:

• -Dotel.exporter.prometheus.host=0.0.0.0 — binds the metrics exporter to all interfaces. See observability.
• -XX:+CrashOnOutOfMemoryError — the process dies rather than degrading when it exhausts the heap.
• -XX:+DisableExplicitGC, -XX:-OmitStackTraceInFastThrow
• -Xlog:gc*,safepoint:$COMPACT_LOG_DIR/compact_gc_%p.log with 10 files of 20 MB
• -Djava.net.preferIPv4Stack=true, -Dio.netty.tryReflectionSetAccessible=true, Netty recycler tuning and leak detection disabled

Sizing

Compaction reads the WAL and writes Parquet, so it is bounded by object-store throughput and by direct memory rather than by heap. walReadRateLimitInBytesPerSecond caps how fast one process reads, defaulting to 50 MiB/s; compactedThreadNum sets how many tasks run at once, defaulting to one less than the available processors.

Leadership

Every compactor process runs tasks, but three duties belong to a single elected leader:

• publishing compaction tasks
• committing compacted output
• cleaning up compacted objects that have fallen behind the mark-deleted offset

Election is a single ephemeral record in the metadata store at /compact/leader, claimed with a create-if-absent write and refreshed every two seconds. A process that loses the record becomes a follower and stops those three duties.

The commit runner is gated on leadership rather than draining on demotion. A demoted leader therefore stops committing promptly, which is what prevents two processes from writing duplicate data files into the same table.

Run more than one process for availability. Exactly one should hold leadership at any moment — see the leader gauge in observability.

Starting up

On start the service bootstraps table catalogs from the configuration, starts its worker pool, then joins the election. Catalog bootstrap is tolerant: if the bootstrap class is absent, it logs a warning and continues, so a process that is not configured for tables still compacts.

How a task fails

A failing task is classified and held back rather than retried immediately. Retryable failures are quarantined for retryableQuarantineInSeconds; the rest for nonRetryableQuarantineInSeconds. Streams that keep failing accumulate in a dead-letter queue, which is replayed on the first round after a restart when replayDLQTasksEnabled is left on.

A stream can also be excluded outright with blackNamespaceOfCompact or blackTopicOfCompact, which is the usual response to one stream blocking progress while its cause is investigated.

Failures are counted and exposed; the DLQ and quarantine gauges are the ones worth alerting on.

Shutting down

CompactionMain registers the scheduler's shutdown as a JVM shutdown hook, so a SIGTERM runs an ordered close: each executor is given ten seconds to finish and then ten more after a forced shutdown, followed by the lock manager, the materialization service, the catalog, the election record and the metadata-store clients.

Allow for that when setting a container's termination grace period — a pod killed sooner can leave its leadership record to expire on its own rather than releasing it.

Credentials in logs

The service logs its configuration at startup with s3AccessKeyId, s3SecretAccessKey, unityCatalogToken, unityCatalogClientSecret and any iceberg.catalog.*.credential value masked.

Other properties are logged as given. Keep secrets that do not appear in that list out of the properties file and supply them through the credential-file properties instead — see how properties are loaded.