Observability

Ursa instruments its storage and compaction paths with OpenTelemetry. Metric names below are the instrument names Ursa registers, under the instrumentation scope io.lakestream.ursa.storage.

Turning metrics on

Instrumentation is inert unless an OpenTelemetry SDK is present. When Ursa is constructed without an explicit OpenTelemetry instance it falls back to the global one, so metrics appear once an SDK is configured — most commonly by attaching the OpenTelemetry Java agent, or by configuring the SDK in the embedding application.

The standalone compactor's launcher sets -Dotel.exporter.prometheus.host=0.0.0.0, so a Prometheus exporter it is configured with binds on all interfaces. Configure the exporter and its port through the usual otel.* system properties or OTEL_* environment variables.

Attributes are attached per stream: ursa.stream.namespace, ursa.stream.name and ursa.stream.partition.

WAL storage

Metric	Type	Meaning
`ursa_storage_wal_putEntry_count_total`	Counter	Entries written to the WAL.
`ursa_storage_wal_putEntry_rejected_count_total`	Counter	Entries rejected on write. A rising count means appends are exceeding the pending-bytes budget.
`ursa_storage_wal_putEntry_duration_seconds_bucket`	Histogram	Time to write an entry.
`ursa_storage_wal_putEntry_pending_duration_seconds_bucket`	Histogram	Time an entry waits in the write buffer before processing.
`ursa_storage_wal_putEntry_cache_duration_seconds_bucket`	Histogram	Time to write an entry into the write cache.
`ursa_storage_wal_getEntries_duration_seconds_bucket`	Histogram	Time to read a batch, whichever tier serves it.
`ursa_storage_wal_getEntry_duration_seconds_bucket`	Histogram	Time to read a single entry.
`ursa_storage_wal_writeCache_flush_duration_seconds_bucket`	Histogram	Time to flush the write cache to storage.
`ursa_storage_wal_readCache_loading_count_total`	Counter	Loads into the read cache from the object store.
`ursa_storage_wal_readCache_eviction_count_total`	Counter	Read-cache evictions.
`ursa_storage_wal_readCache_loading_duration_seconds_bucket`	Histogram	Time to load a WAL object into the read cache.
`ursa_storage_wal_read_cache_missed_total`	Counter	Reads served by neither cache.
`ursa_storage_wal_putEntry_pending_count`	Gauge	Entries currently queued in the write buffer.
`ursa_storage_wal_writeCache_flushCallback_pending_count`	Gauge	Flushes awaiting acknowledgement.
`ursa_storage_wal_readCache_size_bytes`	Gauge	Current read-cache size.

Object storage

Metric	Type	Meaning
`ursa_storage_backend_storage_request_total`	Counter	Requests to the backend, local or cloud.
`ursa_storage_backend_write_duration_seconds_bucket`	Histogram	Backend write latency.
`ursa_storage_backend_read_duration_seconds_bucket`	Histogram	Backend read latency.
`ursa_storage_backend_metadata_read_duration_seconds_bucket`	Histogram	Backend metadata read latency.
`ursa_storage_backend_crc_duration_seconds_bucket`	Histogram	Checksum computation latency.
`ursa_storage_backend_delete_duration_seconds_bucket`	Histogram	Object deletion latency.
`ursa_storage_backend_write_bytes_count_bytes_total`	Counter	Bytes written to the backend.
`ursa_storage_backend_read_bytes_count_bytes_total`	Counter	Bytes read from the backend.

Write cache

Metric	Type	Meaning
`ursa_storage_wal_writeCache_used_bytes`	Gauge	Bytes in use.
`ursa_storage_wal_writeCache_bufferSegment_used`	Gauge	Buffer segments in use.
`ursa_storage_wal_writeCache_cacheSegment_used`	Gauge	Cache segments in use.
`ursa_storage_wal_writeCache_segment_count`	Gauge	Segments allocated.
`ursa_storage_wal_writeCache_capacity_bytes`	Gauge	Configured capacity per segment.

Compacted-object reads

Metric	Type	Meaning
`ursa_storage_lakehouse_read_messages_total`	Counter	Messages read from compacted Parquet files.
`ursa_storage_lakehouse_read_bytes_bytes_total`	Counter	Bytes read.
`ursa_storage_lakehouse_read_request_total`	Counter	Read requests served.
`ursa_storage_lakehouse_read_cache_hit_total`	Counter	Prefetch cache hits.
`ursa_storage_lakehouse_read_cache_miss_total`	Counter	Prefetch cache misses.
`ursa_storage_lakehouse_read_latency_seconds_bucket`	Histogram	Read latency.
`ursa_storage_lakehouse_read_request_queued_latency_seconds_bucket`	Histogram	Time a read waits in the queue before processing.

Compaction service

The signals here describe progress through three stages: publishing a task, converting the WAL range to Parquet, and committing the result.

Metric	Type	Meaning
`ursa_storage_compact_ongoing_topic_count`	Gauge	Streams currently being compacted.
`ursa_storage_compact_ongoing_task_count`	Gauge	Tasks in progress.
`ursa_storage_compact_latest_message_offset`	Gauge	Latest offset per stream.
`ursa_storage_compact_latest_published_offset`	Gauge	Latest offset covered by a published task.
`ursa_storage_compact_last_compacted_offset`	Gauge	Latest offset fully committed.
`ursa_storage_compact_lag`	Gauge	Difference between the latest and last-compacted offsets.
`ursa_storage_compact_bytes_total`	Counter	Bytes processed.
`ursa_storage_compact_messages_total`	Counter	Messages processed.
`ursa_storage_compact_message_end_to_end_duration_seconds_bucket`	Histogram	Append to table commit, end to end.
`ursa_storage_compact_message_from_ursa_to_parquet_duration_seconds_bucket`	Histogram	Append to Parquet write.
`ursa_storage_compact_duration_seconds_bucket`	Histogram	Duration of one task.
`ursa_storage_compact_read_messages_duration_seconds_bucket`	Histogram	Reading the WAL range.
`ursa_storage_compact_write_messages_duration_seconds_bucket`	Histogram	Decoding, converting and writing Parquet.
`ursa_storage_compact_task_commit_duration_seconds_bucket`	Histogram	Committing a task, including the index write.
`ursa_storage_compact_commit_to_lakehouse_duration_seconds_bucket`	Histogram	Committing the table snapshot alone.
`ursa_storage_compact_commit_task_batch_size`	Gauge	Files in one commit batch.
`ursa_storage_compact_published_task_bytes`	Gauge	Bytes covered by one task.
`ursa_storage_compact_committed_parquet_file_bytes`	Gauge	Size of committed Parquet files.
`ursa_storage_compact_tasks_in_init_state`	Gauge	Tasks awaiting work.
`ursa_storage_compact_tasks_in_compacted_state`	Gauge	Tasks converted, not yet committed.
`ursa_storage_compact_tasks_in_prepared_commit_state`	Gauge	Tasks prepared for commit.
`ursa_storage_compact_tasks_in_committed_state`	Gauge	Tasks committed.
`ursa_storage_compact_non_committable_task_histogram_bytes_bucket`	Histogram	Size distribution of non-committable tasks.

ursa_storage_compact_lag is the signal that answers "how far behind are my tables", and ursa_storage_compact_message_end_to_end_duration_seconds_bucket answers "how long until a record appears in one".

What to alert on

These are the signals the source marks as alert-worthy.

Metric	Condition
`compaction_cluster_leaders_total`	Sum across the cluster is not exactly `1`. Zero means nothing is publishing or committing; more than one means split leadership.
`ursa_storage_compact_publish_task_failed_count_total`	Rising. Tasks are not being published.
`ursa_storage_compact_failed_task_count_total`	Rising. Conversion is failing.
`ursa_storage_compact_task_commit_duration_seconds_count` with `response_status="failed"`	Rising. Commits to the table are failing.
`ursa_storage_compact_quarantined_topics_count`	Above zero. Streams are being held back after failures.
`ursa_storage_compact_topics_in_dlq`	Above zero. Streams have been dead-lettered.
`ursa_storage_compact_tasks_in_dlq`	Above zero. Tasks have been dead-lettered.
`ursa_storage_compact_non_committable_task_count`	Above zero. A stream has exceeded the non-committable task threshold.

Table writes and reads

Metric	Type	Meaning
`ursa_storage_lakehouse_writer_before_write_duration`	Histogram	Preparation before a write.
`ursa_storage_lakehouse_writer_write_all_duration`	Histogram	Writing a full batch.
`ursa_storage_lakehouse_writer_write_record_duration`	Histogram	Writing one record.
`ursa_storage_lakehouse_writer_encode_duration`	Histogram	Encoding before writing.
`ursa_storage_lakehouse_reader_seek_duration`	Histogram	Seeking.
`ursa_storage_lakehouse_reader_read_all_duration`	Histogram	Reading a full batch.
`ursa_storage_lakehouse_reader_read_record_duration`	Histogram	Reading one record.
`ursa_storage_lakehouse_reader_decode_duration`	Histogram	Decoding after reading.

Parquet files

Metric	Type	Meaning
`ursa_storage_lakehouse_parquet_write_record_duration`	Histogram	Writing records.
`ursa_storage_lakehouse_parquet_write_metadata_duration`	Histogram	Writing file metadata.
`ursa_storage_lakehouse_parquet_read_record_duration`	Histogram	Reading records.
`ursa_storage_lakehouse_parquet_read_metadata_duration`	Histogram	Reading file metadata.
`ursa_storage_lakehouse_parquet_seek_by_offset_duration`	Histogram	Seeking by offset.
`ursa_storage_lakehouse_parquet_seek_by_secondary_index_duration`	Histogram	Seeking by secondary index.

A starting dashboard

The Ursa repository ships a Grafana dashboard for the compaction service at dashboard/CompactionService.json.

Observability

On this page