Improvement proposals (LIPs)

A Lakestream Improvement Proposal (LIP) is a short design document for a change that other people build on or operate against: the Lakestream API, an SPI contract, a storage format, or the contract between Kafka and Lakestream. Writing the design down before the code lets reviewers and future implementers agree on what is changing, and why, before anyone debates how.

The LIPs for every Lakestream component live in one repository, openlakestream/lips, with one numbering. A LIP is written for Lakestream as a whole, and lists what changes in each component:

This page summarizes the process. The repository's contributing guide is the authoritative version, and its template is where a new LIP starts.

When you need a LIP

Most LIPs change an interface: an API, an SPI, a format, or a configuration surface. Write the LIP for Lakestream as a whole, and list what changes in each component, even when only one component changes.

For the Lakestream API and Ursa, you need a LIP for:

• new or changed public types in lakestream-api
• changes to an SPI contract, such as TableMaterializer or TableMaterializerFactory
• changes to the on-object or WAL formats, or to serialized field numbers and identifiers
• every new materializer, because registering one adds a TableCatalogType to lakestream-api

For UFK, you need a LIP for:

• changes to what a Kafka client can observe on a diskless topic: produce, fetch and offset behavior, errors, or limits
• new or changed configuration keys for diskless storage (the ursa.* broker, controller and topic configurations)
• changes to the contract between Kafka and Lakestream: how a topic maps to a stream, the stream properties Kafka writes, and the payload Kafka writes to the write-ahead log (WAL)
• changes to how the controller creates, grows, deletes or cleans up diskless topics

Changes to how classic topics behave usually belong upstream, as a KIP in Apache Kafka. If a change can't avoid touching classic behavior, it needs a LIP.

You don't need a LIP for bug fixes, internal refactoring, performance work that keeps behavior and formats unchanged, tests, or documentation. If you're not sure, ask where step 1 below says to start.

How it works

1. Start a discussion in the repository of the component you want to change. Describe the problem: what you're trying to build, and what gets in the way today. The first step is agreeing that the problem is worth solving.
   • For lakestream-api or Ursa, open a thread in Ursa's Discussions: Ideas. For a new materializer, open a New materializer proposal issue instead.
   • For UFK, open a thread in its Discussions: Ideas.
   • If the change spans components, start in the repository that owns the interface you're changing, usually Ursa, which holds lakestream-api.
2. Write the LIP. Copy the template to proposals/LIP-NNN-Short-Title.md, and add the LIP to the index in the repository's README.
3. Open a pull request in openlakestream/lips that adds the file with the status Proposed, and link it from the discussion. The design review stays on the pull request, so the document and its review stay together.
4. Review. The code owners of openlakestream/lips review it, together with a maintainer of each component the LIP changes. Mention those maintainers on the pull request. Expect questions about compatibility, rollback, alternatives and testing.
5. Merge. Merging the pull request accepts the LIP. Implementation pull requests, in whichever repository, link to it. A LIP that isn't accepted is closed, with the reasons recorded on the pull request.
6. Keep the status current as the work lands: update the LIP's header and its row in the index in the same pull request.

Governance describes who the code owners and maintainers are.

Status

LIPs that are proposed, accepted or implemented also appear on the roadmap.

Numbering

A new LIP takes the highest existing number plus one. If two open pull requests pick the same number, the one merged second renumbers.

Numbering starts at 161, the number LIP-161 was first published under. LIP-162 and LIP-163 were LIP-001 and LIP-002 in openlakestream/kafka before they moved to openlakestream/lips.

Index

Titles are quoted as each LIP gives them. The status column uses the site's version labels; each LIP's header records the exact release.

Writing a good LIP

• Lead with the problem. A reader should understand why the change matters before reading the design.
• Say what changes in each component, and whether one component has to ship before another.
• Be explicit about compatibility. Say what happens on upgrade and on rollback, and call breaking changes breaking.
• Keep it as short as the change allows. Link to existing documentation instead of repeating it.
• Record the alternatives you rejected, and why. They stop the same debate from happening twice.