Ursa for Apache Kafka (UFK)

This repository, openlakestream/kafka, is Ursa for Apache Kafka (UFK): a Kafka distribution built on the Lakestream API and specification. Diskless topics keep their records on object storage through Ursa; every other topic stays Kafka. UFK is built on Apache Kafka 4.3.1, and its releases are published as the lakestream/kafka Docker image.

Lakestream is an open API and specification for stream storage on object storage, with a stream materialization framework that defines how a stream becomes a lakehouse table. Ursa implements it as a storage engine; UFK is a Kafka distribution built on it. Both are open source under Apache 2.0.

UFK adds two things to Kafka:

• Diskless topics. Topics whose durability comes from object storage instead of broker disks. Brokers hold no partition data, so failover moves no bytes, scaling is a CPU and network decision, and a zone-aware client never pays for cross-AZ replication.
• Lakehouse tables. The compaction pass that consolidates those topics' write-ahead log can also materialize an Iceberg table, queryable from DuckDB, Trino or Spark, with no copy job or Connect sink in between. Kafka consumers keep reading the log; query engines read the table.

Classic topics behave as they do in Apache Kafka. Diskless storage is an additional, per-topic option, so a single cluster can serve latency-sensitive workloads on classic topics and cost-sensitive workloads on diskless topics at the same time.

Both modes share one controller quorum, one metadata log, one set of ACLs, one consumer-group coordinator, and the same client protocol. Consumer offsets stay in __consumer_offsets, which is always a classic topic.

• Nothing changes until you turn it on. ursa.storage.enable defaults to false on both the broker and the topic, so a stock configuration behaves exactly like the Apache Kafka release it is built from (Apache Kafka 4.3.1).
• No protocol changes. Existing producers, consumers, Kafka Streams, Kafka Connect, admin clients, and the bin/ tools work unmodified against diskless topics. Clients cannot tell the difference: the broker still reports a leader, still serves Fetch and ListOffsets, still enforces offsets and idempotent producer semantics.
• Per-topic, not per-cluster. Diskless is a topic config. Migrating means creating new topics with ursa.storage.enable=true; existing topics are untouched.
• Upstream-compatible codebase. Diskless logic sits behind isDisklessTopic() checks; upstream code paths are unchanged when diskless is disabled.

Diskless storage is a bypass layer in ReplicaManager that routes storage operations for enabled topics to Ursa instead of local logs.

          Producers / Consumers
                    │
                    ▼
            ┌───────────────┐
            │   KafkaApis   │
            └───────┬───────┘
                    ▼
          ┌───────────────────┐
          │  ReplicaManager   │
          └─────────┬─────────┘
                    │  DisklessStorageReplicaManagerSupport
          ┌─────────┴──────────┐
      diskless              classic
          │                    │
          ▼                    ▼
   ┌──────────────┐    ┌──────────────────┐
   │ Ursa writer  │    │    Local log     │
   │  and reader  │    │  (.log + ISR)    │
   └──────┬───────┘    └──────────────────┘
          │
   ┌──────┴────────────┬──────────────────┐
   ▼                   ▼                  ▼
Object storage       Oxia            Compaction
(WAL + Parquet)   (catalog +      (WAL → Parquet
                 producer state)   → Iceberg)

• Writes go into an Ursa write-ahead log in object storage. The write buffer flushes on an interval (ursa.storage.write.buffer.flush.interval.ms, default 250 ms) or once ursa.storage.write.buffer.flush.size bytes accumulate — the interval is the produce-latency floor for an idle topic, throughput is the bound for a busy one.
• Reads are served from the same storage, transparently spanning the WAL and the compacted objects that compaction has already written.
• Compaction consolidates the WAL into those compacted objects. It runs as a separate service, and it is required: Kafka retention on a diskless topic issues a soft trim, and WAL objects are only physically deleted up to the oldest un-compacted position across every stream. See Compaction is required.
• Producer state for idempotent producers is snapshotted into Oxia instead of local .snapshot files, so a broker restart does not lose idempotence guarantees.
• Topic lifecycle — creation, partition growth, deletion, and orphan cleanup — is reconciled by the active controller against the stream catalog.
• All storage calls are asynchronous; request-handler threads are never blocked on object storage.

Records written to a diskless topic are compacted out of the WAL, and the same compaction pass can materialize them into an Iceberg table registered in an Iceberg REST catalog, queryable by engines like DuckDB, Trino, or Spark without a copy job or a Connect sink.

Kafka producer ──▶ broker ──▶ Ursa WAL (object storage)
                                       │
                                       ▼
                              compaction (Parquet)
                                  │          │
                                  ▼          ▼
                          compacted     Iceberg table
                           objects      (REST catalog)
                              │               │
                              ▼               ▼
                      Kafka consumers    SQL engines

Compaction runs as a separate service (the Ursa compactor) rather than inside the broker, so it never competes with the request path. Kafka consumers read the compacted objects and query engines read the table: two outputs of one compaction pass, both in your own bucket.

Compaction is not a lakehouse add-on — it is what makes diskless storage reclaimable:

1. Kafka retention on a diskless topic issues a soft trim, which moves the log's first offset. Soft-trimmed entries are logically gone but still occupy object storage.
2. WAL objects are physically deleted by a cleaner that runs inside the broker's storage runtime.
3. That cleaner's delete watermark is the oldest un-compacted position across every stream, and only compaction advances it.

So a deployment with no compaction service has a WAL that only grows: retention.ms and retention.bytes hide records from consumers without freeing a byte. Because the watermark is a global minimum, one stream that never gets compacted also pins reclamation for every other stream.

Registering an Iceberg table is a second, optional sink on the same compaction pass. You can run compaction without any catalog at all.

The fastest way to see it running is the Docker Compose stack — three brokers, Oxia, MinIO, and optionally the compactor plus an Iceberg catalog. The released image, lakestream/kafka:4.3.1.1, carries the brokers, the CLI and the Ursa compactor; pair it with the Compose files from its tag:

git checkout v4.3.1.1
cd docker/examples/docker-compose-files/cluster/ursa
export IMAGE=lakestream/kafka:4.3.1.1   # keep IMAGE exported in every shell that runs make here

make up                 # Oxia + MinIO + 3 brokers + compactor
make create-topic       # create a diskless topic
make demo               # producer/consumer perf demo
make destroy            # tear everything down

To build the image from your checkout instead, leave IMAGE unset and run ./build-image.sh before make up. It builds lakestream/kafka:latest and resolves the compactor (ursa-storage-compact) from Maven Central.

Adding the Iceberg catalog is one more command — it starts Polaris and has the compactor write the external table alongside the compacted objects it already writes:

make lakehouse-demo     # Kafka -> Parquet -> Iceberg -> DuckDB, asserted end to end

See the stack's README for the full set of profiles, ports, and knobs.

You need an object storage bucket, an Oxia service, the Ursa runtime jars in $KAFKA_HOME/ursa-storage/ (the release tarball built by ./gradlew releaseTarGz puts them there), and a running compaction service. An Iceberg catalog is optional on top of that.

Broker/controller configuration:

# Turn the diskless storage system on for this cluster
ursa.storage.enable=true

# Metadata and producer state
ursa.catalog.oxia.service.url=oxia://oxia:6648/default
ursa.oxia.service.url=oxia://oxia:6648/default

# Record storage
ursa.storage.backend.type=S3
ursa.storage.path=ursa/wal
ursa.storage.s3.bucket=my-kafka-bucket
ursa.storage.s3.region=us-east-1
ursa.storage.s3.endpoint=https://s3.us-east-1.amazonaws.com
ursa.storage.s3.access.key=...
ursa.storage.s3.secret.key=...

# Compacted objects
ursa.storage.compaction.bucket=my-kafka-bucket
ursa.storage.compaction.prefix=ursa/compacted

Then create a diskless topic:

bin/kafka-topics.sh --create \
  --topic events \
  --partitions 12 \
  --replication-factor 1 \
  --config ursa.storage.enable=true \
  --bootstrap-server localhost:9092

Every other topic in the cluster stays classic. To flip the default for new, non-internal topics, set ursa.storage.topic.default.enable=true on the brokers.

ursa.storage.enable is immutable after topic creation — changing a topic between modes means deleting and recreating it.

Run bin/kafka-configs.sh or read ServerLogConfigs for the complete list.

To keep diskless traffic inside an availability zone, set broker.rack on each broker and put a zone_id token in the client's client.id:

client.id=orders-service,zone_id=us-east-1a

The broker then picks an owner in the client's zone and reports it as the partition leader, falling back to the full set of live brokers when no in-zone broker is available. Clients need no code changes beyond the client.id. See LIP-163.

• No transactional producers. Transactional produce requests are rejected; idempotent producers are supported.
• No key-based log compaction. cleanup.policy=compact semantics are not implemented for diskless topics; only WAL-to-Parquet compaction in the storage layer.
• Replication factor must be 1. Durability comes from object storage, so Kafka-level ISR replication is bypassed. CreateTopics rejects any other value.
• Internal topics are always classic. __consumer_offsets and __transaction_state stay on local logs.
• Mode is fixed at creation. ursa.storage.enable cannot be altered afterwards.
• Retention needs compaction to free storage. Without a compaction service running, retention.ms and retention.bytes remove records logically but reclaim nothing.

You need JDK 17. The org.openlakestream:ursa-storage artifacts are resolved from Maven Central.

./gradlew jar                                  # build
./gradlew releaseTarGz                         # binary tarball, including ./ursa-storage/ runtime jars
./gradlew test -Pkafka.ci.isolated.tests=exclude   # main test suite
./gradlew test -Pkafka.ci.isolated.tests=only      # diskless integration tests

The standard Apache Kafka build, test, IDE, and code-quality instructions all still apply — see the upstream README. UFK's own conventions live in CONTRIBUTING.md.

• LIP-162: Diskless Storage with Ursa Integration
• LIP-163: Ursa Zone-Aware Owner Selection
• Docker Compose stack
• Materializers: what the compactor's materializers get from Kafka
• AGENTS.md — architecture, module layout, and build conventions
• Apache Kafka documentation for everything inherited from upstream

Upstream Kafka has no pluggable replica manager, so UFK carries the diskless path in its own ReplicaManager.

This repository tracks Apache Kafka and keeps divergence minimal: changes are additive where possible, and the upstream test suite is expected to pass with diskless storage disabled. Bugs in Kafka itself belong upstream; issues with diskless storage or the lakehouse path belong here.

Contributions are welcome. CONTRIBUTING.md covers what belongs here and what belongs upstream, how to build and test, and how to open a pull request. If you use AI tools, read the AI policy. Coding agents start with AGENTS.md. Report security problems privately, as described in SECURITY.md.

Apache License 2.0, the same as Apache Kafka. See LICENSE.