NEW Nisshi is now backed by Solace, and it's still open sourceRead the blog and hear from the creator →

Apache Kafka® compatible broker

PostgreSQL, SQLite or S3 storage

AVRO, JSON or Protobuf schema validation

Iceberg or Delta Lake support

Open Source Apache licensed

broker.rs
Cargo.toml
async move {
Frame::response(
Header::Response { correlation_id },
self.response_for(client_id, body, correlation_id)
.await?,
api_key,
api_version,
)
.map_err(Into::into)
}
.instrument(span)
.await

Introduction

Getting started

Nisshi (née Tansu) is an Apache Kafka®-compatible broker with PostgreSQL, SQLite, S3 or memory storage engines. Without the cost of broker replicated storage for durability. Licensed under the Apache License. Written in 100% safe 🦺 async 🚀 Rust 🦀.

Installation

Step-by-step guides to setting up Nisshi with various storage engines.

Storage Engines

Explore pluggable storage engines supporting PostgreSQL, S3 or memory.


Quick start

Nisshi is published on crates.io, and installed using cargo, part of the Rust systems programming language. You can easily install cargo via rustup.

Nisshi uses cargo features to enable the storage engines:

FeatureDescription
libsqlSQLite storage
postgresPostgreSQL storage
dynostoreS3 compatible and memory storage

Similarly, support for the Apache Iceberg or Delta Lake open table formats, or plain Apache Parquet files, can be enabled via:

FeatureDescription
icebergApache Iceberg open table format support
deltaDelta Lake open table support
parquetApache Parquet file support

We will use the SQLite storage engine for this quick start:

cargo install nisshi --features libsql

To also write schema-backed topics as Iceberg, Delta Lake or Parquet, add those features. They add to the default features, so the default storage engines stay in:

cargo install nisshi --features iceberg,delta,parquet

Nisshi is a single executable, creating a 40-150MB binary depending on the features enabled.

Once cargo has finished building, you can verify which storage engines are enabled with:

nisshi --help

Assuming you've just included SQLite support you will see:

Storage engines: libsql

Now, we will start the broker:

nisshi --storage-engine=sqlite://nisshi.db

Nisshi uses the RUST_LOG, environment variable if you want to enable a lot of logging you can use:

RUST_LOG=debug nisshi --storage-engine=sqlite://nisshi.db

Leaving the broker running, switching to another shell, we can maintain topics using the nisshi topic subcommand:

nisshi topic --help

Using the topic command, we can create, delete or list topics available on the broker.

nisshi topic create test

We can verify the topic has been created on the broker with:

nisshi topic list | jq '.[].name'

We can use the nisshi cat subcommand to produce data to the topic:

echo '{"value": "Hello World!"}' | nisshi cat produce test

We can consume from the topic using:

nisshi cat consume test

Which will output:

[{"key":null,"value":"Hello World!"}]

To read with a consumer group and committed offsets, use any Kafka client: see consumer groups.


Getting help

Thank you for your interest in Nisshi. Have a question? Please Contact us.

Submit an issue

Please raise any bug reports as an Issue on our GitHub.