Apache Kafka® compatible broker

PostgreSQL, SQLite or S3 storage

AVRO, JSON or Protobuf schema validation

Iceberg or Delta Lake support

Open Source Apache licensed

Get startedView on GitHub
broker.rs
Cargo.toml
async move {
    Frame::response(
        Header::Response { correlation_id },
        self.response_for(client_id, body, correlation_id)
            .await?,
        api_key,
        api_version,
    )
    .map_err(Into::into)
}
.instrument(span)
.await

Introduction

Getting started

Nisshi (née Tansu) is a drop-in replacement for Apache Kafka with PostgreSQL, SQLite, S3 or memory storage engines. Without the cost of broker replicated storage for durability. Licensed under the Apache License. Written in 100% safe 🦺 async 🚀 Rust 🦀.

Installation

Step-by-step guides to setting up Nisshi with various storage engines.

Storage Engines

Explore pluggable storage engines supporting PostgreSQL, S3 or memory.

Quick start

Nisshi is published on crates.io, and installed using cargo, part of the Rust systems programming language. You can easily install cargo via rustup.

Nisshi uses cargo features to enable the storage engines:

FeatureDescription
libsqlSQLite storage
postgresPostgreSQL storage
dynostoreS3 compatible and memory storage

Similarly, support for the Apache Iceberg or Delta Lake open table formats can be enabled via:

FeatureDescription
icebergApache Iceberg open table format support
deltaDelta Lake open table support

We will use the SQLite storage engine for this quick start:

cargo install nisshi --features libsql

Alternatively, you can enable all the features with:

cargo install nisshi --all-features

Nisshi is a single executable, creating a 40-150MB binary depending on the features enabled.

Once cargo has finished building, you can verify which storage engines are enabled with:

nisshi --help

Assuming you've just included SQLite support you will see:

Storage engines: libsql

Now, we will start the broker:

nisshi --storage-engine=sqlite://nisshi.db

Nisshi uses the RUST_LOG, environment variable if you want to enable a lot of logging you can use:

RUST_LOG=debug nisshi --storage-engine=sqlite://nisshi.db

Leaving the broker running, switching to another shell, we can maintain topics using the nisshi topic subcommand:

nisshi topic --help

Using the topic command, we can create, delete or list topics available on the broker.

nisshi topic create test

We can verify the topic has been created on the broker with:

nisshi topic list | jq '.[].name'

We can use the nisshi cat subcommand to produce data to the topic:

echo '{"value": "Hello World!"}' | nisshi cat produce test

We can consume from the topic using:

nisshi cat consume test

Which will output:

[{"key":null,"value":"Hello World!"}]

Getting help

Thank you for your interest in Nisshi. Have a question? Please Contact us.

Submit an issue

Please raise any bug reports as an Issue on our GitHub.

Next
Features