Introduction
Getting started
Tansu is a drop-in replacement for Apache Kafka with PostgreSQL, S3 or memory storage engines. Without the cost of broker replicated storage for durability. Licensed under the GNU AGPL. Written in 100% safe 🦺 async 🚀 Rust 🦀.
Installation
Step-by-step guides to setting up Tansu with various storage engines.
Storage Engines
Explore pluggable storage engines supporting PostgreSQL, S3 or memory.
Quick start
A from scratch multi-arch Docker container available from the Github Container Registry (ghcr.io).
Start Tansu
We will start a Tansu broker using memory storage:
docker run \
--detach \
--name tansu \
--rm \
-p 9092:9092 \
--env RUST_LOG=tansu_server::broker=debug \
ghcr.io/tansu-io/tansu \
--kafka-cluster-id="tansu-demo" \
--storage-engine="memory://tansu/" \
--kafka-advertised-listener-url="tcp://host.docker.internal:9092"
host.docker.internal
We are setting the Kafka advertised listener URL to host.docker.internal so that metadata responses from Tansu resolve to the IP of the host. Note that we will alias "host.docker.internal" as "host-gateway" so that this demo script will work with docker on both Mac-OS and Linux.
Apache Kafka® Java CLI
Let's start an interactive terminal session with the official Apache Kafka 3.9.0 command line interface:
docker run \
--tty \
--interactive \
--rm \
--add-host host.docker.internal:host-gateway \
apache/kafka:3.9.0 \
/bin/bash
Your prompt will now look something like:
6f102b2782d6:/$
Lets create a topic. Note that our bootstrap server is Tansu:
/opt/kafka/bin/kafka-topics.sh \
--bootstrap-server host.docker.internal:9092 \
--partitions=3 \
--replication-factor=1 \
--create \
--topic test
Your terminal should respond with:
Created topic test.
We can describe the topic that we have just created with:
/opt/kafka/bin/kafka-topics.sh \
--bootstrap-server host.docker.internal:9092 \
--describe \
--topic test
The response looks like:
Topic: test TopicId: AZSjwe9FdAKotCsOaO3_qQ PartitionCount: 3 ReplicationFactor: 1 Configs:
Topic: test Partition: 0 Leader: 111 Replicas: 111 Isr: 111 Adding Replicas: Removing Replicas: Elr: N/A LastKnownElr: N/A
Topic: test Partition: 1 Leader: 111 Replicas: 111 Isr: 111 Adding Replicas: Removing Replicas: Elr: N/A LastKnownElr: N/A
Topic: test Partition: 2 Leader: 111 Replicas: 111 Isr: 111 Adding Replicas: Removing Replicas: Elr: N/A LastKnownElr: N/A
Each of the partitions in a topic have node 111 as the leader. In Tansu, all brokers are equal. They're all called Spartacus... 111. Replication is the responsibility of the storage provider, not of the brokers. No matter how many replicas a topic has, they're all 111... Forget worrying about in sync replicas.
A demo isn't a demo without "Hello World!":
echo "hello world" | /opt/kafka/bin/kafka-console-producer.sh \
--bootstrap-server host.docker.internal:9092 \
--topic test
Consuming the topic:
/opt/kafka/bin/kafka-console-consumer.sh \
--bootstrap-server host.docker.internal:9092 \
--consumer-property fetch.max.wait.ms=15000 \
--group test-consumer-group \
--topic test
Should result in:
hello world
You can hit control-C a couple of times to break out of the consumer.
In Tansu all brokers are also the group (and transaction) controller. Tansu reduces the ping pong that clients experience looking up the current controller.
Getting help
Thank you for your interest in Tansu. Have a question? Please Contact us.
Submit an issue
Please raise any bug reports as an Issue on our GitHub.