Kafka is a robust event streaming platform that is used by a lot of companies these days. As companies and developers adopt an event-driven architecture, advanced data pipelines, and real-time analytics - Kafka is often the default platform of choice.
The project itself is open-source and I’ve noticed a lot of “managed” Kafka offering around. I myself have been using a flavour of Kafka (Azure Event Hubs ) quite a lot in my projects and startups.
In this blog post, I’ll demonstrate how you can install Kafka on your WSL (Windows Subsystem for Linux). This technique is very useful if you want to test out Kafka locally and simulate messages with your applications. Although it is possible to install Kafka directly on Windows, I find the experience not very intuitive - not to mention it’s good to install it on a Linux flavour as it’s a good testing ground as you deploy Kafka later on a server.
There are at least 2 ways to do it.
- Manual download
- Docker compose
Manual Download
Install Dependenceis
In order to run Kafka, you need to first install all the dependencies.
sudo apt update
sudo apt install default-jre
sudo apt install openjdk-8-jre-headless
Download Kafka
Head over to Apache Kafka Downloads to check for the latest release. Right click on the Binary downloads like and copy the link.
Example: https://downloads.apache.org/kafka/3.2.1/kafka_2.12-3.2.1.tgz
Then download the tgz file and extract it.
wget https://dlcdn.apache.org/kafka/3.2.0/kafka_2.13-3.2.0.tgz
tar -xvzf kafka_2.13-3.2.0.tgz
Note: You can move the extracted file to any preferred location.
Configure your source file
At this point you have everything that you need to use Kafka. However, to make the experience easier later, it’s recommended to pre-configure your source file (.bashrc / .zshrc) so that each time you load the console, all paths are pre-set.
Add these lines to your .bashrc or .zshrc file.
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64
export KAFKA_HOME=~/kafka_2.12-3.2.1/
To load the source file again without doing a re-opening of your WSL, just type:
source ~/.bashrc
Testing Kafka
You need 4 tabs for this running the following (single lines)
- Start zookeeper
$KAFKA_HOME/bin/zookeeper-server-start.sh $KAFKA_HOME/config/zookeeper.properties
- Start Kafka
$KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties
- Use a Producer to publish events
$KAFKA_HOME/bin/kafka-console-producer.sh --topic sample-topic --broker-list localhost:9092
- Use a Consumer to receive events
$KAFKA_HOME/bin/kafka-console-consumer.sh --topic sample-topic --from-beginning --bootstrap-server localhost:9092
Go back to the producer tab and publish some events.
Then go back to the consumer tab and you can see that the events are consumed.
You can also run them side by side to see them in action.
Bonus: Using VSCode
Visual Studio Code supports running WSL via remote server. Instead of just looking at the terminal, using VSCode, you can see the file structure of your WSL and can do right-clicks and drag-drop experience.
Using Docker
Installing Docker
Head over to Install Docker Desktop on Windows | Docker Documentation to install Docker on your Windows machine.
Once installed, make sure you go to your settings and tick the Use the WSL 2 based engine so that the container images that you’ll be using (such as Kafka) will be running on a Linux host (instead of Windows).
Docker Compose Files
Although there are a lot of docker compose files around there, for this sample, let’s use Canonical’s Kafka image . Canonical is the company who is actively maintaining Ubuntu (together with the open source community).
This approach is a good starting point as it’s just an automation of what we’ve done in the manual install section.
Bonus: Other compose files
Run Zookeeper and Kafka
There are two docker images involved here that you need to run locally.
Run the Zookeeper docker image:
docker run -d -p 2181:2181 ubuntu/zookeeper:edge
Now, run the Kafka container locally.
docker run -d --name kafka-container -e TZ=UTC -p 9092:9092 -e ZOOKEEPER_HOST=host.docker.internal ubuntu/kafka:3.1-22.04_beta
You should be able to see them running on the Desktop app:
Testing and Debugging
Launch an interactive shell so that you can run the console applications inside the container.
docker exec -it kafka-container /bin/bash
On one tab, run the producer console:
kafka-console-producer.sh --topic sample-topic --broker-list localhost:9092
On the other tab, run the consumer console:
kafka-console-consumer.sh --topic sample-topic --from-beginning --bootstrap-server localhost:9092
🎉 Congratulations! You can now test Kafka on your local machine using Docker.
Summary
- Kafka is a powerful event streaming platform.
- Setting up Kafka on WSL can be done via manual download or Docker
- These setup allows you to test Kafka locally without the need of a cloud hosted service
References: