In the previous blog, we discussed the introduction, architecture and use cases of Apache Kafka. One of the important use cases of Kafka is producing and consuming messages from Kafka topics, we will dive deep into the practical implementation of Kafka using CLI.
Getting Started with Kafka CLI
The Kafka CLI is a collection of shell scripts that interact with the Kafka server using the Kafka protocol. The scripts are located in the Kafka installation directory under the "bin" folder. Before using Kafka CLI, you must download and install Kafka on our system.
Step 1: Start a Zookeeper service
As we know from the previous blog the zookeeper is the one who tracks the status of nodes in the Kafka cluster and maintains a list of Kafka topics and messages. So we need to, first of all, start the zookeeper service by using the below command after going inside the Kafka folder.
$ bin/zookeeper-server-start.sh config/zookeeper.properties
Step 2: Start a Kafka broker service
After starting the zookeeper, we need to start the Kafka server (this will run a Kafka broker service) on another terminal:
$ bin/kafka-server-start.sh config/server.properties
This command starts the Kafka server and loads the configuration from the "config/server.properties" file.
Step 3: Managing Kafka Topics
One of the primary use cases of the Kafka CLI is managing Kafka topics. Topics are used to organize data in Kafka, and each topic is partitioned across multiple Kafka brokers.
To create a new Kafka topic, you can use the following command:
$ bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic tano_topic
This command creates a new Kafka topic named "tano_topic" with one partition and one replica factor. The "--bootstrap-server" flag specifies the Kafka broker to use for topic creation.
Step 4: Producing messages
To produce messages to a Kafka topic ("tano_topic"), you can use the following command:
$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic tano_topic
This opens up a console where you can type messages to be produced to the topic ("tano_topic"). With every new message produced offset is a count which gets increased. Publishing messages on a topic is also known as writing events on a topic.
Step 5: Consuming messages
To consume messages to a Kafka topic ("tano_topic"), you can use the following command:
$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic tano_topic --from-beginning
This opens up a console where you can consume messages from the "tano_topic" topic starting from the beginning. You can also consume messages from a specific offset by specifying the "--offset" flag.
You can list all the topics in a Kafka cluster using the following command:
$ bin/kafka-topics.sh --list --bootstrap-server localhost:9092
Conclusion
In this blog, we explored the implementation of the Kafka CLI and how it can be used to manage Kafka clusters. We covered the basics of managing Kafka topics, producing and consuming messages, and described some of the commonly used Kafka CLI commands.
The Kafka CLI is a powerful tool for managing Kafka clusters, and it is an essential tool for developers working with Kafka. By mastering the Kafka CLI, developers can easily manage Kafka topics, produce and consume messages, and troubleshoot Kafka clusters.