Setup Kafka without ZooKeeper

From CMEE4K
Jump to navigation Jump to search

System Requirement

YouTube: https://www.youtube.com/watch?v=YGMrVT3a8J4
WebPage: https://www.digitalocean.com/community/tutorials/how-to-install-apache-kafka-on-centos-7

  1. >4G RAM
  2. OpenJDK 1.8+
    • For CentOS 7 & 8, enter yum install java-1.8.0-openjdk.x86_64

Prepare user ID

useradd kafka -m          where '-m' creates the /home/kafka directory
passwd   kafka
usermod  -aG  wheel kafka

Download Kafka 2.8+

https://kafka.apache.org/downloads

Download Scala 2.13 - kafka_2.13-2.8.0.tgz file

Logon to kafka UID, and cd to /home/kafka directory.

curl https://dlcdn.apache.org/kafka/2.8.0/kafka_2.13-2.8.0.tgz -o  Downloads/kafka.tgz
mkdir kafka;  cd kafka
tar -xvzf ~/Downloads/kafka.tgz --strip 1

A set of files are uncompressed in /home/kafka/kafka/... directory.

Configuring three Kafka servers

https://adityasridhar.com/posts/how-to-easily-install-kafka-without-zookeeper

      OR

  • Following the copy / edit Kafka cluster configuration instruction.

For this no ZooKeeper setup

  • The server properties files are in config/kraft directory. It is not using config/properties file.
  • To understand the values in server property file, see this [ How to easily install Kafka without zookeeper ]
  • Generate an UUID for formatting storage: bin/kafka-storage.sh random-uuid
One server Three servers
Use config/kraft/server.properties file which is for one server mode
To format these three storages
bin/kafka-storage.sh format -t {UUID} -c config/kraft/server.properties
How to easily install Kafka without zookeeper explains how to prepare 3 server.properties files in HOME/kafka/config/kraft directory
  • You can download these files from this Git, and edit them (server[1-3].properties files).
  • To format three storages
bin/kafka-storage.sh format -t {UUID} -c config/kraft/server1.properties
bin/kafka-storage.sh format -t {UUID} -c config/kraft/server2.properties
bin/kafka-storage.sh format -t {UUID} -c config/kraft/server3.properties

The Kafka working directories are defined in the server.properties file (the following lines).

log.dirs=/tmp/kraft-combined-logs
  OR
log.dirs=/tmp/server[1-3]/kraft-combined-logs

Starting the kafka servers

Since this command export KAFKA_HEAP_OPTS="-Xmx200M –Xms100M" does not work, so edit bin/kafka-server-start.sh file

if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
    #export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"
    export KAFKA_HEAP_OPTS="-Xmx300M -Xms100M"
fi
One server Three servers
bin/kafka-server-start.sh -daemon config/kraft/server.properties
Since these 3 servers are configured to read (waiting) from each other ports, they must start together.
bin/kafka-server-start.sh -daemon config/kraft/server1.properties
bin/kafka-server-start.sh -daemon config/kraft/server2.properties
bin/kafka-server-start.sh -daemon config/kraft/server3.properties

See the log: tail -f logs/server.log

Create a kafka topic

One server

This creates a topic pams-payment with default a partition and no replica (or 1 replica); and list and describle the topic.

bin/kafka-topics.sh --create --topic pams-payment --bootstrap-server localhost:9092

bin/kafka-topics.sh --bootstrap-server localhost:9092 --list
bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic kraft-test

Three servers

This creates a topic kraft-test with 3 partitions, and 3 replicas for 3 nodes; and list and describle the topic.

bin/kafka-topics.sh --create --topic kraft-test --partitions 3 --replication-factor 3 --bootstrap-server localhost:9092

bin/kafka-topics.sh --bootstrap-server localhost:9092 --list
bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic kraft-test

Exploring the kafka metadata using metadata shell

Similar to zookeeper cli, there is a metadata shell provided by Kafka so that we can read the data in the @metadata internal topic. Open the shell using the following command

bin/kafka-metadata-shell.sh  --snapshot /tmp/kraft-combined-logs/\@metadata-0/00000000000000000000.log
   OR
bin/kafka-metadata-shell.sh  --snapshot /tmp/server1/kraft-combined-logs/\@metadata-0/00000000000000000000.log

>> ls brokers/
>> ls topics/
>> cat topics/pams-payment/0/data
   OR
>> cat topics/kraft-test/0/data
>> cat metadataQuorum/leader
>> exit

Producing and consuming data from kafka

Productor

In terminal 1, start producer shell and enter messages. Note: all of them are using port 9092.

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic pams-payment
   OR
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic kraft-test

>message 1
>message 2
>message 3
>hello
>bye
CTRL-C

Or, using one line mode.

echo "Hello, World" | bin/kafka-console-producer.sh --broker-list localhost:9092 --topic pams-payment > /dev/null
   OR
echo "Hello, World" | bin/kafka-console-producer.sh --broker-list localhost:9092 --topic kraft-test   > /dev/null

Consumer

In terminal 2, start consumer shell to see the messages. Note: all of them are using port 9092

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic pams-payment
   OR
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic kraft-test

message 1
message 2
message 3
hello
bye
CTRL-C

Or, using one command line mode. The --from-beginning always read all the messages from the beginning, otherwise it only read the latest inserted message.

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic pams-payment --from-beginning
   OR
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic kraft-test   --from-beginning
CTRL-C



Appendix

https://www.digitalocean.com/community/tutorials/how-to-install-apache-kafka-on-centos-7

Start Kafka as system service

Apply to above one server mode

  • sudo vi /etc/systemd/system/kafka.service
[Service]
Type=simple
User=kafka
ExecStart=/bin/sh -c '/home/kafka/kafka/bin/kafka-server-start.sh /home/kafka/kafka/config/server.properties > /home/kafka/kafka/kafka.log 2>&1'
ExecStop=/home/kafka/kafka/bin/kafka-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

Using systemctl command to control kafka as daemon service

  • sudo systemctl status kafka
  • sudo systemctl start kafka
  • sudo systemctl stop kafka
  • journal -u kafka

The Kafka server is listening on port 9092

  • sudo systemctl enable kafka
  • sudo systemctl disable kafka

Other installation options

Step 6 — Installing KafkaT (Optional)
KafkaT is a tool from Airbnb that makes it easier for you to view details about your Kafka cluster and perform certain administrative tasks from the command line. Because it is a Ruby gem, you will need Ruby to use it.
Step 7 — Setting Up a Multi-Node Cluster (Optional)
If you want to create a multi-broker cluster using more CentOS machines, you should repeat Step 1, Step 4, and Step 5 on each of the new machines.
Step 8 — Restricting the Kafka User
Now that all of the installations are done, you can remove the kafka user’s admin privileges. Before you do so, log out and log back in as any other non-root sudo user.

__END__