Apache Kafka is a highly scalable distributed message queue. Message queues are middleware that mediates the transfer of data between systems and temporarily holds (queues) data.
This time, I will use Apache kafka's Docker Image and run kafka in the local environment. The version of Apache kafka used is 2.6.0.
$ docker -v
Docker version 19.03.6, build 369ce74a3c
I used a machine with a total of 8GB of memory, but it works fine with about 4GB.
$ cat /etc/issue
Ubuntu 18.04.2 LTS
$ cat /proc/meminfo | grep Mem
MemTotal: 8168284 kB
MemFree: 5540948 kB
MemAvailable: 7472660 kB
Many kafka Docker images are available on the Docker hub. This time, I am using the Docker image prepared here.
$ docker run --name my-kafka blueskyareahm/kafka:2.6.0
The container will start after the Image is downloaded, as shown below.
Unable to find image 'blueskyareahm/kafka:2.6.0' locally
2.6.0: Pulling from blueskyareahm/kafka
75f829a71a1c: Already exists
455e7974a143: Pull complete
a439c4ff44d0: Pull complete
b9eb8968940e: Pull complete
e6979812c85a: Pull complete
7812677cc867: Pull complete
ceacbc38a4c6: Pull complete
70319b3a888f: Pull complete
3fb4c3130215: Pull complete
68dc00f6ba7c: Pull complete
Digest: sha256:802f5d87bfb51fd5af66853a345e0ed65de8acc2f3fd61c259def2d6c05f63d6
Status: Downloaded newer image for blueskyareahm/kafka:2.6.0
/usr/lib/python2.7/site-packages/supervisor-4.2.1-py2.7.egg/supervisor/options.py:474: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security.
'Supervisord is running as root and it is searching '
2020-10-17 04:52:25,057 CRIT Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this message.
2020-10-17 04:52:25,068 INFO RPC interface 'supervisor' initialized
2020-10-17 04:52:25,069 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2020-10-17 04:52:25,069 INFO supervisord started with pid 1
2020-10-17 04:52:26,076 INFO spawned: 'zookeeper' with pid 9
2020-10-17 04:52:26,082 INFO spawned: 'kafka' with pid 10
2020-10-17 04:52:27,091 INFO success: zookeeper entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-10-17 04:52:27,091 INFO success: kafka entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Open another console and run the following command. This command creates Kafka Topic. (Here, the name is "topic1") Kafka exchanges messages via Topic.
$ docker exec -it my-kafka /opt/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic topic1
If the following message is displayed, it is OK.
Created topic topic1.
Execute the following command. This command is starting Kafka Producer. You will be ready to accept messages interactively as shown below.
$ docker exec -it my-kafka /opt/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic topic1
>
Open another console and run the following command. This command is starting Kafka Consumer. When I run it, nothing is displayed, but it is ready to accept messages.
$ docker exec -it my-kafka /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic topic1
Go back to the console where Kafka Producer is running and try typing some message.
Producer side
$ docker exec -it my-kafka /opt/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic topic1
>Hello Kafka!
>
Then, the same message as the one entered on the Kafka Consumer side will be displayed. Consumer side
$ docker exec -it my-kafka /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic topic1
Hello Kafka!
Recommended Posts