Kafdrop: I can see you, Protobuf

 


Kafdrop is a web UI built on top of Spring Boot. It's free and has a good dark theme. (Well, the dark theme is the only theme it provides, actually.) Speaking of kafka web UI, there is another web UI like the Kafka UI - which also has the capability of browsing message, but at the time I write this blog post Kafdrop is the only one that supports viewing protobuf message.

Since I've been dealing a lot with kafka protobuf messages lately, I needed to know how to use Kafdrop with protobuf support. Because you know, it's such a pain in the eye viewing protobuf using kafkacat and protoc:

kafkacat -b kafka:9092 -C -t topic.name -o beginning -e | protoc --decode_raw

Luckily after few days of fooling around, I found that making kafdrop to work with protobuf is not a hard thing to do.

The Kafdrop Readme says that I need to put the protobuf descriptor file (.desc) to a directory, and pass the directory path as a parameter to Kafdrop. Creating a descriptor file from a .proto file is straight forward. I only need to do this:

protoc -o outfile.desc -I import-path-1:import-path-2 --include_imports \
file-1.proto file-2.proto

--include_imports option only necessary when your proto imports another file.

In case you are not familiar with protoc, in Ubuntu protoc can be installed using:

sudo apt update && sudo apt install -y protobuf-compiler

Once I have the .desc file in the right location, there's only one more thing to do. Pass the directory as a protobufdesc.directory parameter to Kafdrop. When I want to peek a message in a remote server, I prefer using Kafdrop jar like the following, but please beware that running Kafdrop jar requires Java 11:

java --add-opens=java.base/sun.nio.ch=ALL-UNNAMED -jar $KAFDROP_JAR \
--kafka.brokerConnect=remote.kafka:9092 --message.format=PROTOBUF \
--protobufdesc.directory=$PROTOBUFDESC_DIR

 Or, with docker-compose.yaml from Kafdrop repo :

services:
  kafdrop:
    ...
    environment:
      KAFKA_BROKERCONNECT: remote.kafka:9092
      CMD_ARGS: "--message.format=PROTOBUF --protobufdesc.directory=/var/protobuf_desc"
    ...
    volumes:
      - ~/protobuf/desc:/var/protobuf_desc

Then, when I navigate my browser to localhost:9000, click the specified topic's link, click the "View Messages" button, and pick PROTOBUF from the Message format dropdown list, pick the .desc file, and last but not least fill in the message name and click the button next to it. I can see the protobuf messages beautifully formatted.

Try this in your machine, clone my git repo here.

Comments

  1. Founded in 2000, its primary website, tombola.co.uk has become the largest bingo web site in the UK, with more players than another company, 메리트카지노 nicely as|in addition to} being primary in Spain and Italy. BETER Live is the live-casino brand of next-gen betting content and information provider BETER, whose other product verticals embody quick sports activities, esports and gaming solutions. This fun-filled game combines the thrills of both lotto and bingo, in a single fun-fuelled experience. You can select to guess with very small stakes and watch as the ball slides round, in hope of your successful numbers lining up. There are some huge bonuses on provide, and should you win, you can to|you probably can} select to increase the amount with a variety of multipliers.

    ReplyDelete

Post a Comment

Popular posts from this blog

Using Value Object as an ActiveRecord Attribute

How to Upgrade PostgreSQL 10 Cluster to 12 in Ubuntu 20.04