In today’s world of observability and real-time monitoring, efficiently collecting and transferring logs is more important than ever. One of the most popular methods for achieving this is by using syslog-ng, a reliable and flexible log forwarder, to gather logs and send them to Apache Kafka, a powerful distributed streaming platform.
In this guide, we will walk you through everything you need to know to set this up—from configuring your environment to testing that logs successfully arrive in Kafka. We’ll be compiling everything from source so you have complete control over the versions and features. And just to clarify, we’re not using Docker here; we’ll be working directly on bare metal or virtual machines.
Whether you’re experimenting in a local lab or building the foundation for a robust production pipeline, this tutorial is here to support you. 🙌
Before we get started, here’s what you will need:
This guide is suitable for:
KafkaWhether you are deploying in a lab environment or preparing a production-ready log pipeline, this hands-on guide provides full control over the components involved—without the use of Docker or containerization.
📌 Note: We will be using Kafka in KRaft mode, which means no ZooKeeper! This is a simpler and more modern approach that works well for test setups or small deployments.
Before we begin compiling anything, it’s essential to ensure that your system has all the necessary tools and libraries. Think of this step as making sure your toolbox is fully stocked before you start building a piece of furniture—this will save you a lot of time and frustration later on.
We’ll be installing a set of packages that are crucial for:
Kafka communicationThe exact command for installation will depend on your operating system. Please select the one that matches your setup.
ubuntu/debian based systems: Open your terminal and run:
sudo apt update
sudo apt install -y git gcc make autoconf automake libtool pkg-config \
libglib2.0-dev libjson-c-dev libcurl4-openssl-dev libssl-dev \
liblzma-dev libsystemd-dev libz-dev uuid-dev flex bison
This grabs compilers (gcc, make), build tools (autoconf, automake, libtool), and a set of development libraries that syslog-ng relies on.
centos/rocky linux (RHEL-based systems): Run the following:
sudo dnf install -y git gcc make autoconf automake libtool pkgconfig \
glib2-devel json-c-devel libcurl-devel openssl-devel libuuid-devel \
flex bison zlib-devel
Same idea here — development headers, compilers, and build essentials tailored for RHEL-based environments.
💡 Why are all these packages necessary?
Syslog-ng is modular and flexible. Compiling it with Kafka support requires pulling in all the essential dependencies manually, including those for crypto, JSON, UUIDs, and networking. Trust us, installing these now will help you avoid a lot of “missing dependency” errors later on.
Once this step is complete, your system will be ready to start building real-time log magic. ✨
Now that your toolbox is ready, it’s time to bring in one of the essential components for this integration: librdkafka.
Librdkafka is the official C/C++ client library for Apache Kafka, developed by Confluent. This library allowssyslog-ngto communicate effectively with Kafka, enabling it to send logs across the network in a partitioned format that can be consumed by any downstream system.
For this setup, we’ll use version 1.9.2, which is a stable and proven release that works well withsyslog-ngversion 3.35.x.
🧠 Important: The compatibility betweensyslog-ngand librdkafka versions is crucial. They depend on specific APIs and features that must align. If you’re using a different version of syslog-ng, it’s advisable to verify which librdkafka release is supported before moving forward.
Let’s grab the source code and build it from scratch:
git clone https://github.com/confluentinc/librdkafka.git
cd librdkafka
git checkout v1.9.2
./configure
make -j$(nproc)
sudo make install
sudo ldconfig
Here’s a clearer explanation of each step:
The compiled libraries usually end up in the directory /usr/local/lib. Thanks to ldconfig, these libraries will be recognized system-wide. This means that whensyslog-ngneeds to link to Kafka, it will know exactly where to find it.
Once this step is complete, you’ll officially have Kafka enabled! Next up is installing another important component: the eventlog library. ✨
Before we dive into compiling syslog-ng, there’s one important piece we need to include: the eventlog library.
This library is the unsung hero behind the scenes. It’s a core dependency of syslog-ng, handling how logs are structured and written internally. Without it, syslog-ng won’t even compile.
Fortunately, building it is quite straightforward! 😊
🛠️ Let’s Build It!
Here’s how to download and install it from the source:
git clone https://github.com/balabit/eventlog.git
cd eventlog
./autogen.sh
./configure
make -j$(nproc)
sudo make install
sudo ldconfig
What’s Happening Here?
💡 Pro Tip: If autogen.sh complains about missing tools like libtool, m4, or autoconf, simply install them using your package manager. These tools help generate the configure scripts from scratch.
Once you’ve installed eventlog, you’re officially ready to build syslog-ng itself — the heart of this whole operation.
Traditionally, running Kafka also required setting up ZooKeeper, which handled coordination and metadata. However, starting from Kafka 3.x, there is a much simpler way to run things: KRaft mode (Kafka Raft Metadata mode). This allows Kafka to manage itself without needing ZooKeeper at all.
In this step, we’ll guide you through setting up Kafka in KRaft mode manually—no containers, no fluff-just pure, hands-on Kafka goodness.
### 4.1 Install Java
Kafka is built on Java, so you’ll need a working Java environment before starting it up. Here’s how to install OpenJDK 11 based on your operating system:
ubuntu/debian based systems:
sudo apt install -y openjdk-11-jdk
centos/rocky linux (RHEL-based systems):
sudo dnf install -y java-11-openjdk
✅ Note: You can verify that Java is installed by running java -version.
### 4.2 Download and Extract Kafka
We will be using Confluent’s Kafka 7.5.0 Community Edition, which includes everything needed to get started quickly.
wget https://packages.confluent.io/archive/7.5/confluent-community-7.5.0.tar.gz
tar -xzf confluent-community-7.5.0.tar.gz
cd confluent-7.5.0
🎯 This will create a confluent-7.5.0/ directory containing all the Kafka tools and scripts we’ll need.
### 4.3 Create a KRaft Configuration File
Kafka requires a configuration file to know how to run. Create a file named custom-kraft-conf.properties and paste the following content:
# KRaft mode - broker config
process.roles=broker,controller
node.id=1
controller.quorum.voters=1@localhost:9093
log.dirs=/tmp/kraft-combined-logs
listeners=PLAINTEXT://localhost:9092,CONTROLLER://localhost:9093
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
inter.broker.listener.name=PLAINTEXT
controller.listener.names=CONTROLLER
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
💡 This configuration is ideal for development or testing on a single node. You can adjust it later for production setups with multiple brokers.
### 4.5 Format KRaft Storage and Start Kafka
Before Kafka can start, it needs to initialize the storage directory using the configuration file you just created.
Once formatted, start the Kafka server using the following command:
bin/kafka-storage format -t $(bin/kafka-storage random-uuid) -c custom-kraft-conf.properties
bin/kafka-server-start custom-kraft-conf.properties
🎉 That’s it! Kafka is now running in KRaft mode with both the broker and controller roles handled by a single process. No ZooKeeper, no external dependencies.
🧪 Up next, we’ll create a topic and prepare Kafka to receive messages from syslog-ng.
Now that Kafka is up and running, it’s time to set up syslog-ng so it can forward log messages directly to our Kafka topic. This step connects log collection with streaming.
Let’s begin by creating a custom configuration file. You can place it wherever you prefer, but a common convention is to use a specific directory for configuration files.
/etc/syslog-ng/conf.d/syslog-ng-custom.conf
Here’s what the contents of the file should look like:
@version: 3.35
@include "scl.conf"
# 1. Source: Accept syslog over TCP on port 5140
source s_netcat {
tcp(ip("0.0.0.0") port(5140));
};
# 2. Destination: `Kafka` sink
destination d_kafka {
kafka(
bootstrap-servers("localhost:9092")
topic("syslog-ng-topic")
message("$(format-json --scope all-nv-pairs)")
);
};
# 3. Log path: Connect source to destination
log {
source(s_netcat);
destination(d_kafka);
};
Let’s break down the configuration:
format-json template and sent to the Kafka topic named syslog-ng-topic.s_netcat) with our Kafka destination (d_kafka).🧩 Note: In the example above, we’ve defined a static Kafka topic (syslog-ng-topic). However, if your use case requires dynamic topic assignment - for example, routing logs based on source IP, hostname, or application - you can achieve that too using rewrite rules, set() functions, and a $(format-json) tricks in syslog-ng. It requires some customization, but it’s definitely possible and worth exploring for more complex log routing scenarios.
Additional Configurations
If you wish to listen on a different port or use UDP instead of TCP, you can easily adjust the source section to accommodate those changes.
Once this configuration is ready, all that’s left is to start syslog-ng with it and begin streaming logs into Kafka — which we’ll cover in the next step! 🚀
Now that our configuration file is ready, it’s time to bring syslog-ng to life and let it start forwarding logs to Kafka. Run syslog-ng using the compiled binary and custom config:
sudo systemctl start syslog-ng
sudo systemctl status syslog-ng
Use -Fvde to run in foreground, with verbose and debug output.
🛠️ Bonus Tip: Test Your Config
Before starting the service, it’s a good idea to check if your config is valid:
syslog-ng -s -f /etc/syslog-ng/conf.d/syslog-ng-custom.conf
If everything is okay, you won’t see any errors.
Once syslog-ng is running and your Kafka broker is already up, you’re all set to send and receive logs in real time. In the next step, we’ll send a test log message using netcat and watch it flow into Kafka like magic! ✨
Now that everything is set up, it’s time to test the pipeline and see if your logs make it from syslog-ng to Kafka. We’ll use a simple yet powerful tool called netcat (nc) to simulate a syslog message sent over TCP.
echo "<13>Sep 15 16:25:00 myhost myapp: This is a test from nc" | nc 127.0.0.1 5140
💡 Why This Works
Your syslog-ng config listens for incoming TCP messages on port 5140, parses them, and forwards them to your Kafka topic (syslog-ng-topic). So when you send this message using nc, it’s picked up by syslog-ng and sent to Kafka—no extra work needed!
✅ Pro Tip: If you have a Kafka consumer running in the background, you should see the message arrive almost instantly, neatly wrapped in JSON format! If you’re unsure how to check the messages, feel free to jump to the next section. 😊
You’ve sent your log—now let’s ensure that it actually made it to Kafka. This is the final (and most satisfying) step: watching your log message appear in real-time like magic! 🪄
To do this, use the built-in Kafka consumer tool to read messages from the topic.
bin/kafka-console-consumer --bootstrap-server localhost:9092 \
--topic syslog-ng-topic --from-beginning
If everything is working correctly, you should see output like this:
{"TRANSPORT":"rfc3164+tcp","SOURCE":"s_netcat","PROGRAM":"myapp","MSGFORMAT":"rfc3164","MESSAGE":"This is a test from nc","LEGACY_MSGHDR":"myapp: ","HOST_FROM":"localhost","HOST":"localhost"}
Let’s quickly recap what we’ve accomplished in this hands-on journey:
Apache Kafka in KRaft mode — no ZooKeeper needed!Kafka topic and connected it with syslog-ngKafka successfullyBy the end of this guide, you now have a real-time logging pipeline using syslog-ng and Kafka — from scratch and without Docker or containers. 🎉 This setup is solid, flexible, and perfect for both experimentation and production use.
🔜 In future posts, we’ll explore more advanced features — like filtering, enriching messages, and partitioning strategies to fine-tune performance.
👀 Looking for more?
You might also enjoy Selim’s excellent article on a similar topic: ➡️ “Ingesting Syslog data to Kafka” (2021) — a great reference if you’re working with older versions of syslog-ng or Kafka.
Have ideas, suggestions, or want to share how you’ve extended this setup? Contributions are always welcome! Feel free to fork the setup, build on it, and tag me when you do.
If you run into issues or just want to chat about logs, Kafka, or infrastructure in general — don’t hesitate to reach out to me on LinkedIn. I’d love to hear what you’re building.
📦 Bonus Coming Soon: If you’re more of an rsyslog fan — don’t worry. A Kafka integration guide for rsyslog is on the way! Stay tuned. 🚀
