If you enjoyed setting up syslog-ng with Kafka, you’re in for another treat. rsyslog—another popular and well-established syslog daemon—offers powerful features that make it an excellent choice for real-time log forwarding. In this guide, we will show you how to stream logs directly from rsyslog into Apache Kafka dynamically, with topics determined on the fly based on the source IP address. Yes, it’s that flexible!
We’ll use a convenient JSON-based lookup table to match incoming IPs with their corresponding Kafka topics. This approach allows you to route logs from multiple clients to different topics without hardcoding each one. It’s clean, efficient, and scalable.
This walkthrough builds on the same Kafka KRaft-mode setup we covered in our syslog-ng + Kafka post. If you’re just joining us and haven’t set up Kafka yet, we recommend reading that section first.
As before, we won’t cover container setups or system-wide installations. There are no Kafka installation steps here. We’re keeping it simple and focused—so let’s dive into the rsyslog side of things!
rsyslog is the default syslog daemon on many Linux distributions, and it’s known for its speed and flexibility. One of its strengths is the ability to dynamically route logs using lookups, which we’ll take advantage of in this setup.
What we’re aiming for in this guide:
Whether you are running a large-scale log pipeline or just testing locally, this setup will provide you with flexibility and insight, allowing you to avoid manually hardcoding every source-to-topic rule.
Let’s dive into setting up rsyslog for dynamic log forwarding to Kafka.
Below is a functional configuration that listens for incoming syslog messages (both UDP and TCP on port 514). It utilizes a JSON lookup file to match IP addresses to Kafka topics, forwarding the raw messages to Kafka while also storing them in structured log files on disk.
# Load necessary modules
module(load="imudp")
module(load="imtcp")
module(load="omkafka")
# Listen on UDP and TCP port 514
input(type="imudp" port="514")
input(type="imtcp" port="514")
# Load the lookup table that maps IPs to topic names
lookup_table(name="allhostnames" file="/home/oyku_home/rsyslog/lookup.json" reloadOnHUP="on")
set $.lookupresult = lookup("allhostnames", $fromhost-ip);
# Define templates for file paths and Kafka topics
$template logfile_path,"/opt/syslog/%$.lookupresult%/%FROMHOST-IP%/%$YEAR%-%$MONTH%-%$DAY%-%$HOUR%.log"
$template kafka_topic,"%$.lookupresult%"
$template rawmsg,"%msg%\n"
# Save logs to local disk using dynamic folders
action(
type="omfile"
dynaFile="logfile_path"
template="rawmsg"
createDirs="on"
)
# Only send to Kafka if the IP matched something in the lookup
if ($.lookupresult != "catchall") then {
action(
type="omkafka"
dynatopic="on"
broker=["localhost:9092"]
topic="kafka_topic"
template="rawmsg"
)
}
🧠 What’s Happening Here?
dynatopic="on".This configuration is clean, efficient, and powerful. It creates a flexible log pipeline where logs are stored and routed intelligently based on their origin. This approach is ideal for scaling across multiple sources without the need to manually define every topic or folder.
To enable all the dynamic features, you’ll need a JSON lookup file that maps incoming IP addresses to Kafka topic names.
Here’s what a simple lookup.json might look like:
{
"version": 1,
"nomatch": "catchall",
"type" : "string",
"table" : [
{"index": "127.0.0.1", "value":"rsyslog-topic"}
]
}
🔍 How It Works:
✨ Pro Tip: You can add as many IP-topic pairs as needed. There’s no need to restart rsyslog when you update the file—simply send a HUP signal (using kill -HUP <pid>), and your changes will be reloaded instantly.
💡 Why This Setup Is Beneficial:
rsyslog is extremely efficient—it’s the default logging system for a reason!This setup is perfect for hybrid environments where you want the flexibility of Kafka pipelines without sacrificing reliability.
Everything is set up—Kafka is running, rsyslog is configured, and your lookup.json file knows where each IP’s logs should go. Now, it’s time to send a test log and watch it arrive in the correct Kafka topic. 🎯
ncRun the following command on your machine to simulate a syslog message:
echo "<13>Sep 15 16:25:00 myhost myapp: This is a test from nc" | nc 127.0.0.1 514
This command sends a simple syslog-formatted message over UDP to port 514. Ensure that the IP 127.0.0.1 is correctly mapped to rsyslog-topic in your lookup file.
In a separate terminal, listen to the Kafka topic where you expect the logs to be routed:
bin/kafka-console-consumer --bootstrap-server localhost:9092 \
--topic rsyslog-topic --from-beginning
If everything is working correctly, you should see the message in the Kafka topic like this:
2025-09-16T16:25:00+00:00 myhost myapp: This is a test from nc
And there you have it—a live log flowing from rsyslog, matched by source IP, and delivered directly into Kafka!
💬 Tip: Try sending test logs from different IPs (or simulate them using iptables and socat) to observe how they get dynamically routed to different topics.
And that’s a wrap! 🎉 You’ve just created a flexible and dynamic log shipping setup using rsyslog and Kafka. Instead of static topics and rigid configurations, you now have:
This architecture is ideal for scaling across environments where logs from different sources need to be grouped, processed, and analyzed separately. Whether you’re developing a centralized logging system, a SIEM pipeline, or a robust observability stack, this pattern provides considerable power with minimal overhead.
💬 If you encounter any issues or simply want to discuss logs, Kafka, or infrastructure in general, feel free to reach out to me on LinkedIn. I’d love to hear about your projects or offer assistance if you’re facing challenges.
Happy logging! 🚀📊
