Splunk Connect for Syslog (SC4S): Installation, Usage & Implementations

This blog outlines the usage of SC4S to import syslog input data into Splunk, addressing various use cases including custom and complex configurations. From the installation of SC4S and establishing connections to ensuring accurate timestamping and field mappings on indexed data, each step is crucial for effective log management and analysis. By providing insights and tips, this article aims to empower users to optimize their syslog integration process, ensuring accurate and actionable insights from their log data within Splunk. If you want to learn more information about SC4S you can check Splunk Connect for Splunk official documentation page, Splunk Connect for Syslog: Extending the Platform .conf2020 presentation and Syslog Data Collection (SC4S) for Splunk and Custom Inputs blog written by Splunk, Mark Bonsack/Ryan Faircloth and Merih Bozbura Türk respectively. After reading this, you may also want to check the GitHub page of SC4S and syslog-ng administration guide to understand and to create different types of filters in the future for your custom or known vendor inputs.

SC4S, Splunk Connect for Syslog is an open-source and Splunk supported solution that utilizes syslog-ng (Open Source Edition) to transport data directly to Splunk via the Splunk HTTP Event Collector (HEC) without reading data from somewhere on the disk [1]. This also implies that there is no requirement for a Universal/Heavy Forwarder or extensive disk space on SC4S server(s) during the data collection process.

Below, you’ll find a list of the use cases covered in this blog. If you prefer, you can skip the installation section and dive directly into the use cases using the provided links. I hope you enjoy reading this article and find it helpful.

Before Starting

Before starting, be sure which ports will be used and that there are no connection issues between SC4S and Splunk Servers. This is basic but one of the most common issues. The SC4S documentation says that there are several pre-defined indexes that Splunk should have if data from known vendors is to be imported, and HTTP Event Collector (HEC) needs to be defined in Splunk. So, be sure both of them are defined correctly for your deployment, and check the SC4S official documentation page, getting started for planning your deployment and other basics.

Notes : If you don’t know how to create HEC inputs, please check the documentation. Below you can find inputs.conf example.

# inputs.conf
[http://sc4sinput]
disabled = 0
token = e8de5f0e-97b1-4485-b111-1191cbf89392
useACK = 0
index = sc4s

To utilize SC4S, you must choose an installation method that suits your needs. The installation can be done offline or online as outlined on the Runtime Configuration page. While installation can be done in any desired method, there will be used two different installation methods in this article as Docker Desktop + Compose (MacOS) and Docker&Podman offline installation.

SC4S Installations:

Docker Desktop will be utilized for both installations. You can refer to the Docker documentation page for proper installation guidance. In this article, we will adhere to the official documentation pages for the two runtime installations as mentioned, without creating any service file. These installations reflect the setup I implemented on my local laptop to test some configurations. If you are installing SC4S in your production environment, you may need to create a service file as outlined in the documentation. Also, don’t forget to check pre-configurations on this page.

  1. At the beginning of the SC4S installation, we need to create a volume, which we will call splunk-sc4s-volume with below command, and some directories (<path-of-sc4s>/sc4s and <path-of-sc4s>/sc4s/local) under other than /opt directory to locate and run the Docker Container in our local system for both method.

    Notes: If you intend to utilize your local /opt directory to create SC4S directories as outlined in the documentation, ensure that the Docker user executing the operation has the necessary read, write, and delete permissions.

    • sudo docker volume create splunk-sc4s-volume

  2. After this creation we need to define directory mappings between the Docker Container and the local to control SC4S configurations from the local with the docker-compose.yml file. The difference between the online and offline methods can be seen in this step. The online method will get the sc4s image from the Splunk repository directly but we need to deploy the image to Docker in the offline method by ourselves.

    Tip: Also, don’t forget target port is your local & published is the Docker Container’s port. If you want, you can change them individually.

    Notes: Only 514 (default) port for UDP/TCP protocols are defined in these YAML files for start. We will add more ports when we talk about the use cases section.

    • Online Docker Compose YAML configuration: Docker Desktop + Compose (MacOS)

      In the online method, the following docker-compose.yml file which can be located under the sc4s directory, can be used for the latest image of SC4S version 3. If you wish to modify the version, you can simply adjust the container3:latest section in the provided YAML file.

         version: "Latest"
     services:
         sc4s:
             deploy:
             replicas: 1
             restart_policy:
                 condition: on-failure
             image: ghcr.io/splunk/splunk-connect-for-syslog/container3:latest
             ports:

             # Default Ports (TCP/UDP):
                
             - target: 514
               published: 514
               protocol: tcp
             - target: 514
               published: 514
               protocol: udp
            
            
             env_file:
             - <path-of-sc4s>/sc4s-latest/env_file
             volumes:
             - <path-of-sc4s>/sc4s-latest/local:/etc/syslog-ng/conf.d/local:z
             - splunk-sc4s-volume:/var/lib/syslog-ng

     volumes:
         splunk-sc4s-volume:

    • Offline Docker Compose YAML configuration: Docker/Podman

      In the offline method, we need to download depending on which version of the sc4s image (oci_container.tar.gz) we want to use from the relevant github repository releases. After this, we need to load the image into Docker with the following command.

      • docker load < oci_container.tar.gz

      After this step, we should list all imported images to determine the ID of the SC4S image and tag it as sc4simage:latest for use in the YAML configuration file provided below.

      • docker image ls

      Once the Image ID for the imported SC4S image is identified, you can utilize the following command. Ensure to replace the placeholder (<image-id>) with the actual Image ID of yours.

      • docker tag <image-id> sc4simage:latest

      Now you can use the following docker-compose.yml file which can be located under the SC4S directory, as the online method.

      Notes: It is not the exact docker-compose.yml file in online method. So, be sure you are using the correct version of the method.

         version: "Latest"
     services:
         sc4s:
             deploy:
             replicas: 1
             restart_policy:
                 condition: on-failure
             image: sc4simage:latest
             ports:

             # Default Ports (TCP/UDP):
                
             - target: 514
               published: 514
               protocol: tcp
             - target: 514
               published: 514
               protocol: udp
            

             env_file:
             - <path-of-sc4s>/sc4s-latest/env_file
             volumes:
             - <path-of-sc4s>/sc4s-latest/local:/etc/syslog-ng/conf.d/local:z
             - splunk-sc4s-volume:/var/lib/syslog-ng

     volumes:
         splunk-sc4s-volume:

  3. We need to create the env_file to define relevant ports and communication information with Splunk for SC4S as below.

    Tips : The SC4S DEFAULT_TIMEZONE option allows you to change the default time zone of your SC4S to which you want. Also, if you are using the default port, there’s no need to define it in the env_file unless you have specific plans for filtering usage.

    Notes: Don’t forget to change <hec-token-that-is-created-in-the-Splunk> as your HEC token.

     SC4S_DEST_SPLUNK_HEC_DEFAULT_URL = https://host.docker.internal:8088
 SC4S_DEST_SPLUNK_HEC_DEFAULT_TOKEN = <hec-token-that-is-created-in-the-Splunk>
 SC4S_DEST_SPLUNK_HEC_DEFAULT_TLS_VERIFY = no

 SC4S_DEFAULT_TIMEZONE=Europe/Istanbul

  4. After completing the previous steps, executing one of the commands below will ensure that Docker properly composes with the SC4S image. You can then check the SC4S logs and Splunk Search Head as shown in Figure 1 and Figure 2 respectively to verify if the connection was successfully established.

    Notes: If you want to use the first command, you need to ensure you are located under the exact directory that includes the docker-compose.yml file.

    • docker-compose up docker-compose.yml

      or

    • docker-compose up -f <path-of-sc4s>/sc4s-latest/docker-compose.yml

    screenshot
    Figure 1: Log outputs of SC4S upon successfully connection.
    screenshot
    Figure 2: Log outputs of SC4S can be viewed in Splunk Search Head upon successful connection.

SC4S Use-Cases:

In this section, we will examine various scenarios I encountered when trying to retrieve syslog data via SC4S. These scenarios fall into two categories: known and custom vendors. So, let’s get started.

1. Known Vendor Use-Cases:

A list of known vendors that have already successfully parsed in the SC4S as default, can be found on the documentation page. These are generally well-known and well-used vendors. Also, most probably, you will use at least one of them in your deployment.

2. Custom Vendor Use-Cases:

Some vendors/apps cannot be parsed by SC4S due to a lack of parsing configurations. Even though, the least known apps for “known vendors” may not be available. Information about all pre-defined parsers can be found on official SC4S documentation which was mentioned in the Known Vendor User-Cases section.

In this section, we will create both simple & complicated configuration files with different examples of custom vendors.

Conclusion

To sum up, SC4S (Splunk Connect for Syslog) is a handy tool for simplifying the process of ingesting syslog-ng data into Splunk. Throughout this blog, we’ve covered the basics of SC4S, from installation to practical applications.

For known vendors, SC4S does the heavy lifting by automatically parsing syslog data. We’ve seen examples with Cisco IOS, where SC4S handles data and ensures compatibility with Splunk’s Common Information Model (CIM) with pre-created apps/add-ons.

Even with custom vendors, SC4S shines. It provides flexibility to adapt to unique logging requirements. By creating custom configurations and metadata files, users can tailor SC4S to their needs. We’ve explored examples showcasing SC4S’s ability to handle syslog data from custom vendors accurately.

In essence, SC4S is a versatile tool that makes syslog data ingestion into Splunk a breeze. Whether dealing with known or custom vendors, SC4S streamlines the process, enabling organizations to make the most of their log data for security and operational monitoring.

Also, thanks for reading such a long article. :)

References:

Seynur Bilgi Teknolojileri A.Ş.
Esentepe Mah. Büyükdere Cad. No:199/6 Levent Şişli İstanbul
+90 212 403 9563 | info@seynur.com
©2022 Seynur | Privacy Policy | Terms & Conditions