OpenBSD does not provide many good options for shipping logs to a remote destination. Well known solutions like Fluentd, fluent-bit, Cribl, etc are just not (yet) available :(

In this blog post I describe how I’m shipping Zeeks logs from my firewall using Rsyslog into my logging infrastructure which currently consists of Cribl and Splunk running on Linux VM’s.

Enable JSON logging in Zeek

The default TSV logging format of Zeek is fine when working with the logs locally with tools like cat, grep and zeek-cut. But when forwarding logs to a SIEM I prefer to use the JSON format. To make Zeek create logs in JSON you have to load the json-logs module in the site local configuration.

cerberus$ tail -1 /etc/zeek/site/local.zeek
@load tuning/json-logs

Install rsyslog

Since Zeek can only write to local logs, another tool is needed to forward these logs to a remote destination. The default syslogd on OpenBSD does not provide this capability but fortunately the awesome rsyslogd is available as a package, so let’s install it!

cerberus# pkg_add rsyslog

Easy, huh?!

Configure rsyslog

Configuring rsyslog can seem daunting due to its extensive configuration options. For this reason I tend to start with my Rsyslog configs repository which provides a good starting point. This also provides an opportunity to revisit, retest and improve the configurations in the repository.

What I need from rsyslog for this use-case is to read a bunch of local files and forward them using syslog/tcp. For this simple setup I’ve thrown all statements in a single configuration file stored at /etc/rsyslog.conf, so no includes.

Some notable things to consider:

  • You need to (recursively) create /var/spool/rsyslog/queue and make it owned by _rsyslogd:_rsyslogd as we’re dropping privileges
  • maxMessageSize is set to 8k which is more than sufficient for the Zeek events
  • We’re loading the imfile module and configure it in polling mode as OpenBSD does not support inotify()
  • Since I’m running on a small AMD based SOC the message queue is limited to 10MB of RAM
  • I’m sending the logs to Cribl so I’ve defined a somewhat easy to parse template
  • Forwarding the logs are done with the omfwd module using a TCP based transport

The complete configuration:

#
# Global settings
# ---------------
#

global(
    # set a liberal umask here - to be overriden in output actions
    umask="0000"

    # working directory - temp/state files are stored here
    workDirectory="/var/spool/rsyslog"
   
    # truncate messages after 8kb
    maxMessageSize="8k"

    # do not interfere with log flow
    dropMsgsWithMaliciousDNSPtrRecords="off"
    preserveFQDN="on"

    privdrop.user.name="_rsyslogd"
    privdrop.group.name="_rsyslogd"

    # set the hostname you want to use for this system (optional)
    #localHostname=""

)


#
# Built-in modules
# ----------------
# Explicitly load builtin modules for completeness.
#

module(
    load="builtin:omfwd"
)

#
# External modules
# ----------------
# Load external modules early
#

module(
    load="imfile"
    mode="polling" 	  # OpenBSD does not support inotify()
    PollingInterval="30"  # Conservative polling (performance)
)


#
# Main message queue (input)
# --------------------------
# There is a single main message queue inside rsyslog. Each input module delivers messages to it.
# The main message queue worker filters messages based on rules specified in rsyslog.conf and 
# dispatches them to the individual action queues. Once a message is in an action queue, 
# it is deleted from the main message queue.
#

main_queue(

    # queue spool directory, must already exist
    queue.spoolDirectory="/var/spool/rsyslog/queue"
    queue.filename="input.q"

    # define queue type (in-memory only)
    queue.type="LinkedList"

    # queue sizing:
    # - size for 10000 messages in-memory queue
    # - assume avg. message size is 1024 bytes
    # - overhead due to pointers: 4 bytes on 32bit systems, 8 bytes on 64bit systems
    # - memory usage: 10000 * 1024 byyes + (10000 * 8 bytes) = 9,84 MB
    queue.size="10000"

    # queue management
    queue.workerthreads="1"                     # out-of-order delivery when > 1
    queue.workerthreadminimummessages="100"     # start a thread for every X messages in q
    queue.dequeuebatchsize="16"

    # save log on exit
    queue.saveonshutdown="on"
)


#
# Monitoring configuration
# ------------------------
#

template(
    name="ZeekTemplate"
    type="string"
    string="host=%hostname%, source=%!zeek_log_file%, app=%syslogtag%, message=%msg%\n"
)


ruleset(
    name="ZeekRuleset"
) {
    set $!zeek_log_file = $!metadata!filename;
    action(
        type="omfwd"
        template="ZeekTemplate"
        target="xxx.xxx.xxx.xxx"
        port="60516"
        protocol="tcp"
    )
}

input(
    type="imfile"
    File="/var/log/zeek/current/*.log"
    Tag="zeek"
    Facility="local0"
    Ruleset="ZeekRuleset"
    Severity="info"
    addMetadata="on"
)

# EOF