Zeekurity Zen – Part III: How to Send Zeek Logs to Splunk
If you're looking for professional services on this topic or interested in other cybersecurity consulting services, please reach out to me via my Contact page to discuss further.
This is part of the Zeekurity Zen Zeries on building a Zeek (formerly Bro) network sensor.
Overview
In our Zeek journey thus far, we’ve:
Now we’ll send our Zeek logs to Splunk, a popular log analysis platform. This will enable us to quickly search through Zeek’s large dataset and build interesting queries and dashboards.
To do this, we’ll walkthrough these steps:
- Configure Zeek to output logs in JSON format for consumption by Splunk.
- Create an index in Splunk for Zeek data.
- Installing and configuring the Corelight For Splunk app to index and parse Zeek logs in Splunk.
- Create a splunk user to run the Splunk Universal Forwarder.
- Installing and configuring a Splunk Universal Forwarder to send Zeek logs to a Splunk instance.
Output Zeek logs to JSON
- Stop Zeek if it is currently running.
zeekctl stop - Edit /opt/zeek/share/zeek/site/local.zeek and add the following.
# Output to JSON @load policy/tuning/json-logs.zeek - Restart Zeek and view the logs in /opt/zeek/logs/current to confirm they are now in JSON format.
zeekctl deploy cd /opt/zeek/logs/current less conn.log
Create an index in Splunk for Zeek data
It’s best practice to create separate indexes for different types of Splunk data.
- Login to your Splunk instance.
- In the top right menu navigate to Settings -> Data -> Indexes.
- In the Indexes page, click on New Index.
- Type “zeek” for Index Name and click Save to create your new index.
[Optional] Install and configure the Corelight For Splunk app
The Corelight For Splunk app is developed by the Corelight team for use with Corelight (enterprise Zeek) and open-source Zeek sensors. We’ll use this app to help parse, index, and visualize Zeek logs. Note that it is completely optional to use this app. You are free to skip this section entirely.
Note that Splunk has also published their own Splunk Add-on for Zeek aka Bro app which helps to ingest Zeek logs but does not feature any sort of dashboards or reports.
- Download and install the Corelight for Splunk app onto your Splunk server. This can either be done within the Splunk server itself or by manually downloading and installing as you would all other Splunk apps.
- You can navigate to the app to verify that it is installed correctly. However, since we have not yet configured our sensor to send data, the dashboards will be blank.
- In the top right menu navigate to Settings -> Knowledge -> Event types.
- In the App dropdown menu, select Corelight For Splunk and click on corelight_idx.
- In the Search string field type index=zeek. This tells the Corelight for Splunk app to search for data in the “zeek” index we created earlier.
Create the splunk user to run the Splunk Universal Forwarder
- Back in the Zeek sensor, create a splunk user and add it to the splunk and zeek groups.
sudo groupadd splunk sudo useradd splunk -g splunk -G zeek - As root/sudo, set a password for the splunk user.
sudo passwd splunk
Install and configure a Splunk Universal Forwarder
- Login to your Splunk account and download the latest Splunk Universal Forwarder. Once logged in, click “Download Now” for the Linux 64-bit .rpm installer. Note that Splunk also generates a convenient wget command you can use from the sensor itself once you accept the license agreement. As of this writing, the latest release is version 8.2.0. If the download URL referenced in the wget command below no longer works, download directly as noted above.
wget -O splunkforwarder-8.2.0-e053ef3c985f-linux-2.6-x86_64.rpm 'https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=8.2.0&product=universalforwarder&filename=splunkforwarder-8.2.0-e053ef3c985f-linux-2.6-x86_64.rpm&wget=true' - Install the forwarder to /opt/splunkforwarder using the rpm command.
sudo rpm -i splunkforwarder-<…>-linux-2.6-x86_64.rpm - As root/sudo set ownership of /opt/splunkforwarder to the splunk user.
sudo chown -R splunk:splunk /opt/splunk - Switch to the splunk user.
su splunk - Start the forwarder to accept the license agreement and create an administrative password.
cd /opt/splunkforwarder/bin ./splunk start --accept-license - Stop the forwarder.
./splunk stop - Remove the default data processing limit. Edit /opt/splunkforwarder/etc/system/local/limits.conf and add the following lines.Note that given the volume of data that Zeek generates, the forwarder may never process all log data if the default limit is not removed.
[thruput] maxKBps = 0 # means unlimited - Edit /opt/splunkforwarder/etc/system/local/inputs.conf to monitor desired Zeek logs. An example inputs.conf is below but may or may not include the logs you wish to ingest. Note that the index and sourcetype fields are leveraging the “zeek” naming scheme to match the “zeek” index we created in Splunk. If you intend to use the Corelight For Splunk app, you’ll want to replace the “zeek” sourcetype prefix with “corelight” as this is what the app is expecting (e.g. replace “zeek_conn” with “corelight_conn”). Modify the index and sourcetype configurations to your needs.
[default] host = sensor [monitor:///opt/zeek/logs/current/conn.log] _TCP_ROUTING = * index = zeek sourcetype = zeek_conn [monitor:///opt/zeek/logs/current/dns.log] _TCP_ROUTING = * index = zeek sourcetype = zeek_dns [monitor:///opt/zeek/logs/current/software.log] _TCP_ROUTING = * index = zeek sourcetype = zeek_software [monitor:///opt/zeek/logs/current/smtp.log] _TCP_ROUTING = * index = zeek sourcetype = zeek_smtp [monitor:///opt/zeek/logs/current/ssl.log] _TCP_ROUTING = * index = zeek sourcetype = zeek_ssl [monitor:///opt/zeek/logs/current/ssh.log] _TCP_ROUTING = * index = zeek sourcetype = zeek_ssh [monitor:///opt/zeek/logs/current/x509.log] _TCP_ROUTING = * index = zeek sourcetype = zeek_x509 [monitor:///opt/zeek/logs/current/ftp.log] _TCP_ROUTING = * index = zeek sourcetype = zeek_ftp [monitor:///opt/zeek/logs/current/http.log] _TCP_ROUTING = * index = zeek sourcetype = zeek_http [monitor:///opt/zeek/logs/current/rdp.log] _TCP_ROUTING = * index = zeek sourcetype = zeek_rdp [monitor:///opt/zeek/logs/current/smb_files.log] _TCP_ROUTING = * index = zeek sourcetype = zeek_smb_files [monitor:///opt/zeek/logs/current/smb_mapping.log] _TCP_ROUTING = * index = zeek sourcetype = zeek_smb_mapping [monitor:///opt/zeek/logs/current/snmp.log] _TCP_ROUTING = * index = zeek sourcetype = zeek_snmp [monitor:///opt/zeek/logs/current/sip.log] _TCP_ROUTING = * index = zeek sourcetype = zeek_sip [monitor:///opt/zeek/logs/current/files.log] _TCP_ROUTING = * index = zeek sourcetype = zeek_files - Edit /opt/splunkforwarder/etc/system/local/outputs.conf to send data to your Splunk server. In the sample file below, replace each instance of splunkserver:9997 with your own server name/IP and port number.
[tcpout] defaultGroup = default-autolb-group [tcpout:default-autolb-group] server = splunkserver:9997 [tcpout-server://splunkserver:9997] - Start the forwarder as the splunk user and confirm it successfully ran. You can check /opt/splunkforwarder/var/log/splunk/splunkd.log for any issues.
cd /opt/splunkforwarder/bin ./splunk start
Up Next
In Part IV of this series, we’ll introduce threat hunting with Zeek data through actionable Splunk queries.
If you like my content and want to support me, I'd greatly appreciate you buying me a coffee. Thanks! 🙏
Hey Eric, thanks for this awesome guide! How would you set up inputs.conf if I intend to shutdown my forwarder and server every day? Monitor just the archived directory and not current? I keep zeek running and collecting , but shut down the PC running the server as to not waste power and shutdown the UF so its not bogging down my network trying to reach and unreachable host. Anyway, zeek archives and compresses the logs in 15min increments, and so the UF never sees the logs in the /current directory when I bring everything back up after work. Also, it seems like Splunk is replicating a lot of the .tar.gz archived files into the main index with weird sourcetypes such as conn-3 and dns-7 . I am not using Corelight, just sending my logs to the zeek index. Thanks again!
Thanks for the kind words, EddG. I’m always impressed that people not only find their way to my little corner of the internet but also find it useful! 🙂
That’s an interesting use case. From your description it sounds like you’re running this in a VM? I’d honestly just run everything on a dedicated low-power server (this is what I do) to avoid this issue altogether. You’d be surprised just how little power it ultimately uses. This also lets you capture activity 24/7.
Otherwise, an alternative is to reconfigure how often Zeek rotates its logs and opt for something much longer to account for your daily monitoring gap.
Didn’t think to set the log rotation for that long, good idea. I am actually just running Splunk on my daily rig and that is why its shutdown most of the day. And Splunk Enterprise is not supported on RPI. Thanks again!
Hello,
When running zeek and forwarding the logs to splunk, the hard drive space on my zeek virtual machine i created are getting filled up by all the logs.
For the logs in /opt/zeek/logs/current ,
I guess all the different logs (conn, dns, dhcp etc) get put into folders for that day/hour. I increased the hard drive space from 500 GB to 1TB, but i guess we just have a lot of logs and it gets filled up so zeek stops working since no more space on hard drive.
So my question is do people usually configure it so somehow older longs after 1 month get deleted automatically ? Or do they just have a lot more space and manually delete the logs once a month or something?
Thanks
It’s up to you how you want to do it — manual or automatic. Though, it sounds like the manual method isn’t working so great for you. 😛
It turns out that Zeek can be configured to remove these automatically for you. If you edit /opt/zeek/etc/zeekctl.cfg and look in the “Logging Options” area you should find the section below. Edit it to your needs. Note that you will need to have zeekctl cron configured and working (it is covered in Part 1).
# Expiration interval for archived log files in LogDir. Files older than this
# will be deleted by “zeekctl cron”. The interval is an integer followed by
# one of these time units: day, hr, min. A value of 0 means that logs
# never expire.
LogExpireInterval = 0
Ahh I see. Thanks for all your help again Eric! I will look into that and see if I can modify the config so what we need.
Thanks
Sure thing, Kevin. Let me know if y’all are interested in contract work. 🙂
Hi Eric,
reading all the documentation and installing the new add-on https://splunkbase.splunk.com/app/5466/#/overview still does not work the log parse with TSV format.
Have you had any similar problem?
Hi Tatiana — I haven’t used that add-on before, so I can’t say what the issue might be. If you’re set on using TSV, you could try reaching out to Corelight directly for support. Otherwise, I’d suggest trying out the JSON format.
Hello Eric,
If we are using Corelight for Splunk App, then for the following steps:
Create an index in Splunk for Zeek Step 4: index name zeek
and
[Optional] Install and configure the Corelight For Splunk app Step 5: index=zeek
Would we put index name corelight and index=corelight instead?
And the input conf file would look something like this:?
[monitor:///opt/zeek/logs/current/conn.log]
_TCP_ROUTING = *
index = corelight
sourcetype = corelight_conn
Thanks again for answering my questions pretty quickly.
Would you also be finished with Part 8 of sending zeek logs to elasticsearch and creating queries in kibana anytime soon?
Yep, you’d use “corelight” instead of “zeek” as that is what the app is expecting. Even then, you’ll likely find the app will have blank charts / queries as zeek sensors are not completely interchangeable with corelight sensors.
I won’t be finished with that one anytime soon, holidays and all. 😛 Do you have both Splunk and Elastic? Curious why you’d be interested in both.
I was wondering which one I should use actually and was wondering if Kibana would be more preferred than Splunk. So I just wanted to compare both of them and see what the differences were.
Ah, got it. As with everything, depends on your requirements and budget. If you’ve already got a Splunk license, I’d stick with that. Otherwise, Elastic is a great alternative.
Splunk looks like we have to create queries to find things, which i’m not great at yet. I’ll have to do more research into splunk queries. I know you provided some already as well.
For Elastic/Kibana, are there advantages to using Kibana? Kibana looks like there’s dashboards and stuff to visual things easier. Is this something Splunk can do as well?
In either Splunk or Kibana you’ll need to create queries. Absolutely, Splunk can do dashboards, visuals, and fully featured “apps.” They were one of the early pioneers of ingesting and visualizing system logs. There’s likely more content on the internet around Splunk queries as well as pre-built apps and dashboards (like the Corelight app) than Kibana with regards to Zeek.
Using Filebeat with Kibana will get you a very basic Zeek dashboard and given that Kibana generally isn’t as fully featured as Splunk (nor is it nearly as pricey), you may find it easier to use.
The Sigma project aims to develop and share queries formatted for popular SIEM tools like Splunk and Kibana. You can start there for ideas on queries. https://github.com/Neo23x0/sigma/
Give both a try and see what you like. Note that Splunk offers a free license for up to 500 MB daily (which in just about any enterprise, is nowhere near what you’ll need) and then requires a paid for license for anything beyond that. Elastic + Kibana started out as a free and open source project that to this day still offers a free basic license that lets you use a good portion of its features. Paid licenses are available that unlock additional features such as machine learning and also provide customer support.
Hello Eric,
I’m having trouble starting splunk. I believed I followed instructions exactly but different version of splunk. I’m not a linux expert and trying to learn so hard for me to troubleshoot issues when following commands exactly as you have here. I previous commented earlier about issue with af_packet install but just gave up on that. Hopefully I can still run zeek without that and it’s not a big impact? or is it?
So when i go to cd /opt/splunkforwarder/bin and run ./splunk start –accept-license, i get
warning: cannot create “/opt/splunk/forwarder/var/log/splunk
warning: cannot create ….. /log/introspection
warning: cannot create …. /log/watchdong
This appears to be your first time running this version of splunk.
Could not open log file “/opt/splunkforwarder/var/log/splunk/first_install.log” for writing (13).
Any ideas? thanks
I’m not a Linux expert by any means either. But you’ll be impressed by how far Google can take you. 🙂
To be clear — what I’m covering here is installing a Splunk Universal Forwarder to send Zeek logs to an *existing* Splunk server. So unless you already have a Splunk server, you do not need to complete this part of the series as it will be completely useless to you. 😛
That being said, it looks like you’ve got a permissions issue. I’ve updated this part to make creating the splunk user and setting permissions more clearly for /opt/splunkforwarder. You’ll also need to first run “sudo chmod 750 /opt/zeek” as I didn’t have this set properly in the first part of the series. Doing this should allow splunk to properly read Zeek’s logs.
Yes, you can certainly run Zeek without AF_PACKET, but depending on the size of the network that you’re monitoring, you may see degraded performance without it.
Hi Eric,
First of all, I want to say thank you of your good work.
When I setup Corelight for Splunk App with index as zeek, I still see error message “Eventtype ‘bro*’ does not exist or is disabled.” I manually entered Eventtype for bro but didn’t fix the problem. Do you have any idea how to fix this? Fyi, mine is Splunk Enterprise 8.0.1.
Thanks for the kind words, Dai!
To your question, it might be easiest to set your inputs.conf to use something like “corelight_conn” instead of “zeek_conn”, as that is what the application is expecting. I mention this in Step 6 of “INSTALL AND CONFIGURE A SPLUNK UNIVERSAL FORWARDER.”
If you still run into issues, you can try reaching out to Corelight directly. Personally, I don’t use the app. 😛
Hope that helps!
Eric
Hello Dai (and any other folks with this same issue),
Just want to note here that I, too, ran into this issue, running zeek 4.0.7 on CentOS 7 for the ‘sensor’ node and with Splunk 9.0.0.1 (free version) on CentOS 7 for my indexer node.
I was able to solve those errors by editing /opt/splunk/etc/apps/CorelightForSplunk/local /macros.conf on the Splunk indexer and modifying both ‘eventtypes’ and ‘sourcetypes’ from using the ‘bro_’ prefix to ‘corelight_’ prefix. Then I restarted splunk. On the ‘sensor’ node, I edited the /opt/splunkforwarder/etc/system/local/inputs.conf and made sure that the sourcetypes listed there also match the ‘corelight_’ prefix. Making these changes such that through both the splunk forwarder and the splunk application have matching eventtypes and sourcetypes, and then restarting everything solved the issue.
2 years later — hope this can help. you or anyone else who runs into this issue.
Thanks Eric for a great series of posts!
Thanks for sharing and for the kind words! Happy Zeeking!
Hi Eric, thanks for putting this together, saved me a lot of time.
There are a few things you might consider reviewing in the post:
1- in the inputs.conf file, all entries have 3 slashes and it is supposed to be 2 slashes -> monitor:/// does not work
2- in splunk, TCP/9997 need to be configured to receive data. some people reading this might have just configured splunk and that is a necessary step that is not enabled by default at least in version 8.0.2
3- the source types you defined in the splunk forwarder input.conf file do not match the corelight app
Hi Freddy,
Thanks for the kind words and feedback! To your points:
1. That’s weird. I’ve got it configured with three slashes and it is working for me. It’s also recommended from Splunk’s documentation directly (see Step 5): https://docs.splunk.com/Documentation/Splunk/8.0.2/Data/Monitorfilesanddirectorieswithinputs.conf. But hey, if 2 slashes works for you, then do what works. 🙂
2. Ah, I haven’t experienced that myself as Splunk’s always had TCP/9997 as the default. Splunk’s documentation also states this is the default port: https://docs.splunk.com/Documentation/Splunk/8.0.2/InheritedDeployment/Ports. My guess is that you needed to open the port on your host/network firewall?
3. You are absolutely correct that they do not match. If you take a look at Step 6 of section “INSTALL AND CONFIGURE A SPLUNK UNIVERSAL FORWARDER,” I note the following — If you intend to use the Corelight For Splunk app, you’ll want to replace the “zeek” sourcetype prefix with “corelight” as this is what the app is expecting (e.g. replace “zeek_conn” with “corelight_conn”). Modify the index and sourcetype configurations to your needs.
Hi Eric,
Yeah, I was puzzled with 1 as well. It only worked for me using “//”, I am running raspbian buster 10.
For 2, yes, that is the default port, but it was not configured under Data -> Forwarding and receiving -> Receive data. I had to do that manually.
For 3. I ended up figuring that out and when I came back to see your response that was exactly what I did. The connections dashboard is not working, some mismatch between the searches and the parsed data. I will manually fix the searches. Weird enough, all other dashboards under data exploration work fine.
1.Hm, I think monitor:// is the required part and the syntax after is the path. Perhaps you’re using a relative path? Either way, if it works, then leave it be. 🙂
2. Got it. I suppose I’m assuming that is already configured. But thanks for the tip!
3. Yeah, I found the Corelight app a bit wonky which is why I mention installing it is optional. My “Part IV: Threat Hunting” post just assumes you didn’t install it at all.
wonky is the right word. 🙂
Started from scratch due to new installs and ran into the same issue. Even though the splunk user is part of the zeek group, it does not have access to any /opt/zeek/* subdirectory or any files. When doing ‘chmod 750 /opt/zeek’ instead of ‘chmod 740 /opt/zeek’ the user does have access and it works for me on Splunk 8.0.
Great tutorial, only one thing missing which I hope you can update. It’ll help the person above as well: your inputs.conf is set to monitor the /opt/zeek/logs/current folder, but the splunk user has no access to that folder so it can’t read the actual log files and nothing will ultimately be indexed.
Once you fix the folder permissions on the Zeek host to allow read access for the Splunk user to that directory and restart the Splunk Forwarder you’re good to go.
Thanks, Patrick! There’s already a step where you add the splunk user to the zeek group, specifically — sudo useradd splunk -g zeek. This *should* give the splunk user read permissions to the /opt/zeek/logs/current folder assuming that you already did the steps in the first part of the series (http://www.ericooi.com/zeekurity-zen-part-i-how-to-install-zeek-bro-on-centos-8/) —
sudo chown -R zeek:zeek /opt/zeek
sudo chmod 740 /opt/zeek
If that isn’t working, definitely let me know!
Thanks, must’ve overlooked the chmod 740 part then as I definitely recall doing the other ones. Now eagerly awaiting part 4 :-).
No worries. Definitely hold me accountable if I miss something!
Appreciate the words of encouragement, I promise I’m writing part 4 and more. 😛
Eric,
I’ve been following your posts, and they are amazing, thanks for sharing your thoughts.
I have a small issue seeing events in the zeek app, I do see splunk events using the index=zeek. I’m testing this in my home lab, so not I’m not sure if is related to the Splunk free version.
Any feedback is appreciated.
Hi Rain1,
I appreciate your kind words and am happy to hear the positive feedback on my posts. It means a lot. 🙂
It shouldn’t matter that you’re using the free Splunk, I’m doing the same as well. A couple things to check — the Corelight app expects to see logs in the “zeek” index, so be sure you completed Step 5 under “INSTALL AND CONFIGURE THE CORELIGHT FOR SPLUNK APP.” The app also expects all sourcetypes to be prefixed with “corelight_” so be sure those are set properly as well.
Note that the Corelight app isn’t required for you to get full use from your zeek logs in Splunk. I view the app as more of an introduction to what’s possible, especially if you’re a brand new zeek / splunk user. I am working on Part IV of this series (really, I promise!) which will cover interesting queries you can use in your own environment.
Hope that helps!
Took a while, but the fourth part in my series is up. Hopefully you figure out your Splunk issues and are ready to start diving into your data!
http://www.ericooi.com/zeekurity-zen-part-iv-threat-hunting-with-zeek/