Reputation Lists and Datasets with Suricata

Another use case we were wondering about is domain IOCs and datasets using the reputation option.

So I exported our domain IOCs from our intel platform into a csv. To transform the IOC domains to sha256, I used this python script.

The values in the original csv are domain,numerical_reputation format, so the script just grabs the domain, converts it and then smashes it back together with its associated reputation and then writes it all out to a new file for use in our Suricata rule.

Victor made some enhancements to the way the suricata.yaml dataset configuration gets parsed and loaded so the suricata.yaml snippet looks a little different than in my original post.

Also after some discussion with Jason Ish on the OISF team, I made a change to the location for our state files and now will be putting all our lists into a common but separate directory from our rules.
This is more or less to make sure nothing gets overwritten by rule set changes if there are naming colllisions.

suricata.yaml snippet:

datasets:
  dns-seen:
    type: sha256
    load: /nsm/lists/topdomains.lst
    hash:
      memcap: 1024mb
      hash-size: 1024mb
      prealloc: 1024mb

  dns-ioc:
    type: sha256
    load: /nsm/lists/domainioc.lst
    hash:
      memcap: 256mb
      hash-size: 256mb
      prealloc: 256mb

The chunk I selected for the domainioc.lst is just shy of 700,000 records. An example of the format for domainioc.lst:

d4c9d9027326271a89ce51fcaf328ed673f17be33469ff979e8ab8dd501e664f,8.0

Our test rule is:
alert dns any any -> any any (msg:"DNS IOC List Test"; dns.query; to_sha256; datarep:dns-ioc, >, 7.0, sid:1234; rev:1;)

alert.json entry sample:

Looking at the suricata load time:
13929] 11/10/2019 -- 19:12:21 - (suricata.c:1078) <Notice> (LogVersion) -- This is Suricata version 5.0.0-dev (a1ee536 2019-10-10) running in SYSTEM mode
[13929] 11/10/2019 -- 19:12:21 - (util-cpu.c:171) <Info> (UtilCpuPrintSummary) -- CPUs/cores online: 40

14078] 11/10/2019 -- 19:21:03 - (source-af-packet.c:1802) <Perf> (AFPComputeRingParamsV3) -- AF_PACKET V3 RX Ring params: block_size=32768 block_nr=3126 frame_size=1600 frame_nr=62520 (mem: 102432768)
[14078] 11/10/2019 -- 19:21:04 - (source-af-packet.c:515) <Info> (AFPPeersListReachedInc) -- All AFP capture threads are running.

It looks like we are at about 8.5 minutes or so. This seems reasonable given how much data we are loading but am curious what others are seeing with their setups. I will run this by the OISF folks as well to get their thoughts.

More Dataset Performance Notes

-- Update: Forgot to mention we have a number of PTResearch rules too --

in the last post we covered the performance of a single rule using a large (10 million record) dataset.

Since it's been running for a few days now I wanted to see what performance was like with a dataset based rule using the same large dataset but also using the our mix of VRT/ET/PTResearch/custom rules.

Here's where things stand.

Our current uptime is:
tail -n 1 stats.json | jq 'select(.event_type=="stats").stats.uptime'
595932 (~6.9 days)


Our current packet status:
tail -n 1 stats.json | jq -c 'select(.event_type=="stats").stats.capture'
{"kernel_packets":102603516231,"kernel_packets_delta":13328638,"kernel_drops":21501885,"kernel_drops_delta":0,"errors":0,"errors_delta":0}

Let's make that a percentage kernel drop:
tail -n 1 stats.json | jq '.stats.capture.kernel_drops / .stats.capture.kernel_packets'
0.0002095900739094382 (0.00021%)

When we loaded the ET Pro set we use suricatasc so we need to see last_reload:
tail -n 1 stats.json-20191011 | jq -c 'select(.event_type=="stats")|.stats.detect.engines'
[{"id":0,"last_reload":"2019-10-07T18:57:51.607027+0000","rules_loaded":31887,"rules_failed":6}]

The failed rules are expected, it's on my list of things to fix :)

So since we loaded our rules, how many rules have we seen fire?

From our alert.json logs we have seen 104,638 alerts since we reloaded our rules. A sample of our dataset alert looks like:

{"timestamp":"2019-10-07T18:58:03.362167+0000","flow_id":1566205101704887,"in_iface":"p4p1","event_type":"alert","vlan":[245],"src_ip":"10.0.0.133","src_port":50201,"dest_ip":"8.8.8.8","dest_port":53,"proto":"UDP","tx_id":0,"alert":{"action":"allowed","gid":1,"signature_id":99070083,"rev":1,"signature":"DNS List Test","category":"","severity":3},"dns":{"query":[{"type":"query","id":30205,"rrname":"detectportal.firefox.com","rrtype":"A","tx_id":0}]},"app_proto":"dns","flow":{"pkts_toserver":1,"pkts_toclient":0,"bytes_toserver":84,"bytes_toclient":0,"start":"2019-10-07T18:58:03.362167+0000"},"stream":0,"packet_info":{"linktype":1},"host":"sensor01"}

So far it looks like performance is within our expectations. Next up is taking our ranked domain IOCs and put them into a reputation list rule.

Performance with a single large dataset and single rule

Our singe large dataset that we loaded in this post has been running for about 4 days. Taking a look at our stats from stats.json.

tail -n 1 stats.json | jq -c 'select(.event_type=="stats").stats.capture'
{"kernel_packets":56282979747,"kernel_packets_delta":12499348,"kernel_drops":9927231,"kernel_drops_delta":0,"errors":0,"errors_delta":0}

So if we do some quick math:

tail -n 1 stats.json | jq '.stats.capture.kernel_drops / .stats.capture.kernel_packets'

0.00017638069349960352

tail -n 1 stats.json | jq -c 'select(.event_type=="stats")|.stats.detect.engines'

[{"id":0,"last_reload":"2019-10-03T20:09:39.014088+0000","rules_loaded":1,"rules_failed":0}]

tail -n 1 stats.json | jq 'select(.event_type=="stats").stats.uptime'

339487

So our sensor has processed 56,282,979,747 packets in the last ~4 days and has dropped 9,927,231 packets which gives us a drop percentage of 0.00017638069349960352%.

It should be noted that the only rule loaded is our dataset based DNS rule, which means that it's time to load the ET Pro rule set and see what performance looks like. :)

How big can a suricata dataset be? take 2..

Continuing work with datasets, I was previously working on my laptop and got time to test this on real hardware.

The real hardware are just off the shelf Dell r700 series with 256G of RAM.

snippet of lscpu output:
Architecture:            x86_64
CPU op-mode(s):     32-bit, 64-bit
Byte Order:              Little Endian
CPU(s):                    40
On-line CPU(s) list: 0-39
Thread(s) per core:  2
Core(s) per socket:  10
Socket(s):                2
NUMA node(s):      4
Vendor ID:              GenuineIntel
CPU family:            6
Model:                     63
Model name:           Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz

So of course the first thing to test is just HOW FAST can we load 10 million records :)

The answer is:

real 7m57.045s

user 7m52.763s

sys 0m3.446s

[16012] 3/10/2019 -- 18:33:55 - (datasets.c:219) <Config> (DatasetLoadSha256) -- dataset: dns-seen loading from '/etc/nsm/eint/lists/topdomains.lst'

[16012] 3/10/2019 -- 18:41:45 - (datasets.c:275) <Config> (DatasetLoadSha256) -- dataset: dns-seen loaded 10000000 records

[16012] 3/10/2019 -- 18:41:45 - (defrag-hash.c:248) <Config> (DefragInitConfig) -- allocated 3670016 bytes of memory for the defrag hash... 65536 buckets of size 56

[16012] 3/10/2019 -- 18:41:45 - (defrag-hash.c:273) <Config> (DefragInitConfig) -- preallocated 65535 defrag trackers of size 160

[16012] 3/10/2019 -- 18:41:45 - (defrag-hash.c:280) <Config> (DefragInitConfig) -- defrag memory usage: 14155616 bytes, maximum: 1350565888

[16012] 3/10/2019 -- 18:41:45 - (stream-tcp.c:399) <Config> (StreamTcpInitConfig) -- stream "prealloc-sessions": 2048 (per thread)

Now on to seeing about reputation lists...

How big can a suricata dataset be?

After poking around with datasets yesterday I was curious how many domain sha256 hashes we can load with the default dataset configuration.

The list came from https://www.domcop.com/top-10-million-domains and I just grabbed chunks with 'head -n' against the list of domains in the csv export (just the 2nd field in the csv, the actual domains).


I was able to load 155,802 hashes/domains!

[11148] 30/9/2019 -- 21:48:12 - (datasets.c:275) <Config> (DatasetLoadSha256) -- dataset: dns-seen loaded 155802 records
[11148] 30/9/2019 -- 21:48:12 - (datasets.c:491) <Debug> (DatasetGet) -- set 0x55d95b6753e0/dns-seen type 3 save ./topdomains.155k.lst load ./topdomains.155k.lst
[11148] 30/9/2019 -- 21:48:12 - (datasets.c:577) <Debug> (DatasetsInit) -- dataset dns-seen: id 0 type sha256
[11148] 30/9/2019 -- 21:48:12 - (datasets.c:591) <Debug> (DatasetsInit) -- datasets done: 0x55d95b577850


At first I was thinking it might be tuneable but the suricata.log entries referencing memcap and hash-size just appear to be utility items from util-thash.c (I'll check with the OISF folks to make sure I am not missing something)

[10211] 30/9/2019 -- 21:12:56 - (conf.c:335) <Debug> (ConfGet) -- failed to lookup configuration parameter 'dns-seen.memcap'
[10211] 30/9/2019 -- 21:12:56 - (conf.c:335) <Debug> (ConfGet) -- failed to lookup configuration parameter 'dns-seen.hash-size'
[10211] 30/9/2019 -- 21:12:56 - (conf.c:335) <Debug> (ConfGet) -- failed to lookup configuration parameter 'dns-seen.hash-size'


I am not sure if there's a use case yet for that many items in a list but nice to know the option is there :)

--- Update October 1, 2019 ---
The hash-size and memcap can be configured per dataset (Thanks Victor!) Example snippet:

datasets:
 - dns-seen:
     type: sha256
     state: topdomains.lst

dns-seen:
  memcap: 1024mb
  hash-size: 1024mb

With the memcap and hash-size additions, I was able to load all 10,000,000 domain hashes. While this is hardly practical, it can be done :)

real    78m25.400s
user    78m2.285s
sys    0m4.419s

[4544] 1/10/2019 -- 21:09:31 - (host.c:276) <Config> (HostInitConfig) -- preallocated 1000 hosts of size 136
[4544] 1/10/2019 -- 21:09:31 - (host.c:278) <Config> (HostInitConfig) -- host memory usage: 398144 bytes, maximum: 33554432
[4544] 1/10/2019 -- 21:09:31 - (util-coredump-config.c:142) <Config> (CoredumpLoadConfig) -- Core dump size is unlimited.
[4544] 1/10/2019 -- 21:09:31 - (util-conf.c:98) <Notice> (ConfigGetDataDirectory) -- returning '.'
[4544] 1/10/2019 -- 21:09:31 - (datasets.c:219) <Config> (DatasetLoadSha256) -- dataset: dns-seen loading from './topdomains.lst'
[4544] 1/10/2019 -- 22:27:16 - (datasets.c:275) <Config> (DatasetLoadSha256) -- dataset: dns-seen loaded 10000000 records
[4544] 1/10/2019 -- 22:27:16 - (defrag-hash.c:245) <Config> (DefragInitConfig) -- allocated 3670016 bytes of memory for the defrag hash... 65536 buckets of size 56
[4544] 1/10/2019 -- 22:27:16 - (defrag-hash.c:272) <Config> (DefragInitConfig) -- preallocated 65535 defrag trackers of size 160

Datasets with Suricata

Suricata recently introduced datasets so I thought I would take a stab at using them and seeing what could maybe be done with them.

From the 5.0.0rc1 release announcement:

"Still experimental at this time, the initial work to support datasets is part of this release. It allows matching on large amounts of data. It is controlled from the rule language and will work with any ‘sticky buffer’. https://suricata.readthedocs.io/en/suricata-5.0.0-rc1/rules/datasets.html
"

A lot of my recent research/work has been around DNS so I figured I would start there.

Step 1:

Get suricata master or 5.0.0rc1 installed, the installation from source instructions are here:
https://suricata.readthedocs.io/en/latest/install.html

Source tarballs can be retrieved from:
https://github.com/oisf/suricata/

If you install from github be sure to clone libhtp into the suricata repo directory, libhtp can be found here:
https://github.com/OISF/libhtp


On a Fedora 30 system installing the build time requirements should be something like:'sudo dnf install gcc gcc-c++ rust cargo libyaml-devel python3-pyyaml libnfnetlink-devel libnetfilter_queue-devel libnet-devel zlib-devel pcre-devel libcap-ng-devel lz4-devel libpcap-devel nspr-devel nss-devel nss-softokn-devel file-devel  jansson-devel GeoIP-devel python3-devel lua-devel autoconf automake libtool'

My particular install steps for Suricata from source (Fedora 30) with all the build requirements installed:


mkdir git
cd git
git clone https://github.com/OISF/suricata.git
cd suricata
git clone https://github.com/OISF/libhtp.git
./autogen.sh; ./configure --enable-gccprotect --enable-pie --disable-coccinelle --enable-nfqueue --enable-af-packet --with-libnspr-includes=/usr/include/nspr4 --enable-jansson --enable-geoip --enable-lua --enable-rust --enable-debug --enable-profiling --enable-rust-debug --enable-lzma
make
sudo make install


Step 2:

I chose to put part of the configuration in the suricata.yaml file since setting up the datasets seems like something that would be easier to do in the yaml with a configuration management setup (chef, ansible, puppet, etc.).

So I tossed the following in the suricata.yaml just under the yaml header/boilerplate stuff:

datasets:
 - dns-seen:
     type: sha256
     state: dns-seen.lst

This is telling Suricata that I want to set up a dataset name 'dns-seen' and that it will contain sha256 values and running and saved information about the dataset will be stored in a file named dns-seen.lst.

So far so good.

Step 3:

Time to write a rule that will use the dataset. Since we are going to look for DNS queries, it only makes sense of course to use the DNS sticky buffer. Time to be creative... :)

alert dns any any -> any any (msg: "dns list test"; dns.query; to_sha256; dataset:isset,dns-seen; sid:123; rev:1;)

What this rule will do is write an alert (in our case to alert.json) for any traffic that the Suricata protocol parser(s) determines is a DNS query for any domain in our dns-seen.lst file.

One thing worth noting here is the to_sha256 keyword, this is what Suricata calls a transform. This keyword will tell suricata to take whatever is in the dns buffer and calculate a sha256 hash. To say the least transforms are quite useful in rule writing!

For more on transforms:
https://suricata.readthedocs.io/en/latest/rules/transforms.html

Step 4:

Okay so we have all the prep work done..well not quite. So generally speaking domain IOCs don't come to us as sha256 hashes. Soo what do we do?

We write some bad python:
https://github.com/jmtaylor90/dns2sha256

This simple script takes a file with domain names and writes out a file with the corresponding sha256 hash values.

In my case I selected, google.com, slashdot.org and reddit.com to use in the dns-seen.lst file.

Step 5:

Now that we have our hashes, we just need a pcap with DNS traffic that contains DNS queries for our domains. Using dig and tcpdump we can generate the traffic and pcap.

something like 'tcpdump -nn -i $activenetworkcard -w dnslisttest.pcap' and then

dig google.com
dig reddit.com
dig slashdot.org

Step 6:

Now we can replay our pcap through suricata to see what happens. I ran the following:


jason@dinosaur suri]$ rm *.json *.log; $(which suricata) -k none -c suricata.yaml -r dnslisttest.pcap

This just makes sure I don't have an old logs laying around and then runs suricata with my configuration file and replays the pcap we captured in Step 5.

If I look at the alert.json file (eve log configured for alerts):
{"timestamp":"2019-09-28T20:38:06.536624-0400","flow_id":2206638181068848,"pcap_cnt":11,"event_type":"alert","src_ip":"172.16.42.7","src_port":38966,"dest_ip":"172.16.42.1","dest_port":53,"proto":"UDP","tx_id":0,"alert":{"action":"allowed","gid":1,"signature_id":123,"rev":1,"signature":"dns list test","category":"","severity":3},"dns":{"query":[{"type":"query","id":16741,"rrname":"google.com","rrtype":"A","tx_id":0}]},"app_proto":"dns","flow":{"pkts_toserver":1,"pkts_toclient":0,"bytes_toserver":93,"bytes_toclient":0,"start":"2019-09-28T20:38:06.536624-0400"},"payload":"QWUBIAABAAAAAAABBmdvb2dsZQNjb20AAAEAAQAAKRAAAAAAAAAMAAoACAI9cKIVTyJM","stream":0,"packet":"8nUnCb+QtLZ2CFMnCABFAABPXUkAAEARcSysECoHrBAqAZg2ADUAO6PpQWUBIAABAAAAAAABBmdvb2dsZQNjb20AAAEAAQAAKRAAAAAAAAAMAAoACAI9cKIVTyJM","packet_info":{"linktype":1}}
{"timestamp":"2019-09-28T20:38:01.959896-0400","flow_id":2191197773342104,"pcap_cnt":5,"event_type":"alert","src_ip":"172.16.42.7","src_port":38358,"dest_ip":"172.16.42.1","dest_port":53,"proto":"UDP","tx_id":0,"alert":{"action":"allowed","gid":1,"signature_id":123,"rev":1,"signature":"dns list test","category":"","severity":3},"dns":{"query":[{"type":"query","id":22527,"rrname":"slashdot.org","rrtype":"A","tx_id":0}]},"app_proto":"dns","flow":{"pkts_toserver":1,"pkts_toclient":0,"bytes_toserver":95,"bytes_toclient":0,"start":"2019-09-28T20:38:01.959896-0400"},"payload":"V\/8BIAABAAAAAAABCHNsYXNoZG90A29yZwAAAQABAAApEAAAAAAAAAwACgAIrWxDkiq\/JNg=","stream":0,"packet":"8nUnCb+QtLZ2CFMnCABFAABRTS4AAEARgUWsECoHrBAqAZXWADUAPe+oV\/8BIAABAAAAAAABCHNsYXNoZG90A29yZwAAAQABAAApEAAAAAAAAAwACgAIrWxDkiq\/JNg=","packet_info":{"linktype":1}}
{"timestamp":"2019-09-28T20:38:10.967322-0400","flow_id":781430593602202,"pcap_cnt":13,"event_type":"alert","src_ip":"172.16.42.7","src_port":39980,"dest_ip":"172.16.42.1","dest_port":53,"proto":"UDP","tx_id":0,"alert":{"action":"allowed","gid":1,"signature_id":123,"rev":1,"signature":"dns list test","category":"","severity":3},"dns":{"query":[{"type":"query","id":11178,"rrname":"reddit.com","rrtype":"A","tx_id":0}]},"app_proto":"dns","flow":{"pkts_toserver":1,"pkts_toclient":0,"bytes_toserver":93,"bytes_toclient":0,"start":"2019-09-28T20:38:10.967322-0400"},"payload":"K6oBIAABAAAAAAABBnJlZGRpdANjb20AAAEAAQAAKRAAAAAAAAAMAAoACHszHT2Q7UUp","stream":0,"packet":"8nUnCb+QtLZ2CFMnCABFAABPaqMAAEARY9KsECoHrBAqAZwsADUAO6btK6oBIAABAAAAAAABBnJlZGRpdANjb20AAAEAAQAAKRAAAAAAAAAMAAoACHszHT2Q7UUp","packet_info":{"linktype":1}}

It fired alerts!

So this is pretty interesting functionality and I expect to test a lot more with it in the upcoming months.