Reputation Lists and Datasets with Suricata

Another use case we were wondering about is domain IOCs and datasets using the reputation option.

So I exported our domain IOCs from our intel platform into a csv. To transform the IOC domains to sha256, I used this python script.

The values in the original csv are domain,numerical_reputation format, so the script just grabs the domain, converts it and then smashes it back together with its associated reputation and then writes it all out to a new file for use in our Suricata rule.

Victor made some enhancements to the way the suricata.yaml dataset configuration gets parsed and loaded so the suricata.yaml snippet looks a little different than in my original post.

Also after some discussion with Jason Ish on the OISF team, I made a change to the location for our state files and now will be putting all our lists into a common but separate directory from our rules.
This is more or less to make sure nothing gets overwritten by rule set changes if there are naming colllisions.

suricata.yaml snippet:

datasets:
  dns-seen:
    type: sha256
    load: /nsm/lists/topdomains.lst
    hash:
      memcap: 1024mb
      hash-size: 1024mb
      prealloc: 1024mb

  dns-ioc:
    type: sha256
    load: /nsm/lists/domainioc.lst
    hash:
      memcap: 256mb
      hash-size: 256mb
      prealloc: 256mb

The chunk I selected for the domainioc.lst is just shy of 700,000 records. An example of the format for domainioc.lst:

d4c9d9027326271a89ce51fcaf328ed673f17be33469ff979e8ab8dd501e664f,8.0

Our test rule is:
alert dns any any -> any any (msg:"DNS IOC List Test"; dns.query; to_sha256; datarep:dns-ioc, >, 7.0, sid:1234; rev:1;)

alert.json entry sample:

Looking at the suricata load time:
13929] 11/10/2019 -- 19:12:21 - (suricata.c:1078) <Notice> (LogVersion) -- This is Suricata version 5.0.0-dev (a1ee536 2019-10-10) running in SYSTEM mode
[13929] 11/10/2019 -- 19:12:21 - (util-cpu.c:171) <Info> (UtilCpuPrintSummary) -- CPUs/cores online: 40

14078] 11/10/2019 -- 19:21:03 - (source-af-packet.c:1802) <Perf> (AFPComputeRingParamsV3) -- AF_PACKET V3 RX Ring params: block_size=32768 block_nr=3126 frame_size=1600 frame_nr=62520 (mem: 102432768)
[14078] 11/10/2019 -- 19:21:04 - (source-af-packet.c:515) <Info> (AFPPeersListReachedInc) -- All AFP capture threads are running.

It looks like we are at about 8.5 minutes or so. This seems reasonable given how much data we are loading but am curious what others are seeing with their setups. I will run this by the OISF folks as well to get their thoughts.