How big can a suricata dataset be? take 2..

Continuing work with datasets, I was previously working on my laptop and got time to test this on real hardware.

The real hardware are just off the shelf Dell r700 series with 256G of RAM.

snippet of lscpu output:
Architecture:            x86_64
CPU op-mode(s):     32-bit, 64-bit
Byte Order:              Little Endian
CPU(s):                    40
On-line CPU(s) list: 0-39
Thread(s) per core:  2
Core(s) per socket:  10
Socket(s):                2
NUMA node(s):      4
Vendor ID:              GenuineIntel
CPU family:            6
Model:                     63
Model name:           Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz

So of course the first thing to test is just HOW FAST can we load 10 million records :)

The answer is:

real 7m57.045s

user 7m52.763s

sys 0m3.446s

[16012] 3/10/2019 -- 18:33:55 - (datasets.c:219) <Config> (DatasetLoadSha256) -- dataset: dns-seen loading from '/etc/nsm/eint/lists/topdomains.lst'

[16012] 3/10/2019 -- 18:41:45 - (datasets.c:275) <Config> (DatasetLoadSha256) -- dataset: dns-seen loaded 10000000 records

[16012] 3/10/2019 -- 18:41:45 - (defrag-hash.c:248) <Config> (DefragInitConfig) -- allocated 3670016 bytes of memory for the defrag hash... 65536 buckets of size 56

[16012] 3/10/2019 -- 18:41:45 - (defrag-hash.c:273) <Config> (DefragInitConfig) -- preallocated 65535 defrag trackers of size 160

[16012] 3/10/2019 -- 18:41:45 - (defrag-hash.c:280) <Config> (DefragInitConfig) -- defrag memory usage: 14155616 bytes, maximum: 1350565888

[16012] 3/10/2019 -- 18:41:45 - (stream-tcp.c:399) <Config> (StreamTcpInitConfig) -- stream "prealloc-sessions": 2048 (per thread)

Now on to seeing about reputation lists...