Getting Splunk onto a NixOS Linode
I really like Splunk1, especially for personal projects where usage can fit comfortably into the free tier. I also really like NixOS, and have recently embarked on getting my web stuff onto NixOS.
Getting Splunk to run on NixOS is a bit fiddly, but straight-forward once you know what you need to do. If you want to run Splunk on NixOS, read on!
Installing NixOS
First, you’ll need to get NixOS installed wherever you’re going to be doing this. Previously I got one set up in DigitalOcean using a minimal NixOS image I build on my computer. Today, I’m putting it on Linode. The process is pretty much identical: just upload your image, and spin up a VPS based on the image.
Linode requires a gzipped .img file, though, so you’ll need to do a quick conversion.
1
2
qemu-img convert -O raw result/nixos[...].qcow2 nixos.img
gzip nixos.img
Then you just upload nixos.img.gz as an Image, put it in whatever datacenter you want to run this in, and then spin up a Linode. Surprisingly, a Nanode (1gb ram) actually works for this in a pinch. Though of course depending on how much data you want to retain, you might want to mount add a data volume.
Finally, navigate to your Linode, go to “Configurations”, edit the disk configuration, and under “Select a Kernel” switch it to Direct Disk.
The Flake
Given that Splunk’s free tier has no authentication at all, we’re going to want to put this thing behind nginx so we can have basic auth. Set up this folder structure in your flake (I’m calling it “splunk”, call it whatever you like).
1
2
3
4
5
6
7
hosts/
splunk/
default.nix
hardware.nix
nginx.nix
sites/
splunk.nix
Primary config
This is our primary configuration for the host, for users, networking, etc. Start with the following, and we’ll talk through it.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
{
inputs,
pkgs,
...
}: {
imports = [
./hardware.nix
./nginx.nix
./sites/splunk.nix
];
nix.registry.nixpkgs.flake = inputs.nixpkgs-splunk;
nix.nixPath = ["nixpkgs=${inputs.nixpkgs-splunk}"];
networking.hostName = "splunk";
services.openssh = {
enable = true;
ports = [22 2222];
settings = {
PermitRootLogin = "prohibit-password";
PasswordAuthentication = false;
};
};
environment.systemPackages = with pkgs; [
htop
git
rsync
ripgrep
];
users.users.root.openssh.authorizedKeys.keys = ["ssh-rsa AAA<your key>"];
users.users.ojensen = {
isNormalUser = true;
extraGroups = ["wheel"];
openssh.authorizedKeys.keys = ["ssh-rsa AAA<your key>"];
};
zramSwap.enable = true;
swapDevices = [
{
device = "/var/swapfile";
size = 2048;
}
];
nix.settings.experimental-features = ["nix-command" "flakes"];
nix.gc = {
automatic = true;
dates = "weekly";
options = "--delete-older-than 30d";
};
system.stateVersion = "26.05";
}
We start by importing the other files. The registry and nixpath directives pin nixpkgs on the host to the revision your flake was built from, so ad-hoc nix shell commands reuse the store instead of pulling down a duplicate set of base libraries. Then we set the hostname, and enable SSH
I’ve found it’s a good idea to have SSH listening on port 22 and 2222, then configure a firewall on your hosting provider to only allow 22 from your IP. Hosts get absolutely hammered with brute-force SSH attacks, which can impact your ability to log in yourself. So default 22 is only for you, and keep 2222 open for when you’re away from home. In practice, scanners don’t bother much with non-standard ports.
Next, add whatever software packages you generally want available. This is a pretty sparse list, but in practice you’re probably not doing much from this shell, and you can just nix shell anything you need, so no reason to go wild.
Set up your users with appropriate SSH keys. Presumably your username isn’t going to be ojensen, and you don’t want to accept my keys, so adjust accordingly.
The swap config is primarily useful for small Linodes, and the rest is basically just nix boilerplate.
Hardware
SSH into your linode to get your disk’s UUID. Run blkid /dev/sda1, then fill out your hardware.nix as follows:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
{modulesPath, ...}: {
imports = [(modulesPath + "/profiles/qemu-guest.nix")];
fileSystems."/" = {
device = "/dev/disk/by-uuid/<your disk uuid>";
fsType = "ext4";
};
boot.loader.grub = {
enable = true;
devices = ["/dev/sda"];
extraConfig = ''
serial --speed=19200 --unit=0 --word=8 --parity=no --stop=1;
terminal_input serial;
terminal_output serial;
'';
};
boot.loader.timeout = 10;
boot.kernelParams = ["console=ttyS0,19200n8"];
nixpkgs.hostPlatform = "x86_64-linux";
}
Nginx
The free tier of Splunk has no authentication at all, so we’ll put this behind Nginx with basic auth. This means we’ll need to define some sane defaults for Nginx.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
{pkgs, ...}: {
networking.firewall.allowedTCPPorts = [80 443];
security.acme = {
acceptTerms = true;
defaults.email = "<your email address>";
};
services.nginx = {
enable = true;
recommendedGzipSettings = true;
recommendedOptimisation = true;
recommendedTlsSettings = true;
recommendedProxySettings = true;
commonHttpConfig = ''
access_log /var/log/nginx/access.log combined;
'';
virtualHosts."_" = {
default = true;
root = "/var/www/empty";
locations."/".return = "404";
};
};
systemd.tmpfiles.rules = ["d /var/www/empty 0755 root root -"];
services.logrotate.settings.nginx = {
frequency = "monthly";
rotate = 12;
dateext = true;
compress = true;
delaycompress = true;
};
}
We’re not going to host anything at that default virtual host, so just shouting 404 at anyone who visits is good enough.
Splunk
OK, here’s where things get interesting.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
{
config,
pkgs,
...
}: {
programs.nix-ld = {
enable = true;
libraries = with pkgs; [
stdenv.cc.cc.lib # libstdc++, libgcc_s
zlib
libxcrypt-legacy # libcrypt.so.1
];
};
services.envfs.enable = true;
boot.kernelParams = ["transparent_hugepage=never"];
users.users.splunk = {
isSystemUser = true;
group = "splunk";
home = "/opt/splunk";
description = "Splunk Enterprise";
};
users.groups.splunk = {};
systemd.tmpfiles.rules = [
"d /opt 0755 root root -"
"d /opt/splunk 0750 splunk splunk -"
"d /var/lib/secrets 0755 root root -"
];
systemd.services.splunk = {
description = "Splunk Enterprise";
wantedBy = ["multi-user.target"];
after = ["network-online.target"];
wants = ["network-online.target"];
path = with pkgs; [
bash
gnugrep
coreutils
gnused
gawk
procps
util-linux
findutils
gzip
];
serviceConfig = {
Type = "forking";
PIDFile = "/opt/splunk/var/run/splunk/splunkd.pid";
User = "splunk";
Group = "splunk";
Environment = [
"SPLUNK_HOME=/opt/splunk"
"SPLUNK_BINDIP=127.0.0.1"
];
ExecStart = "/opt/splunk/bin/splunk start --accept-license --answer-yes --no-prompt";
ExecStop = "/opt/splunk/bin/splunk stop";
Restart = "on-failure";
RestartSec = 30;
TimeoutStartSec = 900;
TimeoutStopSec = 300;
LimitNOFILE = 64000;
LimitNPROC = 16000;
LimitFSIZE = "infinity";
LimitDATA = "infinity";
TasksMax = "infinity";
ProtectSystem = "full";
PrivateTmp = true;
NoNewPrivileges = true;
};
};
services.nginx.virtualHosts."splunk.yourdomain.com" = {
enableACME = true;
forceSSL = true;
locations."/" = {
proxyPass = "http://127.0.0.1:8000";
proxyWebsockets = true;
basicAuthFile = "/var/lib/secrets/splunk-htpasswd";
extraConfig = ''
proxy_read_timeout 300s;
proxy_send_timeout 300s;
'';
};
locations."/services/collector" = {
proxyPass = "http://127.0.0.1:8088";
extraConfig = ''
client_max_body_size 20m;
'';
};
};
}
Splunk ships as a tarball of executables, so we’ll need to work around the fact that they haven’t been packaged for NixOS. nix-ld helps with dynamically linked libraries, and envfs helps with the shebangs that head a lot of the scripts Splunk runs.
Splunk recommends switching off Transparent Huge Pages, so we do so. Then we create our user and some directories that we’ll need to use.
The next large block is the systemd service. We include the bash shell in its path because a lot of Splunk’s scripts invoke #! /bin/bash at the top and this is just the easiest way to deal with that. We’re binding to 127.0.0.1 because nothing should be able to reach Splunk except by going through Nginx. The timeouts and limits are mitigating the fact that we’re on the teeny-tiniest of Nanodes. And finally, a teeny bit of hardening
The last block is the Nginx virtualhost. Set whatever hostname you want this to be at, and don’t forget to set up DNS (since auto-provisioning of TLS certificates depends on DNS being set up). We pass it very large read and send timeouts because searches can take a long time, especially on our winky-dink little machine. We also need to pass in a second location for the HEC (assuming you will use it) so that it isn’t behind basic auth.
First deploy
Wire up something in your flake’s primary file to import all of this. Maybe something like:
1
2
3
4
5
6
7
splunk = nixpkgs-splunk.lib.nixosSystem {
system = "x86_64-linux";
specialArgs = {inherit inputs;};
modules = [
./hosts/splunk
];
};
and then push it all out to your Linode:
1
nixos-rebuild switch --flake .#splunk --target-host root@<hostname>
That’s the declarative stuff done, but obviously we still have some work to do. We haven’t actually put Splunk on the thing yet, after all.
Don’t forget to reboot the Linode so that it picks up its new hostname.
Basic-Auth
Once the deploy has finished, you can visit your hostname and you should get a Basic Auth password prompt. Before we can get past this, we need to set up the htpasswd file we referenced in splunk.nix.
SSH into your box, and generate it:
1
2
3
4
5
nix-shell -p apacheHttpd --run \
'htpasswd -c /tmp/htpasswd ojensen'
sudo mv /tmp/htpasswd /var/lib/secrets/splunk-htpasswd
sudo chown nginx:nginx /var/lib/secrets/splunk-htpasswd
sudo chmod 400 /var/lib/secrets/splunk-htpasswd
Presumably you don’t want your username to be ojensen so adjust accordingly.
Installing Splunk
Next, fetch the latest version of Splunk Enterprise from splunk.com. Go ahead and click through the “free trial” – when it runs out, the license turns into the free license. Make sure to download the .tgz version.
Copy that to your linode, and then extract it into /opt:
1
2
sudo tar xvf splunk.tgz -C /opt
sudo chown -R splunk:splunk /opt/splunk
Before starting it up, we’ll need to configure two things. First, if you’re on a RAM-strapped Nanode, you’ll probably want to make sure KV-stores are not enabled. Secondly, you’ll need to define an initial username and password for Splunk: while the free license is unauthenticated, right now you’re on Splunk Enterprise, which is authenticated.
1
2
3
4
5
6
7
8
9
10
11
12
sudo -u splunk mkdir -p /opt/splunk/etc/system/local
sudo -u splunk tee /opt/splunk/etc/system/local/server.conf <<'EOF'
[kvstore]
disabled = 1
EOF
sudo -u splunk tee /opt/splunk/etc/system/local/user-seed.conf <<'EOF'
[user_info]
USERNAME = admin
PASSWORD = <put something here>
EOF
Finally, bring Splunk up:
1
systemctl start splunk
This will take a few minutes the first time you do it. When it’s done, try accessing it again. After getting past the Basic Auth prompt, you should be presented with Splunk’s login page. Enter the username and password from above, and you should then be signed in.
Data volumes
Depending on how much data you’re processing and how long you want to retain the data for, your little Nanode probably won’t have enough disk space. You can just spin up a Data Volume in Linode and mount it into your Nanode where Splunk puts its data.
Configure the mount point in splunk.nix:
1
2
3
4
5
fileSystems."/mnt/splunk-data" = {
device = "/dev/disk/by-id/scsi-0Linode_Volume_<label>";
fsType = "ext4";
options = ["defaults" "nofail"];
};
Put a folder there that Splunk can own
1
2
sudo mkdir /mnt/splunk-data/splunk-db
sudo chown -R splunk:splunk /mnt/splunk-data/splunk-db
Set the SPLUNK_DB variable in /opt/splunk/etc/splunk-launch.conf accordingly, and restart Splunk.
1
2
3
4
5
[...]
SPLUNK_DB=/mnt/splunk-data/splunk-db
[...]