Nix Distributed Builds

On this post you can find a mostly declarative setup for Nix store distributed builds. The information taken from the NixOS wiki and this two blog posts:

• https://nathan.gs/2024/02/04/nixos-enable-distributed-builds/
• https://sgt.hootr.club/molten-matter/nix-distributed-builds/

Assumptions/Prerequisites:

• System Wide NixOS install on both machine (remote and local)
• High degree of trust to the remote machine
• An SSH connection between the machines
• A bit of experience with NixOS and the Nix language

Type of setup

There I will show a passwordless ssh connection between a user in the local machine and the builder user in the remote machine. Everything will be provided and explained via a NixOS configuration.

As stated in the NixOS wiki it is recommended to have a purposely created user on the remote machine to connect to when using the remote builder. Since the nix store (the place where the builds and packages are stored) is system global, it does not matter if the access is root or not. Following the principle of least privilege, this remote user will not be in the wheel group (an admin) and will not have a home directory.

In the wiki it is also recommended to have the ssh connection from the local machine to be root, so that every local user can access the builder. This is not the case in this guide, therefore to use the remote builder on another local user, we would need to create another ssh key to authorize another connection.

Guide

ssh key creation

Simple run:

ssh-keygen -f ~/.ssh/<name of ssh key file>

It will be store in the $HOME/.ssh of the user that you are on. In my case this is a user in the sudoers group.

DO NOT add a passphrase to the ssh key, as this needs to be a passwordless connection.

Remote configuration

Go the configuration.nix file and insert the following lines into it:

{
	...

	user.users.USER_NAME = {
		description = "nixos remote builder"; # can be anything
		isSystemUser = true;
		createHome = false;
	
		openssh.authorizedKeys.keys = [
			"the .pub ssh key from your local machine"
		];
		openssh.authorizedKeys.files = [
			"or path to a copy of the .pub file"
		];
		# choose one of this two options
	
		uid = 500;
		group = "<USER_NAME>";
		useDefaultShell = true;
	};

	users.groups.USER_NAME = {
		gid = 500;
	};

	nix.settings.trusted-users = [ "<USER_NAME>" ];

	...
}

Here we have created a new user with the name USER_NAME, replace it with the name of your choice. In my case it is nixremote.

This user is inside of a group with the same name. Both user and group IDs are the same. The user is also under the system’s trusted users. This will grant certain privileges to the user like the availability to import unsigned NARs (Nix Archives) or get additional binary caches, as per the NixOS wiki.

What is important here is the authorized ssh keys, the local keys needs to be added to the system in some way, either by a hardcoding it on the Nix config or specifying the public key file.

Also do not forget to have the ssh-daemon enabled in the system:

services.openssh.enable = true;

Local configuration

Go to the configuration.nix file in your local machine and insert the following lines:

{
	...

	nix.buildMachine = [{
		hostName = "<USER>@<HOST>";
		system = "x86_64=linux"; # or your builder system type

		# the private ssh key file name created in the first step
		sshKey = "/home/local_user/.ssh/<SSH_FILE_NAME>"; 

		protocol = "ssh-ng";	
		maxJobs = <NUMBER_THREADS>;
		speedFactor = <ARBITRARY_NUMBER>;
		supportedFeatures = [ "nixos-test" "benchmark" "big-parallel" "kvm" ];
	}];

	nix.distributedBuilds = true;
	nix.extraOptions = ''
		builders-use-substitutes = true
	'';

	programs.ssh.extraConfig = ''
Host <HOST_NAME or IP>
	Port 22
	User <USER_NAME>
	IdentitiesOnly yes
	IdentityFile ~/.ssh/nixremote
'';

	...
}

As before do not forget to enable the ssh-daemon.

In the <USER>@<HOST>, the host name can either be the IP of the machine or the name that it has on your home network. For a truly remote setup, you would need to have port forwarding on your network.

The maxJobs can be all the available threads on your remote machine or a portion of them, its up to you. The same goes for the speedFactor parameter, it is an arbitrary number that states how fast is the remote machine relative to you local machine.

In supportedFeatures I have included everyone of them, as this is also recommended in the wiki tutorial.

The remote system type (in my case x86_64-linux) can be different from your local machine. In this case refer to this other tutorial, where the remote machine is a x86 Linux server and the local machine an ARM Linux Raspberry Pie. As a TLDR, emulation would need to be enabled in the remote server.

All the other configuration options are mandatory and need to be as stated. The ssh config here is done declaratively, unlike the wiki tutorial where the /root/ssh/config is created directly.

Testing

First of all there should be a passwordless ssh connection between the two machines, from the local to the remote.

You should also be able to ping the remote nix store doing the following:

nix store ping --store ssh-ng://USER@HOST

There should be no need to insert a password, and the expected output should be something like this:

Store URL: ssh-ng://USER@HOST
Version: 2.18.5
Trusted: 1

Where the USER and HOST are the ones you specified.

If this command works as expected, congratulations!

Expectations/How it works

When building using Nix (e.g. nix-rebuild, nix-shell, using a flake…) it should offload the work to the remote builder if possible. In my case with speedFactor set to 1, it always happens if something is not in cache.nixos.org.

Furthermore, any build is cached in the remote server. This is very useful, as the store in the local machine can be regularly garbage collected, with the server taking the occupied space burden.

Particular usage case

When updating both my machine’s channel to unstable, the waybar binary needed to be compiled because it was no present on the Nix cache. Since it compiled first on my PC, it was cached there and there was no need for my laptop to compile it. It just grabbed from the ssh connection, very practical.

My use case

As an Artificial Intelligence student, I have to sometimes work locally with images or videos, as I have computer vision classes. The cached builds allow me to always have my Nix development environments quickly as well as building my own programs faster.

Most usually a regular ssh connection would work for a Computer Science student that does not need to run programs locally while being. However while developing an OpenCV application with C++, I prefer building remotely and executing locally.

For Python developers another option would be to simply use Jupyter Lab, as you can view images and other types of files with it.

It just prefer to use Distributed Nix Builds, because as long as I build or compile with Nix, my laptop does not run hot.