My custom lib.nixosSystem

How I came to write my own lib.nixosSystem

Introduction

I’ve been using NixOS for a while now, and my biggest issue was that I have a lot of machines. Which leads to having a lot of different systems on my flake because of that hardware. The normal solution would be to use the module system, but a new issue arises when we add nix-darwin. We suddenly start to fail eval over issues because some modules don’t exist that are in the normal NixOS module system. So my new issue is that I can no longer use my small abstraction over lib.nixosSystem. I would have to expand the abstraction to include lib.darwinSystem since I can no longer unconditionally import all modules I use if they don’t exist in nix-darwin? But what if I don’t want to do that? What if I want to write my own lib.nixosSystem or my own lib.darwinSystem? What if I call it mkSystem. Well that’s exactly what I did. And this article covers how I got to that point and how my custom “builder” later evolved into easy-hosts.

The research

To start we should read the documentation for lib.evalModules. From which we find that there are 4 arguments, but 3 that we really care about. Those being modules, specialArgs and class. The modules argument is a list of modules, which are files, functions or attrsets that will be merged and then evaluated. The specialArgs argument is a attrset of arguments that are passed to the modules, but are not evaluated with the file structure in mind. The class argument is a nominal type that ensures that only compatible modules are imported. This is a very important argument, as it allows us to have different modules for different systems.

However, I’m not much a fan of reading documentation. So instead lets dig into the source code and pick it apart. To find out what we need to get our custom lib.nixosSystem working. To do this I identified 4 main files in the nixpkgs repository:

These files may seem somewhat arbitrary, but they are listed in the order they are called. The flake.nix file has our lib.nixosSystem function that calls our nixos/lib/eval-config.nix file which is a light wrapper around our final lib/modules.nix. So lets walk through those files in order and see what exactly they do.

flake.nix

This file contains our lib.nixosSystem function, which takes args as an argument. The documentation lists a set of known arguments being modules, specialArgs and modulesLocation, it also specifies some additional legacy arguments system and pkgs (both of which are now redundant).

The lib.nixosSystem then imports the nixos/lib/eval-config.nix file whilst passing lib, and the remaining args to it. It also sets system to null as well as adding nixpkgs.flake.source to nixpkgs output derivation.

nixos/lib/eval-config.nix

This file immediately points us to the fact that it is a “light wrapper” around lib.evalModules. This file also has a large collection of arguments most of which will be the defaults. A good example of this is baseModules which defaults to a list of modules from the nixpkgs repo. The most important arguments from this file are specialArgs, lib and modules. For the most part these come from the prior flake.nix file.

As we read down the file we notice that there are two additional modules that are going to be added. These are the pkgsModule and the modulesModule. These appear to be pretty strange names at first, but the pkgsModule will set nixpkgs.system if system was not null, and will set nixpkgs.pkgs if pkgs is not null. The modulesModule will add config._module.args as an attrset of noUserModules, baseModules, extraModules and modules. So now we know some of the arguments that are given to lib.evalModules lets see what that does.

nixos/lib/eval-config-minimal.nix

This file is a small wrapper upon lib.evalModules, but it gives us a little bit of guidance on how to use the class argument. As well as showing us the default for modulePath which is going to be passed as a special arg.

lib/modules.nix

This file won’t add much to our understanding of lib.evalModules, but it still is the definition of the function, so it’s good to keep in mind if we have any issues later down the line.

The implementation

Now that we have our key inputs of class, modules and specialArgs we can start implementing our own lib.nixosSystem.

Getting the basics

Let us start in our very own flake.nix by writing the following code. This will give us a basic template to work with and you, the reader, knows how to start.

{
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";


  outputs = inputs: let
    mkSystem = import ./mksystem.nix { inherit inputs; };
  in {
    nixosConfigurations = {
      mySystem = mkSystem { };
    };
  };
}

Now that we have a very bare bone flake.nix we can get started on the mkSystem function. Let’s also create the mksystem.nix file. We are going to add some basic args that we know we are going to need later such as modules, specialArgs and class. And we are going to add some defaults to these args such that we don’t get any errors if they are not provided.

{
  inputs ? throw "No inputs provided",
  lib ? inputs.nixpkgs.lib,
  ...
}:
{
  modules ? [ ],
  specialArgs ? { },
  class ? "nixos",
}: lib.evalModules {
  inherit modules specialArgs class;
}

Adding modulesPath

Now that we have our basic template down. Let’s start by adding the modulesPath, most people probably recognize this from when they first installed nix and read their hardware-configuration.nix file and saw something along the lines of modulesPath + /installer/scan/not-detected.nix. That is why we are starting with this, so that our hardware-configuration.nix will work.

The key to this is going to be including modulesPath in our specialArgs. This is because specialArgs should only be used with arguments that need to be evaluated when resolving module structure.

{
  inputs,
  lib ? inputs.nixpkgs.lib,
  ...
}:
{
  modules ? [ ],
  specialArgs ? { },
  class ? "nixos",
}: let
  modulesPath = "${inputs.nixpkgs}/nixos/modules";
in
lib.evalModules {
  inherit modules class;


  # here we are merging the user provided specialArgs with the modulesPath
  specialArgs = { inherit modulesPath; } // specialArgs;
}

So close and yet so far

Only adding modulePath is a bit useless however, so we can’t exactly replace our lib.nixosSystem’s yet. So let’s work on that. We are going to start by importing baseModules, this will provide us with a base set of modules that provide abstractions over configuring your system. We can use the modulePath and get the module list.

{
  inputs,
  lib ? inputs.nixpkgs.lib,
  ...
}:
{
  modules ? [ ],
  specialArgs ? { },
  class ? "nixos",
}: let
  modulesPath = "${inputs.nixpkgs}/nixos/modules";


  baseModules = import "${modulesPath}/module-list.nix";
in
lib.evalModules {
  inherit class;


  specialArgs = { inherit modulesPath; } // specialArgs;


  modules = baseModules ++ modules;
}

It actually works?

We now have a mostly functional replacement. Depending on your configuration may actually work as it is now! To keep progressing we are going to have to go back to the modulesModule from earlier. we need this such that some nixpkgs modules will work, one of these is the documentation module which will be a hard module to ignore, when so many people use it.

To fix this we are going to introduce a new module which contains config._module.args and takes a set of attrs that will be passed to each module. I’m sure most of you recognize these when writing a module and adding something like { pkgs, config, ... } to the top of a file. The config._module.args option should be used when trying to pass arguments to all modules, but should not be evaluated with file structure in mind.

{
  inputs,
  lib ? inputs.nixpkgs.lib,
  ...
}:
{
  modules ? [ ],
  specialArgs ? { },
  class ? "nixos",
}: let
  modulesPath = "${inputs.nixpkgs}/nixos/modules";


  baseModules = import "${modulesPath}/module-list.nix";
in
lib.evalModules {
  inherit class;


  specialArgs = { inherit modulesPath; } // specialArgs;


  modules = baseModules ++ modules ++ [
    {
      config._module.args = {
        inherit baseModules modules;
      };
    }
  ];
}

Adding some of our own modules

Even better, now we have completely replaced lib.nixosSystem with our own mkSystem function. But let’s be real. That’s not enough for us. We should start abstracting some common themes between our systems. Some big examples of this are networking.hostName and nixpkgs.hostPlatform. And while were at it lets also re-add the nixpkgs.flake.source from the original lib.nixoSystem, as well as adding inputs as a special arg. As most people do this anyway, I think it’s a safe assumption we should add it. For further reading about passing inputs to modules check nobbz’s blog on getting inputs to flake modules.

{
  inputs,
  lib ? inputs.nixpkgs.lib,
  ...
}:
name:
{
  modules ? [ ],
  specialArgs ? { },
  class ? "nixos",
}: let
  modulesPath = "${inputs.nixpkgs}/nixos/modules";


  baseModules = import "${modulesPath}/module-list.nix";
in
lib.evalModules {
  inherit class;


  specialArgs = { inherit inputs modulesPath; } // specialArgs;


  modules = baseModules ++ modules ++ [
    {
      config. = {
        _module.args = {
          inherit baseModules modules;
        };


        networking.hostName = name;


        nixpkgs.flake.source = inputs.nixpkgs.outPath;
      };
    }
  ];
}

I just added name as an additional argument to our mkSystem function. This allows us to set the hostname of our system. The way I opted to write it allows for us to use mapAttrs on our nixosConfigurations. This will mean that we need to change how the original flake.nix works though.

{
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";


  outputs = inputs: let
    mkSystem = import ./mksystem.nix { inherit inputs; };
  in {
    nixosConfigurations = builtins.mapAttrs mkSystem {
      mySystem = { };
    };
  };
}

Furthermore, notice how I lied about settings nixpkgs.hostPlatform. If your curious why, maybe you should read my last blog post about it. (Shameless plug)

Darwin compatibility

One of the main reasons I wanted to make this was to support lib.darwinSystem. So let’s address that now.

To introduce Darwin support we are allowing users to set the class argument to darwin from there we can determine what modules to import. As a result of this you may notice that Darwin has a different set of modules which introduced some new options to set for this system type. This includes nixpkgs.source and darwinVersionSuffix and darwinRevision. Some of these are for commands like darwin-version.

You may also notice that we had to add system = eval.config.system.build.toplevel back into the final eval produced by our Darwin eval. This is required to swap to the configuration, otherwise it won’t work at all. This is because the system output is used by darwin-rebuild, to identify the final build output.

{
  inputs,
  lib ? inputs.nixpkgs.lib,
  ...
}:
name:
{
  modules ? [ ],
  specialArgs ? { },
  class ? "nixos",
}: let
  # this is new? what is it?
  # I'm glad you asked, this is a nice way of checking if we have our darwin and nixpkgs inputs
  nixpkgs = inputs.nixpkgs or (throw "No nixpkgs input found");
  darwin = inputs.darwin or inputs.nix-darwin or (throw "No nix-darwin input found");


  modulesPath = if class == "darwin" then "${darwin}/modules" else "${nixpkgs}/nixos/modules";


  baseModules = import "${modulesPath}/module-list.nix";


  eval = lib.evalModules {
    inherit class;


    specialArgs = { inherit inputs modulesPath; } // specialArgs;


    modules = baseModules ++ modules ++ [
      {
        config. = {
          _module.args = {
            inherit baseModules modules;
          };


          networking.hostName = name;


          nixpkgs.flake.source = nixpkgs.outPath;
        };
      }
    ] ++ lib.optionals (class == "darwin") [
      {
        config = {
          nixpkgs.source = nixpkgs.outPath;


          system = {
            checks.verifyNixPath = false;


            darwinVersionSuffix = ".${darwin.shortRev or darwin.dirtyShortRev or "dirty"}";
            darwinRevision = darwin.rev or darwin.dirtyRev or "dirty";
          };
        };
      }
    ];
  };
in
  if class == "darwin" then (eval // { system = eval.config.system.build.toplevel; }) else eval;

The final touch

The final and maybe the best bit is adding inputs'. For those who are unaware of flake-parts, you probably are not aware of the greatness that is inputs'. The diff below shows the advantage of using inputs' over inputs for accessing packages.

inputs.input-name.packages.${pkgs.stdenv.hostPlatform.system}.package-name
inputs'.input-name.packages.package-name

Is that not awesome? So how can we replicate that for ourselves?

What we will need to do is map over all inputs, and their outputs and select the output dependent on the host platform, if a system dependent output exists, otherwise it will leave it as is. We can achieve that with the following code:

inputs' = lib.mapAttrs (_: lib.mapAttrs (_: v: v.${config.nixpkgs.hostPlatform} or v)) inputs;

Or if you are using flake-parts, you may prefer using the following code instead:

withSystem config.nixpkgs.hostPlatform ({ inputs', ... }: { inherit inputs'; });

So let’s add that to our mkSystem function.

{
  inputs,
  lib ? inputs.nixpkgs.lib,
  ...
}:
name:
{
  modules ? [ ],
  specialArgs ? { },
  class ? "nixos",
}: let
  nixpkgs = inputs.nixpkgs or (throw "No nixpkgs input found");
  darwin = inputs.darwin or inputs.nix-darwin or (throw "No nix-darwin input found");


  modulesPath = if class == "darwin" then "${darwin}/modules" else "${nixpkgs}/nixos/modules";


  baseModules = import "${modulesPath}/module-list.nix";


  eval = lib.evalModules {
    inherit class;


    specialArgs = { inherit inputs modulesPath; } // specialArgs;


    modules = baseModules ++ modules ++ [
      ({ config, ... }: {
        config = {
          _module.args = {
            inherit baseModules modules;


            inputs' = lib.mapAttrs (_: lib.mapAttrs (_: v: v.${config.nixpkgs.hostPlatform} or v)) inputs;
          };


          networking.hostName = name;


          nixpkgs.flake.source = nixpkgs.outPath;
        };
      })
    ] ++ lib.optionals (class == "darwin") [
      {
        config = {
          nixpkgs.source = nixpkgs.outPath;


          system = {
            checks.verifyNixPath = false;


            darwinVersionSuffix = ".${darwin.shortRev or darwin.dirtyShortRev or "dirty"}";
            darwinRevision = darwin.rev or darwin.dirtyRev or "dirty";
          };
        };
      }
    ];
  };
in
  if class == "darwin" then
    (eval // { system = eval.config.system.build.toplevel; })
  else
    eval;

Conclusion

And that’s it! We have a fully functional mkSystem function that can replace both lib.nixosSystem and lib.darwinSystem. This was quite the task, and although this blog post seems to reduce the quite simple. I’ve spent a lot of time on this, both when researching how to create the custom builder and writing and maintaining the latest rendition in the form of a flake module called easy-hosts. If you enjoyed this post, please consider donating on ko-fi or github sponsors.

© Blog post licensed under CC BY-NC-SA 4.0