UP | HOME

Building your own project with nix

Table of Contents

1. Motivation

I've found plenty of getting-started guides for working with nix. Most take the perpective of "a reproducible apt-get": they cover how to use nix to assemble/modify external software packages.

Less common is instruction on how best to package local (non-nixpkgs) software, in particular if you have multiple packages with non-trivial interdependencies.

The sequel summarizes my current understanding of how best to make this work (thanks to Jade Lovelace's article on flakes and nixpkgs, link below)

2. Lessons

2.1. Keep .nix files and project source in separate repos

On the face of it, it seems attractive and natural to keep nix files for a project in the same repo! That's what we do with build instructions (CMakeLists.txt etc)

2.1.1. Why not

  1. .nix files (default.nix, shell.nix etc) likely need to contain revision hashes. Want to avoid situation where you've modified default.nix, and need it to record a hash that wil be determined… after you've committed the file to git
  2. You might want to incorporate a repo child into another repo parent as a submodule. The .nix files associated with child probably don't do what you want in that context.
  3. Even stronger reason to avoid this: nix flakes. Flakes will break (at least as of Dec 2023) if a directory that contains a flake.lock file contains another directory with a flake.lock file. The flake system can't exit from a state in which directory tree is dirty. This nixes composing project trees that contain flakes.
  4. The nixpkgs authors have reached the same conclusion. In nixpkgs, instructions for building a package are expressed as a function, to be invoked from callPackage. The nix-pkgs provides for features like parameterization and cross-compilation; tradeoff is that they then don't work directly from 'nix-build'

2.1.2. Do this instead

For a project foo, create a foo-nix repo that uses fetchurl or cousins to fetch and build foo.

2.2. Setup your build the nixpkgs way

In other words, use callPackage!

  1. callPackage supports cross-compilation (out of the box)
  2. callPackage supports parameterization (i.e. overrides)
  3. If you already have nix build setup for nixpkgs, then there's nothing new to do if you want to upload into nixpkgs (or incorporate into your fork thereof) later

2.3. Don't put build instructions in flakes

Do use flakes! They're great for version pinning and specifying configurations

However, if you put build instructions in flakes (i.e. call outside callPackage), then you lose the ability to easily compose and adapt that the nixpkgs tools give you.

For packaging our own software we will use flake to do these things (and nothing else):

  1. pin nixpkgs.
  2. fetch repos for local projects (things we aren't getting from nixpkgs)
  3. specifying dependendency sets for local projects.

In particular we exclude from flake:

  1. stdenv.mkDerivation calls
  2. specifying buildInputs or cousins

3. Example Flake Setup

Here's an example, following advice above This example condensed from cmake-examples-nix. We have a local project cmake-examples, with a bunch of nixpkgs dependencies.

We'll setup a nix build in cmake-examples-nix. Will need just two files:

  • flake.nix, a nix flake
  • pkgs/ex23.nix, build instructions as in nixpgks.

3.1. File cmake-examples-nix/pkgs/ex23.nix:

{
  # dependencies
  stdenv, doxygen, cmake, catch2, pkg-config, python3Packages, boost, zlib, # ... other deps here

  # args
  #   someconfigurationoption ? false
  pybind11 ? python3Packages.pybind11,
  sphinx ? python3Packages.sphinx,
  breathe ? python3Packages.breathe,
} :

stdenv.mkDerivation (finalattrs:
  {
    name = "cmake-examples-ex23";

    # note: ../flake.nix will override this
    src = fetchGit {
      url = "https://github.com/rconybea/cmake-examples";
      ref = "ex23";     # branch
      # rev = "12345";  # hash
      sha256 = "";      # must supply if not overridden from parent
    };

    # run unit tests
    doCheck = true;

    buildInputs = [ cmake pkg-config  pybind11 sphinx doxygen breathe catch2 boost zlib ];

    buildPhase = ''
      make
      make doxygen
    '';
  })

3.2. File cmake-examples-nix/flake.nix:

{
  description = "Flake for cmake-examples project";

  # pinning nixpkgs
  inputs.nixpkgs.url = "https://github.com/NixOS/nixpkgs/archive/4dd376f7943c64b522224a548d9cab5627b4d9d6.tar.gz";
  # inputs.nixpkgs.url = "github:nixos/nixpkgs/23.11";  # to use particular stable version

  inputs.flake-utils.url = "github:numtide/flake-utils";

  # branch 'ex23' from cmake-examples
  inputs.cmake-examples-ex23-path = { type = "github"; owner = "Rconybea"; repo = "cmake-examples"; flake = false; ref = "ex23"; };

  outputs = { self,
              nixpkgs,
              flake-utils,
              cmake-examples-ex23-path
            } :
              let
                out = system :
                  let
                    pkgs = nixpkgs.legacyPackages.${system};
                    appliedOverlay = self.overlays.default pkgs pkgs;
                  in
                    {
                      # 1 line for each of our own packages
                      packages.cmake-examples-ex23 = appliedOverlay.cmake-examples-ex23;
                    };
              in
                flake-utils.lib.eachDefaultSystem out // {
                  overlays.default = final: prev: (
                    let
                      # configuration choices
                      boost = prev.boost182;                     # boost 1.82
                      python3Packages = prev.python311Packages;  # python 3.11

                      # configuration choices we're making here
                      extras = { boost = boost; python3Packages = python3Packages; };
                    in
                      {
                        cmake-examples-23 =
                          (prev.callPackage ./pkgs/ex23.nix { boost = boost;
                                                              python3Packages = python3Packages; })
                            .overrideAttrs(old: { src = cmake-examples-ex23-path; });
                      });
                };
  }

3.3. Explanation

Diving into some of the contents of these two .nix files:

3.3.1. List derivations to build

In flake.nix we have:

let
  pkgs = nixpkgs.legacyPackages.${system};
  appliedOverlay = self.overlays.default pkgs pkgs;
in
  {
    # 1 line for each of our own packages
    packages.cmake-examples-ex23 = appliedOverlay.cmake-examples-ex23;
  };

Here packages.cmake-examples-ex23 represents a derivation to be built by this flake.

3.3.2. Build for host architecture

flake-utils.lib.eachDefaultSystem out // ...

establishes builds compatible with host architecture (not cross-compiling). out is target architecture. The // substitutes attributes from its RHS, discussed below:

3.3.3. Overlay to choose configuration

#+begin-src overlays.default = final: prev: (….); #+end_src specifies an overlay, in "lazy-converging-to-a-fixpoint style" (which is apparently a thing).

When nix invokes overlays.default:

  • prev refers to a guess (at entire altered-nixpkgs-expression?) from prior iteration.
  • final refers to current iteration.

After first iteration, when prev and final are the same, nix:

  • recognizes that it has reached a fixpoint,
  • declares victory
  • uses the derivations from either of the converged arguments (since they're the same).

The iteration is initiated by nixpkgs for selected architecture twice (as prev and final):

pkgs = nixpkgs.legacyPackages.${system}
appliedOverlay = self.overlays.default pkgs pkgs;

We make configuration choices here:

boost = prev.boost182;
python3Packages = prev.python311Packages;

If the LHS names matches something in nixpkgs, then that line can be omitted to adopt whatever default nixpkgs offers.

For example:

$ nix-env -qaP | grep nixpkgs.boost
nixpkgs.boost175    boost-1.75.0
nixpkgs.boost177    boost-1.77.0
nixpkgs.boost178    boost-1.78.0
nixpkgs.boost       boost-1.79.0
nixpkgs.boost180    boost-1.80.0
nixpkgs.boost181    boost-1.81.0
nixpkgs.boost182    boost-1.82.0

So our example configuration chooses boost-1.82.0 instead of nixpkgs default boost-1.79.0

3.3.4. Delegate build instructions and respect flake-mediated pinning

We finally introduce a derivation for our own package (cmake-examples-23) here:

cmake-examples-23 =
  (prev.callPackage ./pkgs/ex23.nix { boost = boost;
                                      python3Packages = python3Packages; })
    .overrideAttrs(old: { src = cmake-examples-ex23-path; });

Here

{ boost = boost;
  python3Packages = python3Packages; }

introduces our overrides (we're using flake.nix for the approved purpose of making configuration choices) to arguments of the top-level function in ./pkgs/ex23.nix

Meanwhile

.overrideAttrs(old: { src = cmake-examples-ex23-path; })

tells nix to override the src attribute in ./pkgs/ex23.nix's argument to stdenv.mkDerivation

In other words, it substitutes for:

src = fetchGit {
  url = "https://github.com/rconybea/cmake-examples";
  ref = "ex23";     # branch
  # rev = "12345";  # hash
  sha256 = "";      # must supply if not overridden from parent
};

3.4. Use

Use result just like a regular flake

Verify flake:

$ cd cmake-examples-nix
$ nix flake check

Example flake.lock:

{
  "nodes": {
    "cmake-examples-ex23-path": {
      "flake": false,
      "locked": {
        "lastModified": 1709699013,
        "narHash": "sha256-Polpd2+DiZF615sih6HLOtzj0LaaFkQxrN6Jobdnq7M=",
        "owner": "Rconybea",
        "repo": "cmake-examples",
        "rev": "6d14f4146f2c7fc2011dd6790947ea261575304e",
        "type": "github"
      },
      "original": {
        "owner": "Rconybea",
        "ref": "ex23",
        "repo": "cmake-examples",
        "type": "github"
      }
    },
    "nixpkgs": {
      "locked": {
        "narHash": "sha256-mBXQ65IrCJbNgTrj0+6xdXpD9/U31AWPKdwGlOufhtI=",
        "type": "tarball",
        "url": "https://github.com/NixOS/nixpkgs/archive/4dd376f7943c64b522224a548d9cab5627b4d9d6.tar.gz"
      },
      "original": {
        "type": "tarball",
        "url": "https://github.com/NixOS/nixpkgs/archive/4dd376f7943c64b522224a548d9cab5627b4d9d6.tar.gz"
      }
    },
    ...
  },
  "root": "root",
  "version": 7
}

Build (from pinned revisions)

$ nix build -L --print-build-logs .#cmake-examples-ex23

Result

$ tree ./result
result
├── 3.11
│   ├── pyzstream.cpython-311-x86_64-linux-gnu.so
│   └── zstream.py
├── bin
│   ├── hello
│   └── myzip
├── include
│   ├── compression
│   │   ├── base_zstream.hpp
│   │   ├── buffer.hpp
│   │   ├── buffered_deflate_zstream.hpp
│   │   ├── buffered_inflate_zstream.hpp
│   │   ├── compression.hpp
│   │   ├── deflate_zstream.hpp
│   │   ├── hex.hpp
│   │   ├── inflate_zstream.hpp
│   │   ├── span.hpp
│   │   └── tostr.hpp
│   └── zstream
│       ├── xfilebuf.hpp
│       ├── zstream.hpp
│       └── zstreambuf.hpp
├── lib
│   ├── libcompression.so -> libcompression.so.2.3
│   ├── libcompression.so.2
│   └── libcompression.so.2.3 -> libcompression.so.2
└── share
    └── doc
        └── cmake-examples
            └── html
                 ...

3.5. Links

Resources

Author: Roland Conybeare

Created: 2024-09-08 Sun 18:01

Validate