Building your own project with nix
Table of Contents
1. Motivation
I've found plenty of getting-started guides for working with nix. Most take the perpective of "a reproducible apt-get": they cover how to use nix to assemble/modify external software packages.
Less common is instruction on how best to package local (non-nixpkgs) software, in particular if you have multiple packages with non-trivial interdependencies.
The sequel summarizes my current understanding of how best to make this work (thanks to Jade Lovelace's article on flakes and nixpkgs, link below)
2. Lessons
2.1. Keep .nix files and project source in separate repos
On the face of it, it seems attractive and natural to keep nix files for a project
in the same repo! That's what we do with build instructions (CMakeLists.txt etc)
2.1.1. Why not
.nixfiles (default.nix,shell.nixetc) likely need to contain revision hashes. Want to avoid situation where you've modifieddefault.nix, and need it to record a hash that wil be determined… after you've committed the file to git- You might want to incorporate a repo child into another repo parent as a submodule.
The
.nixfiles associated with child probably don't do what you want in that context. - Even stronger reason to avoid this: nix flakes. Flakes will break
(at least as of Dec 2023) if a directory that contains a
flake.lockfile contains another directory with aflake.lockfile. The flake system can't exit from a state in which directory tree is dirty. This nixes composing project trees that contain flakes. - The
nixpkgsauthors have reached the same conclusion. Innixpkgs, instructions for building a package are expressed as a function, to be invoked fromcallPackage. Thenix-pkgsprovides for features like parameterization and cross-compilation; tradeoff is that they then don't work directly from 'nix-build'
2.1.2. Do this instead
For a project foo, create a foo-nix repo that uses fetchurl or cousins to
fetch and build foo.
2.2. Setup your build the nixpkgs way
In other words, use callPackage!
callPackagesupports cross-compilation (out of the box)callPackagesupports parameterization (i.e. overrides)- If you already have nix build setup for nixpkgs, then there's nothing new to do if you want to upload into nixpkgs (or incorporate into your fork thereof) later
2.3. Don't put build instructions in flakes
Do use flakes! They're great for version pinning and specifying configurations
However, if you put build instructions in flakes (i.e. call outside callPackage), then
you lose the ability to easily compose and adapt that the nixpkgs tools give you.
For packaging our own software we will use flake to do these things (and nothing else):
- pin nixpkgs.
- fetch repos for local projects (things we aren't getting from nixpkgs)
- specifying dependendency sets for local projects.
In particular we exclude from flake:
stdenv.mkDerivationcalls- specifying
buildInputsor cousins
3. Example Flake Setup
Here's an example, following advice above
This example condensed from cmake-examples-nix.
We have a local project cmake-examples, with a bunch of nixpkgs dependencies.
We'll setup a nix build in cmake-examples-nix.
Will need just two files:
flake.nix, a nix flakepkgs/ex23.nix, build instructions as innixpgks.
3.1. File cmake-examples-nix/pkgs/ex23.nix:
{
# dependencies
stdenv, doxygen, cmake, catch2, pkg-config, python3Packages, boost, zlib, # ... other deps here
# args
# someconfigurationoption ? false
pybind11 ? python3Packages.pybind11,
sphinx ? python3Packages.sphinx,
breathe ? python3Packages.breathe,
} :
stdenv.mkDerivation (finalattrs:
{
name = "cmake-examples-ex23";
# note: ../flake.nix will override this
src = fetchGit {
url = "https://github.com/rconybea/cmake-examples";
ref = "ex23"; # branch
# rev = "12345"; # hash
sha256 = ""; # must supply if not overridden from parent
};
# run unit tests
doCheck = true;
buildInputs = [ cmake pkg-config pybind11 sphinx doxygen breathe catch2 boost zlib ];
buildPhase = ''
make
make doxygen
'';
})
3.2. File cmake-examples-nix/flake.nix:
{
description = "Flake for cmake-examples project";
# pinning nixpkgs
inputs.nixpkgs.url = "https://github.com/NixOS/nixpkgs/archive/4dd376f7943c64b522224a548d9cab5627b4d9d6.tar.gz";
# inputs.nixpkgs.url = "github:nixos/nixpkgs/23.11"; # to use particular stable version
inputs.flake-utils.url = "github:numtide/flake-utils";
# branch 'ex23' from cmake-examples
inputs.cmake-examples-ex23-path = { type = "github"; owner = "Rconybea"; repo = "cmake-examples"; flake = false; ref = "ex23"; };
outputs = { self,
nixpkgs,
flake-utils,
cmake-examples-ex23-path
} :
let
out = system :
let
pkgs = nixpkgs.legacyPackages.${system};
appliedOverlay = self.overlays.default pkgs pkgs;
in
{
# 1 line for each of our own packages
packages.cmake-examples-ex23 = appliedOverlay.cmake-examples-ex23;
};
in
flake-utils.lib.eachDefaultSystem out // {
overlays.default = final: prev: (
let
# configuration choices
boost = prev.boost182; # boost 1.82
python3Packages = prev.python311Packages; # python 3.11
# configuration choices we're making here
extras = { boost = boost; python3Packages = python3Packages; };
in
{
cmake-examples-23 =
(prev.callPackage ./pkgs/ex23.nix { boost = boost;
python3Packages = python3Packages; })
.overrideAttrs(old: { src = cmake-examples-ex23-path; });
});
};
}
3.3. Explanation
Diving into some of the contents of these two .nix files:
3.3.1. List derivations to build
In flake.nix we have:
let pkgs = nixpkgs.legacyPackages.${system}; appliedOverlay = self.overlays.default pkgs pkgs; in { # 1 line for each of our own packages packages.cmake-examples-ex23 = appliedOverlay.cmake-examples-ex23; };
Here packages.cmake-examples-ex23 represents a derivation to be built by this flake.
3.3.2. Build for host architecture
flake-utils.lib.eachDefaultSystem out // ...
establishes builds compatible with host architecture (not cross-compiling).
out is target architecture. The // substitutes attributes from its RHS,
discussed below:
3.3.3. Overlay to choose configuration
#+begin-src overlays.default = final: prev: (….); #+end_src specifies an overlay, in "lazy-converging-to-a-fixpoint style" (which is apparently a thing).
When nix invokes overlays.default:
prevrefers to a guess (at entire altered-nixpkgs-expression?) from prior iteration.finalrefers to current iteration.
After first iteration, when prev and final are the same, nix:
- recognizes that it has reached a fixpoint,
- declares victory
- uses the derivations from either of the converged arguments (since they're the same).
The iteration is initiated by nixpkgs for selected architecture twice (as prev and final):
pkgs = nixpkgs.legacyPackages.${system} appliedOverlay = self.overlays.default pkgs pkgs;
We make configuration choices here:
boost = prev.boost182; python3Packages = prev.python311Packages;
If the LHS names matches something in nixpkgs, then that line can be omitted to adopt
whatever default nixpkgs offers.
For example:
$ nix-env -qaP | grep nixpkgs.boost nixpkgs.boost175 boost-1.75.0 nixpkgs.boost177 boost-1.77.0 nixpkgs.boost178 boost-1.78.0 nixpkgs.boost boost-1.79.0 nixpkgs.boost180 boost-1.80.0 nixpkgs.boost181 boost-1.81.0 nixpkgs.boost182 boost-1.82.0
So our example configuration chooses boost-1.82.0 instead of nixpkgs default boost-1.79.0
3.3.4. Delegate build instructions and respect flake-mediated pinning
We finally introduce a derivation for our own package (cmake-examples-23) here:
cmake-examples-23 = (prev.callPackage ./pkgs/ex23.nix { boost = boost; python3Packages = python3Packages; }) .overrideAttrs(old: { src = cmake-examples-ex23-path; });
Here
{ boost = boost;
python3Packages = python3Packages; }
introduces our overrides (we're using flake.nix for the approved purpose of making configuration choices)
to arguments of the top-level function in ./pkgs/ex23.nix
Meanwhile
.overrideAttrs(old: { src = cmake-examples-ex23-path; })
tells nix to override the src attribute in ./pkgs/ex23.nix's argument to stdenv.mkDerivation
In other words, it substitutes for:
src = fetchGit { url = "https://github.com/rconybea/cmake-examples"; ref = "ex23"; # branch # rev = "12345"; # hash sha256 = ""; # must supply if not overridden from parent };
3.4. Use
Use result just like a regular flake
Verify flake:
$ cd cmake-examples-nix $ nix flake check
Example flake.lock:
{
"nodes": {
"cmake-examples-ex23-path": {
"flake": false,
"locked": {
"lastModified": 1709699013,
"narHash": "sha256-Polpd2+DiZF615sih6HLOtzj0LaaFkQxrN6Jobdnq7M=",
"owner": "Rconybea",
"repo": "cmake-examples",
"rev": "6d14f4146f2c7fc2011dd6790947ea261575304e",
"type": "github"
},
"original": {
"owner": "Rconybea",
"ref": "ex23",
"repo": "cmake-examples",
"type": "github"
}
},
"nixpkgs": {
"locked": {
"narHash": "sha256-mBXQ65IrCJbNgTrj0+6xdXpD9/U31AWPKdwGlOufhtI=",
"type": "tarball",
"url": "https://github.com/NixOS/nixpkgs/archive/4dd376f7943c64b522224a548d9cab5627b4d9d6.tar.gz"
},
"original": {
"type": "tarball",
"url": "https://github.com/NixOS/nixpkgs/archive/4dd376f7943c64b522224a548d9cab5627b4d9d6.tar.gz"
}
},
...
},
"root": "root",
"version": 7
}
Build (from pinned revisions)
$ nix build -L --print-build-logs .#cmake-examples-ex23
Result
$ tree ./result
result
├── 3.11
│ ├── pyzstream.cpython-311-x86_64-linux-gnu.so
│ └── zstream.py
├── bin
│ ├── hello
│ └── myzip
├── include
│ ├── compression
│ │ ├── base_zstream.hpp
│ │ ├── buffer.hpp
│ │ ├── buffered_deflate_zstream.hpp
│ │ ├── buffered_inflate_zstream.hpp
│ │ ├── compression.hpp
│ │ ├── deflate_zstream.hpp
│ │ ├── hex.hpp
│ │ ├── inflate_zstream.hpp
│ │ ├── span.hpp
│ │ └── tostr.hpp
│ └── zstream
│ ├── xfilebuf.hpp
│ ├── zstream.hpp
│ └── zstreambuf.hpp
├── lib
│ ├── libcompression.so -> libcompression.so.2.3
│ ├── libcompression.so.2
│ └── libcompression.so.2.3 -> libcompression.so.2
└── share
└── doc
└── cmake-examples
└── html
...
3.5. Links
Resources
- https://jade.fyi/blog/flakes-arent-real/ wonderful Jade Lovelace blog – inspiration for this article!
- https://github.com/vlktomas/nix-examples Tomas Vlk nix examples
- https://nixos.org/guides/nix-pills/ Nix pills (nix tutorial, from the ground up)
- https://ianthehenry.com/posts/how-to-learn-nix/ Ian Henry's "nix travel diary"
- https://ryantm.github.io/nixpkgs/stdenv/stdenv Nix standard environment docs
- https://github.com/Rconybea/cmake-examples my progressive series of cmake examples, using as example of a local project
- https://github.com/Rconybea/cmake-examples-nix nix build for cmake-examples