Building your own project with nix
Table of Contents
1. Motivation
I've found plenty of getting-started guides for working with nix. Most take the perpective of "a reproducible apt-get": they cover how to use nix to assemble/modify external software packages.
Less common is instruction on how best to package local (non-nixpkgs) software, in particular if you have multiple packages with non-trivial interdependencies.
The sequel summarizes my current understanding of how best to make this work (thanks to Jade Lovelace's article on flakes and nixpkgs, link below)
2. Lessons
2.1. Keep .nix files and project source in separate repos
On the face of it, it seems attractive and natural to keep nix files for a project
in the same repo! That's what we do with build instructions (CMakeLists.txt
etc)
2.1.1. Why not
.nix
files (default.nix
,shell.nix
etc) likely need to contain revision hashes. Want to avoid situation where you've modifieddefault.nix
, and need it to record a hash that wil be determined… after you've committed the file to git- You might want to incorporate a repo child into another repo parent as a submodule.
The
.nix
files associated with child probably don't do what you want in that context. - Even stronger reason to avoid this: nix flakes. Flakes will break
(at least as of Dec 2023) if a directory that contains a
flake.lock
file contains another directory with aflake.lock
file. The flake system can't exit from a state in which directory tree is dirty. This nixes composing project trees that contain flakes. - The
nixpkgs
authors have reached the same conclusion. Innixpkgs
, instructions for building a package are expressed as a function, to be invoked fromcallPackage
. Thenix-pkgs
provides for features like parameterization and cross-compilation; tradeoff is that they then don't work directly from 'nix-build'
2.1.2. Do this instead
For a project foo
, create a foo-nix
repo that uses fetchurl
or cousins to
fetch and build foo
.
2.2. Setup your build the nixpkgs way
In other words, use callPackage
!
callPackage
supports cross-compilation (out of the box)callPackage
supports parameterization (i.e. overrides)- If you already have nix build setup for nixpkgs, then there's nothing new to do if you want to upload into nixpkgs (or incorporate into your fork thereof) later
2.3. Don't put build instructions in flakes
Do use flakes! They're great for version pinning and specifying configurations
However, if you put build instructions in flakes (i.e. call outside callPackage
), then
you lose the ability to easily compose and adapt that the nixpkgs
tools give you.
For packaging our own software we will use flake to do these things (and nothing else):
- pin nixpkgs.
- fetch repos for local projects (things we aren't getting from nixpkgs)
- specifying dependendency sets for local projects.
In particular we exclude from flake:
stdenv.mkDerivation
calls- specifying
buildInputs
or cousins
3. Example Flake Setup
Here's an example, following advice above
This example condensed from cmake-examples-nix.
We have a local project cmake-examples
, with a bunch of nixpkgs
dependencies.
We'll setup a nix build in cmake-examples-nix
.
Will need just two files:
flake.nix
, a nix flakepkgs/ex23.nix
, build instructions as innixpgks
.
3.1. File cmake-examples-nix/pkgs/ex23.nix
:
{ # dependencies stdenv, doxygen, cmake, catch2, pkg-config, python3Packages, boost, zlib, # ... other deps here # args # someconfigurationoption ? false pybind11 ? python3Packages.pybind11, sphinx ? python3Packages.sphinx, breathe ? python3Packages.breathe, } : stdenv.mkDerivation (finalattrs: { name = "cmake-examples-ex23"; # note: ../flake.nix will override this src = fetchGit { url = "https://github.com/rconybea/cmake-examples"; ref = "ex23"; # branch # rev = "12345"; # hash sha256 = ""; # must supply if not overridden from parent }; # run unit tests doCheck = true; buildInputs = [ cmake pkg-config pybind11 sphinx doxygen breathe catch2 boost zlib ]; buildPhase = '' make make doxygen ''; })
3.2. File cmake-examples-nix/flake.nix
:
{ description = "Flake for cmake-examples project"; # pinning nixpkgs inputs.nixpkgs.url = "https://github.com/NixOS/nixpkgs/archive/4dd376f7943c64b522224a548d9cab5627b4d9d6.tar.gz"; # inputs.nixpkgs.url = "github:nixos/nixpkgs/23.11"; # to use particular stable version inputs.flake-utils.url = "github:numtide/flake-utils"; # branch 'ex23' from cmake-examples inputs.cmake-examples-ex23-path = { type = "github"; owner = "Rconybea"; repo = "cmake-examples"; flake = false; ref = "ex23"; }; outputs = { self, nixpkgs, flake-utils, cmake-examples-ex23-path } : let out = system : let pkgs = nixpkgs.legacyPackages.${system}; appliedOverlay = self.overlays.default pkgs pkgs; in { # 1 line for each of our own packages packages.cmake-examples-ex23 = appliedOverlay.cmake-examples-ex23; }; in flake-utils.lib.eachDefaultSystem out // { overlays.default = final: prev: ( let # configuration choices boost = prev.boost182; # boost 1.82 python3Packages = prev.python311Packages; # python 3.11 # configuration choices we're making here extras = { boost = boost; python3Packages = python3Packages; }; in { cmake-examples-23 = (prev.callPackage ./pkgs/ex23.nix { boost = boost; python3Packages = python3Packages; }) .overrideAttrs(old: { src = cmake-examples-ex23-path; }); }); }; }
3.3. Explanation
Diving into some of the contents of these two .nix
files:
3.3.1. List derivations to build
In flake.nix
we have:
let pkgs = nixpkgs.legacyPackages.${system}; appliedOverlay = self.overlays.default pkgs pkgs; in { # 1 line for each of our own packages packages.cmake-examples-ex23 = appliedOverlay.cmake-examples-ex23; };
Here packages.cmake-examples-ex23
represents a derivation to be built by this flake.
3.3.2. Build for host architecture
flake-utils.lib.eachDefaultSystem out // ...
establishes builds compatible with host architecture (not cross-compiling).
out
is target architecture. The //
substitutes attributes from its RHS,
discussed below:
3.3.3. Overlay to choose configuration
#+begin-src overlays.default = final: prev: (….); #+end_src specifies an overlay, in "lazy-converging-to-a-fixpoint style" (which is apparently a thing).
When nix invokes overlays.default
:
prev
refers to a guess (at entire altered-nixpkgs-expression?) from prior iteration.final
refers to current iteration.
After first iteration, when prev
and final
are the same, nix:
- recognizes that it has reached a fixpoint,
- declares victory
- uses the derivations from either of the converged arguments (since they're the same).
The iteration is initiated by nixpkgs for selected architecture twice (as prev
and final
):
pkgs = nixpkgs.legacyPackages.${system} appliedOverlay = self.overlays.default pkgs pkgs;
We make configuration choices here:
boost = prev.boost182; python3Packages = prev.python311Packages;
If the LHS names matches something in nixpkgs
, then that line can be omitted to adopt
whatever default nixpkgs
offers.
For example:
$ nix-env -qaP | grep nixpkgs.boost nixpkgs.boost175 boost-1.75.0 nixpkgs.boost177 boost-1.77.0 nixpkgs.boost178 boost-1.78.0 nixpkgs.boost boost-1.79.0 nixpkgs.boost180 boost-1.80.0 nixpkgs.boost181 boost-1.81.0 nixpkgs.boost182 boost-1.82.0
So our example configuration chooses boost-1.82.0
instead of nixpkgs default boost-1.79.0
3.3.4. Delegate build instructions and respect flake-mediated pinning
We finally introduce a derivation for our own package (cmake-examples-23
) here:
cmake-examples-23 = (prev.callPackage ./pkgs/ex23.nix { boost = boost; python3Packages = python3Packages; }) .overrideAttrs(old: { src = cmake-examples-ex23-path; });
Here
{ boost = boost; python3Packages = python3Packages; }
introduces our overrides (we're using flake.nix
for the approved purpose of making configuration choices)
to arguments of the top-level function in ./pkgs/ex23.nix
Meanwhile
.overrideAttrs(old: { src = cmake-examples-ex23-path; })
tells nix to override the src
attribute in ./pkgs/ex23.nix
's argument to stdenv.mkDerivation
In other words, it substitutes for:
src = fetchGit { url = "https://github.com/rconybea/cmake-examples"; ref = "ex23"; # branch # rev = "12345"; # hash sha256 = ""; # must supply if not overridden from parent };
3.4. Use
Use result just like a regular flake
Verify flake:
$ cd cmake-examples-nix $ nix flake check
Example flake.lock
:
{ "nodes": { "cmake-examples-ex23-path": { "flake": false, "locked": { "lastModified": 1709699013, "narHash": "sha256-Polpd2+DiZF615sih6HLOtzj0LaaFkQxrN6Jobdnq7M=", "owner": "Rconybea", "repo": "cmake-examples", "rev": "6d14f4146f2c7fc2011dd6790947ea261575304e", "type": "github" }, "original": { "owner": "Rconybea", "ref": "ex23", "repo": "cmake-examples", "type": "github" } }, "nixpkgs": { "locked": { "narHash": "sha256-mBXQ65IrCJbNgTrj0+6xdXpD9/U31AWPKdwGlOufhtI=", "type": "tarball", "url": "https://github.com/NixOS/nixpkgs/archive/4dd376f7943c64b522224a548d9cab5627b4d9d6.tar.gz" }, "original": { "type": "tarball", "url": "https://github.com/NixOS/nixpkgs/archive/4dd376f7943c64b522224a548d9cab5627b4d9d6.tar.gz" } }, ... }, "root": "root", "version": 7 }
Build (from pinned revisions)
$ nix build -L --print-build-logs .#cmake-examples-ex23
Result
$ tree ./result result ├── 3.11 │ ├── pyzstream.cpython-311-x86_64-linux-gnu.so │ └── zstream.py ├── bin │ ├── hello │ └── myzip ├── include │ ├── compression │ │ ├── base_zstream.hpp │ │ ├── buffer.hpp │ │ ├── buffered_deflate_zstream.hpp │ │ ├── buffered_inflate_zstream.hpp │ │ ├── compression.hpp │ │ ├── deflate_zstream.hpp │ │ ├── hex.hpp │ │ ├── inflate_zstream.hpp │ │ ├── span.hpp │ │ └── tostr.hpp │ └── zstream │ ├── xfilebuf.hpp │ ├── zstream.hpp │ └── zstreambuf.hpp ├── lib │ ├── libcompression.so -> libcompression.so.2.3 │ ├── libcompression.so.2 │ └── libcompression.so.2.3 -> libcompression.so.2 └── share └── doc └── cmake-examples └── html ...
3.5. Links
Resources
- https://jade.fyi/blog/flakes-arent-real/ wonderful Jade Lovelace blog – inspiration for this article!
- https://github.com/vlktomas/nix-examples Tomas Vlk nix examples
- https://nixos.org/guides/nix-pills/ Nix pills (nix tutorial, from the ground up)
- https://ianthehenry.com/posts/how-to-learn-nix/ Ian Henry's "nix travel diary"
- https://ryantm.github.io/nixpkgs/stdenv/stdenv Nix standard environment docs
- https://github.com/Rconybea/cmake-examples my progressive series of cmake examples, using as example of a local project
- https://github.com/Rconybea/cmake-examples-nix nix build for cmake-examples