Announcing fc-userscan

They link them against libraries they have installed before by nix-env. This works well... until something is updated! On the next change anywhere down the dependency chain, libraries get new hashes in the Nix store, the garbage collector removes old versions, and user applications break until recompiled. In this blog post, I would like to introduce fc-userscan. This little tool scans (home) directories recursively for Nix store references and registers them as per-user roots with the garbage collector. This way, dependencies will be protected even if they cease to be referenced from "official" Nix roots like the current-system profile or a user's local Nix profile. After registering formerly unmanaged references with fc-userscan, one can fearlessly run updates and garbage collection. This problem would not be there if everyone would just write Nix expressions for everything. This is, for various reasons, not realistic. Some users find it hard to accommodate with Nix concepts ("I just want to compile my software and don't care about your arcane Linux distro!"). Others use deploy tools like zc.buildout which are explicitly designed for in-place updates and don't really distinguish compile time from run time. With fc-userscan, you can get both: Nix expressions for superb reproducibility, and an acceptable degree of reliance for impurely compiled applications. fc-userscan is published as open source software under a 3-clause BSD license. Get the Rust source code or the latest pre-compiled binaries for x86_64. Pull requests are welcome.

Example: How to get stale Nix store references

Imagine we have Python installed in our Nix user profile. When creating a Python virtual environment, we get unmanaged references into the Nix store:

$ nix-env -iA nixpkgs.python35
$ ~/.nix-profile/bin/pyvenv venv
$ ls -og venv/bin/python3.5
lrwxrwxrwx 1 71 Sep 29 11:19 venv/bin/python3.5 -> /nix/store/8lh4pxhhi4cx00pp1zxpz9pqyy44kjm6-python3-3.5.4/bin/python3.5

The garbage collector does not know anything about this reference. This is OK as long as nothing gets deleted from the Nix store. But eventually channel updates arrive. After an seemingly innocent run of nixos-rebuild or nix-env --upgrade, chances are high that the Python package's hash has changed. The next run of nix-collect-garbage will trash the Python binary still referenced from our virtualenv. Although we use Python for illustration, the problem is universal. You can easily run into similar trouble with C programs, for example.

Protecting unmanaged references

We run fc-userscan to detect and register Nix store references:

$ fc-userscan -v venv
fc-userscan: Scouting venv
2 references in /nix/var/nix/gcroots/profiles/per-user/ck/home/ck/venv
Processed 741 files (8 MB read) in 0.011 s
fc-userscan: Finished venv

Now the exact Python interpreter version is registered as a garbage collection root:

$ ls -og /nix/var/nix/gcroots/profiles/per-user/ck/home/ck/venv
lrwxrwxrwx 1 57 Sep 29 11:46 8lh4pxhhi4cx00pp1zxpz9pqyy44kjm6 -> /nix/store/8lh4pxhhi4cx00pp1zxpz9pqyy44kjm6-python3-3.5.4
[...]

The Python interpreter and all its dependencies are now protected. The file layout in the per-user directories is organized in such a way that is possible to re-scan arbitrary directories and always get correct results.

Feature overview

Include/exclude lists

To reduce system load, fc-userscan can be instructed to skip files that are unlikely to contain store references. Specify include/exclude globs from

1. a default ignore file ~/.userscan-ignore if existent (in gitignore(5) format)
2. a custom ignore file with -­-exclude-from (in gitignore(5) format)
3. the command line with -­-exclude, -­-include.

A related option is -­-quickcheck which causes fc-userscan to stop scanning a file when there is no Nix store reference in the first n bytes of that file.

Cache

fc-userscan is able to preserve scan results between runs to avoid re-scanning unchanged files. Simply add -­-cache FILE. It uses the ctime inode attribute to determine if a file has been changed or not. The cache is stored as compressed messagepack file so that it does not take much space even on large installations.

Unzip on the fly

Some runnable application artefacts are stored as ZIP files and may contain Nix store references. The most prominent instances are compressed Python eggs and JAR files. fc-userscan has support to transparently decompress files that match glob patterns passed with -­-unzip. Note that scanning inside compressed ZIP archives is significantly slower than scanning regular files.

List mode

It is possible to scan directory hierarchies and just print all found references to stdout instead of registering them with -­-list. The output format is script friendly. These options and many more are explained in the fc-userscan(1) man page which ships with the release.

Interaction with nix-collect-garbage

How does fc-userscan interact with nix-collect-garbage now? We at Flying Circus run fc-userscan for all users and start nix-collect-garbage directly afterwards. fc-userscan exits unsuccessfully if there was a problem (e.g., unreadable files). We recommend not to run nix-collect-garbage if fc-userscan found a problem. Otherwise, not all relevant files might have been checked for Nix store references and nix-collect-garbage might delete too much. Here is an example script which scans all users' home directories:

failed=0
while read user home; do
  sudo -u $user -H -- \
    fc-userscan -v 2 \
    --exclude-from /etc/userscan.exclude \
    --cache $home/.cache/fc-userscan.cache \
    --unzip '*.egg'
    $home || failed=1
done < <(getent passwd | awk -F: '$4 >= 100 { print $1 " " $6 }')

if (( failed )); then
  echo "ERROR: fc-userscan failed (see above)"
  exit 1
else
  nix-collect-garbage --delete-older-than 3d
fi

Conclusion

While in theory it would be better to drive all software installations on a NixOS system via derivations, fc-userscan brings us flexibility to protect impurely installed applications.