Sneaky Nix Channels
How I found Nix 22.11 hiding on my system and the quest to remove it.TLDR
- Check your nix installation with
nix-info -m- If not using
nixpkgs.lib.nixosSystem, make sure to setnixpkgs.flake.sourcemanually usingnixpkgs.flake.source = self.inputs.nixpkgs.outPath;- Disable channels with
nix.channel.enable = false;- You may need to manually delete existing channels from your gcroots
Imagine my surprise when I found out that my nix environment was still running 22.11!
➜ nix-info -m
- system: `"x86_64-linux"`
- host os: `Linux 6.18.5, NixOS, 26.05 (Yarara), 26.05pre-git`
- multi-user?: `yes`
- sandbox: `yes`
- version: `nix-env (Nix) 2.31.2+2`
- channels(root): `"nixos-22.11"`
- nixpkgs: `/nix/store/111wqhhcjs7a08iy95by4gg4g8za4x1w-nixos-22.11/nixos`
Doing a lookup on that store path, I see it's used in the root user's channels:
➜ nix-store -q --referrers-closure /nix/store/111wqhhcjs7a08iy95by4gg4g8za4x1w-nixos-22.11/nixos
/nix/store/111wqhhcjs7a08iy95by4gg4g8za4x1w-nixos-22.11
/nix/store/skln9fzqj1bwhypgifkpbb29z29n0mmc-env-manifest.nix
/nix/store/kjlw7w1gpfp8ndw6gpgdqasg1i0lz9dr-user-environment.drv
/nix/store/rqglmxyw54qg9k6dzsx3r2pyvx24d4j5-user-environment
➜ echo $NIX_PATH
nixpkgs=/nix/var/nix/profiles/per-user/root/channels/nixos nixos-config=/etc/nixos/configuration.nix /nix/var/nix/profiles/per-user/root/channels
➜ cat /nix/var/nix/profiles/per-user/root/channels/manifest.nix
[ { meta = { }; name = "nixos-22.11"; out = { outPath = "/nix/store/111wqhhcjs7a08iy95by4gg4g8za4x1w-nixos-22.11"; }; outPath = "/nix/store/111wqhhcjs7a08iy95by4gg4g8za4x1w-nixos-22.11"; outputs = [ "out" ]; system = "builtin"; type = "derivation"; } ]
Yikes!
If I do some searching, it looks like the intention is for nixpkgs.flake.source to be set. Which, in turn, happens when you use nixpkgs.lib.nixosSystem. Problem is, nixpkgs.lib.nixosSystem isn't documented anywhere. I use colmena, I'm sure many people here use colmena or deploy.rs or bento or some other deployment system that doesn't necessarily use nixpkgs.lib.nixosSystem. So let's check if it's set. In the Nix repl I load up my system configuration for the flake:
{
__toString = «lambda __toString @ /nix/store/kfcxqcxb9hcq6x33sg4cmwakbb1ifwg9-source/lib/modules.nix:1126:20»;
_type = "option";
declarationPositions = [ ... ];
declarations = [ ... ];
default = null;
defaultText = "if (using nixpkgsFlake.lib.nixosSystem) then self.outPath else null";
definitions = [ ... ];
definitionsWithLocations = [ ... ];
description = "The path to the nixpkgs sources used to build the system. This is automatically set up to be\nthe store path of the nixpkgs flake used to build the system if using\n`nixpkgs.lib.nixosSystem`, and is otherwise null by default.\n\nThis can also be optionally set if the NixOS system is not built with a flake but still uses\npinned sources: set this to the store path for the nixpkgs sources used to build the system,\nas may be obtained by `fetchTarball`, for example.\n\nNote: the name of the store path must be \"source\" due to\n<https://github.com/NixOS/nix/issues/7075>.\n";
example = "fetchTarball { name = \"source\"; sha256 = \"sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=\"; url = \"https://github.com/nixos/nixpkgs/archive/somecommit.tar.gz\"; }";
files = [ ... ];
highestPrio = 100;
isDefined = true;
loc = [ ... ];
options = [ ... ];
type = { ... };
value = "/nix/store/kfcxqcxb9hcq6x33sg4cmwakbb1ifwg9-source";
valueMeta = { ... };
}
➜ cat /nix/store/kfcxqcxb9hcq6x33sg4cmwakbb1ifwg9-source/.version
26.05
That looks fine, what gives?! (off screen I confirmed setNixPath and setFlakeRegistry are also correct, both true)
I also confirmed my nix.nixPath was set correctly:
[
"nixpkgs=flake:nixpkgs"
"/nix/var/nix/profiles/per-user/root/channels"
]
but that clearly doesn't match my real $NIX_PATH.
I can fix this by sidestepping nixpkgs.flake and go straight to nix.nixPath, right?
nix.nixPath = [ "nixpkgs=/run/current-system/nixpkgs" ];
➜ nix-info -m
- system: `"x86_64-linux"`
- host os: `Linux 6.18.5, NixOS, 26.05 (Yarara), 26.05pre-git`
- multi-user?: `yes`
- sandbox: `yes`
- version: `nix-env (Nix) 2.31.2+2`
- channels(root): `"nixos-22.11"`
- nixpkgs: `/nix/store/111wqhhcjs7a08iy95by4gg4g8za4x1w-nixos-22.11/nixos`
Wrong!
➜ echo $NIX_PATH
nixpkgs=/run/current-system/nixpkgs
At least the nix path is set! So now my nix-info appears to be ignoring my $NIX_PATH, I'm not sure what's going on here. Let me run my magic garbage collection command (leave out devenv if you don't use devenv)
sudo nix-collect-garbage -d && nix-collect-garbage -d && devenv gc
And check nix info again:
➜ nix-info -m
- system: `"x86_64-linux"`
- host os: `Linux 6.18.5, NixOS, 26.05 (Yarara), 26.05pre-git`
- multi-user?: `no`
- sandbox: `yes`
- version: `nix-env (Nix) 2.31.2+2`
- channels(root): `"nixos-22.11"`
- nixpkgs: `not found`
Much better! But why are there still channels? Let's go back to nixpkgs.flake... Reading the source code, I see there's an additional nix.channel.enable setting. I certainly don't want channels, so let's set it to false:
nix.channel.enable = false;
Reboot and run GC again... Still no good. And now my $NIX_PATH is empty! I'd like to be able to still use nix-shell -p. One step at a time... My channel files still clearly exist in /nix/var/nix/profiles/per-user. Let's clean that up so nix can garbage collect it:
sudo rm -r /nix/var/nix/profiles/per-user/root/channels*
now run GC. And finally:
➜ nix-info -m
- system: `"x86_64-linux"`
- host os: `Linux 6.18.5, NixOS, 26.05 (Yarara), 26.05pre-git`
- multi-user?: `no`
- sandbox: `yes`
- version: `nix-env (Nix) 2.31.2+2`
- nixpkgs: `not found`
Looking good! But $NIX_PATH is now broken:
➜ echo $NIX_PATH
~
The docs for nixpkgs.flake.setFlakeRegisty says something interesting:
Whether to pin nixpkgs in the system-wide flake registry (/etc/nix/registry.json) to the store path of the sources of nixpkgs used to build the NixOS system.
Let's take a look.
{"flakes":[],"version":2}
Well there's the problem! Something tells me this shouldn't be empty!
Looking at the source code, it appears setFlakeRegisty will modify nix.registry.nixpkgs.to. Let's see what happens
nix-repl> nixosConfigurations.my-machine.config.nix.registry.nixpkgs
{
exact = true;
flake = null;
from = { ... };
to = { ... };
}
nix-repl> nixosConfigurations.my-machine.config.nix.registry.nixpkgs.to
{
path = "/nix/store/kfcxqcxb9hcq6x33sg4cmwakbb1ifwg9-source";
type = "path";
}
nix-repl> nixosConfigurations.my-machine.config.nix.registry.nixpkgs.from
{
id = "nixpkgs";
type = "indirect";
}
I don't think null is correct. Going to the description for nix.registry.from and clicking through to the sourcecode, I see this is how our file is generated:
config = mkIf cfg.enable {
environment.etc."nix/registry.json".text = builtins.toJSON {
version = 2;
flakes = mapAttrsToList (n: v: { inherit (v) from to exact; }) cfg.registry;
};
};
It actually looks like null doesn't matter, just a red herring.
Let's search the config in the repl directly:
nix-repl> nixosConfigurations.my-machine.config.environment.etc."nix/registry.json".text
"{\"flakes\":[{\"exact\":true,\"from\":{\"id\":\"nixpkgs\",\"type\":\"indirect\"},\"to\":{\"path\":\"/nix/store/kfcxqcxb9hcq6x33sg4cmwakbb1ifwg9-source\",\"type\":\"path\"}}],\"version\":2}"
At this point I'm starting to think I've been doing something wrong this whole time. Something I'm sure you've been screaming at me if you're paying close attention: I'm using Colmena, so I shouldn't be checking nixosConfigurations. Let's repl something else:
nix-repl> colmenaHive.nodes.my-machine.config.nixpkgs.flake.source
null
The culprit is found! colmena.lib.makeHive is not setting nixpkgs.flake.source, while nixosSystem does! I hang onto nixosConfigurations as a backup, but I primarily deploy with colmena. I was searching the wrong output the whole time!
nixpkgs.flake.source = self.inputs.nixpkgs.outPath;
And now:
{"flakes":[{"exact":true,"from":{"id":"nixpkgs","type":"indirect"},"to":{"path":"/nix/store/kfcxqcxb9hcq6x33sg4cmwakbb1ifwg9-source","type":"path"}}],"version":2}
This is good!
➜ echo $NIX_PATH
~
😭
I don't know what could be going wrong here...
When in doubt, reboot!
➜ echo $NIX_PATH
nixpkgs=flake nixpkgs
~
➜ nix-shell -p nix-info --run "nix-info -m"
- system: `"x86_64-linux"`
- host os: `Linux 6.18.5, NixOS, 26.05 (Yarara), 26.05pre-git`
- multi-user?: `yes`
- sandbox: `yes`
- version: `nix-env (Nix) 2.31.2+2`
- nixpkgs: `/nix/store/kfcxqcxb9hcq6x33sg4cmwakbb1ifwg9-source`
Finally!