NixOS USB Recovery Skill

NixOS USB Recovery

Use this skill when repairing an installed NixOS system from a NixOS USB/live environment. The goal is to recover the existing installation in place: mount the target system, inspect generations/configuration, rebuild or repair, then reboot safely.

When to use

The machine is booted into a NixOS installer/live USB for rescue work.
A NixOS generation, boot entry, activation script, or package change broke boot.
You need to build a new generation for the installed system without reinstalling.
You need to repair bootloader entries, profile symlinks, or /boot contents.

Safety principles

Do not reinstall unless the user explicitly asks.
Do not assume device names, pool names, hostnames, or users.
Prefer inspection before mutation: identify disks, filesystems, mountpoints, and boot entries first.
Keep a note of every mount and symlink you change so you can undo or explain it.
Use pinned flake refs or the user's requested source when rebuilding.
Treat secrets and SSH keys carefully; do not print private key contents.

Quick recovery flow

Confirm live environment and network access
```
hostname
ip addr
lsblk -f
findmnt
```
If working remotely, confirm SSH stays available before starting a long rebuild.
Identify the installed system

Inspect disks and filesystems without assuming names:
```
lsblk -f
sudo blkid
sudo zpool import 2>/dev/null || true
sudo btrfs filesystem show 2>/dev/null || true
```
Look for the installed root filesystem, Nix store filesystem, home datasets, and EFI System Partition.

Import or unlock storage if needed

ZFS example, replacing placeholders with discovered names:

sudo zpool import -N -R /mnt <pool>
sudo zfs list

LUKS example:

sudo cryptsetup open /dev/disk/by-uuid/<uuid> <name>

Mount the installed system under /mnt

Use discovered devices/datasets. Examples:

sudo mount /dev/disk/by-uuid/<root-uuid> /mnt
sudo mkdir -p /mnt/nix /mnt/boot /mnt/home
sudo mount /dev/disk/by-uuid/<nix-uuid> /mnt/nix        # if separate
sudo mount /dev/disk/by-uuid/<efi-uuid> /mnt/boot       # or /mnt/boot/efi

ZFS example:

sudo zfs mount <pool>/<root-dataset>
sudo mkdir -p /mnt/nix /mnt/boot
sudo zfs mount <pool>/<nix-dataset>
sudo mount /dev/disk/by-uuid/<efi-uuid> /mnt/boot

Verify before continuing:

findmnt -R /mnt
test -e /mnt/etc/NIXOS || echo "Warning: /mnt does not look like NixOS root"
test -d /mnt/nix/store || echo "Warning: Nix store is not mounted"

Inspect generations and boot entries

ls -l /mnt/nix/var/nix/profiles/system*
find /mnt/boot -maxdepth 3 -type f \( -name '*.conf' -o -name 'loader.conf' \) -print
cat /mnt/boot/loader/loader.conf 2>/dev/null || true
ls /mnt/boot/loader/entries 2>/dev/null || true

If the task is a rollback, update the boot default and system profile only after confirming the target generation exists.

Prepare for nixos-enter or rebuild

Ensure DNS works inside the installed root if the rebuild fetches inputs:
```
sudo rm -f /mnt/etc/resolv.conf
sudo cp -L /etc/resolv.conf /mnt/etc/resolv.conf
```
Enter the installed environment when activation scripts or installed paths matter:
```
sudo nixos-enter --root /mnt
```
Rebuild or repair

For a new generation from a flake:
```
sudo nixos-enter --root /mnt -c \
  'nixos-rebuild boot --refresh --flake <flake-ref>#<host>'
```
If private flake inputs are fetched over SSH, pass an installed deploy key or agent deliberately:
```
sudo nixos-enter --root /mnt -c \
  'env GIT_SSH_COMMAND="ssh -i /home/<user>/.ssh/<key> -o IdentitiesOnly=yes -o StrictHostKeyChecking=accept-new" \
   nixos-rebuild boot --refresh --flake <flake-ref>#<host>'
```
Use boot for rescue work when you want the new generation selected on next boot without switching live services in the USB environment. Use switch only when you intentionally want activation inside the entered system.

Verify the installed result

readlink -f /mnt/nix/var/nix/profiles/system
cat /mnt/boot/loader/loader.conf 2>/dev/null || true
ls -lt /mnt/boot/loader/entries 2>/dev/null | head
sudo nixos-enter --root /mnt -c 'nixos-version; systemctl --version | head -1'

If the repair was for a specific command or package, test it from nixos-enter:

sudo nixos-enter --root /mnt -c '<command> --help | head'

Unmount and reboot safely
```
sync
sudo umount -R /mnt || true
sudo zpool export <pool> 2>/dev/null || true
sudo reboot
```
If a pool export says it is busy, do not panic. Run findmnt -R /mnt, close shells using /mnt, run sync, and document the busy export if rebooting anyway.

Common repair patterns

Build a fixed generation instead of rolling back

Use this when the user has already fixed the flake/config and wants that fix deployed:

sudo nixos-enter --root /mnt -c \
  'nixos-rebuild boot --refresh --flake <flake-ref>#<host>'

Then verify the new generation number, boot default, and the fixed command/package.

Select a known-good generation

Use only when rollback is requested or rebuilding is not possible:

ls -l /mnt/nix/var/nix/profiles/system-*-link
sudo ln -sfn /nix/var/nix/profiles/system-<N>-link /mnt/nix/var/nix/profiles/system
sudo sed -i 's/^default .*/default nixos-generation-<N>.conf/' /mnt/boot/loader/loader.conf

Verify the entry exists before changing the default:

test -f /mnt/boot/loader/entries/nixos-generation-<N>.conf

Repair systemd-boot entries

If the installed profile is correct but boot entries are missing or stale, rebuild boot files from the installed environment:

sudo nixos-enter --root /mnt -c 'nixos-rebuild boot --install-bootloader --flake <flake-ref>#<host>'

Post-boot checks

After removing the USB and booting from disk, SSH into the installed system and check:

readlink /nix/var/nix/profiles/system
systemctl is-system-running || true
systemctl --failed --no-pager
systemctl is-active sshd tailscaled 2>/dev/null || true

If a service initially failed during boot but later recovered, use systemctl reset-failed only after confirming the service is now healthy.

Final report checklist

Summarize:

How the installed system was identified and mounted.
What generation or flake ref was deployed.
Any bootloader/profile changes made.
Verification performed before reboot and after boot.
Any remaining failed units or follow-up work.

Agent Skills: NixOS USB Recovery

Install this agent skill to your local

Skill Files