horcrux-be: /boot snapshot copies content-addressed, retention tied to ZFS snapshots and free space; never fails on a full /boot #19

Merged
CanuteTheGreat merged 3 commits from fix/be-boot-space into main 2026-11-05 18:55:53 -08:00
Owner

Incident 2026-11-05: hogshead /boot 97% (8 byte-identical 75 MB kernel copies in /boot/be/.snap, create-* never pruned, no space check).

  • hard-linked content store /boot/be/.objects (identical kernels free; atomic replace)
  • destroy-snapshot/prune remove the /boot copy with the ZFS snapshot; orphans cleaned; KEEP covers pre-upgrade-* and create-*
  • /boot kept <= BOOT_MAX_USED_PCT (80) by destroying the OLDEST automatic snapshots first (never user-named, clone origins, BEs)
  • a full /boot never fails a snapshot/upgrade: copy skipped with a message
  • tests: 12 unit tests (fake zfs) + real 512 MB ext4 loop test (30 snapshots, worst 61.6%) wired into ci-run.sh
  • test-machine allowlist: net-misc/ctg-ssh-ca (-r3)
Incident 2026-11-05: hogshead /boot 97% (8 byte-identical 75 MB kernel copies in /boot/be/.snap, create-* never pruned, no space check). - hard-linked content store /boot/be/.objects (identical kernels free; atomic replace) - destroy-snapshot/prune remove the /boot copy with the ZFS snapshot; orphans cleaned; KEEP covers pre-upgrade-* and create-* - /boot kept <= BOOT_MAX_USED_PCT (80) by destroying the OLDEST automatic snapshots first (never user-named, clone origins, BEs) - a full /boot never fails a snapshot/upgrade: copy skipped with a message - tests: 12 unit tests (fake zfs) + real 512 MB ext4 loop test (30 snapshots, worst 61.6%) wired into ci-run.sh - test-machine allowlist: net-misc/ctg-ssh-ca (-r3)
Every BE snapshot copied the whole kernel set (~75 MB) into /boot/be/.snap/<be>@<snap>, identical kernels included, with no
space check; create-* snapshots were never pruned. A lab laptop's 2 GB /boot hit 97% (Rustinion critical).

- /boot/be/.objects/<sha256> content store; every kernel/initramfs under be/ is a hard link to it (copy when links are
  unavailable): identical kernels cost nothing, an object nobody links to is deleted (st_nlink==1), replacement is atomic
  so a new BE kernel never writes through to a snapshot's link
- destroy-snapshot / prune remove the /boot copy with the ZFS snapshot; copies of snapshots destroyed elsewhere are removed
  (unless ZFS cannot be queried); KEEP now covers pre-upgrade-* and create-* (never user-named, never a clone's origin)
- before saving a kernel, /boot is kept under BOOT_MAX_USED_PCT (80) / BOOT_MIN_FREE_MB (32): oldest automatic snapshots are
  destroyed first; if even that cannot make room the copy is skipped with a message
- a full /boot never fails a snapshot, an auto-snapshot or a kernel sync: the ZFS snapshot stands
- horcrux-be dedupe converts old plain copies; tests: unit (fake zfs) + real 512 MB ext4 loop test, 30 snapshots <= 80%
test machines: allow net-misc/ctg-ssh-ca (host SSH CA trust, same family as -fleet) in world.allowlist (-r3); hx-invariants kept failing on it
Some checks failed
CI / Mobile theme (light/dark) (pull_request) Has been cancelled
CI / Check (pull_request) Has been cancelled
CI / Test (pull_request) Has been cancelled
CI / Build (pull_request) Has been cancelled
CI / Build-1 (pull_request) Has been cancelled
CI / Build Web UI (pull_request) Has been cancelled
CI / Security Audit (pull_request) Has been cancelled
CI / Gentoo Docker Build (pull_request) Has been cancelled
CI / USE-flag feature builds (pull_request) Has been cancelled
CI / UI E2E (browser) (pull_request) Has been cancelled
CI / Gentoo USE-flag matrix (ebuild install) (pull_request) Has been cancelled
CI / Web UI theme (light/dark) (pull_request) Has been cancelled
5709f4f9e0
horcrux-be: a repeated sync/snapshot of an unchanged kernel left *.new temp links (rename of two links to one inode is a no-op); test
All checks were successful
CI / Mobile theme (light/dark) (pull_request) Successful in 4s
CI / Check (pull_request) Successful in 1m37s
CI / Web UI theme (light/dark) (pull_request) Successful in 4s
CI / Build Web UI (pull_request) Successful in 2m18s
CI / Security Audit (pull_request) Successful in 2m44s
CI / USE-flag feature builds (pull_request) Successful in 13m29s
CI / UI E2E (browser) (pull_request) Successful in 9m31s
CI / Gentoo Docker Build (pull_request) Successful in 23m55s
CI / Test (pull_request) Successful in 35m12s
CI / Build-1 (pull_request) Successful in 4m34s
CI / Build (pull_request) Successful in 6m33s
CI / Gentoo USE-flag matrix (ebuild install) (pull_request) Successful in 55m17s
063c104d7f
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
CanuteTheGreat/horcrux!19
No description provided.