Sol 98 Storage Resilience

Architecture

The live flattened VM win98-stable-2026-07-19 runs from the internal NVMe:

/home/david/VirtualBox VMs/win98-stable-2026-07-19

The registered snapshot-chain archive win98 and recovery backups remain on
the external Sol data SSD under /mnt/sol-data. A USB disconnect therefore
removes backup availability but not the live guest's writable disk.

The user service sol98-storage-guard.service checks every five seconds that:

• the registered configuration and active VDI exist;
• the VDI is still on the internal root filesystem;
• the filesystem is writable and has at least 12 GiB available;
• a periodic write-and-fsync probe succeeds.

If live storage becomes unsafe while the VM is running, the guard pauses the
VM and sends a desktop notification. It does not auto-resume. Loss of the
external backup SSD is logged but does not pause the internally hosted VM.

When the VM is powered off or saved and the external SSD is mounted writable,
the guard creates a quiescent rsync backup. It rechecks the VM state and source
fingerprint after copying before promoting a backup, keeps three generations,
and atomically updates the latest symlink.

The udev rule /etc/udev/rules.d/80-sol-sandisk-no-autosuspend.rules forces
USB runtime power control to on for SanDisk device 0781:558c. This reduces
power-management disconnect risk but cannot compensate for a bad cable,
physical removal, enclosure reset, or device failure.

Operations

bash
systemctl --user status sol98-storage-guard.service
python3 ~/.local/lib/sol98/sol98_storage_guard.py --status
python3 ~/.local/lib/sol98/sol98_storage_guard.py --once
python3 ~/.local/lib/sol98/sol98_storage_guard.py --backup-now

--backup-now only runs while the VM is powered off or saved. Never copy a
running VDI as a recovery backup.

After a guard-triggered pause, inspect the state file and storage first:

bash
cat ~/.local/state/sol98-storage-guard/state.json
VBoxManage showvminfo win98-stable-2026-07-19 --machinereadable
findmnt -T "$HOME/VirtualBox VMs/win98-stable-2026-07-19"

Resume only after the recorded hazard is corrected and --once passes.

2026-07-21 Incident

At 01:53:59 CDT the SanDisk Extreme SSD disconnected from USB. Linux returned
write I/O errors for sdf, aborted the ext4 journal on sdf1, and remounted
the stale filesystem read-only. VirtualBox ATA-0 writes then failed and
Windows 98 displayed Disk Write Error for drive C:.

Recovery powered off the failed VM without saving state, ran offline e2fsck,
remounted the SSD, created a SHA-256-verified recovery copy, moved the active VM
to internal NVMe storage with VBoxManage movevm, and allowed Windows ScanDisk
to complete. The guest then reached its desktop and private network without new
VirtualBox storage errors.