# Sol 98 Storage Resilience

## Architecture

The live flattened VM `win98-stable-2026-07-19` runs from the internal NVMe:

`/home/david/VirtualBox VMs/win98-stable-2026-07-19`

The registered snapshot-chain archive `win98` and recovery backups remain on
the external Sol data SSD under `/mnt/sol-data`. A USB disconnect therefore
removes backup availability but not the live guest's writable disk.

The user service `sol98-storage-guard.service` checks every five seconds that:

- the registered configuration and active VDI exist;
- the VDI is still on the internal root filesystem;
- the filesystem is writable and has at least 12 GiB available;
- a periodic write-and-fsync probe succeeds.

If live storage becomes unsafe while the VM is running, the guard pauses the
VM and sends a desktop notification. It does not auto-resume. Loss of the
external backup SSD is logged but does not pause the internally hosted VM.

When the VM is powered off or saved and the external SSD is mounted writable,
the guard creates a quiescent rsync backup. It rechecks the VM state and source
fingerprint after copying before promoting a backup, keeps three generations,
and atomically updates the `latest` symlink.

The udev rule `/etc/udev/rules.d/80-sol-sandisk-no-autosuspend.rules` forces
USB runtime power control to `on` for SanDisk device `0781:558c`. This reduces
power-management disconnect risk but cannot compensate for a bad cable,
physical removal, enclosure reset, or device failure.

## Operations

```bash
systemctl --user status sol98-storage-guard.service
python3 ~/.local/lib/sol98/sol98_storage_guard.py --status
python3 ~/.local/lib/sol98/sol98_storage_guard.py --once
python3 ~/.local/lib/sol98/sol98_storage_guard.py --backup-now
```

`--backup-now` only runs while the VM is powered off or saved. Never copy a
running VDI as a recovery backup.

After a guard-triggered pause, inspect the state file and storage first:

```bash
cat ~/.local/state/sol98-storage-guard/state.json
VBoxManage showvminfo win98-stable-2026-07-19 --machinereadable
findmnt -T "$HOME/VirtualBox VMs/win98-stable-2026-07-19"
```

Resume only after the recorded hazard is corrected and `--once` passes.

## 2026-07-21 Incident

At 01:53:59 CDT the SanDisk Extreme SSD disconnected from USB. Linux returned
write I/O errors for `sdf`, aborted the ext4 journal on `sdf1`, and remounted
the stale filesystem read-only. VirtualBox `ATA-0` writes then failed and
Windows 98 displayed `Disk Write Error` for drive `C:`.

Recovery powered off the failed VM without saving state, ran offline `e2fsck`,
remounted the SSD, created a SHA-256-verified recovery copy, moved the active VM
to internal NVMe storage with `VBoxManage movevm`, and allowed Windows ScanDisk
to complete. The guest then reached its desktop and private network without new
VirtualBox storage errors.