Skip to content

Preprod crash 2021 12#

What happens#

  • zfs storage on OVH2 was full
  • this blocks the restart of two VM : dockers (200) and dockers-prod (201)

Why this happens#

  • probably because of resize of partition the day before on both machine, this makes snapshot diverge and take much more space. (see 2021-12-21-disk-extension)
  • out of space on zfs, blocks machine because proxmox needs to snapshot memory

What was done#

The same day:

  • rollback VM dockers (200) to its snapshoted version
  • removed dockers-prod (201) snapshot
  • extension of disk space for this VM dockers (200) was reduced from 400G to 300G (this is still a 50G improvment over the previous size for dockers VM)

The day after:

  • removed the efficientnet.tar.gz in /home/off/robotoff-ann-net as it was already untared in the ann_data folder
  • hard reboot of VM dockers (200)
  • resized partition to 300G following previous operating mode