Top Menu

Blacknight Hosting Network Status data corruption issue [resolved]

Summary: Additional cores were added to this node due to high load around 14:04 today Friday the 7th of April. Approx 90 minutes after the reboot we began getting calls about missing blog posts and other similar issues.

What Happened: Upon investigation it appears that the live image had somehow rolled back to March 8th @ 23:30. We then discovered that what had transpired was an old Vhdx had been attached to this VM some days prior to restore a customers website. When the VM was rebooted by us at 14:04 it randomly chose which disk to boot from, it booted from the old one.

Findings: We’ll take better care to remove restored vhdx’s from restores in future. A SOP change will occur to ensure that the number of images attached to a VM is checked prior to any reboot to ensure that there is only 1.

N.B. We would like to apologise to customers for losing their data from between 14:04 today and 16:10~ today due to this issue. This data is not recoverable.



April 7, 2023 @ 16:10

Notification Type:

Emergency Incident

Affected Services:

Shared Hosting,