We’re currently experiencing an issue with our Qmail shared email hosting service, affecting mail.blacknight.com. Engineers are actively investigating.
Update: 01:10: Mail services are back online. The NFS server crashed around 22:35~ which meant the storage for the mail servers stopped working. This meant we stopped being able to accept new email (which was queued and has been delivered since) and also that email clients (Such as outlook, thunderbird, ox.blacknight.com, altmail etc) would have stopped working.
What exacerbated the issue was an unusually prolonged import of the zpool. It took over 40 minutes to import properly. This is usually a 2-3 minute process. We suspect that due to the crash, ZFS had to perform some data integrity actions, but as this wasn’t in any of the logs we weren’t sure. So we ended up stopping the import process twice and starting again which added to the recovery time.
As of 01:11am on March 17th service is back. The systems cache (temporary storage in memory) will take some time to fill up which may make it feel a little sluggish. Things should return to normal in a few hours.