Summary: Additional cores were added to this node due to high load around 14:04 today Friday the 7th of April. Approx 90 minutes after the reboot we began getting calls about missing blog posts and other similar issues. What Happened: Upon investigation it appears that the live image had somehow rolled back to March 8th […]
Author Archive | Paul Kelly
Anycast DNS outage
Summary: We started seeing sporadic responses from our AnyCast DNS servers last night. We contacted AfiliasDNS who operate that service for us. As of yet we haven’t had a reply or a resolution from them that gives us any insight as to what the problem is. What impact is there on customers: Typically, none. But […]
broadband database backend migration
Summary: Wednesday 16/11 @ 10:00am we’ll be migrating the database backend for our broadband control systems to new infrastructure. What: We’re running a GaleraDB cluster on MariaDB 10.1 and we want to migrate this MariaDB 10.6. Rather than doing this in-situ and having no roll back plan we made new VMs and a new GaleraDB […]
plesk001 and plesk002 are down due to packaging issues
Summary: A problem was noticed with the Plesk control panel related to a mariadb package. The package in question was updated which appears to have caused further updates. Both machines are being rolled back to their last snapshots (approx 3am) and will be back online around 09:30 this morning. We’re keeping the original disks for […]