We have scheduled a maintenance window to remediate a critical electrical power issue in the Carlow Data Centre for Friday 30th October from 11:00 PM to 3:00 AM Saturday morning. The work requires a complete shutdown of electrical power to the Data centre for safety reasons because the electrical engineers will be replacing the Main Circuit Breakers on both the A-side and B-side feeds and power needs to be fully isolated.
It is essential that we restore full confidence in the power feeds to the data centre as soon as possible and to ensure we do not have a reoccurrence of the A-side breaker tripping unexpectedly. We have commissioned 2 new custom designed ABB circuit breakers to replace the existing breakers and these will be fitted as soon as power is isolated. The electrical engineers have estimated 2 hours to carry out the works, however we will need to shut down all servers and network equipment in advance of the power down and it will take some time to reboot everything and verify services are fully operational. There will be a team of electrical and Systems Engineers onsite to ensure the work is carried out as quickly as possible, but we are allowing a window of 4 hours for contingency.
Services affected during the window are as follows
- CloudCW Platform
- NG Shared hosting platform
- pemlinng001-0024
- Legacy shared platform on Direct Admin
- Igraine
- Cador
- Ector
- Iseult
- Ragnell
- Windows shared hosting on pemwinweb59 -62
- Cloud2 Platform and control panel
- Jelastic Platform
UPDATE 03:30 Remedial works have been completed and services are returning to normal. There is an issue with the Cloud2 storage platform and our engineers are currently investigating it. We will update this post shortly.
UPDATE 04:10 We have had to contact our storage vendor as the Solidfire storage platform is not behaving properly. We have raised a call with them and had their oncall senior engineer raised to look at the issue.
Update: 05:40 Solidfire have applied a patch to our environment. This was some sort of known bug. Further info to follow. However all VMs are now starting and will be back online shortly.