Summary: At 18:51 Irish time we began seeing alerts from some sections of our network in Suite 1, BK3. While customer traffic is flowing the command and control aspects of the management system, the UI within our control panel etc is unavailable. It appears to be a switch related issue. Engineers are working on the problem and hope to have it resolved as soon as possible.
Whats affected: Our old control panel (https://cp.blacknight.com) and management of our Cloud3 VM platform. In addition spampanel.blacknight.ie and the 2 nodes involved in mail relay in and out are down.
Update: 20:21: Our engineers continue to work on this issue. We have engaged senior network and systems engineers to resolve this issue.
Update: 20:55: All services are back online. The issue was a specific port being down between our bgw-sw01 and acc-sw03 in BK3. This prevented some management traffic in specific VRFs and VLANs on specific switches from working correctly. More importantly, our monitoring system icinga was down for the duration of this. Erroneous emails and SMS messages were sent to customers as a result but none of their services were actually down.