We observed sporadic I/O performance spikes to customer virtual machines in our VEN1 VPC site. The issue has been isolated as a fault within one of the Storage array components connectivity which is being replaced.
Our engineers were alerted to an issue earlier in the day that caused sporadic issues with storage performance for some customers in our VEN1 site.
We have been working with the vendor through the day and isolated the issue with connectivity between one of the storage controllers and access switch. We have performed a controlled failover to the second controller on this device at 20:33 on 16/02/2019 which restored services. The controller is currently considered “At-Risk” while we replace potentially faulty hardware.
We have identified the faulty components and they have been replaced. We’ve monitored the stability of the device and confirmed the issue has been resolved. Both controllers are in service and full resilience has been resumed.
An incident report is due to be issues on Monday 19th February with more details.
If your services were running from our VEN1 VPC platform, your virtual machine may have seen spikes in Disk I/O latency which could make reading/writing to disk appear slow for brief periods of time. As of 20:33 those spikes would have been resolved. The remedial work to restore resilience should have no further impact on customer running workloads.
