We are performing emergency security update to our VMware platforms
| Site: | All Cloud Sites and Customer Private Clouds |
|---|---|
| Date: | Wed 22 Aug – Fri 24 Aug |
| Time: | Between 0900 and 1730 BST |
| Works: | L1 Terminal Fault Security Patching |
We are applying security patching to mitigate the risks around the recently announced L1 Terminal Fault Speculative Execution vulnerability to our VMware platforms.
The first stage requires security updates applied to the hosts running virtual machines. A second stage will follow requiring additional configuration to enable enhanced mitigation.
This should be considered “At-Risk” only. Customer virtual machines should continue to run without impact and the VPC Portal will remain available.
This is a newly discovered vulnerability in the same family as the Spectre and Meltdown vulnerabilities. As with the previous cases, this is a vulnerability in the CPU architecture and Intel, VMware and vendors have provided patches that work around the risks.
The vulnerability, while difficult to exploit, can lead a malicious program being able to access data in the CPU’s L1 cache. As such we are applying the security patches under emergency maintenance windows to ensure our customer workloads continue to enjoy the high levels of protection and isolation in our Cloud environments.
Customers might experience some disruption connecting to the VPC Portal during the update process
Maintenance Window Extension?
Having successfully validated a stable environment we will continue the updating process between the dates of Tue 28 Aug – Fri 31 Aug
We have identified an issue during the security update process and are currently working with our partner, VMware, to resolve this issue. We are now suspending the security update process until the issue is resolved.
This is a non-impacting issue and therefore your environment will continue to run as normal.
We will re-schedule this Emergency Maintenance once the issue has been completely fixed.
Maintenance Window Extension
We will continue the updating process between the dates of Tue 11 Sep – Fri 14 Sep
Customers in our NGD site may be experiencing issues controlling their virtual machines due to an issue during the upgrade of the NGD systems. We are working with our vendors to resolve this issue as soon as possible to restore services.
This will affect the ability to power on/power off, create or remove virtual machines as well as make changes to firewall rules/edge gateways. Virtual machines already running are unaffected and existing firewall rules will still be applying, this is only affecting the ability to manage entities from the portal.
We are still working through the issues with the NGD site, servers are working but responding slowly. We will continue to work closely with VMware on providing a resolution.
The NGD vSphere environment has now returned to a stable state, however it is currently disconnected from the VPC Portal. Customers will still experience issues managing their VMs and Firewalls in the NGD site while we resolve this last stage.
Services are currently restored for our NGD VPC Portal. We are continuing to monitor for stability but our tests of VPC Portal activities are all succeeding in a timely fashion. Our engineers will continue to keep an eye on services, please do report any issues to the Service Desk if you experience any issues.
We are extending the maintenance window from Wed 19 Sep 0500 until Fri 21 Sep 1730