Amazon Web Services (AWS) has alerted users of its EC2 platform that around 10 percent of the service will be taken offline to apply a security update. However, customers have complained this warning has come too late for them to be able to react as required to ensure services are not affected.
The update is required to fix a security issue with the open source Xen virtualisation platform used by AWS. The vulnerability will be made public on 1 October so AWS said it has to fix it before that date.
The company explained that while usually it applies software updates without having to reboot its servers, this update will require a hard reboot. Chief evangelist for AWS Jeff Barr explained why this was necessary in a blog post.
“The instances that need the update require a system restart of the underlying hardware and will be unavailable for a few minutes while the patches are being applied and the host is being rebooted.”
Barr said that the company would stagger the reboots so no two regions of availability zones (AZ) would be impacted at the same time. All servers will restart with all saved data and all automated configuration intact, Barr added.
Barr said that “most customers” should experience no “significant issues” during the process, but admitted that some will face problems.
“We understand that for a small subset of customers the reboot will be more inconvenient; we wouldn’t inconvenience our customers if it wasn’t important and time-critical to apply this update,” he said.
Despite reassuring customers that it has done all it can to minimise the impact, many firms have taken to the AWS forum to complain that they have been given far too little time to prepare for the downtime.
One forum user, called zavenb, wrote that their firm will not have the time to get staff in place to monitor the impact of the downtime.
“This is most definitely a problem for us as well. We currently have more than 100 instances scheduled for reboot, and we too cannot scramble the staff on short notice to monitor the many services that this will impact.”
“In fact, entire service clusters of machines are scheduled to be rebooted at once, although they are in different AZs, amounting to an event equivalent to the loss of an entire region. On two days’ notice we can’t possibly prepare for this.”
Another questioned why AWS was scheduling the update for Monday and Tuesday next week, rather than a weekend.
AWS had responded to most customers’ comments, attempting to answer questions where possible. It also warned customers not to stop/start their instances to try and avoid the update, as this would not work and posed security risk.
“An important note about stopping/starting instances: this will exchange your machine instance with another randomly selected machine instance. It is not guaranteed that the replacement will already be patched.
“For this reason, launching new machines or stop/starting machines is not a recommended way to avoid the scheduled reboot.”
The security issue AWS is fixing is not linked to the recently discovered Bash vulnerability, which the company said it has reviewed and determined does affect some of its systems.