From Thursday June 15th through Sunday June 18th we upgraded our customer-hosting systems and patched the following XSAs:

The primary reasons we patched were the privilege escalation related ones XSA-222 and XSA-224. While we also patched all of the above XSAs, we were not vulnerable to all of them.

The average delay from the beginning of the window to a given VM coming back up was 38 minutes. The longest delay was 54 minutes, while the shortest delay was 21 minutes. Our shortest time was 6 minutes longer than our previous maintenance window, while our longest server was 13 minutes shorter. The average downtime was largely unchanged.

In the retrospective from our last maintenance window, we stated we’d evaluate live patching. With this maintenance window, we’ve upgraded to Xen 4.8 and enabled LivePatch which allows some code patches to be applied without reboots. We’re working on a pull request to Xen4CentOS so that other Xen4CentOS users can also upgrade to 4.8 with live patching.

Live patching is not possible for every security update, but an estimate as of a year ago is that 90% of them can be handled with live patching. Xen is supposed to be easier to live patch than Linux because the context switching is more simple. But the same caveats as for kpatch on what changes can be made generally apply for Xen as well. We’ll evaluate other options for avoiding downtime when live patching is not a good idea.

Another change we made was to also enable live patching for Linux though we have not tested it to verify it is actually possible with the kernel we built. There are commercial options we may be able to use instead but have not made a final decision yet. In order to enable live patching for Linux we installed a kernel built for CentOS 7 and added a shim package to provide the single /usr/sbin/new-kernel-pkg dependency.