Mon, 27 Jan 2020 15:00:00 -0800 - Sarah Newman
None of our systems are vulnerable to the CacheOut Attack otherwise known as L1D Eviction Sampling/CVE-2020-0549/INTEL-SA-00329.
Tue, 14 Jan 2020 11:05:00 -0800 - Sarah Newman
For a long time, we only had PayPal and BitPay as payment processors. This Saturday we added Square for processing one-time credit card payments, as Square does not support recurring payments via its API. As a workaround for the lack of recurring payments, customers who want to use Square may either switch to annual billing or prepay using the “Make Payment” button from https://billing.prgmr.com. Adding Square also should have added support for Apple and Google Pay, though we haven’t tested either yet.
Since our experience may be of interest to current and future customers, we’ll describe it below.
Creating an account with Square was reasonably painless, and we completed signing up with them in a single day. One complication was our accountant wasn’t immediately able to sign in, and we don’t know why. Another complication was that despite disabling transactional emails for our main account, we were receiving an email for each API transaction until we found the list-unsubscribe option in the email. We think this is because our support email is not otherwise associated with the account. It would also be nice if there was a global setting for preventing any non-transactional emails from being sent to customers.
To summarize, we found adding Square as a payment option to be relatively painless, though until they support recurring payments via their API they are not a standalone solution for us.
Fri, 10 Jan 2020 12:15:00 -0800 - Chris Brannon
We will perform a software upgrade for our billing system, billing.prgmr.com, during a two-hour maintenance window starting Saturday January 11, 2020, 19:00 UTC. Downtime for billing.prgmr.com is expected to be less than 5 minutes.
If you have any questions or concerns, please write us at firstname.lastname@example.org.
Tue, 17 Dec 2019 14:00:00 -0800 - Sarah Newman
We are patched or otherwise not vulnerable for the following advisories released in the last few months:
- Xen Security Advisories 296 through 311 (the latest is disclosed 2019-12-11)
- CVE-2018-12207 Improper invalidation for page table updates
- CVE-2019-14607 Unexpected Page Fault In Virtualized Environment
All privilege escalations were patched within 4 days of public disclosure. Normally we patch privilege escalations more quickly. In this case we judged 4 days to be an acceptable amount of delay due to the practical difficulty of an exploit and because it allowed us to reduce our future exposure to certain types of bugs, as we describe below.
Almost 2 months ago we were notified of several Xen security advisories:
- XSA-296 VCPUOP_initialise DoS
- XSA-298 missing descriptor table limit checking in x86 PV emulation
- XSA-299 Issues with restartable PV type change operations
XSA-299 was not easily live-patched, so we decided to live-migrate virtual machines to patched host servers to address these issues.
On Saturday Oct 26 06:10:13 UTC 2019, one of these servers crashed with a Xen backtrace. We decided to continue with moving virtual machines that night, but without using live migration, as we thought that the bug was most likely related to live migration based on how often various functionality in Xen is used.
Within the next 24 hours we attempted to reproduce the bug on our test systems. We had a hypothesis that it was related to NetBSD. The only reason for this is XSA-280, which we found as soon as a NetBSD 5.2 host was booted. NetBSD uses virtual memory in a different way than Linux does, and so it exercises parts of the Xen code that Linux does not. NetBSD is also a less common operating system than Linux.
We found a bug with NetBSD guests that crashed the host, and sent this on to the Xen security team. We very quickly got back a patch that fixed the immediate issue, which generated version 3 of XSA-299.
However, further testing led to at least two additional distinct crashes. Our ability to reproduce this resulted in XSA-309 as at least one of the crashes had been publicly reported but not reproduced. While we again received proposed fixes extremely quickly, we did not have time to do adequate testing of them in advance of public disclosure.
Because of the additional two types of crashes we had found, we decided to delay patching the remainder of our systems until after the embargo had lifted for the above XSAs. This allowed us to disable the feature that lets NetBSD run directly on Xen. Disabling this feature is not allowed under embargo because it can be detected by a virtual machine. Our NetBSD guests are now running in a “shim” which is a virtualized version of Xen.
During the last night we were live migrating guests, we had another crash. We then realized that the original bug we had found was not exactly the same, though it happened during the same operation. There was nothing in the backtrace tying it directly to live migration - it was a bug that happened when the virtual machine was shut down.
It took us a long time to reproduce. Starting and shutting down virtual machines tens of thousands of times did not do it. Live migrating any old virtual machine hundreds of times did not do it. What did it was migrating a 32-bit Linux virtual machine with more than 4GiB RAM about 60 or more times. The live migration failed for whatever reason, and something about shutting down the virtual machine before it had finished migrating triggered the bug. Once we were able to reliably reproduce, we found that some of the patches we had received in response to the second round of NetBSD testing actually fixed this bug. Eventually this resulted in XSA-310.
Once we were confident that we were able to both reliably reproduce the bug and that we had a fix for it, we generated a livepatch for it along with a few other Denial of Service (DoS) fixes.
We tested reboots with these patches tens of thousands of times and performed a couple of thousand live migrations. We also tested the NetBSD related fixes with a few thousand virtual machine reboots. We are now confident that we should see no further crashes from these bugs.
Wed, 11 Dec 2019 12:00:00 -0800 - Chris Brannon
We’ve added a new distribution image, CentOS 8 Stream. From the Release Notes: “CentOS Stream is a rolling-release Linux distro that exists as a midstream between the upstream development in Fedora Linux and the downstream development for Red Hat Enterprise Linux (RHEL).”
The major difference between CentOS 8 and CentOS 8 Stream is a newer kernel.
We are not offering a netboot installer for CentOS 8 Stream, on account of this upstream bug. We’ll add it as soon as package updates are pulled from the 8-stream release channel. However, an installable tarball is available from the “install new OS image” menu of the management console.
We also rebuilt our Arch Linux tarball and install ISO with the 2019.12.01 update from Arch Linux.
All of the updates are only available for VPSes with HVM virtualization. The VPS virtualization is displayed under “system details” when logged into the management console. If the system virtualization is not HVM, please contact support for a conversion.