• Distributions Update: Alpine Linux 3.11.3, NetBSD 9.0 Added, Fedora 29 Removed

    Wed, 19 Feb 2020 16:30:00 -0800 - Chris Brannon

    The following distributions/installers have been updated:

    Additionally, Fedora 29 has been removed, since it is end-of-life.

    Notable Changes to Alpine Linux

    The following changes were made in the 3.11 series which may be of interest to operators of a prgmr.com VPS. See the Alpine 3.11.0 release announcement for more.

    • Linux 5.4 kernel (linux-lts)
    • Rust is available on all architectures except s390x

    Notes for Upgrading to Alpine Linux 3.11.x

    • linux-vanilla has been removed. Install linux-lts when upgrading.
    • Python 2 is deprecated. Majority of Python 2 packages was removed and will be completely removed in next release.
    • Packages now use /var/mail instead of /var/spool/mail, in accordance with FHS
    • clamav-libunrar is no longer a hard dependency of clamav and needs to be manually installed.

    Notable NetBSD Changes

    Here are some of the NetBSD improvements mentioned in the release notes linked above, which may be of interest to people with a prgmr.com VPS.

    • Support for Kernel ASLR, on x86 64-bit, via the new GENERIC_KASLR kernel configuration file.
    • Support for KLEAK, a new feature able to detect kernel memory disclosures, with initial support for amd64.
    • Support for Kernel Address Sanitizer (KASAN), on amd64 and aarch64. This feature allows the kernel to detect illegal memory accesses, such as buffer overflows, stack overflows and use-after-frees.
    • Support for Kernel Undefined Behavior Sanitizer (KUBSAN), this feature allows the kernel to detect several classes of undefined behavior.
    • Support for Kernel Coverage (KCOV), on amd64. This drivers allows fuzzers to collect kernel coverage to improve fuzzing inputs.
    • Support for userland sanitizers, with new configurations allowing to run the entire userland stack with sanitizers.
    • Kernel Heap Hardening, making it harder to exploit several classes of memory bugs.
    • Audited network stack, bringing more confidence in the networking components of the kernel.
    • Many improvements in NPF, including new features, bug fixes, better documentation, and increased performance with a new lookup algorithm (thmap). NPF is now enabled by default.
    • Updated ZFS. This is the first release with ZFS usable for daily use, but there is no support for booting from ZFS nor using ZFS as root filesystem yet.

    Note that Alpine Linux is only available as a netboot installer to customers with a VPS using HVM virtualization. NetBSD is only available to customers with a VPS using PV virtualization. To check your virtualization mode, use the “system details” option of the main menu of your management console, and look for the “virtualization mode” line.

    Our distribution images and netboot installers are available from the management console of any Prgmr.com VPS.

  • Not Vulnerable to CacheOut Attack

    Mon, 27 Jan 2020 15:00:00 -0800 - Sarah Newman

    None of our systems are vulnerable to the CacheOut Attack otherwise known as L1D Eviction Sampling/CVE-2020-0549/INTEL-SA-00329.

  • Adding Square as a Payment Processor

    Tue, 14 Jan 2020 11:05:00 -0800 - Sarah Newman

    For a long time, we only had PayPal and BitPay as payment processors. This Saturday we added Square for processing one-time credit card payments, as Square does not support recurring payments via its API. As a workaround for the lack of recurring payments, customers who want to use Square may either switch to annual billing or prepay using the “Make Payment” button from https://billing.prgmr.com. Adding Square also should have added support for Apple and Google Pay, though we haven’t tested either yet.

    Since our experience may be of interest to current and future customers, we’ll describe it below.

    Integrating with Square was very easy. We use Blesta as our billing system, which already has support for Square Checkout. Square Checkout is the hosted version of the Square API, where all the credit card processing is done on Square’s own servers. Per our privacy policy we made some local modifications to minimize the amount of data sent to Square, such as not pre-filling the buyer address or email. We also set the merchant_support_email field.

    Creating an account with Square was reasonably painless, and we completed signing up with them in a single day. One complication was our accountant wasn’t immediately able to sign in, and we don’t know why. Another complication was that despite disabling transactional emails for our main account, we were receiving an email for each API transaction until we found the list-unsubscribe option in the email. We think this is because our support email is not otherwise associated with the account. It would also be nice if there was a global setting for preventing any non-transactional emails from being sent to customers.

    To summarize, we found adding Square as a payment option to be relatively painless, though until they support recurring payments via their API they are not a standalone solution for us.

  • Scheduled Maintenance for Billing System

    Fri, 10 Jan 2020 12:15:00 -0800 - Chris Brannon

    We will perform a software upgrade for our billing system, billing.prgmr.com, during a two-hour maintenance window starting Saturday January 11, 2020, 19:00 UTC. Downtime for billing.prgmr.com is expected to be less than 5 minutes.

    If you have any questions or concerns, please write us at support@prgmr.com.

  • (Relatively) Recent Security Updates

    Tue, 17 Dec 2019 14:00:00 -0800 - Sarah Newman

    Summary

    We are patched or otherwise not vulnerable for the following advisories released in the last few months:

    All privilege escalations were patched within 4 days of public disclosure. Normally we patch privilege escalations more quickly. In this case we judged 4 days to be an acceptable amount of delay due to the practical difficulty of an exploit and because it allowed us to reduce our future exposure to certain types of bugs, as we describe below.

    Timeline

    Almost 2 months ago we were notified of several Xen security advisories:

    XSA-299 was not easily live-patched, so we decided to live-migrate virtual machines to patched host servers to address these issues.

    On Saturday Oct 26 06:10:13 UTC 2019, one of these servers crashed with a Xen backtrace. We decided to continue with moving virtual machines that night, but without using live migration, as we thought that the bug was most likely related to live migration based on how often various functionality in Xen is used.

    Within the next 24 hours we attempted to reproduce the bug on our test systems. We had a hypothesis that it was related to NetBSD. The only reason for this is XSA-280, which we found as soon as a NetBSD 5.2 host was booted. NetBSD uses virtual memory in a different way than Linux does, and so it exercises parts of the Xen code that Linux does not. NetBSD is also a less common operating system than Linux.

    We found a bug with NetBSD guests that crashed the host, and sent this on to the Xen security team. We very quickly got back a patch that fixed the immediate issue, which generated version 3 of XSA-299.

    However, further testing led to at least two additional distinct crashes. Our ability to reproduce this resulted in XSA-309 as at least one of the crashes had been publicly reported but not reproduced. While we again received proposed fixes extremely quickly, we did not have time to do adequate testing of them in advance of public disclosure.

    Because of the additional two types of crashes we had found, we decided to delay patching the remainder of our systems until after the embargo had lifted for the above XSAs. This allowed us to disable the feature that lets NetBSD run directly on Xen. Disabling this feature is not allowed under embargo because it can be detected by a virtual machine. Our NetBSD guests are now running in a “shim” which is a virtualized version of Xen.

    During the last night we were live migrating guests, we had another crash. We then realized that the original bug we had found was not exactly the same, though it happened during the same operation. There was nothing in the backtrace tying it directly to live migration - it was a bug that happened when the virtual machine was shut down.

    It took us a long time to reproduce. Starting and shutting down virtual machines tens of thousands of times did not do it. Live migrating any old virtual machine hundreds of times did not do it. What did it was migrating a 32-bit Linux virtual machine with more than 4GiB RAM about 60 or more times. The live migration failed for whatever reason, and something about shutting down the virtual machine before it had finished migrating triggered the bug. Once we were able to reliably reproduce, we found that some of the patches we had received in response to the second round of NetBSD testing actually fixed this bug. Eventually this resulted in XSA-310.

    Once we were confident that we were able to both reliably reproduce the bug and that we had a fix for it, we generated a livepatch for it along with a few other Denial of Service (DoS) fixes.

    We tested reboots with these patches tens of thousands of times and performed a couple of thousand live migrations. We also tested the NetBSD related fixes with a few thousand virtual machine reboots. We are now confident that we should see no further crashes from these bugs.