• Support for ED25519 SSH Keys Added, Miscellaneous Other Improvements

    Fri, 27 Mar 2020 07:45:00 -0700 - Sarah Newman

    Last Saturday, as announced earlier, we deployed new versions of our billing and maintenance console software.

    The major public facing feature is that support for ED25119 SSH keys were added. We now support RSA, ECDSA, and ED25119 SSH keys.

    Here are the other public facing changes:

    • QR codes for two-factor authentication previously were generated by Google Charts, which is the OTP QR code implementation shipped by our billing system provider. We now generate the OTP QR codes locally using a thin library on top of phpqrcode. We have released this library for public use. The motivation for this change was to reduce data sent to third parties and to have control over the image caching policy.
    • We fixed a bug where sometimes transactions from Square were not being registered with the billing system.
    • We fixed another bug where paying too many invoices at once with Square would not work.

    Internal improvements included switching to Python 3 from Python 2 and using SHA256 based TLS certificates for all of our internal components, not just a subset.

    If you encounter any issues, please contact support.

  • Recursive DNS Resolvers Service Interruption

    Wed, 25 Mar 2020 12:40:00 -0700 - Chris Brannon

    At Wed Mar 25 16:04:57 UTC 2020, the majority of our recursive resolvers stopped working. The overall recursive resolver downtime was one hour and twenty minutes. We are in the process of issuing prorated credits for VPS customers.

    As part of this DNS outage, we identified a weakness in our monitoring infrastructure: we have an external monitor of our primary monitoring system, but it doesn’t guarantee that the DNS resolver is working on the primary monitoring system, only that the system is up with an active network. Therefore when both of our local resolvers stopped working simultaneously we were not paged as we should have been. We’ve added an external resolver where needed and long term will have the primary monitoring system initiate connections to the external monitor to prevent this in the future.

    The root cause was that our DNSSEC settings were stale and had not been revisited. Our bind configuration contained the following line:

    dnssec-lookaside auto;
    

    DNSSEC Lookaside Validation, specified in RFC 5074, is a now-historical mechanism for publishing DNS Security trust anchors outside of the DNS delegation chain. This mechanism was using a public registry provided by the Internet Systems Consortium (ISC) at dlv.isc.org.

    In September of 2017, the ISC retired dlv.isc.org, replacing it with an empty zone, so as not to break existing resolver configurations. The signatures for dlv.isc.org expired earlier today on accident, causing all queries from our resolvers to fail with broken trust chain errors. The expiration resulted in this error message:

    named:   validating : dlv.isc.org NSEC: verify failed due to bad signature (keyid=64263): RRSIG has expired
    

    The fix was to replace dnssec-lookaside auto with dnssec-lookaside no in our resolver configuration and restart the resolvers.

  • Distributions Update: Arch Linux 2020.03.01 Added, Various Bug Fixes

    Mon, 16 Mar 2020 16:30:00 -0700 - Chris Brannon

    We have updated our Arch Linux image and the Arch Linux install ISO to 2020.03.01 and made minor modifications to the remainder of the Linux based images.

    The Arch Linux image comes with systemd version 245. Version 245 has two notable changes. The first is systemd-homed, which can safely be ignored. The second is systemd-repart, which is capable of growing disk partitions or adding disk partitions at boot. The boot message for systemd-repart is: “Starting Repartition Root Disk”.

    Due to a customer request, our Debian and Ubuntu images received a round of internal cleanup. The following files were removed from the pre-installed prgmr-pv-domu package in favor of modifying them from preseed:

    • /etc/default/grub
    • /etc/ssh/sshd_config
    • /etc/hosts
    • /etc/hostname
    • /etc/network/interfaces

    The default CentOS 6 sshd configuration was updated to more closely comply with Mozilla’s guidelines.

    Finally, we eliminated a cause of boot delays for HVM Linux guests. By default, Linux loads a driver called xen_kbdfront. During initialization, the driver waits up to 30 seconds to detect a keyboard before giving up. Since a prgmr.com VPS has no keyboard, this probing unnecessarily delays the boot sequence.

    Where possible, we have blacklisted the xen_kbdfront module. This works for Arch Linux, Fedora, and CentOS. Debian and Ubuntu build the module into their kernels. In those cases, we use the initcall_blacklist kernel command line parameter to prevent the driver’s initialization function from executing. This effectively blacklists a built-in driver.

    Our distribution images and netboot installers are available from the management console of any Prgmr.com VPS.

    Note that Arch Linux is only available to customers with a VPS using HVM virtualization. To check the virtualization mode of existing services, log in to the management console, select “system details”, and check the virtualization mode.

  • Scheduled Maintenance for Billing System and Maintenance Console

    Sun, 15 Mar 2020 12:00:00 -0700 - Sarah Newman

    We will perform a software upgrade for our billing system and console, billing.prgmr.com and *.console.xen.prgmr.com, during a two-hour maintenance window starting Saturday March 21 18:00:00 UTC. As part of this we are refreshing our internal TLS certificates and certificate authority. Downtime is expected to be less than 20 minutes.

    If you have any questions or concerns, please write us at support@prgmr.com.

  • Distributions Update: Alpine Linux 3.11.3, NetBSD 9.0 Added, Fedora 29 Removed

    Wed, 19 Feb 2020 16:30:00 -0800 - Chris Brannon

    The following distributions/installers have been updated:

    Additionally, Fedora 29 has been removed, since it is end-of-life.

    Notable Changes to Alpine Linux

    The following changes were made in the 3.11 series which may be of interest to operators of a prgmr.com VPS. See the Alpine 3.11.0 release announcement for more.

    • Linux 5.4 kernel (linux-lts)
    • Rust is available on all architectures except s390x

    Notes for Upgrading to Alpine Linux 3.11.x

    • linux-vanilla has been removed. Install linux-lts when upgrading.
    • Python 2 is deprecated. Majority of Python 2 packages was removed and will be completely removed in next release.
    • Packages now use /var/mail instead of /var/spool/mail, in accordance with FHS
    • clamav-libunrar is no longer a hard dependency of clamav and needs to be manually installed.

    Notable NetBSD Changes

    Here are some of the NetBSD improvements mentioned in the release notes linked above, which may be of interest to people with a prgmr.com VPS.

    • Support for Kernel ASLR, on x86 64-bit, via the new GENERIC_KASLR kernel configuration file.
    • Support for KLEAK, a new feature able to detect kernel memory disclosures, with initial support for amd64.
    • Support for Kernel Address Sanitizer (KASAN), on amd64 and aarch64. This feature allows the kernel to detect illegal memory accesses, such as buffer overflows, stack overflows and use-after-frees.
    • Support for Kernel Undefined Behavior Sanitizer (KUBSAN), this feature allows the kernel to detect several classes of undefined behavior.
    • Support for Kernel Coverage (KCOV), on amd64. This drivers allows fuzzers to collect kernel coverage to improve fuzzing inputs.
    • Support for userland sanitizers, with new configurations allowing to run the entire userland stack with sanitizers.
    • Kernel Heap Hardening, making it harder to exploit several classes of memory bugs.
    • Audited network stack, bringing more confidence in the networking components of the kernel.
    • Many improvements in NPF, including new features, bug fixes, better documentation, and increased performance with a new lookup algorithm (thmap). NPF is now enabled by default.
    • Updated ZFS. This is the first release with ZFS usable for daily use, but there is no support for booting from ZFS nor using ZFS as root filesystem yet.

    Note that Alpine Linux is only available as a netboot installer to customers with a VPS using HVM virtualization. NetBSD is only available to customers with a VPS using PV virtualization. To check your virtualization mode, use the “system details” option of the main menu of your management console, and look for the “virtualization mode” line.

    Our distribution images and netboot installers are available from the management console of any Prgmr.com VPS.