• Distributions Update: Alpine Linux 3.10.3, Arch Linux 2019.11.01, Fedora 31, FreeBSD 12.1, and Ubuntu 19.10

    Fri, 15 Nov 2019 15:00:00 -0800 - Chris Brannon

    The following distributions/installers have been updated:

    Update Highlights

    Fedora 31

    Fedora 31 has enabled cgroups version 2. The python command has also switched to python3. There are many package updates, including glibc 2.30.

    FreeBSD 12.1

    From the FreeBSD 12.1 release announcement:

    • BearSSL has been imported to the base system.
    • The clang, llvm, lld, lldb, compiler-rt utilities and libc++ have been updated to version 8.0.1.
    • OpenSSL has been updated to version 1.1.1d.
    • Several userland utility updates.
    • And more…

    Ubuntu 19.10

    Ubuntu 19.10 has updated to a 5.3 kernel and added experimental support for a ZFS root. We have not tested it, but in theory it should be possible to configure a ZFS root using the netboot installer.

    Ubuntu has switched its kernel to use LZ4 compression. This is said to yield faster boot times. However, this kernel could not be made to boot under paravirtualized Xen. Ubuntu 19.10 is not available for instances that use PV virtualization.

    Installation

    All distributions other than Alpine Linux can be selected while ordering or can be reinstalled from the prgmr.com VPS management console. Alpine Linux is only available as a netboot installer from the prgmr.com VPS management console.

    All of the updates are only available for VPSes with HVM virtualization. The VPS virtualization is displayed under “system details” when logged into the management console. If the system virtualization is not HVM, please contact support for a conversion.

  • OpenBSD 6.6 Now Available as a Netboot Installer

    Thu, 17 Oct 2019 10:00:00 -0700 - Chris Brannon

    The netboot installer for OpenBSD was bumped to version 6.6, following today’s release announcement from OpenBSD. A list of changes can be found at the OpenBSD 6.6 release page. Items of interest to prgmr.com customers include:

    • Further and improved mitigations against Spectre side-channel vulnerability in Intel CPUs built since 2012.
    • Mitigations for Intel’s Microarchitectural Data Sampling vulnerability, using the new CPU VERW behavior if available or by using the proper sequence from Intel’s “Deep Dive” doc in the return-to-userspace and enter-VMM-guest paths. Updated vmm(4) to pass through the MSR bits so that guests can apply the optimal mitigation.

    If you’re interested in running OpenBSD on prgmr.com, please see our wiki page on OpenBSD.

    Note that the OpenBSD installer is only available for VPSes with HVM virtualization. The virtualization type can be checked from the ‘system details’ option of the management console. If your VPS uses PV and you are interested in a conversion to HVM, please write support@prgmr.com.

  • NixOS 19.09 Now Available as a Netboot Installer

    Wed, 09 Oct 2019 14:00:00 -0700 - Chris Brannon

    The netboot installer for NixOS has been bumped to version 19.09. Some highlights are that php was upgraded to 7.3 and the installer user is now nixos instead of root. More information can be found in the release notes.

    If you’d like to run NixOS, you can use this installer alongside notes from our Wiki page on NixOS to perform the installation. The netboot installer is available from the management console of any Prgmr.com VPS.

    Note that the NixOS installer is only available for VPSes with HVM virtualization. The virtualization type can be checked from the ‘system details’ option of the management console. If your VPS uses PV and you are interested in a conversion to HVM, please write support@prgmr.com.

  • The networking problem that wasn't

    Thu, 03 Oct 2019 05:00:00 -0700 - Sarah Newman

    Nobody likes to get paged. This is especially true of pages for mysterious network issues that have cleared up by the time you have a chance to look at them. This has happened to us very occasionally over the last few months.

    We ran through the obvious culprits - were there error counters we missed? Usage spikes? There was nothing visible. So we tried to figure out what common element there was, if any, for the hosts we were getting paged for.

    One clue was that all the hosts were on the same physical layer 2 network, but on different layer 3 networks. We also weren’t seeing any failures with IPv6, only IPv4.

    From one manual run of ping we saw output like:

    ....
    From 10.2.2.2: icmp_seq=1 Redirect Host(New nexthop: 10.1.1.1)
    ....
    27 packets transmitted, 27 received, +1 errors, 0% packet loss, time 26018ms
    

    So where does this “Redirect Host” message come from?

    When host A sends traffic to host B, if host A and B aren’t on the same network, i.e. they can’t talk directly to each other, host A will send the traffic to host B via a third party, a router. If you’ve ever manually configured a network and had to enter a gateway IP address, it is the router that this gateway address belongs to.

    If a router has the gateway IPs for host A and host B on the same physical interface, it may send what is called an ICMP redirect packet when host A sends traffic for host B via the router. The redirect packet lets host A know that it can contact host B directly without going through the router. This specific type of redirect is a “shared media” redirect and is described in RFC1620. Only the gateway router is allowed to send this message. In Linux, sending and receiving this message is controlled by the ip sysctl settings send_redirects, accept_redirects, and shared_media.

    Seeing that error was odd, but was not a smoking gun. Eventually we captured output similar to:

    CMD: /usr/bin/ping -n -U -W 10 -c 5 example.com
    Output: PING example.com (10.1.1.1) 56(84) bytes of data.
    Output: From 10.2.2.2 icmp_seq=1 Redirect Host(New nexthop: 10.1.1.1)
    Output: From 10.2.2.2: icmp_seq=1 Redirect Host(New nexthop: 10.1.1.1)
    Output: 64 bytes from 10.1.1.1: icmp_seq=1 ttl=63 time=0.490 ms
    Output: 64 bytes from 10.1.1.1: icmp_seq=2 ttl=63 time=0.613 ms
    Output: 64 bytes from 10.1.1.1: icmp_seq=3 ttl=63 time=0.565 ms
    Output: From 10.2.2.2 icmp_seq=4 Redirect Host(New nexthop: 10.1.1.1)
    Output:
    Output: --- example.com ping statistics ---
    Output: 4 packets transmitted, 3 received, +2 errors, 25% packet loss,
    time
    3000ms
    Output: rtt min/avg/max/mdev = 0.490/0.556/0.613/0.050 ms
    PING CRITICAL - Packet loss = 25%, RTA = 0.56
    ms|rta=0.556000ms;100.000000;500.000000;0.000000 pl=25%;2;6;0
    100.000000:2% 500.000000:6%
    

    This reported packet loss of 25% must have been why we were seeing problems.

    After we turned off sending redirects on our routers, which was on by default, we looked into why this was occurring within ping.

    One odd thing about ping is that if a count of packets is specified with -c, the count specifies the number of packets to receive, not the number of packets to send. In the above output, receiving a redirect as the 5th packet meant there was no response received for the last probe sent.

    By code inspection and pulling up ping in a debugger, we found that a message for the ICMP redirect was being inserted into the socket error queue. The socket error queue is read by calling the function recvmsg with the MSG_ERRQUEUE flag. We tried patching ping to not increment the number of errors when a message of type ICMP_REDIRECT was received, and that worked.

    However, it was curious that this patch was needed. By testing other distributions we found that there was no error received for ICMP redirects, and that if we replaced the stock CentOS 7 kernel with an alternative one, the errors went away. A ticket has been filed with CentOS.

  • CentOS 8 Now Available

    Fri, 27 Sep 2019 21:00:00 -0700 - Chris Brannon

    We have added CentOS 8 to our supported distributions. Here are the release notes for RHEL 8, which CentOS 8 is based on. Notable changes from CentOS 7 include:

    • The package manager has changed to dnf. The yum command is still available but it is just a backward compatible wrapper around dnf.

    • Python 3.6 is the default Python implementation now. Upstream CentOS does not come with Python installed automatically, but we have added the python3 package to our build.

    • The nftables framework is now used under the hood for packet filtering, rather than iptables. CentOS 8 still uses firewalld as the user-facing frontend to firewall management, and only the backend has changed.

    • NetworkManager is used for managing network configuration instead of ifup and ifdown. It is capable of running the network-scripts used by previous CentOS releases, so the network configuration is still kept under /etc/sysconfig/network-scripts.

    If from the management console, the option ‘system details’ says ‘Virtualization mode: Paravirtualized’ then you will not be able to install CentOS 8 automatically. Please contact support for assistance, or order a replacement VPS. Otherwise, the pre-installed image and netboot installer are available from the management console of any Prgmr.com VPS.