Thu, 17 Oct 2019 10:00:00 -0700 - Chris Brannon
The netboot installer for OpenBSD was bumped to version 6.6, following today’s release announcement from OpenBSD. A list of changes can be found at the OpenBSD 6.6 release page. Items of interest to prgmr.com customers include:
- Further and improved mitigations against Spectre side-channel vulnerability in Intel CPUs built since 2012.
- Mitigations for Intel’s Microarchitectural Data Sampling vulnerability, using the new CPU VERW behavior if available or by using the proper sequence from Intel’s “Deep Dive” doc in the return-to-userspace and enter-VMM-guest paths. Updated vmm(4) to pass through the MSR bits so that guests can apply the optimal mitigation.
If you’re interested in running OpenBSD on prgmr.com, please see our wiki page on OpenBSD.
Note that the OpenBSD installer is only available for VPSes with HVM virtualization. The virtualization type can be checked from the ‘system details’ option of the management console. If your VPS uses PV and you are interested in a conversion to HVM, please write firstname.lastname@example.org.
Wed, 09 Oct 2019 14:00:00 -0700 - Chris Brannon
The netboot installer for NixOS has been bumped to version 19.09. Some highlights are that php was upgraded to 7.3 and the installer user is now
root. More information can be found in the release notes.
If you’d like to run NixOS, you can use this installer alongside notes from our Wiki page on NixOS to perform the installation. The netboot installer is available from the management console of any Prgmr.com VPS.
Note that the NixOS installer is only available for VPSes with HVM virtualization. The virtualization type can be checked from the ‘system details’ option of the management console. If your VPS uses PV and you are interested in a conversion to HVM, please write email@example.com.
Thu, 03 Oct 2019 05:00:00 -0700 - Sarah Newman
Nobody likes to get paged. This is especially true of pages for mysterious network issues that have cleared up by the time you have a chance to look at them. This has happened to us very occasionally over the last few months.
We ran through the obvious culprits - were there error counters we missed? Usage spikes? There was nothing visible. So we tried to figure out what common element there was, if any, for the hosts we were getting paged for.
One clue was that all the hosts were on the same physical layer 2 network, but on different layer 3 networks. We also weren’t seeing any failures with IPv6, only IPv4.
From one manual run of ping we saw output like:
.... From 10.2.2.2: icmp_seq=1 Redirect Host(New nexthop: 10.1.1.1) .... 27 packets transmitted, 27 received, +1 errors, 0% packet loss, time 26018ms
So where does this “Redirect Host” message come from?
When host A sends traffic to host B, if host A and B aren’t on the same network, i.e. they can’t talk directly to each other, host A will send the traffic to host B via a third party, a router. If you’ve ever manually configured a network and had to enter a gateway IP address, it is the router that this gateway address belongs to.
If a router has the gateway IPs for host A and host B on the same physical interface, it may send what is called an ICMP redirect packet when host A sends traffic for host B via the router. The redirect packet lets host A know that it can contact host B directly without going through the router. This specific type of redirect is a “shared media” redirect and is described in RFC1620. Only the gateway router is allowed to send this message. In Linux, sending and receiving this message is controlled by the ip sysctl settings
Seeing that error was odd, but was not a smoking gun. Eventually we captured output similar to:
CMD: /usr/bin/ping -n -U -W 10 -c 5 example.com Output: PING example.com (10.1.1.1) 56(84) bytes of data. Output: From 10.2.2.2 icmp_seq=1 Redirect Host(New nexthop: 10.1.1.1) Output: From 10.2.2.2: icmp_seq=1 Redirect Host(New nexthop: 10.1.1.1) Output: 64 bytes from 10.1.1.1: icmp_seq=1 ttl=63 time=0.490 ms Output: 64 bytes from 10.1.1.1: icmp_seq=2 ttl=63 time=0.613 ms Output: 64 bytes from 10.1.1.1: icmp_seq=3 ttl=63 time=0.565 ms Output: From 10.2.2.2 icmp_seq=4 Redirect Host(New nexthop: 10.1.1.1) Output: Output: --- example.com ping statistics --- Output: 4 packets transmitted, 3 received, +2 errors, 25% packet loss, time 3000ms Output: rtt min/avg/max/mdev = 0.490/0.556/0.613/0.050 ms PING CRITICAL - Packet loss = 25%, RTA = 0.56 ms|rta=0.556000ms;100.000000;500.000000;0.000000 pl=25%;2;6;0 100.000000:2% 500.000000:6%
This reported packet loss of 25% must have been why we were seeing problems.
After we turned off sending redirects on our routers, which was on by default, we looked into why this was occurring within ping.
One odd thing about ping is that if a count of packets is specified with
-c, the count specifies the number of packets to receive, not the number of packets to send. In the above output, receiving a redirect as the 5th packet meant there was no response received for the last probe sent.
By code inspection and pulling up ping in a debugger, we found that a message for the ICMP redirect was being inserted into the socket error queue. The socket error queue is read by calling the function
MSG_ERRQUEUEflag. We tried patching ping to not increment the number of errors when a message of type
ICMP_REDIRECTwas received, and that worked.
However, it was curious that this patch was needed. By testing other distributions we found that there was no error received for ICMP redirects, and that if we replaced the stock CentOS 7 kernel with an alternative one, the errors went away. A ticket has been filed with CentOS.
Fri, 27 Sep 2019 21:00:00 -0700 - Chris Brannon
We have added CentOS 8 to our supported distributions. Here are the release notes for RHEL 8, which CentOS 8 is based on. Notable changes from CentOS 7 include:
The package manager has changed to
yumcommand is still available but it is just a backward compatible wrapper around
Python 3.6 is the default Python implementation now. Upstream CentOS does not come with Python installed automatically, but we have added the
python3package to our build.
nftablesframework is now used under the hood for packet filtering, rather than
iptables. CentOS 8 still uses
firewalldas the user-facing frontend to firewall management, and only the backend has changed.
NetworkManager is used for managing network configuration instead of ifup and ifdown. It is capable of running the network-scripts used by previous CentOS releases, so the network configuration is still kept under
If from the management console, the option ‘system details’ says ‘Virtualization mode: Paravirtualized’ then you will not be able to install CentOS 8 automatically. Please contact support for assistance, or order a replacement VPS. Otherwise, the pre-installed image and netboot installer are available from the management console of any Prgmr.com VPS.
Fri, 20 Sep 2019 19:00:00 -0700 - Chris Brannon
We have updated our CentOS 7 build to CentOS 7.7, released on September 17, 2019. Major changes include the addition of python 3.6 and updates to bind 9.11 and chrony 3.4.
The pre-installed image and netboot installer are available from the management console of any Prgmr.com VPS.