Nobody likes to get paged. This is especially true of pages for mysterious network issues that have cleared up by the time you have a chance to look at them. This has happened to us very occasionally over the last few months.
We ran through the obvious culprits - were there error counters we missed? Usage spikes? There was nothing visible. So we tried to figure out what common element there was, if any, for the hosts we were getting paged for.
One clue was that all the hosts were on the same physical layer 2 network, but on different layer 3 networks. We also weren’t seeing any failures with IPv6, only IPv4.
From one manual run of ping we saw output like:
.... From 10.2.2.2: icmp_seq=1 Redirect Host(New nexthop: 10.1.1.1) .... 27 packets transmitted, 27 received, +1 errors, 0% packet loss, time 26018ms
So where does this “Redirect Host” message come from?
When host A sends traffic to host B, if host A and B aren’t on the same network, i.e. they can’t talk directly to each other, host A will send the traffic to host B via a third party, a router. If you’ve ever manually configured a network and had to enter a gateway IP address, it is the router that this gateway address belongs to.
If a router has the gateway IPs for host A and host B on the same physical interface, it may send what is called an ICMP redirect packet when host A sends traffic for host B via the router. The redirect packet lets host A know that it can contact host B directly without going through the router. This specific type of redirect is a “shared media” redirect and is described in RFC1620. Only the gateway router is allowed to send this message. In Linux, sending and receiving this message is controlled by the ip sysctl settings
Seeing that error was odd, but was not a smoking gun. Eventually we captured output similar to:
CMD: /usr/bin/ping -n -U -W 10 -c 5 example.com Output: PING example.com (10.1.1.1) 56(84) bytes of data. Output: From 10.2.2.2 icmp_seq=1 Redirect Host(New nexthop: 10.1.1.1) Output: From 10.2.2.2: icmp_seq=1 Redirect Host(New nexthop: 10.1.1.1) Output: 64 bytes from 10.1.1.1: icmp_seq=1 ttl=63 time=0.490 ms Output: 64 bytes from 10.1.1.1: icmp_seq=2 ttl=63 time=0.613 ms Output: 64 bytes from 10.1.1.1: icmp_seq=3 ttl=63 time=0.565 ms Output: From 10.2.2.2 icmp_seq=4 Redirect Host(New nexthop: 10.1.1.1) Output: Output: --- example.com ping statistics --- Output: 4 packets transmitted, 3 received, +2 errors, 25% packet loss, time 3000ms Output: rtt min/avg/max/mdev = 0.490/0.556/0.613/0.050 ms PING CRITICAL - Packet loss = 25%, RTA = 0.56 ms|rta=0.556000ms;100.000000;500.000000;0.000000 pl=25%;2;6;0 100.000000:2% 500.000000:6%
This reported packet loss of 25% must have been why we were seeing problems.
After we turned off sending redirects on our routers, which was on by default, we looked into why this was occurring within ping.
One odd thing about ping is that if a count of packets is specified with
-c, the count specifies the number of packets to receive, not the number of packets to send. In the above output, receiving a redirect as the 5th packet meant there was no response received for the last probe sent.
By code inspection and pulling up ping in a debugger, we found that a message for the ICMP redirect was being inserted into the socket error queue. The socket error queue is read by calling the function
recvmsg with the
MSG_ERRQUEUE flag. We tried patching ping to not increment the number of errors when a message of type
ICMP_REDIRECT was received, and that worked.
However, it was curious that this patch was needed. By testing other distributions we found that there was no error received for ICMP redirects, and that if we replaced the stock CentOS 7 kernel with an alternative one, the errors went away. A ticket has been filed with CentOS.