• Transient network outage

    Tue, 06 Mar 2018 23:47:00 -0800 - Sarah Newman

    Because of a miscommunication in a maintenance notice from our data center provider we unexpectedly had multiple short outages starting at about 2018-03-06 22:03 -0800 before we turned off our BGP connection using the given cross connect at about 22:25. Since then we have gotten confirmation that there should be no more outages on that cross connect and we’ve turned that BGP connection back on.

  • Public-facing Memcached Instances May Be Used in an Amplification Attack

    Wed, 28 Feb 2018 11:00:00 -0800 - Chris Brannon

    We recently learned that public-facing memcached instances have been used in amplification attacks. Here is a relevant quotation from Arbor Networks [1]:

    We have observed a considerable uptick in memcached reflection/amplification attacks ranging in size from a few hundred mb/sec up to 500gb/sec and larger. The amplified attack traffic is sourced from UDP/11211, with a packet size of 1428 bytes (1442 bytes with layer-2 Ethernet framing included), and no fragmentation (memcached segments large responses at layer-7, as does ntp). The attacker typically ‘primes’ a given set of memcached reflectors/amplifiers with arbitrary-length key/value pairs, and then issues memcached queries for those key/value pairs, spoofing the IP addresses of targeted hosts/networks. Both the priming queries and the attack-stimulus queries can be directed from source ports of the attacker’s choice to UDP/11211 on abusable reflectors/amplifiers, meaning that the attacker has full control of which destination port is targeted on the destination hosts/networks.

    It should also be noted that memcached priming queries can also be directed towards TCP/11211 on abusable memcached servers. TCP is not currently considered a high-risk memcached reflection/amplification transport as TCP queries cannot be reliably spoofed.

    According to the sources I have read, legitimate memcached traffic across the public Internet is rare, and the simplest fix for this issue is to block both inbound and outbound UDP and TCP traffic on port 11211.

    If you are running memcached you may wish to consider binding it to localhost, rather than the default behavior of binding it to all ports. John at Nuclearfallout Enterprises, Inc. has shared the following tip for doing so:

    On CentOS/RHEL,

    1. Open /etc/sysconfig/memcached in your favorite text editor.
    2. Change the line currently reading OPTIONS=”” to OPTIONS=”-l”
    3. Save the file and exit the editor.
    4. Restart memcached with this command: /etc/init.d/memcached restart

    On Ubuntu/Debian,

    1. Open /etc/memcached.conf in your favorite text editor.
    2. Locate the line containing the -l parameter and adjust it to read “-l”. If there is no line with a -l parameter, add one at the end of the file.
    3. Save the file and exit the editor.
    4. Restart memcached. To do this, you may need to use command “service memcached restart” or “systemctl restart memcached” depending on your version of Ubuntu.

    Further Reading

    1. memcached Reflection/Amplification Description and DDoS Attack Mitigation Recommendations
    2. The memcached amplification attacks reaching 500 Gbps
    3. Open Memcached in Zimbra 8.6.0_GA_1153
    4. In-the-wild DDoSes use new way to achieve unthinkable sizes
  • Xen Security Advisory Announced Today

    Tue, 27 Feb 2018 10:30:00 -0800 - Chris Brannon

    The following Xen Security Advisory was announced today:

    All of our affected systems have been live-patched to address these issues. None are vulnerable.

  • Scheduled downtime for internal systems

    Tue, 16 Jan 2018 09:30:00 -0800 - Alan Post

    On Friday January 19th at 04:00 UTC we’ll perform scheduled maintenance on our internal systems. The website (including the blog and billing), management console, and the support ticket tracking system will be unavailable for up to town hours.

    We’ll provide real-time updates in #prgmr on Freenode.

  • Updates on speculative information disclosure

    Tue, 09 Jan 2018 22:10:00 -0800 - Sarah Newman

    We expect to have a usable PV-in-HVM shim within the next couple of days to patch meltdown, at which time we will ask all PV users to reboot their domains. We will send an email when this is ready. Customers who wish to convert to HVM or provision a new HVM VPS instead should write support.

    For Spectre variant 2 we are still waiting on patches from Xen, and for many CPUs we are still waiting on microcode updates. Our best estimate for when everything may be ready is the weekend of the 20th, but we have little confidence that every single microcode update will be available by then. If we are able to patch some but not all hosts, we will patch the hosts we can.

    PV VPS kernels are not affected by meltdown, and most updates so far apply to meltdown only.

    The following distributions have released updates and we’ve tested whether they boot:

    • CentOS 6 2.6.32-696.18.7.el6.x86_64 - This works for HVM but not PV. If your VPS is in PV mode, modify “default” in /boot/grub/menu.lst to avoid this kernel. Please contact support if you wish to have your service migrated to HVM in advance of the bug being fixed.
    • CentOS 7 3.10.0-693.11.6.el7.centos.plus.x86_64 - this works for HVM but not PV. If your VPS is in PV mode, modify grub.cfg or menu.lst to avoid this kernel. Please contact support if you wish to have your service migrated to HVM in advance of the bug being fixed.
    • Debian 7 Wheezy 64-bit - 3.2.96-3
    • Debian 8 Jessie 64-bit - 3.16.51-3+deb8u1
    • Debian 9 Stretch 64-bit - 4.9.65-3+deb9u2
    • Fedora 26 64-bit - 4.14.11

    The following distributions have released updates but we have not had a chance to test every version on both PV and HVM:

    The following distributions do not have updates yet as of 2018-01-05 8:30 UTC:

    We have not yet updated any of our distribution images, but we are working on testing them. Updates to the kernel can always be applied via your distributions usual update mechanism.