• scheduled maintenance for PDU replacement

    Sat, 25 Jul 2020 17:00:00 -0700 - Alan Post

    We will be replacing a Power Distribution Unit (PDU) in one of our racks during a 4 hour window starting Saturday August 1st at 20:00 UTC. We do not expect any downtime during this maintenance window.

  • Support Diary: the LAN of 16 Million Hosts

    Fri, 17 Jul 2020 06:00:00 -0700 - Paul Scott


    The history of the Internet is marked by stacked innovations. Ideas that seemed good and necessary in their day are often superseded by better ideas that come along later.

    And yet, remnants often remain.

    We had a customer running NetBSD who was having trouble exchanging emails with us. They could contact other people just fine, but not us.

    Investigation and debugging eventually determined that their service was attempting to communicate directly with our mail server rather than through a gateway. The service was using a manually added IP address. We determined that when they added the address via ifconfig, they did not supply a netmask. NetBSD then assigned it a default network prefix of /8. A /8 would include 16 million other hosts, which is nonsensical.

    We initially thought that it was a default behavior of ifconfig, but some legwork showed that the default was built into the BSD kernel, and also the Linux kernel. It all traced back to a system called “Classful Networking,” which was developed in 1981 and was superseded in 1993 by Classless Interdomain Routing, or CIDR. CIDR, in turn, uses variable length subnet masks (VLSM), whereas in Classful Networking subnet masks were fixed.

    NetBSD was assigning the network prefixes associated with the default subnet masks for the old classful system. Our network was set up with a variable netmask, and hence a different network prefix. This put our customer in a different subnet than we were in, and FUBARed our communications.

    So what is Classful Networking? Why did it ever exist in the first place? And why are modern kernels still defaulting to a system that was made obsolete almost 30 years ago?

    History of Classful Networks

    The internet ran into scaling problems early on. The way an internet address is written, the first iteration was able to identify only 256 unique networks. The network was defined by the first 8 bits of the 32-bit IPv4 address, and individual machines were defined by the other 24 bits, or the remaining three octets in quad-decimal notation.

    It was fine to have only 256 huge networks in the early days, when the only networks around were giants like ARPANET (Network Number 10) and SATNET (Network Number 3). But it soon became clear that there would eventually be much more than 256 networks, and the vast majority of them would not need the 16,777,216 individual nodes allowed by those three trailing octets.

    The solution to that was to divide the IPv4 address domain into “classes,” and give them different scaling capacities. There were five classes, assigned letters A through E.

    Class 1st IP Last IP Netmask CIDR Prefix
    A /8
    B /16
    C /24
    D NA NA
    E NA NA

    Classes A-C were regular networks. Class D was assigned multicast addresses, while Class E was reserved for special uses.

    This system was introduced by RFC 791 in 1981. It was replaced in 1993 by CIDR, under specifications RFC 1518 and RFC 1519. Classful Networking lasted for 12 years, but it was only a stopgap. The problem was the huge leap in size from Class C networks (256 nodes) to Class B networks (65,536 nodes). Many emerging networks needed much more than 256 individual machines, so they were assigned to Class B. Yet very few really needed more than 65,000 machines, so the pool of Class B addresses was being wasted.

    CIDR solved this by making the netmask length variable, and assigning a network prefix that was derived from (and so defined) the netmask in a format that was easier to write and remember. The network prefix is simply the number of leading 1 bits in the netmask address, with all other bits set to 0. The old classful netmasks were fixed, and so received the network prefixes defined in the table above.

    CIDR, with its variable netmasks, has served since 1993, and it still the addressing system used today.

    Carryover to BSD and Linux

    The kernels for BSD and Linux were in active development right about the time that the internet was making the transition from Classful Networking to CIDR. Accordingly they incorporated the Classful Networking IP assignments as part of their default behavior.

    This means that when you assign an IP address using ifconfig, if you don’t specify the netmask and/or the network prefix, the kernel assigns the IP address to one of the five classes and gives it the default netmask and prefix for that class.

    Changing these defaults in modern kernels would break backwards compatibility. And so they remain to this day.

    However, warning the user if there is no netmask supplied to a network configuration tool should not break backwards compatibility. If you would be interested in a year of complimentary service in exchange for adding this warning to any of the distributions we officially support, please contact us.

    Special thanks to sigjuice, cpach and others at Hacker News, for invaluable information on the history of ifconfig and the Linux kernel.

  • Possible Data Corruption on Debian Buster

    Wed, 15 Jul 2020 19:00:00 -0700 - Paul Scott

    We’ve received reports from two users on two distinct physical hosts about severe file corruption after they upgraded to Debian Buster. For one of them, the filesystem corruption showed up approximately a week after the upgrade. For another it was several weeks.

    These specific users were running an older file system called ext3, but we’re still gathering data to attempt to isolate the issue. Right now we think it may be related to the latest Debian kernel version 4.19.118+2+deb10u1 and the Xen hypervisor, or it may be specific to ext3.

    We believe that if the problem is specific to Xen, linux-image-4.19.0-8-amd64/4.19.98-1 should be safe. This is the version present in our current image for Debian Buster.

    This article offers some information about what to do if you are using an EXT3 filesystem, have installed Buster, or are considering either of these choices.

    For customers currently running Buster, please take note of the “Information Wanted” section.

    Regardless of your choice of distribution, if you are using EXT3, you might want to consider migrating your filesystem to EXT4. EXT3 is quite old, and incompatibility issues are more likely to occur using it.

    Planning to install Buster?

    Before installing, please do the following.

    Check your file system

    Check to see if your filesystem is ext3. Use the command:

    $ df -T

    Sample Output:

    Filesystem Type 1K-blocks Used Available Use% Mounted on
    udev   devtmpfs   15080720   1508072   0% /dev
    tmpfs  tmpfs   306488 1072305416   1% /run
    **/dev/sda1  ext4  20509264 17468740   1975668  90% /**

    Check the “Type” column for the filesystem that shows “/” in the “Mounted ON” column. The corresponding line is bolded in the above sample. In this case, the filesystem is ext4.

    Not EXT3?

    If your filesystem type shows as EXT4, ZFS, or any other filesystem that isn’t EXT3, you can continue installing Buster.

    If you have ext3, upgrade to ext4

    It is strongly suggested that you switch to ext4 before upgrading to Buster. Follow this procedure:


    Post-update or install changes

    To downgrade your kernel, the following procedure should suffice. You should reboot immediately once it is complete.

    sudo apt-get install linux-image-4.19.0-8-amd64
    sudo apt-get remove linux-image-4.19.0-9-amd64

    Select “no” when the package manager asks if you would like to abort removal of the running kernel.

    Once you have downgraded and performed a reboot, verify with:

    uname -a |grep '4\.19\.0-8'

    Information Wanted

    If you already have Buster installed, we would greatly appreciate your assistance in tracking down when this bug occurs. We would like to know your kernel version and some filesystem metadata. Please run the following commands:

    df -T > busterdata
    uname -r >> busterdata
    uptime >> busterdata
    /sbin/ip -4 addr list eth0 >> busterdata

    Send it to us.

    Send the output file to support@prgmr.com, with your host name and “Buster Data” in the subject line.

    And of course, if you have any problems with a Buster installation or anything else, let us know.

  • Recent Security Updates

    Mon, 13 Jul 2020 10:00:00 -0700 - Sarah Newman

    We have patched, or are otherwise not vulnerable for, the following advisories announced on Tuesday 2020-07-07:

    XSA-317 is a denial of service attack that occurs when the system runs out of free memory. It is not present in the default xen configuration. The vulnerability is caused by not handling all possible error codes returned by a function. It was introduced in 2018.

    XSA-319 is a denial of service attack from HVM systems using “shadow” paging, which is not really used anymore. It also requires a virtual display device to be in active use. The vulnerability is caused by an inverted error check. It may have been introduced by a copy/paste or search/replace error during a refactor. It was introduced in 2016.

    XSA-321 is a privilege escalation vulnerability. It applies only to Intel and requires both PCI passthrough and page table sharing to be enabled. It is caused by not flushing memory caches when required. It was likely introduced after the PCI passthrough and page table features were added.

    XSA-327 is an Arm-only denial of service attack - a missing alignment check for an address. It has been present since 2013. It may have been introduced due to x86 code being made common without the code being reviewed for additional Arm restrictions.

    XSA-328 is a privilege escalation specific to Intel and HVM systems using hardware assisted paging. This vulnerability depends on compiler behavior. It is doubtful that any version of GCC produces vulnerable code. The vulnerability has likely been present since before 2010, when hardware-assisted paging was added. From the code comments, it may have been introduced due to an assumption that writes to memory happen in source-code order. Even when there is no reordering in hardware, it is not safe to assume this in all cases without explicitly adding compiler memory barriers. The problem was fixed by writing to a temporary variable which is then written atomically.

    We thank the Xen security team for providing fixes for these issues.

  • Setting Up an IRC Bouncer on Ubuntu Linux

    Tue, 23 Jun 2020 11:00:00 -0700 - Paul Scott

    Introduction: IRC Bouncers and Why You Want One

    Internet Relay Chat (IRC) is a venerable online chat protocol, dating back to the late 1980s, but it is still widely used today, especially in the world of computing. There are IRC clients for every major operating system and dozens of IRC networks with thousands of chat channels. There’s a heavy emphasis on tech but nearly every topic is represented. When using IRC, you only receive messages when you are connected–a message is discarded after it is sent and history is not stored on the server.

    Enter the IRC bouncer. As the IRC client is responsible for storing and replaying history, it is divided between two pieces of software: the display application (GUI, TUI) and the bouncer. With a bouncer, you still use your favorite IRC client, but you connect to the bouncer (for example bouncer.example.com) instead of directly to the IRC network (such as chat.freenode.net). Your bouncer can remain connected when you’re logged out. When you log back in, your IRC bouncer presents you with the logged chat and messages that you otherwise would miss. Like an IRC server, an IRC bouncer runs on an always connected computer in order to maintain a persistent connection to an IRC network.

    In this article, I’ll show you how to set up ZNC, which is a widely used IRC bouncer application. We’ll go through the process of installing ZNC, setting up a user account, configuring it and logging in to IRC through the bouncer. We’ll also address some basic security issues.

    We’ll be following the process for Ubuntu, but the basics are similar for most Linux distributions.

    Install ZNC

    Ubuntu has a ZNC package in it’s repository, and you can install it with apt-get.

    $ sudo apt-get install znc

    Next you’ll add a dedicated user account for ZNC. This is a good practice when using any application that remains open to the Internet, since it provides a measure of protection for other accounts on your server (in particular your root account).

    I’m doing several things with the following command: I’m creating a new user named “znc-admin”. I’m setting the account up without a password (since this account will never log in) and I’m defining a home directory for the account. We recommend using /var/znc as your ZNC home directory, but you can use any directory you like (except your root directory!). Likewise you can choose a different account name if you want to.

    $ sudo adduser --disabled-password --home /var/znc znc-admin

    Configure ZNC

    With your new account set up, you’re ready to configure ZNC. Switch to the new account, go to the ZNC home directory, and run ZNC’s configuration routine.

    $ sudo su znc-admin
    $ cd ~
    $ znc --makeconf

    ZNC will present you with various options. Here are our recommendations for how to set them up. Note that ZNC presents the default options [in brackets]. If you like the default then just hit return

     -- Global settings --  
    Listen on port (1025 to 65534): 6697  
    Listen using SSL (yes/no) [no]: yes  
    Listen using both IPv4 and IPv6 (yes/no) [yes]: yes  

    We highly recommend using SSL and IPv6 for your traffic.

    ZNC will then create a PEM file at /var/znc/.znc/znc.pem. Next it will ask you to define the username and password that you will use to log in to your IRC bouncer. You will also define the nick and username that you want to use to connect to your IRC bouncer. Note this does not have to be the nickname you use on IRC, but it can be and is easier if it is.

    -- Admin user settings --  
    Username (alphanumeric): <username here>  
    Enter password:  
    Confirm password:  
    Nick [<username>]:  
    Alternate nick [<nick>_]:  
    Ident [<username>]:   
    Real name (optional): 
    Bind host (optional): 

    We recommend not binding to a host unless you have a good reason. The next set of variables configures your connection to the IRC network. Here Freenode.

    Set up a network? (yes/no) [yes]:  
    -- Network settings --  
    Name [freenode]:  
    Server host [chat.freenode.net]: 
    Server uses SSL? (yes/no) [yes]: yes  
    Server port (1 to 65535) [6697]: 6697  
    Server password (probably empty):  
    Initial channels:   

    If you already have some preferred IRC channels in mind then enter them above. Remember to precede the channel name with a hash mark (#) and separate them with a space.

    ZNC will write the config file (/var/znc/.znc/configs/znc.conf) and you’re all set up.

    Allow IRC on your Firewall

    Now that your bouncer is running, it’s time to allow that port on your firewall. You’ll use a utility called firewalld to make sure the correct port is open through your firewall. If you don’t have it already, just install it with apt-get. Note that firewalled will permit SSH by default, so you shouldn’t lose the connection to your server. Even if something goes wrong, if you are using a Prgmr.com system, you can use the management console to get back in and fix things.

    $ sudo apt-get install firewalld

    Now use it to configure the port. We recommend using port 6697, which is the standard port for encrypted IRC traffic.

    $ sudo firewall-cmd --add-port=6697/tcp
    $ sudo firewall-cmd --runtime-to-permanent

    Sign In to Your Bouncer

    You’ll need an IRC client if you don’t already have one. Popular choices include mIRC, Hexchat, and Weechat. Normally you would set up your client to connect directly to IRC, but here you’ll connect to the bouncer, and the bouncer will connect to IRC using the host and user credentials that you set up when you configured ZNC.

    To connect to your bouncer, launch your client. For most clients, in the message field enter the following command:

    $ /server add -tls <znc_server> +6697 <password> <username>

    Or for weechat, if you want your server available via the name znc, your command will look like:

    $ /server add znc <znc_server>/6697 -ssl -username=<username> -password=<password> -nicks=<username>

    For use your Prgmr.com server address. So if your server is named "foo" then use foo.xen.prgmr.com as the ZNC server address.

    If you receive SSL errors, and are using a certificate generated by znc, you can basically whitelist that certificate via its fingerprint. Take the output from the below command:

    $ sudo cat /var/znc/.znc/znc.pem | openssl x509 -sha256 -fingerprint -noout | cut -d '=' -f 2- | sed 's/://ig'

    And add it to your irc client:

    $ /server modify <znc_server> -tls_pinned_cert <above_output_line>

    Or for weechat:

    $ /set irc.server.znc.ssl_fingerprint <above_output_line>

    If you ever want to change these settings then you can do so by interacting with the *status user. Type help for a list of options.

    Further Reading

    More documentation on ZNC can be found at the ZNC wiki.