• Xen Security Advisories Announced Today

    Thu, 12 Oct 2017 13:00:00 -0700 - Sarah Newman

    None of our production systems are vulnerable to any of the Xen security advisories announced today. The majority of our host servers were live patched to either fix or work around all the problems found which affected the VPSs running on them.

    However, during testing in advance of the maintenance windows we found that disabling linear (nested) page tables, which was a workaround for XSA-240, caused NetBSD VPSs to crash despite them explicitly being called out as not affected in the original advisory announcement. Unfortunately a live patch can only be applied for disabling linear page tables, not for fixing the vulnerability. Therefore host servers running NetBSD VPSs could not be safely live patched.

    We believed we were able to identify all NetBSD VPSs in advance of the maintenance windows because in the Xen key-store database called xenstore, those VPSs set a key called ‘vifname’. It turned out that NetBSD 5, which has been out of support as of 2014, did not set this key and two VPSs had to be moved to a different host server after they crashed. We took the unusual step of parsing console output and identified one additional NetBSD 5 VPS in advance of the remaining maintenance windows. We strongly encourage all customers running NetBSD 5 to upgrade.

  • Website Refresh

    Tue, 26 Sep 2017 13:45:00 -0700 - Will Crawford

    Earlier this month we rolled out a new website. This post will tell you our reason for refreshing it, our process and testing, and what we think its advantages are.

    Prior to the refresh, prgmr.com’s website was a bunch of files sitting in /var/www/html. We had no version control beyond renaming old versions something like index.back.YYYY-MM-DD, which meant we had little in the way of a history of changes. Similarly, we had no way to update content centrally without logging in to the actual web server and mucking about with it.

    We had little code deduplication for things like headers and footers, but we did have some of that present thanks to Server Side Includes. (These were state of the art back in 1996.)

    It wasn’t great, but it was manageable. Unfortunately, it also looked terrible at low resolutions, including mobile browsers. It’s 2017; it’s time to support mobile devices.

    So, we moved to Jekyll - the same static-site generator that GitHub uses - and made a lot of changes. This isn’t our first time using Jekyll; we moved our blog to Jekyll back in 2015. Now, all of our static web-facing content is managed with Jekyll in a Git repository.

    Unlike our old Movable Type blog, the rest of our website was just as static before as it is now, so there’s much in the way of a reduced attack surface, but it does afford us some other advantages.

    Moving from a bunch of stitched-together Server Side Includes was pretty painless. In fact, if you really want to, you can replicate SSIs using Liquid’s templating system to {% include %} things. That’s exactly what my first draft did. Since then, I’ve revised our design to take advantage of Jekyll’s ‘layout’ system which is a big boon for maintenance. Though we could already make changes in one place to affect multiple pages with SSIs, now we can better track where those changes are and what they’ll affect.

    We made some other changes as well. As you can see, there’s a new theme for the website. After doing several mockups, we went hunting for resources. The current instantiation of our website is based pretty heavily on one of the w3 templates. We hoped by using an existing template we’d have fewer display bugs when it came to device testing.

    New to the site is a nav header. Turning this menu into an icon you can expand with a click is a common problem in front-end work. For the most part, it’s a solved problem, albeit a bit of a hacky one - especially if you want to support no javascript AND browsers as old as IE8. We do both… though not at the same time. Fortunately, I don’t know of any mobile devices that surf the web with IE8 and no JS, so I think we have most of our bases covered.

    To support folks with NoScript or JS turned off, we use a newer feature in CSS that selects for the toggle state of an element. This works great on more recent browsers, but some older browsers don’t support this feature. We support those browsers with a more conventional JavaScript-based toggle to control the state of the element. However, we use media queries to detect when the menu toggle should replace the top nav bar, which IE8 doesn’t support. To remedy this (and HTML5 elements) we also use JavaScript - there are some extant polyfills HTML5Shiv and Respond.js that do this job very well.

    Sticky footers are another solved problem - the new CSS feature “FlexBox” handles this really well for footers with a variable height. Older browsers don’t support FlexBox natively, unfortunately, and we had a lot less luck with flexibility.js, the premiere polyfill that adds Flexbox support to old browsers. There’s a table-based hack that seems to have good cross browser compatibility, but it seems to exploit a bug. This might result in the site breaking unpredictably down the road, so I went with Flexbox in the end. Some of the truly mystifying bugs on some older mobile browsers were solved with Philip Walton’s invaluable advice.

    Once the site was feature complete, we needed a way to test on a variety of platforms and browsers. We used CrossBrowserTesting. This is not news to any seasoned web developer, but being able to test changes quickly on a variety of browsers - especially old ones and mobile browsers - makes it easy to test new changes and detect exactly what broke when something inevitably breaks. (This was extremely helpful with the sticky footer bugs I encountered - some of them produced very bizarre rendering errors that I wouldn’t have attributed to the footer had I not tested before and after.)

    I learned during screenshot testing that it wasn’t easy to test a site without Javascript. I anticipated it would be - CBT alleges support for Selenium scripts, and it should be relatively easy to disable JS with Selenium. Unfortunately, they require JSONified Selenium scripts built with Selenium Builder, an extension for Firefox that requires Firefox version <49.

    One thing screenshot testing didn’t help with was testing screenreader support. I haven’t worked on screenreader support before. Most of it seems to be doable - at least in our case - by using tags aria-label and aria-hidden on appropriate elements. Unfortunately, the topnav menu doesn’t work very well with mobile screenreaders. According to our resident screenreader user Chris Brannon, who was kind enough to help me out with testing my attempts to support screenreaders better (Thanks Chris!) one link cluster is better than two, so I added all of the links that are in the header but not the footer to the footer using a hack I found on StackOverflow - specifically, the second option, using “Position + Clip + Collapse.”

    Last, we wanted to use Font Awesome for some sigils, but we wanted to self-host the font. One of the advantages of using a CDN is often people will already have the content cached after downloading it at another location. We don’t use very many sigils, though, and we’d rather self-host our website, so I was able to off-set this for the most part by putting together a custom font using only the five glyphs we’re using with Icomoon - a savings of 73kb.

    Moving to a new theme is a pretty big change for us: we haven’t updated our website’s look and feel in a significant way since about 2009. We hope the changes we’ve made make it even easier for users to find out more about our services no matter what device they’re using to discover prgmr.com.

  • Upgrade with caution: CentOS 7.4 (1708) does not support PV

    Thu, 14 Sep 2017 15:00:00 -0700 - Alan Post

    If you are running CentOS 7 on PV you will not be able to boot after an upgrade to CentOS 7.4.

    Yesterday, Wednesday 13 September, CentOS 7.4 (1708) was released. While this is only a minor release (with CentOS 7.3 having been released in August 2017), the release notes contain a critical erratum:

    If you are running CentOS-7 as a Xen domU in ParaVirtualization (PV) mode, an upgrade to CentOS-7 (1708) will cause the VM to not be able to boot. You must use HVM (full emulation) or PV-on-HVM mode to run this version of CentOS with the Xen hypervisor.

    You can find discussion of this issue on CentOS-Virt.

    To determine whether you are running a PV system, log in to the management console. If you have a legacy console you are on a PV system. On the latest management console, the system details (option ‘a’) screen will display: “Virtualization mode: PVM” if you are running PV.

    You will need to upgrade to an HVM VPS to run CentOS 7.4. For those wishing to self-service this migration, you will need to order a new HVM VPS. We will give you up to a month’s credit while you migrate your PV system to HVM. You can migrate your existing IP address by writing support@prgmr.com before you cancel your PV VPS. We can also migrate an existing PV VPS in-place if you’re confident in your ability to replace pv-grub with grub. You will need to change your serial console from hvc0 to ttyS0.

    If you wish to continue running PV on CentOS 7.4 you will need to run an alternate kernel. The CentOS Plus kernel works. Here are step by instructions on how to install it:

    yum remove $(rpm -qa | grep kernel-3.10.0-693)
    yum install yum-utils
    yum-config-manager --save --setopt=centosplus.enabled=1
    yum-config-manager --save --setopt=centosplus.includepkgs='kernel*'
    yum-config-manager --save --setopt=base.exclude='kernel*'
    yum-config-manager --save --setopt='updates.exclude=kernel*'
    yum update
    yum install kernel-plus
    # For grub2
    /sbin/grubby --grub2 --set-default=$(ls /boot/vmlinuz*centos.plus*)
    /sbin/grubby --grub2 --default-kernel # verify this has centos.plus in it
    # For pv-grub
    /sbin/grubby --grub --set-default=$(ls /boot/vmlinuz*centos.plus*)
    /sbin/grubby --grub --default-kernel # verify this has centos.plus in it

    If you have already upgraded and your host will not boot, you may be able to boot using your previous kernel using grub. Grub can be accessed via option 1 (out of band console) on your Management Console.

    If you have any questions are need more information, please write support@prgmr.com.

  • XSA 231-234 security patches applied

    Thu, 14 Sep 2017 12:53:00 -0700 - Alan Post

    All of our systems have been patched to address the following XSAs made public Tuesday, all of which are privilege escalation vulnerabilities:

    In June, when we patched XSA 216-225, we deployed LivePatch support. While it was not possible to use LivePatch in our August maintenance window, we were able to LivePatch these XSAs, which let us avoid rebooting most of our machines. Unfortunately, we did reboot one server due to live patching bug in Xen. That bug has now been fixed. We’ll have a separate follow-up post with details on that bug.

  • Partial network outage

    Wed, 06 Sep 2017 22:17:00 -0700 - Sarah Newman

    There was a partial network outage starting at 21:17 -0700 where HE.net was reachable but not routing traffic to a substantial portion of the internet. We stopped sending traffic to them until the issue appeared resolved. We have not yet heard anything regarding a root cause.