March 2013 Archives

UPDATE: We are pushing back the first possible breakage of the old ip addresses to no earlier than April 24th (2013-04-24) to give people more time to switch over. If you need more time please email The absolute drop-dead date for switching to a new IP is May 16th.

Some time ago, I got my own allocation from ARIN.
If your IP is in that block? ignore this, it doesn't have anything to do with you. (I mean, other than the bits where I try to distract you from the pain of renumbering by talking about my network upgrades)

For many reasons, (none of them good) I have not given most of you new IP addresses until just last night. But, as of last night, I have.

I've emailed people their IP, but you can also check for the existence of the dns record [username] If such a record exists, it points at the new ( direct allocation) IP address for your guest. My hand is now forced, as you can see from our contract end dates wiki page[1], our contract with EGI is up on 2013-05-17. We won't be renewing. Instead, I signed a 5 year lease at coresite Santa Clara[2]. Now, my reasons for doing so can be questioned, but it is a /much/ nicer data centre, and we are getting a somewhat better deal. It is dramatically closer to where I sleep (and our new office) so there should be dramatically better hardware response times to outages during rush hour. And it's brand new. Did I say it was really nice? I am a little bit impressed; the place has cold row isolation and really big racks, with a lot of room in back. For the first time ever, I have vertical PDUs that are properly mounted (no zipties!) And they are nice PDUs, too; Avocent PM3001V-401 units, with per-outlet metering. The network is going to be way better, too. I've got the new PFY lacing cat6a as we speak. It looks like the network is going to be force10 based, but we will have ten gigabit uplinks in many places on the network, and to some of the servers (most will still be gigabit-only, but the intra-switch connections will all be 10G or better, when this upgrade is complete.) The network is being built out to be strong enough to handle not only increased network quotas, but also to handle some sort of NAS or SAN device, so I can finally start letting people have storage independent of ram. But uh, yeah. The upshot is that on 2013-05-17, the IP addresses that are reassigned to by EGI will stop working. I will temporarily break those IP addresses several times before that (not before 2013-04-20, though.) to get the attention of people who don't check email. Sometime after 2013-04-20 I will alter your DHCP settings to point at the new IPs rather than the old IPs, but most of you statically assign your IP addresses, so you will need to change your IP manually.(really, you probably want to set your IP manually anyhow... you want to control this switch over, not me.) Note, the new subnet mask is, and the default gateway is the first three octets of your IP address; the fourth octet is 1. Please note: you may need to do a hard reboot before the new antispoof rules take effect and allow you to use the new IP. But, aside from that? changing the IP or adding a new IP should be the same as doing so on a regular server. If you are on a RedHat based distro, your IP address is likely set in the file: /etc/sysconfig/network-scripts/ifcfg-eth0 and the default gateway is likely set in /etc/sysconfig/network If you are on a Debian-based distro, your IP address is likely set in the file: /etc/network/interfaces There is nothing at all wrong with running both your old IP and this new IP at the same time. On RedHat-based distros, create a new file, /etc/sysconfig/network-scripts/ifcfg-eth0:1 (it's easiest to copy from ifcfg-eth0 and change the IP) If you wish to change the rdns, email and we'll take care of it. The interface to change rdns expects only 1 ip per guest, and through May, at least, you all have at least two now. can also help with problems changing IP addresses. [1] [2]

update on pillar

| | Comments (0)
19:38 <@prgmrcom> man, pillar hung
19:43 <@prgmrcom> mm.  if this comes up, I'm running out for breakfast.

19:52 <@prgmrcom> ok, blog up.
19:52 <@prgmrcom> food, I guess.
20:19 <@prgmrcom> getty fixed on pillar
20:29 < customer> about that...i'm on pillar and still having problems.  when i ssh, i get "Error:  Unable
                    to connect to xend: No such file or directory. Is xend running?"  then i get the menu, but the same message when i
                    choose console, and no console.  and the VPS itself is down as far as i can tell.  private msg me if you want the
                    actual account name
21:02 <@prgmrcom> picomancer: uh,
21:03 <@prgmrcom> oh no
21:08 < customer> I'm seeing the same thing (also on pillar).
21:08 <@prgmrcom> yeah.
21:08 <@prgmrcom> I screwed something. hangon
21:16 <@prgmrcom> [root@pillar ~]# rpm -qa |grep xen
21:16 <@prgmrcom> kernel-xen-2.6.18-348.3.1.el5
21:16 <@prgmrcom> kernel-xen-2.6.18-348.2.1.el5
21:16 <@prgmrcom> prgmrxen-dist-4.0.1-1
21:16 <@prgmrcom> god damnit.
21:16 <@prgmrcom> that should not be possible
21:16 <@prgmrcom> oh
21:16 <@prgmrcom> no shit
21:17 <@prgmrcom> it has the el5 'kernel-xen' packages, but not the el5 'xen' packages... thus no conflict with the old prgmrxen thing

Anyhow, it's coming up for real now, I added pv-grub and the permissions fix.

(the irc channel is #prgmr on

pillar hang/crash

| | Comments (0)
so pillar hung.  consistent with a bad consumer disk, but it could be other things, too.  dono what it was; pillar has 4x RE4 'enterprise' disks, so that shouldn't be it.  Running disk diagnostic now. 

Meanwhile, you should be back up.  sorry for the outage. 

cauldron crashed

| | Comments (0)
cauldron crashed, probably due to an old version of kernel-xen and xen that we compiled ourselves.  We updated it to the latest kernel-xen and xen from centos and the guests should be coming back up now.

We are going to be updating the other servers that run our custom builds to the stock centos packages in the next few days (Was going to be Tuesday, but we are moving it up due to the problems they have been causing lately).  I'll send out an email to users on the affected servers tonight.