Prgmr.com provides low-cost, high-flexibility webhosting using Linux, NetBSD, and Xen.This file is no longer maintained. For current status, see our Xen hosting blog.
there is something of a gap in prgmr.com history; in late 2007, it looked like I was going out of business. I found a new home for my co-location customer at the time, vuvox.net, and began shutting things down. In December of 2007, my situation changed and I started ramping things back up.
What follows was the prgmr.com status page from January of '06, when I started this, until October of '07, when we started shutting things down.Prgmr.com Xen hosting News
Folsom (220.127.116.11) crashed this afternoon and was rebooted. If you are having trouble accessing your VPS, drop an email to firstname.lastname@example.org. All customers will be moved off folsom onto a new server at some point in the near future, as folsom looks unreliable.
Folsom crash and reboot
Folsom (18.104.22.168) crashed this morning and was rebooted. If you are having trouble accessing your VPS, drop an email to email@example.com
Folsom crash and reboot
the reboot/upgrade will happen next saturday, 2007-07-07, as all of us are busy with other things today.
Coloma Reboot- we are delaying
The crash earlier today due to a bug in the debian Xen packages. We will be rebooting Coloma (on purpose this time) on June 30 to upgrade the kernel packages to prevent this from happening again.
Coloma Reboot- we found the cause
Coloma crashed and took close to 2 hours to be rectified. If you have any troubles with your VPS on that machine (console IP 22.214.171.124) drop an email to firstname.lastname@example.org and we'll get things sorted out.
Both Folsom (126.96.36.199.) and Coloma (188.8.131.52) were successfully rebooted and all VPS on those machines should be running again. Coloma customers got an extra reboot due to a bad cable.
We now have Debian Etch (4.0) and Ubuntu Feisty Fawn (7.04) available for new orders. There are ways to update exsiting customers (dist-upgrade works for debian 3.1 to 4.0), email support if you'd like us to make a spot backup before attempting.
More Image types Added!
Folsom (184.108.40.206) and Coloma (220.127.116.11) will be rebooted on June 9th for a hardware change. Reboot will likely happen later on in the evening (All times ever given are Pacific).
Scheduled Reboots for Coloma and Folsom
Prices for all of our VPS services have been lowered. Sign up Now while we have capacity.
Ram amounts bumped!
Both Folsom and Coloma had some issues while performing a hardware upgrade to another machine, and were rebooted. Things should be up and running as usual.
Subscriptions for new customers are open once again! Visit our Xen Hosting page and signup for your own Xen VPS now.
Subscriptions active once again!
Customers on Coloma (18.104.22.168) will be rebooted in the next 24 hours for a hardware upgrade. Downtime should be minimal. SAN changes will also be done and may affect other machines and VPS.
Coloma will be rebooted
If you've not signed up for a paypal recurring payment or made payment via eGold please do so before 18May 2007. Beginning 19May 2007, we will begin shutting down VPS that have not been paid for. If you are unsure of whether of not you have paid email email@example.com and ask!
adding a new fibre-channel disk to the zone caused some unexpected badness in (we think) the linux SCSI, LVM, or MD subsystems.
Both folsom and auburn had a driver mismatch in the volume manager snapshot setup, locking access to currently unopened logical volumes. Things would work fine until you rebooted. To clear this both machines were rebooted, auburn came up clean, folsom had further issues necessitating a longer downtime. So why did this all happen? We're doing backups! All backups are stored offsite and are created from creating a snpashot of each logical volume, one at a time and using rsync to only update the data that has changed since the last use. Doing backups this way resulting in 0 downtime for a VPS, and doesn't use any bandwidth on the VPS either.
Folsom (22.214.171.124) and Auburn (126.96.36.199) reboots
Some customers have received emails from support with links to a page for monthly PayPal subscriptions. We're calling everything even as of today, so there's no need to worry about missed back payments, it wasn't your fault we have a severe lack of billing and accounting knowledge. Things should now proceed smoothly.
Monthly PayPal subscriptions, bandwidth graphs now available
Also, if you're interesting in knowing how much bandwidth your using and don't want to worry about setting up SNMP and the needed software, we now offer it for free! Email firstname.lastname@example.org and we'll get you the login info for watching how much bandwidth you're using. (Cusomters with console IPs of 188.8.131.52/auburn: we're having issues getting the machine to output names for each user, so we can't offer this for you quite yet.
Finally, there's rumours floating about that we'll soon be accepting signups quite soon. Keep an eye here for further news.
Auburn (184.108.40.206) was rebooted this morning to clear some lingering troubles with the SAN and consle access. Folsom (220.127.116.11) suffered a failed disk, that caused the system to become unstable. A reboot and rebuild on to another disk was done with all VPS on the machine halted to prevent any data loss. If you are unable to access your VPS, or notice data loss, please email email@example.com. (Auburn and Folsom customers will not be charged for the month of April)
Auburn reboot, Folsom disk rebuild
The console menu has been updated, adding VPS status and a link to the kernel modules for that particular Dom0. If your VPS is moved, this will change along with it. Additionally, the reboot listed below did not happen, no need to fret.
Console Menu Additions
on 2007-03-03, all servers will be rebooted and the kernel will be upgraded to xen 3.0.4 in an effort to resolve some of the issues we've been having with domains randomly crashing, and with servers crashing after a month of uptime.
all servers reboot
we have multiple issues on the server that hosts users on .104. We will be taking it down tonight around 9:00PM PST for a reboot.
A minor networking blip took the DHCP server offline for longer than the lease time (both of which have been repaired, the lease time is now a day instead of 10 minutes.) If you are having trouble accessing your VPS via its IP address, console the machine and either reboot it, or run ifdown eth0 ; ifup eth0 to get stuff working again.
Routing Blips and DHCP oddities
the cutover is complete; please e-mail support with any further problems you see.
Xen2 users moved to xen3
all users with a console server ending in .2 experienced a reboot late sunday night.
reboot of folsom
our only xen2 host went down hard; we brought it back up and will be migrating all users off of it (this is also why the prgmr.com website was down yesterday, and why support was bouncing until this morning.) Of course, all users on that server get a free month. (and will be upgraded to a 2.6 linux kernel or a 3.1 netbsd kernel)
Xen2 host down hard
we will be rebooting the Xen2 hosts in order to deal with some disk problems.
Xen2 hosts reboot
Some customers have reported losing the default gateway setting for their interfaces. A quick reboot or ifdown eth0 ; ifup eth0 as root should set things properly.
No default gateway?
Machine locked up, requiring a reboot.
After the reboot in the the previous note, some users may have light data corruption. Console your Xen to see if you need to perform an fsck of your disk. If you are unable to fix it via conosle, email support for further assistance (we can fsck your disk, but only if your xen is offline.)
reboot to fix some MD issues... details on our 'gone wrong' troubleshooting page http://wiki.xen.prgmr.com/index.php/Gone_wrong
We have now determined the extent of the data loss for customers on 18.104.22.168, and it's massive. If you cannot currently access your xen via ssh or console, the sad news is that we were not able to recover any useable data for your xen VPS. This leaves you with a choice: Remain with prgmr.com and get 2 additional months of free service (as mentioned below), or leave prgmr.com and get your monthly fee refunded. Email support with your signup email address (or IP address of your VPS), and what distribution you would like to have your VPS recreated with.
Folsom/22.214.171.124 Data loss
Users on 104 have a new kernel, modules available from us here: Link-a-tude
Auburn/126.96.36.199 kernel modules
We are recovering what data we can
Folsom/188.8.131.52 Dom0 data loss recovery
We trashed our MD disks on folsom; more details on the wiki once it's back up.
all effected customers will recieve either 2 free months or a refund. Sorry.
Folsom/184.108.40.206 Dom0 data loss
Not to left out, folsom also was rebooted.
Folsom/220.127.116.11 Dom0 crash
we are swapping out the hardware as we speak.
104 Dom0 crash and down hard
Customers with console IPs ending in 104 will have noticed a 30 minute downtime when the Dom0 machine crashed, and was rebooted. If you are still having problems at this time, please email firstname.lastname@example.org.
104 Dom0 crash and reboot, again
We are once again acceptnig new orders for the Xen VPS service.
Orders being accepted again
Customers with console IPs ending in 104 will have noticed a 3hrs downtime when the Dom0 machine crashed, and did not recover as expected. If you are still having problems at this time, please email email@example.com.
104 Dom0 crash and reboot
Many customers have been asking for certain kernel modules for their linux DomU VPS. You can pick them up from This link or simply do: wget http://prgmr.com/xen/modules-2.6.13-xen.tgz on the command line to grab the file as well.
Kernel Modules for Xen3 DomUs
we don't appear to have what it takes to maintain the 99.9% uptime we are currently promising. The SLA has been lowered to 99.5%. this means that we can have 216 minutes of downtime a month. that is 3.6 hours. I think this is realistic.
There have been problems. The problems all seem to stem from the fact that we didn't include swap on the dom0 host machines, and they ran themselves out of addressable memory space, with nowhere to go. Older customers previously using 18.104.22.168 for their console IP (not those recently moved!) will be brought back up in the next day or so, with our greatest apologies. None of the downtime for the last week has been planned, nor do we hope to ever repeat it with any regularity
Cusotmers who have been using 22.214.171.124 as a console IP should start using 126.96.36.199. We are still in the process of moving and setting up the machine after this move, and connectivity is currently spotty while we finish up.
Consle IP changed for some
until we fix our capacity issues.
new orders disabled
reports of "no data loss" were premature. apparently a mirror
broke sometime last week; when we brought everything back online, we rebuilt from the week-old mirror. this effects some customers on Folsom. I'm really
sorry, and aside from the free month I will be giving customers that stay something extra. We have figured out the SAN issue; a cable got bumped, and we've prevented this by multipathing our array. We will be keeping a much closer eye on our mirrors in the future, as well.
Users with a console IP ending in 2 will be back up with no data loss when this can be read.
EXP-500 offlinefor some reason, our EXP-500 full of 73G disks is offline; this means that many customers are hung. We've got no idea what happened; our san switch simply can't see the disks. Neal is on his way to the co-lo to check it out now. It looks like
planned downtime on 250250 reboot was slightly less successful with a downtime of 40 minutes.
planned downtime on 104the planned reboot of the hosts on 104 was successful, with a downtime of only 20 minutes.
Unplanned downtimeAll users that have a serial console ending in .2 experienced disk issues and an unplanned reboot around noon. most of you should be back up, a few others will be up in a few minutes.—UPDATE - All customers are now back up and functional. Email firstname.lastname@example.org if you still have trouble accessing your Xen.
Downtime noticeall customers that use serial consoles ending in .104 and .250 will experience a reboot Saturday night, 2006-09-16. we will do our best to make this downtime as short as possible. our Service Level Agreement will be honored. advance notice is no excuse.
2006-09-11FreeBSD-current and NetBSD-3.1-rc2 Xen3 DomU kernels are in testing. E-mail email@example.com if you are interested in participating in the beta.
2006-09-10Jin Chong has joined the prgmr.com team. Chinese language support is now available through firstname.lastname@example.org
2006-09-09Customers will be receiving emails from staff asking them to reply when they are ready to have their xen moved, especially customers with console IPs that end with 104 and 137. Also, coming soon: Recurring PayPal payments.
2006-09-01disk space increased for all plans with 512M ram and up. Existing customers; arrange for how you want us to give you the disk next time you renew; we can use growfs, alternately we can give you another partition with the space. If you are currently on a Xen 2 server and are running Linux, we will ask to move you to a Xen3 server during the upgrade (this involves upgrading to a 2.6 linux kernel)
2006-09-01some of our customers had an interruption in network services around 10AM this morning.
Aug 31 05:40:19 DebianXenHost250 kernel: peth0: can't fill rx buffer (force 0)!ifconfig peth0 down; ifconfig peth0 up seemed to fix it... we are researching a real fix, but for now, Neal has implemented a script that tries to ping the gateway once a minute; if it can not ping the local gateway, it cycles peth0. This should automatically fix the problem if it happens again before we find the root cause.
2006-08-27if you are having difficulty with swap, swapon -a fixes it
2006-08-27Ordering is back up for our Linux images. Also, the prices on the 512M and 1024M plans have been lowered; This effects existing customers as well as new customers.
2006-08-27after much pain and suffering, Folsom, our new 2x2Ghz xeon box w/4G ram is up. We added the first customer tonight. We should be able to get through the backlog fairly quickly and will begin accepting new customers shortly.
2006-08-24the 'order' paypal button has been disabled until I get some more capacity up; also NetBSD has been removed from available systems- we are working as fast as we can on both, considering our limited time.
2006-08-23the reports of the new xen 3 dom0 coming online are premature; we are having hardware issues with the box. we will be digging a replacement out of the kitchen in the next two days. this means we are out of capacity; new customers need to wait until the new box comes on line. If we don't have this box online by Friday or if the queue exceeds the capacity of the replacement box I will take down the order link.
2006-08-23A new Xen 3 Dom0 (machine that hosts customer images) has been brought online, bringing the total number of Xen machines to 4 (2 running Xen 2, 2 running Xen 3). Those that have been waiting shouldn't have to wait much longer. --Neal K
2006-08-22One of our old xen 2 NetBSD boxes crashed hard today; the box is back up now. only users who's console IP ends in 104 were effected. all customers ought to be up now. If your xen is still down, email support. the prgmr.com web and e-mail servers were on that box, which explains the bounces.
2006-08-15The cutover from Cogent to Rippleweb was messy, and resulted in nearly a full day's down time This was completely my fault, (That is, Luke Crawford's) as I set the cut off date for the old service to early. Rippleweb moved quickly to help me recover; the rest of the down time was due to a faulty cross-connect. All customers get one month of credit for this, as this has been a rough month (and we are not entirely out of the woods yet.) On the upside, we have avoided renumbering. Neal has survived the baptism by fire, and we can expect prgmr.com services to improve in reliability now that I have help again.
2006-08-13Neal Krummell is joining the prgmr.com team. Neal and I both worked at jps.net back in the day.
2006-08-12Thuy Vu has left prgmr.com to pursue a graduate degree in computer science at the University of Michigan. I just want to say that without her, prgmr.com would have disappeared a long time ago.
2006-07-17Luke Crawford has resorted to full-time contract work to pay the bills.
2006-06-20Ram allocations have doubled for all Xen products. For $5, you now get 64MB ram, 128MB swap, 1GB 0+1 SCSI disk, and 10Gbit network transfer.
2006-06-19In celebration of the new prgmr.com SAN, the price for 0+1 SCSI disk space have dropped to $1.50/GB,
2006-06-19I am speaking at the Davis Linux users group meeting tonight. I'll be giving away 10 32M Xen Images, and ranting about the importance of having multiple providers.
2006-06-12We are having some issues with one of our xen dom0 servers. Website will bounce a few times and paying customers will be moved to a more reliable server. problem was insufficient MAXUSERS and NMBCLUSTERS on Dom0.
2006-06-10we now accept paypal, e-gold, checks and money orders.
2006-06-09I have the jailtime.org imgages working with my system. they are now available to all Xen VPS customers
2006-06-01Prgmr.com Xen-based virtual servers are now available to the general public. I can provide NetBSD images, or you can upload your own image and kernel.
2006-05-01singin.com is another music-related managed- hosting customer. Singin.com provides a way for creators to distribute their music over the internet.
2006-04-10Michael Crawford's Ogg Frog Free Music Torrents has migrated to one of our NetBSD Xen images.
2006-03-02Thuy Vu and Luke Crawford are now both working full time on on prgmr.com projects. We are actively seeking consulting gigs.
2006-02-20Our NetBSD xen images are in free beta right now... the beta is limited to people who know a prgmr.com employee. Email us if you are interested.