- Chris Takemura and Luke S. Crawford . The Book of Xen. no starch press. 312.
System administration skills
Configuration Management systems, devops tools: puppet (2 years), Ansible (2 years), custom shell, perl and python systems (15 years), spacewalk (6 months), cobbler (1 year), PXE, DHCP and TFTP (10 years), RedHat Kickstart (started from pxe/tftp) (7 years), Debian preeseed (1 years), libvirt (2 years).
Programming languages I have used to write and/or modify production applications: python (5 years), C (10 years), PHP (5 years), Perl (13 years), SQL (10 years).
Operating Systems I have managed in a production environment: FreeBSD (7 years), Linux (13 years), BSDI (1 year), NetBSD (5 years), Mac OSX (1 year).
Virtualization technologies I have managed in a production environment: Xen on Linux Dom0 (10 years), Xen on a NetBSD Dom0 (1 years), KVM (2 years), VMware (2 months), FreeBSD jails (5 years).
revision control systems I am familar with: rcs (15 years), cvs (10 years), git (3 years), perforce (2 years).
Database and Directory servers I have managed in a production environment: PostgreSQL (10 years), MySQL (13 years), OpenLDAP (6 years).
Network services I have managed and supported in a production environment: SMTP (10 years), pop3 (10 years), DNS (10 years), SSH (7 years).
on site at google (through prgmr.com, through ERG)
At Google, I was responsible for supporting, maintaining, expanding,
configuring and repairing a small EDA cluster.
I wrote an LSF ELIM, using Python to import license usage data from a flexlm license management system to prevent batch jobs from crowding out real-time users.
I wrote several monitoring tools using the C API to pull data out of LSF and into the google propritary monitoring system.
I diagnosed puppet conflicts and wrote puppet config files to manage package and configuration-file installs.
I increased NFS throughput by more than 30% through simple NFS tuning.
I brought up a SLURM cluster for a competitive evaluation vs LSF
I set up benchmarks to evaluate and compare the various block device
caching schemes that were compatable with the corporate-approved
version of linux we used.
I wrote numerous monitoring plugins for the in-house monitoring system.
I built custom .deb packages, sometimes using dpkg-divert to override existing packages in order to fix NFS problems that upstream was not interested in.
I used our puppet configuration management to install packages and
modify systems configurations across the cluster
I managed three employees and several contractors, while
also acting as the primary operations engineer. I was responsible
for setting up monitoring and responding to emergencies, in
addition to my role designing new systems.
- data center power management:
I designed a system to manage our Avocent and ServerTech
brand PDUs using SNMP and serial.
- layer 2 datacenter network management:
I managed a heterogeneous network, using switches from Cisco, Force10,
Woven, Extreme, and Foundry on layer two. I built a storage network
using inexpensive 1gbe Woven switches; aggregating them to a 10gbe
force10 using LACP across their four cx4 10gbe ports.
- layer 3 network management:
I managed several brands of routers. I have years of experience
running BGP under quagga, including mitigating incoming DDoS attacks
using a blackhole community. I also have experience with Vyatta,
cisco, and HP layer three gear.
- 24x7 pager responsibilities:
As the senior member of the engineering operations team, I
was either primary or secondary on pager most of the time.
I know my limits and know when to ask for help, but I /am/
willing to carry a pager. I take the responsibility of pager
duty very seriously.
- Xen virtual hosting platform:
I designed, implemented and still maintain a system for hosting
xen-based virtual servers. See http://prgmr.com/xen/
- FreeBSD jails:
I created a similar system using FreeBSD jails as the underlying
technology. To get around IPC issues that were causing problems
with customer PostgreSQL instances, I applied Marco Zek's
VIMAGE patches to FreeBSD 4.11 and ran the resulting system in production.
- San on a budget:
I implemented a 1gigabit Fibre-channel san, using Brocade
SilkWorm 2400 switches, a Dell 224F, and a Sun storedge
5200. I used zones on the SilkWorm to mediate access to the
disks. Prgmr.com later returned to using local disk, but it was
an interesting learning experience.
Drawing from our experiences at prgmr.com, Chris and I wrote The
Book of Xen, published by No-starch Press.
I built prgmr.com from zero to more than two thousand customers
and more than a third of a million dollars a year in revenue.
Xen and Linux Consultant
june 2008-june 2009
I reprised my role as a Xen and Linux consultant at Electronic arts.
- Ldap single-sign on:
I implemented a LDAP authentication system using pam and nss_ldap.
I was able to directly import the password hashes from the NIS system
into LDAP. I also implemented the OpenSSH-LPK schema and patched OpenSSH
- Hardware benchmarking:
I benchmarked various rackmount and blade servers, and various
storage hardware in an attempt to find the most cost effective
- Large scale Nagios rollout:
I participated in the design and implentation of a large scale
nagios monitoring system.
june 2007-july 2008
I responded to pages from the monitoring sytem, diagnosed
and repaired system problems on the (massive) Yahoo! search cluster.
I used DNS to handle some failover, both bind and djbdns.
- memtest86 modifications:
I recompiled memtest86 to output over serial, then I modified
the pass/fail messages to make it easier to automatically
'screen scrape' the results from the serial console.
- semi-automated hardware failure detection:
Through automated and manual processes I identified
thousands of bad disks, and many other bad components in
servers in the Yahoo search cluster.
Xen and Linux Consultant
july 2006-may 2007
I designed and implemented a virtualized development
environment with existing hardware. I also acted in an emergency break-fix role, troubleshooting hardware and interfacing with hardware vendors when necessairy.
- qemu modification:
I wrote a small patch to qemu in order to force a 'reboot' of the
virtual machiene to cause qemu to exit
- dev environment virtualization:
I built a production-quality Xen Dom0 using a RHEL4 userland. (this was before RHEL5 was out)
Account Manager and SysAdmin
January 2003-July 2004
Sawed-off Software was a partnership between myself and another programmer, to engage in consulting activities and to create web applications.
- Financial website:
We were contracted by a stockbroker to build a website to display real-time stock quotes. I acted as the account manager, the database administrator, the System Administrator and the Project Manager, while Wayne managed the application programming. We moved the database backend from Microsoft SQL to MySQL, and the server from IIS to Apache on FreeBSD, taking the site from 'unusably slow' to 'rather fast'
- book price search automation software:
I built software that scraped prices from amazon.com, half.com, powels.com, and several other booksellers. the user would enter an ISBN of a book they wanted, and my system would return the price from various vendors. We used Perl, PHP, MySQL and Apache.
CIS Data Systems/ ihouse2000.com
March 2002-march 2006
ihouse2000.com is a ASP that provides websites to realtors. I was the
first (non-founder) technical hire.
As a systems administratior, I was responsible for maintaining our
Linux and FreeBSD servers, our MySQL servers, and our network.
I was involved at all layers of infrastructure; Once, building maintanince
cut our T1. I was able to effect a temporary fix using my leatherman and
some cat5. (We later had the phone company fix it properly, but the point is that I got us back up and running quickly.) I participated in the pager rotation.
- Kerberos and LDAP single sign on:
I implemented a single sign on system for OSX using mit Kerberos
- Dspam mailhub:
I implemented a mailhub setup with Postfix, MySQL, Dspam and
ClamAV to provide our customers with spam filtering abilities.
This project was never moved out of the testing phase, due
to problems with quarantines with more than 7000 messages in
the test case.
- mbox conversion to Courier maildir:
I wrote a patch for courier to allow us to migrate our
current unix mbox mailboxes to maildir without altering
the UIDL values for our customers that 'leave mail on server'
- data center setup and management:
I was instramental in setting up and managing several
co-located data centers.
- server assembly:
I built servers from parts.
I lowered the failure rate of the servers we built from
well over 25% to under 2% by instituting a ESD safety program.
I Implemented a system to boot and install our FreeBSD servers
off the network using PXE.
Mail Abuse Prevention System
MAPS (now TrendMicro) http://mail-abuse.org operated the Realtime
Blackhole List, one of the first and most well-known anti-spam
dns-based blacklists. I was a programmer and System Administrator.
I was involved with the Distributed Checksum Clearinghouse
project. My contributions included a sendmail .mc m4 macro that allowed
sendmail to reject at SMTP time based on the 'bulkness'
returned by the DCC server.
- easy CVS interface:
I created a system that gave the lawyers an easy-to-use web
interface to see our website at various points in history. I
pulled the historical data out of RCS files.
I joined JPS.net, a Sacramento-area ISP, shortly after it was bought
by OneMain.com. I was hired on as a technical phone support person,
but got myself promoted to network operations within a month.
- Dynamically Configured Mass Virtual Hosting:
I patched mod_vhost_alias to make both www.sub.domain.com and
sub.domain.com go to the same place with an arbitrary number
of subdomains. This made our hosted websites quite a bit faster,
without disturbing our domain.co.uk customers, and others with
an odd number of subdomains.
- pop3 proxy modifications:
I patched the in-house POP3 proxy jps.net used to map users
to the proper mail servers so that it used a MySQL backend.
Before, it simply opened a file on the NAS that corresponded
to the user's user name. Response times during peak hours went
from 3-10 seconds to under one second. Filesystems are not databases; at least they were not databases in 1998.
- MS Access ODBC conversion:
Without altering the existing codebase, I was able to convert our internal customer tracking database from the default Access FileServer backend to a MySQL backend using MyODBC.
help desk technician
American River College
At ARC, I performed maintenance tasks on a wide variety of desktop-class computers.
help desk technician
Yolo County Department of Public Health.
I maintained a wide variety of workstations for the Yolo County Department of Public Health. I did some work on the novell network, and changed the backup tapes for the ArkServ backup system. I worked with Windows for Workgroups 3.11, Windows95, and DOS.
jr. hardware technician
R and R Computers
I repaired computers for a small computer repair business. I worked with ISA, VLB, PCI, and EISA bus computers.
Copyright © 2010 by .