Single hosts dont scale
Alternate title: But damn, FreeBSD is pissing my off good this time around.
One thing about my job that really spoils me is the shear size of it. Where I’m at, we have more machines down for maintenance than most folks have in service company wide. We don’t bundle up too many services on any one box - less things to go wrong when a box fails. And, boxes do fail - a fairly predictable amount fail every day, like clockwork. I tell people I plan for failure and they look at me funny. But what I mean is, I know things will fail - we can build to accommodate it. The other thing with the kind of scale at where I work, is we develop ways to install OS’s on hosts that scale. We don’t use the stock installers. We network boot things, boot partitions get installed with an image we approved, and any last minute stuff gets done to hook into our processes. You pretty much don’ need to know anything about installing the OS in question using tradition tools, since we’ve tossed them out the window.
Now, things are a different story when you’re looking at a personal machine. Especially one that’s got 50 or so users on it. Pretty much boils down to a single box running everything under the sun. One that you install an OS on, on average, every 3 years. I don’t have the luxury of moving things from one box to another, there is no second box. And that, is the crux of this weekend’s upgrade, with major downtime (most of Saturday for nearly all services).
Now, if this were the end of the story, I’d be a happy camper. But, now I’ll rant a bit about how FreeBSD is pissing me off.
Existing partitions. Part of my plan this weekend was to take my mirrored file systems, and just break the mirror. Move the directories asside on one of the two copies; and install over. That way I could just (in a few seconds) rename things back into place, saving hours of copy time. As it turns out, I could tell the installer about those partitions - but it somehow managed to mount them. End result: FreeBSD unpacked onto the miniature boot slice instead, and filled it up just a few seconds into the installation.
Ethernet port turnups. Depending on where your host is, when you first turn up your interface and give it an IP address, the switch you’re plugged into will block your packets for about 10-15 seconds. This is so that it could look for signs that perhps you looped the network. This is all fine and good - except it seems that FreeBSD only sends out one DNS query, doesn’t try again. And that first query was before the host is allowed to talk on the network. Workaround: ctrl-c, tell the installer start over. Next time around, the IP has no problems since it has been up for a couple minutes by the time you fill it all out - but .. again, partitions do not get mounted, and we fill the boot slice instead.
mirroring support. gmirror is actually good stuff. but when you rarely install it (did I mention single hosts don’t scale?). You *really* need to follow dru’s advice for gmirror to the letter. Of course, I didn’t. I installed the OS, rebooted, enabled mirroring, rebooted, and things were thrashed. You *must* enable gmirror while you are booted from CD, instead of the hard disk. (By this point I was becoming expert at reinstallation..).
gmirror has also been giving me unreliable service with respect to pulling disks from the boot mirror set. I’m pretty much relegated to only pulling root drives out with the host shutdown. At least they are covering my ass in case of a drive failure, but I can’t make mirror/remove a bootable drive.
Oh, and while I continue to rant on gmirror, here is another fun one - unplanned resets (read: not cleanly shut down) mean you have to check the disks on boot. gmirror _also_ checks the disks in the mirror. Now, this all sounds good - until you realize both are very long processes when there is nothing going on. But both running at the same time, that can take several hours. After a few hours I ended up aborting the scans Thursday, and let the OS come up without those disks - then finished handling them from the hotel instead of more sitting at the data center.
32 bit on 64 bit machines. Generally speaking various OS’s let you install 32 bit apps on your 64 bit OS, and run them. FreeBSD techically allows it - but the release does not include the libraries. You instead have to build them from source. That actually requires a “make buildworld”, which on my gear runs about an hour.
My own stupidity. The data center gives me plenty of IP space, but only one network cable. I pay for one cable and 7 vertical inches of space. Of course, I forgot to bring an ethernet switch with me. No laptop access for me!
On the positive… gigo.com is now running 64 bit. That means I can actually use all the memory as I see fit, either with one app that needs 2-3 gigs, or I can make huge ram drives, whatever is desirable for the task at hand. And, now that we’re on 7.0, there should be no major upgrades any time soon (maybe some minor dot releases, but those are usually prety smooth).