Garbage In, Garbage Out

ipv6

2008-05-09T05:56:52Z

I'm enabling ivp6 for all dns addresses now. Let me know if I broke something.

1-877-4FESLER if it's urgent.

This blog will have a green header if you're visiting from ipv4, and a blue header from ipv6.

ipv6; bind

2008-05-06T03:37:35Z

I've made some changes on the system; more are coming. Specifically so far:

goat.gigo.com is now accessible on ipv6.
irc.gigo.com is now accessible on ipv6.
mail primary MX is on ipv6; secondaries will be soon.
I changed from djbdns to bind. Lemme know if you see anything funky. If you edit zones here on gigo.com, I'll only install them if it passes a basic "will it load" test (using named-checkzone).
ns3 and ns4.gigo.com are deprecated; they are currently aliased to ns1.gigo.com
I'm not publishing IPv6 NS records - waiting for word on whether I can get "glue" records published to .com for that.

Do I host your DNS?

2008-05-05T22:15:02Z

I'm in the process of simplifying my DNS setup. I'm reducing the number of DNS servers I maintain down to two. Most of the domains are hosted by me for web/email as well - and if the host is down, the DNS isn't gonna matter quite so much. If *.gigo.com servers are hosting your dns domain, please update your domains to point at these two name servers:

ns1.gigo.com (Located in Fremont, CA using HE.NET)
ns2.gigo.com (Located in Sacramento, CA using CWO.COM)

I will still keep ns3 and ns4 around, but they won't be unique from the first two.

Apple's "telnet" is broken.

2008-05-04T20:43:54Z

"telnet" is a program that lets you make an unencrypted login to another host on an arbitrary port. It is a useful testing tool, for making sure that the firewall you're working on is working; or for making sure the service you are working on is answering. Normally you give it a hostname and a port number. Simple. Only, with apple, if you give it port 25 ("SMTP"), it decides not to do a regular host to IP lookup, but instead... does a MX record lookup. As in, it goes somewhere other than where you told it to go.

Jason-Feslers-computer:~ jfesler$ telnet gigo.com 22
Trying 216.218.228.114...
Connected to gigo.com.
Escape character is '^]'.
SSH-2.0-OpenSSH_4.5p1 FreeBSD-20061110
^]
telnet> close
Connection closed.
Jason-Feslers-computer:~ jfesler$ telnet gigo.com 25
Trying 216.218.228.118...
Connected to mx2.gigo.com.
Escape character is '^]'.
220 goat.gigo.com ESMTP secondary mx only defers/rejects. postmaster@gigo.com
^]
telnet> close
Connection closed.
Jason-Feslers-computer:~ jfesler$

Update: This happens for ssh, nc as well. As it turns out, Apple's doing this on quite a bit..

http://blog.jungledisk.com/2007/10/31/leopard-dns-issues-and-work-around/

Bitlbee 1.2 challenge/response update

2008-04-11T16:34:29Z

I have updated my previous bitlbee challenge/response patch, to work on bitlbee 1.2. You can get it at http://gigo.com/ftp/pub/src/bitlbee-challenge-response.patch . Please let me know if you use it and find it useful.

Motherboard Failure

2008-03-29T00:26:32Z

Guys, the outage from 3/27/2008 23:45 to 3/28/2008 17:00 was due to a motherboard failure. More info is at http://status.gigo.com/ for those that care. -jason

Jabber XMPP server changed

2008-03-27T22:34:41Z

I've moved to using OpenFire for the jabber server. If you want to use an @gigo.com jabber address, contact me. Some comparisons:

OpenFire only supports a single domain. ejabberd supported all domains I hosted.
OpenFire is java, and sucks memory like crazy.
OpenFire was insanely easy to setup - unlink ejabberd and erlang.

Single hosts don't scale.

2008-03-26T04:00:00Z

Alternate title: But damn, FreeBSD is pissing my off good this time around. One thing about my job that really spoils me is the shear size of it. Where I'm at, we have more machines down for maintenance than most folks have in service company wide. We don't bundle up too many services on any one box - less things to go wrong when a box fails. And, boxes do fail - a fairly predictable amount fail every day, like clockwork. I tell people I plan for failure and they look at me funny. But what I mean is, I know things will fail - we can build to accommodate it. Existing partitions. Part of my plan this weekend was to take my mirrored file systems, and just break the mirror. Move the directories asside on one of the two copies; and install over. That way I could just (in a few seconds) rename things back into place, saving hours of copy time. As it turns out, I could tell the installer about those partitions - but it somehow managed to mount them. End result: FreeBSD unpacked onto the miniature boot slice instead, and filled it up just a few seconds into the installation. Ethernet port turnups. Depending on where your host is, when you first turn up your interface and give it an IP address, the switch you're plugged into will block your packets for about 10-15 seconds. This is so that it could look for signs that perhps you looped the network. This is all fine and good - except it seems that FreeBSD only sends out one DNS query, doesn't try again. And that first query was before the host is allowed to talk on the network. Workaround: ctrl-c, tell the installer start over. Next time around, the IP has no problems since it has been up for a couple minutes by the time you fill it all out - but .. again, partitions do not get mounted, and we fill the boot slice instead. mirroring support. gmirror is actually good stuff. but when you rarely install it (did I mention single hosts don't scale?). You *really* need to follow dru's advice for gmirror to the letter. Of course, I didn't. I installed the OS, rebooted, enabled mirroring, rebooted, and things were thrashed. You *must* enable gmirror while you are booted from CD, instead of the hard disk. (By this point I was becoming expert at reinstallation..). gmirror has also been giving me unreliable service with respect to pulling disks from the boot mirror set. I'm pretty much relegated to only pulling root drives out with the host shutdown. At least they are covering my ass in case of a drive failure, but I can't make mirror/remove a bootable drive. Oh, and while I continue to rant on gmirror, here is another fun one - unplanned resets (read: not cleanly shut down) mean you have to check the disks on boot. gmirror _also_ checks the disks in the mirror. Now, this all sounds good - until you realize both are very long processes when there is nothing going on. But both running at the same time, that can take several hours. After a few hours I ended up aborting the scans Thursday, and let the OS come up without those disks - then finished handling them from the hotel instead of more sitting at the data center. 32 bit on 64 bit machines. Generally speaking various OS's let you install 32 bit apps on your 64 bit OS, and run them. FreeBSD techically allows it - but the release does not include the libraries. You instead have to build them from source. That actually requires a "make buildworld", which on my gear runs about an hour. My own stupidity. The data center gives me plenty of IP space, but only one network cable. I pay for one cable and 7 vertical inches of space. Of course, I forgot to bring an ethernet switch with me. No laptop access for me! On the positive... gigo.com is now running 64 bit. That means I can actually use all the memory as I see fit, either with one app that needs 2-3 gigs, or I can make huge ram drives, whatever is desirable for the task at hand. And, now that we're on 7.0, there should be no major upgrades any time soon (maybe some minor dot releases, but those are usually prety smooth).]]>

Mail redelivery happening now

2008-03-23T23:10:40Z

Over Saturday and Sunday, a fair bit of mail was mishandled due to the dovecot installation not installing /usr/local/bin/deliver (mea culpa). I sent notices to those of you affected.

In most cases it looks like we *did* keep the mail though - and I'm in the process of remailing that now

with "cat mailbox | formail procmail -d username".

If you're from NANOG or another such mailing list and you saw some duping.. I saw it as quickly as it happened and killed SMTP ASAP. I appologize for the mistake.

gmirror kinda sucks.

2008-03-21T07:07:53Z

So, in advance of upgrade day (Saturday), I stopped at the colo this evening for a quick 5 minute hard drive swap. The intent: Swap the boot drive, so as to have a offline seperate bootable disk that we can fallback to if the upgrade sucks. Unfortunately, the moment I removed one of the two drives that acts as the mirror for gigo.com, I/O froze on the system, entirely. Poof. After I reset, the system did not want to boot. I broke the mirror, and the first drive of couse was the one I removed. Getting it to boot without the mirror was impossible - booting /dev/ad4 instead of /dev/mirror/gm0 wasn't happening, since the boot drive was told to forget about being gm0. After a lot of hassle I got the system to the point where it would start fsck'ing. The beauty of this is, is the other mirrored file system was checking itself, end to end, while trying to fsck. after 3 hours, I aborted it, and let the rest of the system come up. Backups won't run tonight - that's the file system I'm fsck'ing now. Everything else is back to normal.

Mail to yahoo.com

2008-03-18T18:43:50Z

Mail from gigo.com (including mailing lists I host) is being blocked by Yahoo.com. Despite being an employee there, I've got no recourse to quickly get this resolved. I've filled out their form, we'll see how long it takes.

Procmail for the phone

2008-03-17T00:32:58Z

I hate telemarketers. And, despite the do not call list, or perhaps in spite of, they've gotten much meaner and naster as of late - bogus or missing caller id, no identification when the predicitve callers call you, and if you do get a human, the moment you utter DNC lists, actually _before you finish_, you get a click. One was so rude as to tell me *I* had the wrong number, before the click. I finally ran across the device at http://interceptorid.com/. It is known as a few different names but is really the same device. Plug it into the phone line between your phone and the wall. It intercepts all phone calls, finds the caller id after that first ring, then either sends it to your phone, or to an answering machine. For my own setup, calls with valid caller id, coming from "local" area codes (basically, northern California) all ring my phone directly; *everything* else (especially toll free #'s) ring the answering machine instead. Best part is, those screened calls, don't ring me at all. Good stuff, but hard to find at this time - looks like they are preparing to change the design some. This was definately a version 1.0 product - a bit clunky. But, it definately works as advertise.

New: http://status.gigo.com

2008-03-06T16:05:43Z

In the event gigo.com is down and not coming back up I've created http://status.gigo.com. You might want to bookmark it. If you have an RSS reader, consider bookmarking the RSS feed. It will in particular come in handy during the maintenance planned for 3/22/08 and 3/22/08. :-)

Proposed downtime 3/22, 3/23

2008-03-06T00:47:37Z

I am looking at upgrading from FreeBSD 6 to FreeBSD 7. Unfortunately this means downtime. Additionally, as I'll be moving to a 64 bit OS, I can't just build the "next" gigo.com at home without buying a 64 bit capable spare machine that's only gonna be needed for a few days. What this means is, I need to actually bring gigo.com down in a big way to do this upgrade. I expect it to take a weekend. What I'm proposing is 3/22 to 3/23 being declared as "maintenance". I'll obviously try and limit how long mail and web are down, but .. this upgrade is unfortunately going to take time. If this time does not work for you, please let me know. I expect 1 day of major impact, 1 day of minor impact. The priority order on what I'd get back up and running would be:

firewalls, dns, ssh (then work from a hotel)
mailing lists (delayed, until brought back up)
greylisting,spamd,regular mail,imap (delayed, until brought back up)
mysql, web,webmail (flat out offline until brought back up - sorry)
irc, jabber
bitlbee, rsync
nagios

I apologize that this is so soon after last August's update - unfortunately, FreeBSD 7 was only just now released. Minor upgrades are not nearly as big of a deal (usually just a minor install and a reboot). But a major upgrade, those are a bit more painful (especially changing from 32 bit to 64 bit at the same time). Before anyone asks: Yes, in theory, I could move *everything* to another site, somewhere else, maybe even volunteered space, but the overhead in doing so is too much, for the amount of stuff here. Given my limited free time, that's not an option. But, thanks in advance for thinking about it.

server back up

2008-03-04T23:28:42Z

Gigo crashed. Back up now...