Garbage In, Garbage Out

ipv6

Thu, 08 May 2008 22:56:52 -0800

I'm enabling ivp6 for all dns addresses now. Let me know if I broke something.

1-877-4FESLER if it's urgent.

This blog will have a green header if you're visiting from ipv4, and a blue header from ipv6.

ipv6; bind

Mon, 05 May 2008 20:37:35 -0800

I've made some changes on the system; more are coming. Specifically so far:

goat.gigo.com is now accessible on ipv6.
irc.gigo.com is now accessible on ipv6.
mail primary MX is on ipv6; secondaries will be soon.
I changed from djbdns to bind. Lemme know if you see anything funky. If you edit zones here on gigo.com, I'll only install them if it passes a basic "will it load" test (using named-checkzone).
ns3 and ns4.gigo.com are deprecated; they are currently aliased to ns1.gigo.com
I'm not publishing IPv6 NS records - waiting for word on whether I can get "glue" records published to .com for that.

Do I host your DNS?

Mon, 05 May 2008 15:15:02 -0800

I'm in the process of simplifying my DNS setup. I'm reducing the number of DNS servers I maintain down to two. Most of the domains are hosted by me for web/email as well - and if the host is down, the DNS isn't gonna matter quite so much. If *.gigo.com servers are hosting your dns domain, please update your domains to point at these two name servers:

ns1.gigo.com (Located in Fremont, CA using HE.NET)
ns2.gigo.com (Located in Sacramento, CA using CWO.COM)

I will still keep ns3 and ns4 around, but they won't be unique from the first two.

Apple's "telnet" is broken.

Sun, 04 May 2008 13:43:54 -0800

"telnet" is a program that lets you make an unencrypted login to another host on an arbitrary port. It is a useful testing tool, for making sure that the firewall you're working on is working; or for making sure the service you are working on is answering. Normally you give it a hostname and a port number. Simple. Only, with apple, if you give it port 25 ("SMTP"), it decides not to do a regular host to IP lookup, but instead... does a MX record lookup. As in, it goes somewhere other than where you told it to go.

Jason-Feslers-computer:~ jfesler$ telnet gigo.com 22
Trying 216.218.228.114...
Connected to gigo.com.
Escape character is '^]'.
SSH-2.0-OpenSSH_4.5p1 FreeBSD-20061110
^]
telnet> close
Connection closed.
Jason-Feslers-computer:~ jfesler$ telnet gigo.com 25
Trying 216.218.228.118...
Connected to mx2.gigo.com.
Escape character is '^]'.
220 goat.gigo.com ESMTP secondary mx only defers/rejects. postmaster@gigo.com
^]
telnet> close
Connection closed.
Jason-Feslers-computer:~ jfesler$

Update: This happens for ssh, nc as well. As it turns out, Apple's doing this on quite a bit..

http://blog.jungledisk.com/2007/10/31/leopard-dns-issues-and-work-around/

Bitlbee 1.2 challenge/response update

Fri, 11 Apr 2008 09:34:29 -0800

I have updated my previous bitlbee challenge/response patch, to work on bitlbee 1.2. You can get it at http://gigo.com/ftp/pub/src/bitlbee-challenge-response.patch . Please let me know if you use it and find it useful.

Motherboard Failure

Fri, 28 Mar 2008 17:26:32 -0800

Guys, the outage from 3/27/2008 23:45 to 3/28/2008 17:00 was due to a motherboard failure. More info is at http://status.gigo.com/ for those that care. -jason

Jabber XMPP server changed

Thu, 27 Mar 2008 15:34:41 -0800

I've moved to using OpenFire for the jabber server. If you want to use an @gigo.com jabber address, contact me. Some comparisons:

OpenFire only supports a single domain. ejabberd supported all domains I hosted.
OpenFire is java, and sucks memory like crazy.
OpenFire was insanely easy to setup - unlink ejabberd and erlang.

Single hosts don't scale.

Tue, 25 Mar 2008 21:00:00 -0800

Alternate title: But damn, FreeBSD is pissing my off good this time around. One thing about my job that really spoils me is the shear size of it. Where I'm at, we have more machines down for maintenance than most folks have in service company wide. We don't bundle up too many services on any one box - less things to go wrong when a box fails. And, boxes do fail - a fairly predictable amount fail every day, like clockwork. I tell people I plan for failure and they look at me funny. But what I mean is, I know things will fail - we can build to accommodate it.

Mail redelivery happening now

Sun, 23 Mar 2008 16:10:40 -0800

Over Saturday and Sunday, a fair bit of mail was mishandled due to the dovecot installation not installing /usr/local/bin/deliver (mea culpa). I sent notices to those of you affected.

In most cases it looks like we *did* keep the mail though - and I'm in the process of remailing that now

with "cat mailbox | formail procmail -d username".

If you're from NANOG or another such mailing list and you saw some duping.. I saw it as quickly as it happened and killed SMTP ASAP. I appologize for the mistake.

gmirror kinda sucks.

Fri, 21 Mar 2008 00:07:53 -0800

So, in advance of upgrade day (Saturday), I stopped at the colo this evening for a quick 5 minute hard drive swap. The intent: Swap the boot drive, so as to have a offline seperate bootable disk that we can fallback to if the upgrade sucks. Unfortunately, the moment I removed one of the two drives that acts as the mirror for gigo.com, I/O froze on the system, entirely. Poof. After I reset, the system did not want to boot. I broke the mirror, and the first drive of couse was the one I removed. Getting it to boot without the mirror was impossible - booting /dev/ad4 instead of /dev/mirror/gm0 wasn't happening, since the boot drive was told to forget about being gm0. After a lot of hassle I got the system to the point where it would start fsck'ing. The beauty of this is, is the other mirrored file system was checking itself, end to end, while trying to fsck. after 3 hours, I aborted it, and let the rest of the system come up. Backups won't run tonight - that's the file system I'm fsck'ing now. Everything else is back to normal.

Mail to yahoo.com

Tue, 18 Mar 2008 11:43:50 -0800

Mail from gigo.com (including mailing lists I host) is being blocked by Yahoo.com. Despite being an employee there, I've got no recourse to quickly get this resolved. I've filled out their form, we'll see how long it takes.

Procmail for the phone

Sun, 16 Mar 2008 17:32:58 -0800

I hate telemarketers. And, despite the do not call list, or perhaps in spite of, they've gotten much meaner and naster as of late - bogus or missing caller id, no identification when the predicitve callers call you, and if you do get a human, the moment you utter DNC lists, actually _before you finish_, you get a click. One was so rude as to tell me *I* had the wrong number, before the click. I finally ran across the device at http://interceptorid.com/. It is known as a few different names but is really the same device. Plug it into the phone line between your phone and the wall. It intercepts all phone calls, finds the caller id after that first ring, then either sends it to your phone, or to an answering machine. For my own setup, calls with valid caller id, coming from "local" area codes (basically, northern California) all ring my phone directly; *everything* else (especially toll free #'s) ring the answering machine instead. Best part is, those screened calls, don't ring me at all. Good stuff, but hard to find at this time - looks like they are preparing to change the design some. This was definately a version 1.0 product - a bit clunky. But, it definately works as advertise.

New: http://status.gigo.com

Thu, 06 Mar 2008 08:05:43 -0800

In the event gigo.com is down and not coming back up I've created http://status.gigo.com. You might want to bookmark it. If you have an RSS reader, consider bookmarking the RSS feed. It will in particular come in handy during the maintenance planned for 3/22/08 and 3/22/08. :-)

Proposed downtime 3/22, 3/23

Wed, 05 Mar 2008 16:47:37 -0800

I am looking at upgrading from FreeBSD 6 to FreeBSD 7. Unfortunately this means downtime. Additionally, as I'll be moving to a 64 bit OS, I can't just build the "next" gigo.com at home without buying a 64 bit capable spare machine that's only gonna be needed for a few days. What this means is, I need to actually bring gigo.com down in a big way to do this upgrade. I expect it to take a weekend. What I'm proposing is 3/22 to 3/23 being declared as "maintenance". I'll obviously try and limit how long mail and web are down, but .. this upgrade is unfortunately going to take time. If this time does not work for you, please let me know. I expect 1 day of major impact, 1 day of minor impact. The priority order on what I'd get back up and running would be:

firewalls, dns, ssh (then work from a hotel)
mailing lists (delayed, until brought back up)
greylisting,spamd,regular mail,imap (delayed, until brought back up)
mysql, web,webmail (flat out offline until brought back up - sorry)
irc, jabber
bitlbee, rsync
nagios

I apologize that this is so soon after last August's update - unfortunately, FreeBSD 7 was only just now released. Minor upgrades are not nearly as big of a deal (usually just a minor install and a reboot). But a major upgrade, those are a bit more painful (especially changing from 32 bit to 64 bit at the same time). Before anyone asks: Yes, in theory, I could move *everything* to another site, somewhere else, maybe even volunteered space, but the overhead in doing so is too much, for the amount of stuff here. Given my limited free time, that's not an option. But, thanks in advance for thinking about it.

server back up

Tue, 04 Mar 2008 15:28:42 -0800

Gigo crashed. Back up now...