Entries tagged with “maintenance” from Garbage In, Garbage Out

File space usage has grown significantly since our last server upgrade. I expected the current disk space to hold us for 4 years, which is about what I budget for the overall hardware. Alas, it is looking like perhaps I should upgrade the storage sooner.

The fun part is, gigo.com currently is comprised of 5 disks, but realistically only 250 gig usable space to end users. We spend a fair bit on redundancy, in case of catastrophy. Here is how it is broken out:

• disks 1+2: main system, this is where the main files are stored, and served from. Anything we do happens here. This is continously mirrored, so that if either disk fails, the system can quickly recover and keep running. And, I can put in a replacement to restore redundancy "hot".
• disks 3+4+5: Backups. At any given time, 2 disks are hot and mirrored; and 1 is cold (offsite, my house). Periodically, I take the cold disk, stop at the colo, swap out one hot disk for the cold one. The server will resync the mirror, and the disk I have in my hand goes back home - with a copy of several days worth of our files. And, total time in the colo is <10 minutes to sign in and swap a disk.

With that in mind, if I do upgrade storage, I'm not upgrading just one disk, but realistically all 5. Ooof!

What I'm looking at doing is:

• 2 enterprise class SATA 1GB disks - $160 each + the governor's ransom - matched set for mirroring.
  
• 3 desktop class SATA 1.5GB disks - $120 each + the governor's ransom - matched set for mirroring.
  

The backups can be desktop class; they get hit with less work, don't need to be as fast, and we can afford a failure there without a serious panic. They should however be larger than the main system drives, since we backup multiple days worth of changes (currently we back up ~20 days worth of changes; this number varies based on space available and number of changes made in a day).

I'm looking to try and help raise about half this cost - so a target of $375. If you're a significant user of gigo.com and can help, please contact me. Lady Visa will be covering the gap; I'm aiming to do this hardware changeout over the xmas break.

Guys, the outage from 3/27/2008 23:45 to 3/28/2008 17:00 was due to a motherboard failure. More info is at http://status.gigo.com/ for those that care.

-jason

I am looking at upgrading from FreeBSD 6 to FreeBSD 7. Unfortunately this means downtime. Additionally, as I'll be moving to a 64 bit OS, I can't just build the "next" gigo.com at home without buying a 64 bit capable spare machine that's only gonna be needed for a few days.

What this means is, I need to actually bring gigo.com down in a big way to do this upgrade. I expect it to take a weekend.

What I'm proposing is 3/22 to 3/23 being declared as "maintenance". I'll obviously try and limit how long mail and web are down, but .. this upgrade is unfortunately going to take time. If this time does not work for you, please let me know. I expect 1 day of major impact, 1 day of minor impact.

The priority order on what I'd get back up and running would be:

• firewalls, dns, ssh (then work from a hotel)
• mailing lists (delayed, until brought back up)
• greylisting,spamd,regular mail,imap (delayed, until brought back up)
• mysql, web,webmail (flat out offline until brought back up - sorry)
• irc, jabber
• bitlbee, rsync
• nagios

I apologize that this is so soon after last August's update - unfortunately, FreeBSD 7 was only just now released. Minor upgrades are not nearly as big of a deal (usually just a minor install and a reboot). But a major upgrade, those are a bit more painful (especially changing from 32 bit to 64 bit at the same time).

Before anyone asks: Yes, in theory, I could move *everything* to another site, somewhere else, maybe even volunteered space, but the overhead in doing so is too much, for the amount of stuff here. Given my limited free time, that's not an option. But, thanks in advance for thinking about it.

Folks, I'm making a "notify" mailing list -- I'll be using that to communicate the status of the server upgrade and such. I also intend to keep the web site up to date with that regard, so if you're reading this via web or RSS, you're find. However, if you're not, and I haven't subscribed you to the notify mailing list, you can subscribe at http://gigo.com/mailman/listinfo/notify.

The new server to replace vette.gigo.com has arrived from ixSystems. So far, so good. I'm working on a base FreeBSD install on it now. I expect to put this thing live some time late August.

If you're a "local user" on gigo.com, and you're making use of php/mysql/perl/etc, you might want to take stock of what you depend on, and see what gotchas there will be when the latest versions of those tools are used.

POP/IMAP servers are being upgraded. Dovecot has had a 1.0 release out for a while now and no major reports of problems. pop3/imap servers will be bounced to upgrade the apps.
-jason

[This replaces a few earlier posts, as well as a message sent to local users mailboxes]

Several mail server changes this weekend.

SpamAssassin, our spam filter, has been upgraded. Lots of new rules added. 2000+ rules now being checked (at some cost, too - 0.6 to 0.8 CPU seconds for every 1 message). Due to the cost of such rules, we do lots of other things to slow things down before it reaches the spam filtering step.
A reminder, you can tweak spam filter scores at mail.gigo.com.

Greylisting. While this is not new, there is a slight behavior change. Previously, the first time a sender contacted a recipient from a distinct IP addres, we would tell them "try later". Legitimate servers do that, usually 10-20 minutes later. We're only introducing this delay for "new" connections. Reminder, if you don't like greylisting, send me email, and I'll put in a pattern to bypass your mail. (We have been greylisting for 2 years now..).

The actual change to the behavior is thus: If the mail goes to our primary receiver first, then to our backup mail receiver, we'll accept it on the spot. That means the mail application is in general following the rules. We'll reward this by not delaying it any further. (This may increase the spam some, for spam applications that actually follow the rules).

Secondary mail receiver: A second mail receiver is now running. This is running on the same host, and the purpose really is to act as a decoy to the spammers that go after secondary mail receivers. If we don't recognize the sender+recipient+host combination, we'll simply say "try later". Legitimate servers will retry the primary later. Hit and run spammers that try only back up mail receivers when found, will hopefully be decoyed.

Attachments: Spam includes trojans and virii. We are now blocking many types of 'executable' attachments. We are still allowing zip files, image files, and the like. Config files can be perused if you wish at pcre.jimsun.header_checks.txt and pcre.jimsun.body_checks.tx. Regretfully this can only be applied globally (or not) - a few of you who are whitelisted on spam filters is still having this filter applied.

Messages with such attachments will be refused. Legitimately sent mail by people you know with these attachments should still notify them (the sending mail server however is responsible for that).

ZIP files are still allowed. However, they are not 100% safe. Do you have antivirus software installed? Why not? Espcecially those of you running any version of Windows..

Web server upgraded to PHP5. Removed mod_perl. Lemme know if you have problems.

Main fix: squirrelmail (mail.gigo.com) is now showing proper local time zone data if you select a time zone under your personal options.

the main server had to go through a couple reboots tonight, and a total of about 1h downtime. sorry about that folks!

The mail stuff is done. Horray!

• POP3 supports SSL/TLS on the POP3S port, 995
• IMAP supports SSL/TLS on the IMAPS port, 993
• SMTP supports SSL/TLS as an option on the SUBMISSION port, 587. If your ISP is blocking port 25, you can use port 587 to send mail, if you're authenticated (SMTP with username/password).
• SMTP also supports a fulltime wrapped port on port 465 (some older apps will want this).
• SMTP lastly will still honor STARTTLS on the standard SMTP port.
• SMTP AUTH is working after the upgrades. I still need to provide a password change tool. Until I do so, poke me directly.

SSL should "just work" as long as you use the name mail.gigo.com. We're using a paid-for certificate that should have root trusts. I tested Thunderbird and Mail.App (apple), and they worked. I also got pine working (with these instructions).

For those who are curious, the new combination of software on the server is now: OpenSSL, Dovecot POP3/IMAP, Postfix (with Dovecot SMTP AUTH), Squirrelmail, and Apache with SSL. Filtering is provided by both Dovecot LDA + libsieve, and via procmail, depending on user preferences.

Bookmark this URL; if you have problems getting your gigo.com mail (or any other mail hosted at gigo.com), this page should have information you may need.

Status: Done. Everything is back online now.

Latest Apache 1.x, PHP4, and related modules upgraded. Hollar if anything broke..

IMAP, POP3, SMTP all upgraded for security reasons. I appologies for any interruptions this has caused gigo.com users.

We are now running SpamAssassin 3.0, which has several changes to it. You may notice a bit of a difference on the spam that is reported, and how it looks. Several optional rulesets were also added; and some of the features on the old system were lost due to copyright issues.

You can edit your settings at
http://mail.gigo.com
if you would like to fine tune your spam filter settings.

Web mail now has a "Filters" command. SquirrelMail will let you manage server side filtering. Now, you can easily say "any mail that matches this, put it into this new folder..". You *should* do this if X-Spam-Status: says "Yes" (I'll try and make this happen automatically for the people I am converting - most everyone did not use procmail for much).

I will be converting most users on my box to Sieve scripts; a few power users on my system I'll leave with procmail. I'll contact you guys individually to see if you want to convert over to Sieve or not.