Recently in Announcements Category

Hard drives upgraded

| | Comments (0)
Dec 22-23 2009:   Hard drives replaced; now 678 gigs usable free space for home directories and web sites.

Note, access may be a bit slow for the next 48 hours as mirroring and other activities bog the machine down.  However, at this time I don't anticipate any further reboots or crashes.

-jason

GIGO.COM Maintenance - Dec 22, Dec 23

| | Comments (0)
Sometime Tuesday afternoon on Dec 22, gigo.com will be going offline, to upgrade the hard drives.
Unfortunately, I can't do this with a live system.  Downtime will be a maximum of 24 hours, however, I expect it to be far less - perhaps a few hours.

I will maintain status info at http://status.gigo.com .


Partial HW update over xmas

| | Comments (0)

File space usage has grown significantly since our last server upgrade. I expected the current disk space to hold us for 4 years, which is about what I budget for the overall hardware. Alas, it is looking like perhaps I should upgrade the storage sooner.

The fun part is, gigo.com currently is comprised of 5 disks, but realistically only 250 gig usable space to end users. We spend a fair bit on redundancy, in case of catastrophy. Here is how it is broken out:

  • disks 1+2: main system, this is where the main files are stored, and served from. Anything we do happens here. This is continously mirrored, so that if either disk fails, the system can quickly recover and keep running. And, I can put in a replacement to restore redundancy "hot".
  • disks 3+4+5: Backups. At any given time, 2 disks are hot and mirrored; and 1 is cold (offsite, my house). Periodically, I take the cold disk, stop at the colo, swap out one hot disk for the cold one. The server will resync the mirror, and the disk I have in my hand goes back home - with a copy of several days worth of our files. And, total time in the colo is <10 minutes to sign in and swap a disk.

With that in mind, if I do upgrade storage, I'm not upgrading just one disk, but realistically all 5. Ooof!

What I'm looking at doing is:


  • 2 enterprise class SATA 1GB disks - $160 each + the governor's ransom - matched set for mirroring.
  • 3 desktop class SATA 1.5GB disks - $120 each + the governor's ransom - matched set for mirroring.

The backups can be desktop class; they get hit with less work, don't need to be as fast, and we can afford a failure there without a serious panic. They should however be larger than the main system drives, since we backup multiple days worth of changes (currently we back up ~20 days worth of changes; this number varies based on space available and number of changes made in a day).

I'm looking to try and help raise about half this cost - so a target of $375. If you're a significant user of gigo.com and can help, please contact me. Lady Visa will be covering the gap; I'm aiming to do this hardware changeout over the xmas break.

SSH server abuse

| | Comments (0)
Something started about 2 hours ago here; ssh scans for user "root" hitting all public IP's for gigo.com.
Looking at it with a packet sniffer, all IP's get hit in parallel; sometimes without port randomization from the other side.  Looking at the hosts, looks like they all have old sshd's running.  Can't even blame windows this time.

Expect connections to gigo.com to be spotty - sshd is getting overran.  gigo.com users:  I'd like your feedback on whether or not moving the SSH port would be a big impact to you. If it would be.. what if port 22 was open to specific subnets (ie where you work); or having a web CGI that re-enables port 22 for your current IP?

Spam problem @ gigo

| | Comments (0)
Since midnight something has been posting spam via the web server; I've killed everything in the queue that was going outbound spam. Outbound mail to AOL, YAHOO, etc mail be delayed considerably (They blacklisted us due to the spam).  The entry point for it has been neutralized (and the vulnerable software is now on the ban list here).


imap server updated

| | Comments (0)
..for security reasons.  

Hollar if anything is still amiss.

More SSH scanning countermeasures

| | Comments (3)

This is to users to ssh to gigo.com.

Last year, I enabled countermeasures to help keep the SSH hack attempts against gigo.com down to a minimum. We automatically block the IP address of systems trying to log in to gigo.com and repeatedly failing.

The problems are getting worse; as such I'm making these changes:


  • Attempts to log in as a valid ID: unchanged; 10 attempts and you're banned 15 minutes
  • Unknown accounts and daemon accounts: immediately blocked for 150 minutes

If you log in from another account with a different account name, make sure that you always remember to specify your gigo.com account name (correctly!) or the machine you are connecting from will be blocked for everything but the web server, for 150 minutes. If this regularly affects you, I *can* whitelist specific IP addresses to be immune to this behavior.

Mail changes - planned

| | Comments (0)
[Updated Sunday 2:00p]  -  Changes are now in effect.

Condensed version:

Attachments are allowed now (most are, at least).  Incoming viruses or otherwise "dangerous" files will be made a bit safer (the message is turned into an attachment; and the message you see, will say [VIRUS ATTACHED] in the subject line). 

Attachments (in and out) are scanned for viruses using an open-source virus checker.  This is not a good substitute for you running your own protection.  We could (collectively) pay for a commercial virus checker.  A better use for your money is to make sure your workstation is running antivirus of its own.

Spam filtering has changed a bit; please make sure you are filtering on "X-Spam-Flag: YES".  If you can only filter by subject, then look for SPAM in the subject line.   A limited amount of customization is still available at https://mail.gigo.com/ - primarily affecting whitelists, blacklists, and a minimum score to consider a message as SPAM.

Finally, @gigo.com mail is being signed with DKIM.  Some providers will give us a a positive nudge when they do their own spam checking when the mail is properly signed.  I can do this for other domains as well - let me know if you want this.  

I'm reachable as jfesler@gigo.com or (if you have problems with that for any reason) postmaster@gigo.com .





Mail server changes

| | Comments (0)
As of this morning I've enabled some of the blacklists from SpamHaus.  Anyone who has their mail rejected mailing our system will be given a url in the bounce message; as well as postmaster@gigo.com's email address for exceptions.  Note that postmaster is never filtered here - even if we generally block their mail when they mail me as postmaster, it'll still come through.

I'm watching the logs and looking at the rejects looking for signs of false positives.

This change in behavior is to cut down the resources it takes to fight spam - the greylisting was the primary barrier previously, but that is becoming costly.  If you want me to turn off antispam measures at the mail server level to your address or domain, let me know.

gallery 2.2.6 update

| | Comments (0)
those of you running gallery on my server - I've updated all of them for a security update.