It was the best of times, it was the worst of times (or why we’re with Rackspace)

I’m not particularly fond of quoting Dickens, but it seems apt this (southern) spring day after coming off at least a week, but closer to a month of server purgatory: It was the best of times, it was the worst of times.

Regular readers would know about the issues we’ve had with servers over September and into October. When I got back from the States at the end of August, the CPU load headed north. I thought I knew what it was, and I stripped some stuff out, stuff that in September at least cost us in terms of Google juice and traffic. My only focus was on the server, and I couldn’t see the light from the trees. It’s like an all consuming sickness, one that slowly chips away at all that is rational and logical, until you do even the most insane things trying to fix it.

It turns out we had an Apache problem. Whether that was WordPress related or not I’m not 100% sure, but likewise as of yesterday the 5th person on oDesk I’ve hired (and the first competent server guy) identified it, be it by his work shots and subsequent research in those shots.

Here’s what I did know: as September progressed, and no matter what cache I put in place, no matter that all but the thumbnails on this site, and a pile of js and css calls are on Amazon S3, the CPU load just got worse and worse. Turns out that once we started, it was always going to get worse, so I’m told. We needed a fresh start, a fresh WordPress install, and more importantly more than a “you’re in charge” from a service ticket for our US$750 a month in server costs.

So partially by default, because after 3 days straight with very little sleep, I accidentally pointed the name servers for The Inquisitr to rackspace when I’d meant to point the nameservers for my personal blog there in a pre-test, we’re now signed up with Rackspace Cloud Sites.

Don’t get me wrong on one front: I’ve been mostly a happy customer of MediaTemple for the life of this site, but the difference between MT and Rackspace is chalk and cheese. We had to upgrade to the MT Nitro dedicated server about 12 months ago after we killed our cloud based VPS, and as anyone who has had a server knows, that means all responsibility falls on to you.

I can roughly get around the basics. At b5media in the early days, I didn’t have much choice until we finally got help. My vi and ssh is rough, but it came flooding back to me. But likewise my 18mths doing a B. Business with a double major in Information Systems and Marketing didn’t mean much (I finally graduated in 2006 at another university in Ecommerce and Marketing), because really when it comes to Linux servers, I don’t know much at all.

Abandoning MediaTemple for Rackspace sounds a bit extreme, but let me explain: the Rackspace Cloud Sites package is core site hosting without server management, which really is my idea of heaven.

So we’re there now, after a few hiccups. The “rock solid support” meant their live support telling me to submit a ticket that takes 24-48 hours to action, even though all I wanted was for them to import into my MySQL database the dump for this site. I’d have done it myself, but there’s no SSH access, and the DB was too big to be handled in phpmyadmin. In the end, using terminal with root access into the original MediaTemple hosted db I exported every table alone, all 34 of them (don’t ask why so many, I don’t know.) The biggest was 5.6 mb…which then caused the big three into phpmyadmin to time out 34 uploads and 3-4 hours later (and this stage, Rackspace could have solved it for me in under 1 minute because the full dump was on the server.) I then manually split the big files up, recalling my rough SQL semester in 97…but I managed to muck that up as well, because in the end some of the db were screwed (we’ll get to how I know that in a minute.)

At this point, I’d managed, after over 4 hours to get the sql db uploaded in full, and the WP install picking it up. Then I got some weird message about nodes. There’s nothing in the Rackspace Knowledge base about this, but online I found a combination of “server issues” and SQL issues. Ran through and repaired each db, still no luck. Hit Rackspace live help, and I got a cut and paste of what it might be with a message that it’s probably a script I’m running…and here’s the best thing, I’m taxing the service, despite the sales page saying they’ll scale it to fit you.

There was however a blue moon tonight, and Robert Scoble was following the drama on Twitter and offered to find some help. This could be the last time for a long time I heap praise on Scoble, but god bless his sole, because he saved me tonight, and I can never be grateful enough.

Kyle Gelato from Rackspace came to help (@kylegato), and fixed the db errors. As of when he did that maybe 4 hours ago as I write this, all was well in the universe, and the site works, name server propagation issue on occasion aside.

The site isn’t 100% perfect, for example every now and then the front page delays loading, and I don’t know why, that and I’m still uploading the archive of images. But that aside, it’s mostly purring. When I wake up in the morning (presuming Steven Hodson doesn’t call me at 4am again to tell that the server is down) all will work, and we’ll be back on track.

It was the best of times, and worst of times. I just hope to god I don’t have to go through it again. I’ve not got that much hair left to be able to repeat the process ?

I should note though in postscript that I don’t fell ill will to MediaTemple, and I may well downgrade our account to something cheaper and keep it for some minor sites. They’re very well at what they do, but they don’t offer the scaling opportunities a cloud based solution like Rackspace cloud does, and unless you’ve got a tech team to support your server, Rackspace Cloud offers the better package once you get big.