I managed to munge the server trying to upgrade the web server, and lost all services for the majority of yesterday!

Not "all services," but everything I was trying to upgrade hit the tank, and hit it hard.

I'm pretty good at upgrading software. I've been doing it for a long time, and I'm pretty careful about getting it done. There are occasions when things get out of hand and I'm not entirely sure why, and yesterday was just such a day.

The core of the problem is that I'm trying to upgrade the Apache web server on this very server to their new v2.2.0 relase. The release it has, v2.0.54, isn't so far out of date, but I've made multiple revision jumps before, and normally that's tougher than the inconvenience of promptly updating. Another key to this is that I'm trying to add SSL support to the services we offer, and as long as I have to recompile the server anyway...

So I download the source, configure the build with the new features and to install Apache into a new directory, and build the source. It goes well, and the Apache server software is now on my server hardware. The documentation says that the new Apache software will work with the same modules that the old software worked with, but that they'd need to be recompiled. Sure, things changed in source, and they'll need to be relinked; got it.

I checked and didn't have the source used to make mod_perl or PHP still expanded on the system. Usually I try to leave the source for the last builds there, in case I need to rebuild for whatever reason. I download the new PHP (5.1.2) and mod_perl code.

PHP won't build 'cause of a lot of link issues, although it compiles just fine. I remove some options I didn't think were used (IMAP, for example), but it still fails. I find a support forum entry that says the PHP source won't build if MySQL is 64-bit, which mine is. So I download a 32-bit version of MySQL, which happens to be a new version, too. That installs just fine, and PHP stops complaining when told to use that library, but still fails to link to other libraries. I download and upgrade those libraries, but still no linking occurs.

I decide to stay with the current, working PHP, and compile, link, and install that into the new Apache. Well, most of the PHP goes into the same directory into which it was installed before, but it builds and installs an Apache module in the new Apache server's directory.

I build and install mod_jk, so the Apache server can work with Tomcat just fine. Although Tomcat has been updated to 5.5.16, I'm happily plugging along with 5.5.12. Plenty has changed, and I've discovered some new configuration tips that'll help future upgrades go more smoothly (changes between 4 and 5 I didn't know at the time of upgrading that path), but I'll finish other upgrades first.

Then I start in on mod_perl. I need to use v2 of mod_perl to work with Apache 2.2.0, according to their site. I download and start building. First it complains that there's a mod_perl v1 present, which previously was required when we ran Apache 1.x, but the previous installation of mod_perl v2 didn't complain. I follow the directions and remove the offending files (thankfully by copying them away instead of deleting). Then it seems to build and install correctly.

Apache's compiled and installed. The connector modules are compiled and installed. All looks good. I altered the Apache configuration files in the new Apache directory. I double-checked the files and installations, and decide all looks good.

I stopped the running Apache server, change the remaining configuration files and try to start the new server. No good. The error log showed some invalid directives. I check and see that I'd not done the Subversion modules. Since I know I'm the only one to use Subversion on the server, I correct the configuration files to skip those bits and I comment the Subversion modules from the LoadModule section. I tried again; still no go. The error log showed that mod_perl can't find some PERL modules.

I spent a bit of time trying to reload the PERL modules. Some were updated, and some were missing. I noticed that the PERL include path included the module directories for the old PERL installations (currently 5.8.7, and, yes, there's a 5.8.8 out there!), so I moved them away, too, as I had the mod_perl stuff previously. That made it worse, but I figured that there were just required modules in the directories I'd moved, so I spent way too long adding them back.

This started taking too long, so I reverted the configuration files and tried to fire up the old Apache server. This, too, failed. The PERL damage seemed to affect that installation, too. Since I was going to have to solve the problem one way or another, I pressed on.

After far too long I gave up and set about trying to restore the Apache v2.0.54 (yes, there is a 2.0.55, but today I'm not interested). This proved to be troublesome as the PHP seemed to change some libraries in its installation, so I had to recompile it with the old Apache directory. This seemed to be the flaw in my plan as I failed to get the new PHP module to work with the new Apache server and tried to settle on rebuilding the old one; evidently the PHP installation links to Apache server modules, and since the Apache versions were different, there were understandable conflicts.

After almost two hours of recovery efforts, the server is back and running as it had been before I started. Except now there's additional source out there, and a new MySQL installation waiting to be used.