Lightning Storm and Timely Upgrade
Just a few nights ago lightning struck near enough that we thought we'd been hit.
No evidence in the neighborhood in the morning, but immediately after the day-bright flash was the explosion of thunder. Immediately after that our smoke detectors all alarmed (they're connected so if one triggers they all sound). Very shortly after they started they stopped. As the wife and I looked at each other in amazement I realized the light in the garage was on. It was not left on, mind you, but the lightning caused something to ignite the florescent light to glow bright for a few minutes.
A quick search of the house showed no other immediate damage or quirks. The microwave clock was still set as were the bedside alarm clocks. None of the computers seemed harmed. No other lights were illuminating without me toggling its switch.
The next day all looked right as we prepared and left for work. No remains indicated a terrible storm. No repair crews, no burn spots, no fallen limbs. Not even little branches with clumps of leaves that fall so often in heavy winds. All seemed well.
All was not well. I noticed while at work I could not reach one of the servers in the house. I could connect to two, but not the third, and from the two I could ping everything else, so it seemed just the one had a problem.
When I got home I tried checking the downed server via the terminal, but the monitor would not alight, no matter what key I hit. The power light was on, and the hard drive flickering, so I hated to trip the power, but I saw no other option. The machine rebooted immediately and promptly. It seemed to be doing well, but a quick check showed the LAN adapter wass not being recognized. After some double-checking cables and probing in the BIOS I gave up and plugged in a PCI card that happened to be right there. It was found immediately and functioned. I adjusted the static IPs, rebooted again, and it seemed to be behaving.
That was Thursday...then Friday happened.
Friday I woke to an alert that one of the servers couldn't see the other. The one that couldn't be seen is on its own network, hiding behind our VOIP router, as that router didn't play nice when readdressed to participate in the network. It worked well as a stand-alone server, so I left it alone. This stand-alone server happens to be one I've been slowly working to replace with the server with the now new NIC.
A quick check showed the server was running, but the router was off. Last week the same router had a power transformer fail, so I found another one and plugged it in. All seemed well. The phone didn't work, but sometimes there is a required reauthorization with the router that I didn't have time for.
I worked on the servers again for part of the day, getting user accounts and mail data transferred from that server to the new one. After lunch I returned to find that I couldn't connect again to the older server. I figured something may have happened to the router again.
Not too long after, I couldn't get to the web server, the one that this software runs on. I took some time tweaking the installed software on the one server I could get to.
Upon returning home, I found that the VOIP router was again off, but the other router was not off although it wasn't working. A closer inspection revealed a warped section on the top of the VOIP router, and a quick sniff test led me to believe that the router had popped some circuitry. It is kaput. I reset the other router, and again the machines could see each other. I did a quick rewire and reboot of the old mail server and it was now participating on the LAN.
I set about finishing the task of migrating the mail services to the new system. I got all of the files updated, made the requisite DNS changes, and set about to just doing some other maintenance tasks.
After a while I again noticed that the web server was again inaccessible. More investigation showed that while the router would power-on, it wouldn't, well, route.
Some additional rewiring, a little system readdressing, and now the new server is outside the hardware firewalls, being protected by a software firewall, and the web server is behind the one remaining hardware firewall/router I have. The old mail server, which I intended to remove from service after some days, is still running, but inaccessible from the Internet.
I wasn't thinking, since it was creeping on 3AM. Instead of readdressing the router to use the same IP as the now removed router, I instead changed the DNS for the server to the new address. That means a potential delay for users until caches expired and propagation could complete.