It took some tinkering, a little trust, and acceptance that not all will be the same, but the new hard disk in the old desktop-turned-server is finally in use.

It didn't take long to get the new drive installed, partitioned, and formatted. The OS booted and saw it right away. Then I started trying to move data around.

The primary goal was to move the /var folder off the root partition drive and onto its own disk. It seemed to be the biggest folder on the partition, so it made sense to move it away.

The machine isn't huge, with a Ryzen 7, commodity GPU (originally to supplement the CPU's integrated GPU to drive a second monitor), now 64GB of RAM, and 3 HDD. The 1TB drive has a small, 100GB boot partition, and the other 900GB is the original /home. The 2TB drive was mounted as /archive, where my "time machine" kind of back-ups from the other servers go, and also where I store a bunch of bigger data used on other systems. The new drive is a 4TB, chosen for its lower price than the larger drive I was tempted to get. There's about 2TB free across all the original drives, just not spread evenly.

When I started downloading and playing with Ollama and a few LLMs in Docker, I quickly discovered that the Docker install was using a /var/lib folder for its containers and volumes. This, and some previous Docker building and playing when the now server was my then primary desktop, had a ton of stuff in the Docker folder. I cleaned that up and moved the LLM volumes to the /archive drive. That helped, but still left the partition close to full, so I decided to do the thing I usually do, and throw more hardware at it.

Mentioned in a previous post, the machine had 32GB of RAM in it, so I tossed in a couple more sticks, giving it 64GB. I thought about tossing in 64GB to make it 96GB, but the price difference on the 32GB sticks was too much for a hobby project. Perhaps this will come back to haunt me, as now it's a toss-away thing to swap out the sticks for more. At the same time, if I decide I do need enough more to warrant that swap, I'll probably want to build a beefier system anyway, so it'll be newer and bigger RAM on a newer motherboard with a newer processor, so I'm OK waiting.

While I was waiting for the fella to fetch the RAM (he had a 2x16GB 32GB kit in the back that was discounted a few dollars over the 2x16GB single stick kits), I peeked around a few aisles. I'd like to toss in a glorious GPU, but the TensorFlow cards are monstrously priced for my poking around, so I'll stick with CPU AI for now.

I also noticed there were some deals on HDD, and opted to save $100 by grabbing a 4TB instead of a 6TB or larger drive. That'll probably come back to haunt me, too, because I noticed when I installed it that I have no more free SATA ports on the motherboard that's in there! I might have missed something, but it looks like there are 4, and one is used by the DVD drive (it used to be a workstation, right?) and now with the 3 HDD. There are two M2 slots, and I thought I had an M2 SSD given to me by the local Micro Center waiting to be used, but I couldn't find it while I was poking around. Maybe I'll add one (or two) of those later and rearrange the system again.

After installing the bits, I partitioned and formatted the new drive, and set about trying to copy the /var stuff off the root partition. My /var should never approach 4TB, so I thought I'd move the /archive there, and maybe move the /home to the 2TB drive, and put /var on the 1TB partition. There are 900GB of things in the /var folder, so I decided to not push its out of space again and put it on the 2TB drive. The space in the /home is unlikely to grow, and I can spend time cleaning it up (probably putting it into the /archive, like I've done with other machines as I promote them to server...) I can gain that back for another noisy space.

So I started with rsync. I use rsync to time machine things from the servers to this /archive space. I mounted the new drive (as /var2, because I thought to use it as /var when I partitioned it the first time, but will change it later) and started the copy. Alas, I didn't do it right, and the hard links that I use in my time machine tree (so that yesterday's unchanged file is the same as today's when the copy is made) weren't being kept as hard links, and instead the drive exploded with new copies of the same files. I learned I needed to add the -H flag, which worked better, but it was so slow, and things were getting the wrong dates. When rsync made a new folder to copy things into, it had a current timestamp, where in the source folder they tended to have their created dates instead. A small thing, but I knew it'd be annoying to look into a 2024/12/1 folder and see a 2024-12-15 date.

I switched to tar. I made a zipped TAR file of the /archive contents on the /var partition. Hard links are loved and preserved by tar, as are permissions and dates. The creation took a while, and then I exploded the TAR onto the new partition. It created everything as expected, and the sizes of the drives varied only by the size of the TAR file, and some rounding, probably because a time machine or few had run during the process. The TAR exploded into a folder named archive, so I moved its contents to the root of the partition, and ran another rsync, with a --delete parameter, and it copied just a bunch of files, and removed the archive. The file systems matched!

I edited my fstab file and remounted the partitions so the 4TB /archive was in place, and the /var2 was now the old /archive drive. I did one last rsync with a -n (which doesn't copy but shows what it would do if it ran) parameter, and there were still no changes. I purged the /var2 files and did an rsync of /var to /var2. I'm a lot less concerned about the dates of an active file system, nor do I expect nearly as many hard linked files, and rsync does a good enough job with permissions and symlinks. When it finished, I confirmed the contents were the same enough, edited the fstab file to mount the 2TB drive as /var, and rebooted the machine. I doubted the system's ability to handle swapping /var while it was running.

After the reboot, I ran a quick apt update, which ran a dpkg, and everything seemed to work. I ensured the Docker containers were running. I quick hit the Docker commands to remove the old llama3.2 LLM and loaded the larger llama3.3 LLM that started this whole idea.

As a fun tip, I learned about bind mounting, too! I used mount --bind / /tmp/root and was able to access the /var folder hidden by the /var mounted partition! I ran a couple quick maintenance things to make sure the system was behaving, and then removed the near TB from the root's var folder. The root drive is still pretty full, but I removed some snap packages and temporary files, and now it has about 15% of its space free again.

As a reminder to future me, the next time I promote a desktop to become a server, it'll be an archive off of the data and a wipe of the systems. The desktop was originally built a few generations of the OS ago, so the 100GB partition seemed plenty for the root, and probably would have stayed plenty if some of the other folders were moved off earlier. I have removed the desktop packages and some of the software I used, but I'm sure there's still wasted space by software that won't run on the system again. And all of the day-to-day cruft that builds up in a persons home directory is probably largely unnecessary, and also includes software that won't run again. There are iterative builds and a bevy of temporary files that are taking up space on a file system that doesn't need to stick around.

Still, everything I set out to do over the weekend is done now.