Goodbye, Pcie Passthrough

So I finally migrated my Gaming and work stations off my homelab, and removed it’s GPUs. While it was a really fun project, I decided that it was no longer worth the effort or stress to have to deal with.

Description of the Original Project

I wanted a single system that I could do all my processing on a single system. I spent a few quarters’ worth of bonus money getting all the parts together, and built an incredible system. The parts list is available to view in the “My Homelab Link” at the top of my page.

The expectation was that I would have a stable system that I could run 2 “stations” off of. One for gaming, and one for work. While largely successful, there were issues which made this difficult, which I’ll go into more detail on below.

What worked

Having everything on a 32 core system with 256gb of ram was great. I could spin up and down vms at will, and was only limited by the speed of the actual cores and memory. It made my life much easier when I was designing and building new bosh releases, as I could save on overall cloud cost and speed. It was nice being able to really push and see what I could do in the environment. Given that I was running of 10GBe NAS and booting from a USB stick, meant that upgrading was super simple. Can’t ever go wrong there. The system in it’s current form is pretty darn close to rock solid. It is a compute dream.

What didn’t work

This one is a bit more complicated. Because my original expecatation was to be able to run 2 workstations off a single pc, I targetted the Nvidia Quadro RTX series graphics cards, as they had built-in USB that could be passed through to the VM. All-in-one solution. Great, right? Wrong. The USB wasn’t great. Not sure what chipset they’re using, but it was flakey at best in passthrough mode, and would constantly disconnect at inconvenient times. Eventually it got to the point where I was having to reconnect my USB hubs almost hourly. On top of that I was also getting random crashes as a result of the actual GPUs themselves sending error codes to the motherboard, making stability a problem after a while. The actual cards themselves were running cool, but for some reason they were causing system instability. Plus, in linux passthrough, the GPUs just weren’t as performant as an actual physical system.

The Solution

After finally giving up on AMD platform for benchmarking, along with replacing my 3090 ftw3 card, I had a lot of extra hardware sitting around. I had picked up a 5800x off a newegg deal to see if I could get better performance out of the system if everything was on the same die, instead of being split between infinity fabric. Unfortunately, the tests were coming out about the same, and I eventually decommissioned the AMD motherboards and cpus.

They had been sitting in their boxes for a month, and I wanted a new project. I decided I would spend a little money and build 2 new systems. A flashy gaming system with watercooling, reusing parts I already had, and a new workstation for myself to replace the linux vm I’d been using for work since Mid-november.

Now it’s time to do the actual builds.

Parts list

Most of the technical decisions I made on hardware was for the sake of simplicity, and what was available.

Gaming PC

• Corsair 4000D case - Great case! well-built and relatively inexpensive. Also was available at Best Buy.
• AMD 5900x
• Asus ROG Crosshair VII Dark Hero
• 16GB DDR4 4000 RAM (c15) - The leftovers binned out from the racecar. Can’t get them to really overclock, but that’s fine for this build.
• EVGA 3090 FTW3 ultra gaming with EK block
• 360mm Radiator
• 3x Arctic 120MM fans
• 2x Corsair fans that came with the case.
• EK flat pump/res combo
• EK CPU block
• Sexy black tubing and quick disconnects.
• Avermedia GC553 USB and GC573 PCIE for video capture.
• 750w PSU

The system is fast, pretty, and exactly what I want/need to play games and stream. Works great, has lots of connectivity, and does the job exactly how I would want it done. While I think a 3090 is complete overkill, I know that it’s future-proofed and should last me for at least the next 4-5 years before I have to consider upgrading. That’s good enough for me.

Workstation PC

• Corsair 4000D case
• AMD 5800x
• MSI X590-A something-something motherboard
• 64GB DDR4 3600 RAM (4x16, c18) - it was cheap and on sale at Best Buy.
• Simple CPU air cooler. I believe it’s a coolermaster. It was 20 bucks and on sale.
• 1TB WD Black m.2 PCIE SSD - Again, on sale.
• 3x Arctic 120mm PWM fans
• 2x corsair fans that came with the case
• Quadro RTX 4000
• 850w PSU

Now you’re probably asking yourself, “Self, why did he go with a larger psu on this system than the other?” The answer is simple, it’s the parts that they had available at the time. Either PSU would have worked fine in either build, but I was contemplating wanting to also put the quadro rtx 5000 in this system at a later time for ML/AI processing, so I opted to put the 850w psu into the workstation. I didn’t go super sexy with the build, as I wanted this to be a work horse. No need for sexy water cooling as air would do the job well enough. The quadro rtx 4000 is perfect for running my dual 4k monitors on my work desk, and decodes 4k videos off youtube without stutter. I spent for the extra memory as a buffer in case I ever convert this system into an esxi host down the line. While not the fastest, it’s fine for a linux desktop machine.

The builds

While I had originally wanted to build these the day BlizzCon 2021 started, I ended up only getting one system done. The other was a few weeks later. I had already committed to re-streaming the event using windows vms for each of the different panels for our internal groups to watch together. One for the main stage, warcraft, diablo, overwatch, hearthstone, and other. I ran out of time and kind of rushed getting the initial build together, making the cables and pump just kind of “in” the system so I could get everything up and running for the event. Was hacky, but worked.

A couple of weeks later, I finally got better tubing in and got my actual runs right, and everything looks much better. After I finished that build I did the linux workstation.

I used Clonezilla to transfer my workstation vm from the old pc to the new. Once the process was finished, I fixed the grub boot partition using an ubuntu bootable usb and everything came up fine. The system was relatively stable other than the wired network driver having disconnect issues. I installed the r8169-dkms package and everything’s good to go. Nice and fast, XMP enabled, booted right up and had no complaints since.

I do still use the USB hub, as my system is sitting on top of the rack a few feet away from my desk, but I’ve not had disconnect issues since the upgrade. Even got to finally stop using a bluetooth USB dongle for audio from my headset.

The windows system, I didn’t even bother to migrate. It’s old and needed to be rebuilt anyhow. Ended up just getting any personal data off the disk and wiping it. Most of it was already synchronized to NAS and encrypted at rest anyhow.

I do keep the windows VM around though, primarily for transcoding.

Conclusion

My homelab system is still amazing, and I’ll use it until the components die. It’s been rock solid since I removed the GPUs from it. This also freed up 64GB of ram for more virtual machines, which I’ve been using to quickly test pcf deployment automation pipelines as of late.

While the project will always be near and dear to my heart, I probably wouldn’t repeat it. I needed something to take my mind off my shoulder and show me that I still had the ability and skill to do something that very few people had ever done before. I got 2 incredible stations set up, with 2 different uses. I also have more near-silent compute power than most people will ever need in their house.

I will not forget the time and effort I put into the system, and how the build helped me through a difficult time in my life. It was worth the effort, and I am thankful for the experience.

The last 6 months have taught me that Linux is indeed ready to be a permanent workstation and replacement for my mac laptop. Windows is most definitely not a very good development platform unless you’re working specifically with windows.

Hrm. What should I build next?