GPU Render Farm - Small Studio

[Samuel]

Hi CG community,
First of all thanks Alex for the impressive [Ultimate Guide] on this topic as there's a real lack of information out there.

I'm about to set up a CGI Studio, things we'll scale up as we grow but I need a 'proper' render farm as a starting point as we already feel the need for it.

I'm roughly familiar with the subject but it has been a while since I've set up a build, I would appreciate any constructed feedback that would help me make the right decision on which bits of hardware would suit my needs best within my budget. (£15000)

Ideally, I'd like to start with a 4xGPU rig which I would update to an 8xGPU rig at some point in the future.
Day by day projects are using C4D and rendering with Redshift/Octane.

Here's what I had in mind in terms of setup:


CPU: AMD Ryzen Threadripper 3960X
GPU: 4x GIGABYTE GEFORCE RTX 3090 TURBO
MOTHERBOARD: Open for the suggestion, any PCIe x8 or x16 that would work?
RAM: Corsair Vengeance LPX 32GB
SSD: Samsung SSD 860 QVO 1TB
PSU: Open for suggestion

NVLINK: Is there any way to use NVLINK to bridge all the GPU? (Aiming at scaling up the memory)

Cooling & Case:
That's where I'm a bit lost - I've got room to be used exclusively for the render farm - although there's already another server in and it's generating a lot of hit - I'm wondering what's the best solution.

Self container case + water cooling system?
Open rig (Mining) + Fans?
Other options?

Any reason why any mining rack won't work with that build?

Render manager:
Any idea what would be the best render manager to get this up? Thinkbox deadline? We're mainly working with freelancers around the world so it would need to be accessible from the internet + local network - almost exclusively working from C4D using Redshift and Octane with the occasional project rendered through Houdini.

Thanks in advance.

Best,
Samuel

 

[Alex Glawion]

Hey Samuel,
Be sure to take a look at our renderfarm guide as it should answer most of the questions you listed.

For 4x 3090 I'd get a PSU with at least 2000W. Not many choices here, the EVGA Supernova comes to mind.

A Motherboard for 4x dual-slot GPUs (be sure to get blower style gpus that are dual-slot, not open air cooled gpus) is the GIGABYTE TRX40 Designare. I am not sure about the sWRX8 Boards that support Threadripper PRO CPUs, though even though they have 8 pcie-slots they don't seem to be two slots apart to be able to install 8 gpus in there without custom cooling.

Your Setup suggestion looks good to me. You can't link 4 GPUs with NVLINK, two is the maximum - You can link 2x x2 and render a frame per double-linked GPU to make full use of the VRAM.

Any Case with 8 slots and room for an eatx or xlatx board will do just fine. I have a couple of Corsair 600q where I set up my rendernodes with, or you can get something like the Fractal Design Define XL R2.

Blower style gpus are pretty good at exhausting heat out of the case, you won't see that much of an improvement to have them run in a open rig. If you space them out with risers in an open rig, you might even see performance loss due to the risers. A Mining rack would sure work, but you have to make sure the motherboard is not mining-focused, as mining-boards tend to have insufficient pcie-lanes / bandwidth for GPU rendering.

Deadline - no questions about that. I tried a variety of renderers, and it's the best there is out there in my opinion. Free for up to 10(?) nodes, too.