Building a High-Performance Inference Rig with 8x RTX 4090D and 192GB VRAM: Insights and Lessons Learned
In our recent workshop, we assembled a custom on-premise inference server designed to handle demanding AI workloads with high throughput and control. Sitting now beside the workbench, this rig is operational and serving our team efficiently. While the hardware specifications are impressive—featuring eight RTX 4090D GPUs with a combined 192GB of VRAM—the real value lies in the lessons learned during its development, which can be crucial for others considering similar setups.
The Hardware Configuration
- Graphics Processing Units: 8x NVIDIA RTX 4090D, totaling 192GB VRAM
- CPUs: Dual AMD EPYC 9004 Genoa processors
- Motherboard: ASRock Rack GENOA2D24G-2L+
- Power Supply: Four 2000W units on a distribution board, summing up to 8,000W
- Chassis: Custom CNC-machined 4U enclosure (off-the-shelf options proved inadequate)
- Cooling: Twelve identical case fans managed via a single hub, ensuring front-to-back airflow
- Power Consumption: Approximately 4,600W under load during inference
This configuration supports multiple workflows, including:
- Production: Tensor-parallelized 70-billion-parameter models
- Staging: Fine-tuning 32-billion-parameter models concurrently
- Operational Tasks: Running quantized DeepSeek-V3 models for automation
The system operates with no reload penalties, rate limits, or API costs, providing a significant advantage for high-volume, low-latency inference.
Why Dual-Socket EPYC Genoa? Understanding PCIe Lane Allocation
Selecting dual AMD EPYC Genoa processors was primarily driven by PCIe lane requirements rather than core count. Each EPYC 9004 CPU offers 128 PCIe Gen5 lanes; dual socket configurations facilitate even greater connectivity once the interconnect lanes (xGMI) are considered.
- Single-Socket: 128 PCIe Gen5 lanes
- Dual-Socket (Default 4-link xGMI): 128 PCIe lanes, with some bandwidth allocated to inter-socket communication
- Alternative Configurations: 3-link xGMI yields approximately 160 PCIe lanes for device connectivity
- I/O Lanes: An additional 12 PCIe Gen3 lanes from the I/O die support storage and networking
Our motherboard, the ASRock Rack GENOA2D24G-2L+, exposes 20 MCIO connectors—each capable of 8 lanes, aggregating to 160 lanes in the 3-link xGMI setup. This effectively dedicates the PCIe bandwidth where it’s most needed: directly to the GPUs.
Lane allocation for the rig:
- 8 GPUs × 16 lanes each = 128 lanes
- Remaining 32 lanes + 12 Gen3 lanes accommodate storage, network interfaces, and platform overhead
For those designing similar builds, understanding and managing these PCIe lane configurations is critical to ensure optimal GPU performance without bottlenecks.
The MCIO Cable: The Hidden Trap
The MCIO connectors on the motherboard are unique. They look symmetrical but are not interchangeable. Each cable has a designated host and device end, marked by a small embossed triangle. During initial assembly, confusion can lead to misorientations:
- Symptoms of misconnection: GPUs enumerate at PCIe Gen1 x4 speeds, drastically reducing throughput
- Diagnostic tip: Confirm correct orientation at both ends before mounting the GPU risers; once installed, the connectors are hidden and challenging to inspect
In our case, misplugging two cables caused the affected GPUs to operate at only 10% of expected throughput, highlighting the importance of meticulous wiring and careful referencing of the manual.
Power and Thermal Management
High-performance GPUs, especially eight RTX 4090D units, demand significant electrical and thermal management:
- Power Draw: Approximately 3,400W from GPUs alone, with an overall system draw of around 4,600W during inference
- Cooling: Customized airflow with 12 case fans ensures adequate thermal dissipation; inadequate airflow can lead to throttling and throughput loss
- Power Supply Redundancy: Using four 2000W PSUs allows continued operation even if one unit fails, adding resilience
The custom chassis was engineered specifically to accommodate these components—off-the-shelf server chassis either lack the necessary space or are excessively loud. Our design balances performance, noise, and form factor for operational practicality in a workshop environment.
Operational Considerations and Cost-Benefit Analysis
While building such an aggressive setup isn’t suitable for every team, the strategic benefits are noteworthy:
- Data Sovereignty: Operating inference hardware locally keeps sensitive data within your infrastructure
- Cost Efficiency: As models and workloads scale, the cost of cloud API usage can surpass the hardware investment, especially with high throughput demands
- Control & Flexibility: No API rate limits, reload penalties, or vendor-driven restrictions
The hardware investment amortizes well when serving high volumes of tokens or models, with the added intangible benefit of sovereignty and data control.
Final Thoughts and Community Engagement
Building a high-end inference rig is a complex but rewarding endeavor. It requires meticulous planning—particularly around PCIe lane allocation, cable management, and thermal design. For teams exploring transitioning from cloud APIs to on-prem solutions, considerations often revolve around cost, control, and scalability.
If your team has already made the switch from cloud to local infrastructure, what factors influenced your decision? Was it cost savings, data sovereignty, or operational flexibility? Conversely, for those still on hosted services, what barriers prevent moving in-house?
Sharing experiences and technical insights helps advance the community’s collective understanding. Whether you’re optimizing a similar setup or contemplating future hardware investments, continuous dialogue fosters innovation.
Interested in more detailed technical discussions or sharing your own experiences? Feel free to leave a comment or reach out.
