DxPU: Large-scale Disaggregated GPU Pools in the Datacenter

Traditionally, all hardware resources (e.g., CPU, GPU, and memory) are tightly coupled with host servers. However, as different hardware exhibits diverse iteration cycles, this server-centric architecture makes it hard to upgrade hardware resources independently. Recently, there emerges a paradigm shift towards resource disaggregation in the datacenter [3, 31, 33]. By decoupling these resources and grouping them into corresponding pools, it is easier to maintain, upgrade them, and deploy cutting-edge hardware technologies in time. For example, based on the abstraction of a Transport layer, NVMe over Fabrics (NVMe-oF) [9] enables memory operations over various kinds of interconnects (e.g., Fibre Channel, RDMA, or TCP). With the help of NVMe-oF, the storage resource can be maintained, and upgraded independently.

As mentioned above, the tight coupling of host servers and GPUs causes resource fragments and restricts the independent maintenance, upgrade of hardware resources. To overcome these problems, GPU disaggregation is proposed. Specifically, GPU workloads on the host server are encapsulated and redirected to disaggregated GPUs. For example, DGSF [35] makes use of API remoting to realize GPU disaggregation for the serverless environment and supports live migration at the same time. In general, we can implement GPU disaggregation at the software level (user-mode library, kernel-model driver) and hardware level (PCIe level). As the software solutions are based on API hooking, the applicable scenarios and GPU performance are greatly affected. By contrast, exploring forwarding strategies at the PCIe level can overcome these problems easily. However, existing solutions at the PCIe level fall short of the expected disaggregation scope and capacity. In the following paragraphs, we will first explain why we implement GPU disaggregation at the PCIe level (Section 2.2.1). Then, we summarize why existing solutions at the PCIe level fall short (Section 2.2.2), which motivates the design of DxPU. Table 1 makes a comprehensive comparison between existing GPU disaggregation systems.

Figure 2(a) shows the overall architecture of DxPU. Compared with the non-disaggregated architecture, which attaches GPUs to the host server via PCIe endpoints, GPUs are physically decoupled from host servers and put into separate GPU boxes. Correspondingly, each GPU box contains several GPUs. To enable the remote access to GPUs from the host server, a proxy, named DxPU_PROXY, is added on both the server and the GPU side. It acts as medium for the interactions between the host server and GPUs. The DxPU_PROXY is responsible for the conversion between PCIe TLP and network packets. On the one hand, DxPU_PROXY provides PCIe interfaces to get connected with the host server or GPUs. On the other hand, DxPU_PROXYs are interconnected via a network fabric protocol. Specifically, when receiving PCIe TLP packets, DxPU_PROXY encapsulates them into network packets and sends out these packets. Correspondingly, on the other side, DxPU_PROXY extracts the original TLP packets from the received network packets. Therefore, the conversion between PCIe TLP and network packets is transparent to both the host server and GPUs. From the perspective of the host server, GPUs are still attached locally via the PCIe link. The combination of DxPU_PROXYs at both sides works as a PCIe switch (Figure 2(b)). Moreover, DxPU_PROXY provides configuration interfaces to enable the dynamic distribution of GPUs for the cloud orchestration system.

The key observation that motivates the adoption of the network fabric is that nowadays network cables can provide the same (or even faster) data transmission speed when compared with PCIe links. For example, a PCIe Gen 3 interface offers about 7.87 Gbps per lane at the physical layer [43], and the PCIe interface provided by a GPU has no more than 16 lanes, which can be supported easily with two 100 GbE ethernet cables. What is more, the adoption of the network fabric can perfectly solve the scope and capacity problems mentioned above.

Supporting Datacenter-scale GPU Disaggregation. In DxPU, servers and GPUs are connected to the network fabric through network cables, whose maximum length can be up to 100 meters [12]. Hence, the scope of GPU disaggregation is no longer limited to the rack scale. GPU nodes can be allocated to host servers from anywhere in the datacenter, enabling fine-grained and efficient GPU provisioning [36].

Supporting Large-capacity GPU Pools. Driven by the developments of network technologies, nowadays, the number of ports [10, 15] in a datacenter-level network is enough for DxPU to hold more GPU nodes. Specifically, compared with the PCIe fabric, data transfer in the network fabric is more reliable in its error check (e.g., Receive Errors, Send Errors, and Collisions) and packet retransmission. So, the network fabric is more tolerant to transmission error and expansible, where we can simply cascade the switches to build a larger network. With its help, a GPU pool in DxPU can accommodate hundreds of nodes.

It is to be noted that, in DxPU, GPUs are dedicated to the host server during use. The binding relationships between GPUs and the host server will not be changed unless users free them.

With respect to GPU Boxes, there are two kinds of design schemes. The first one learns from NVIDIA DGX Systems [19], which specialize in GPGPU acceleration. Under this scheme, GPUs and the motherboard containing NVSWITCHes are welded together, leaving PCI interfaces for DxPU_PROXY s. Thus, NVLINKs can be utilized in DxPU to support more efficient GPU communications. The second one is a simple PCIe switch that we have mentioned above. All kinds of GPUs that connected via PCI interfaces can be deployed and provisioned in this design. The former design is suitable for users requesting multiple GPUs and high compute capability, where users should be allocated with nodes connected by NVLINKs. While the latter one is appropriate for single GPU request scenarios.

In DxPU, DxPU_PROXY acts as a PCIe virtual switch for the interactions between the host server and GPUs. From the host server and GPU side, DxPU_PROXYs are inserted into the PCIe slot of the host server and GPU Box. And DxPU_PROXYs are interconnected via a network fabric protocol. During the boot stage of host server, BIOS program only enumerates the PCIe virtual switch displayed by DxPU_PROXY. So, it would reserve enough memory space for disaggregated GPUs, which would be hotplugged into the host server later.

In this section, we start by explaining how network latency incurs performance overhead. To measure this impact and guide the implementation of DxPU, we build up a performance model, which is based on AI workloads. It is to be noted that, here, the performance denotes GPU compute capability in DxPU compared with the native one, which is different from the model prediction accuracy in AI scenarios.

Despite the comparable data transmission rate with a network cable, DxPU still introduces non-negligible latency in the PCIe TLP transfer between the host server and GPUs. Furthermore, the introduced latency may affect the throughput of the data transfer between them for some PCIe TLP transactions.

PCIe TLP transactions are categorized into non-posted (e.g., Memory reads) and posted transactions (e.g., Memory Writes), depending on whether the Request (i.e., the issuer of the transaction) expects completion TLPs in a transaction. In non-posted transactions, the Requester will not finish the request until they receive one or more completion TLPs from Completer (i.e., the receiver of the transaction request). Therefore, the Requester needs to trace the status of all in-flight non-posted transactions. To this end, the Requester assigns each non-posted transaction with an identification (named tag) and is responsible for ensuring the uniqueness of tag for each in-flight non-posted transaction to distinguish them from each other. When there are enormous data transfers, the number of tags supported, which is device-specific, is generally enough, and new non-posted transactions will not be blocked (Figure 3(a)). However, as shown in Figure 3(b), the latency introduced by DxPU makes the tags used up quickly. New transactions are postponed until a free tag is available. As a result, the throughput of data transfer based on non-posted TLP transactions (e.g., DMA read issued by GPUs, which is implemented via PCIe Memory Read) would decrease significantly.

As we have discussed, DxPU brings the longer latency for PCIe TLP transfers ❶ and lower throughput for non-posted PCIe TLP transactions ❷. The next question arising is: What is the performance impact of DxPU for GPU workloads? To answer this question, we build up a performance model via API hooking techniques to simulate the introduced latency by DxPU (❶❷). Moreover, our performance model targets AI workloads, since they cover the most GPU application scenarios in the cloud.

According to results of performance model, we notice that the training performance of typical ResNet-50 model can still remain above 90% with about 8 us \(RTT_{DxPU}\) . This motivates us to further prototype the design of DxPU. There are two kinds of straightforward options for us to build the implementation system of DxPU. The first option is leveraging off-the-shelf PCIe-over-TCP commercial products. However, these products target low-speed devices and thus are not suitable due to the performance requirement mentioned above. The second option is building by ourselves. Obviously, building such a product that achieves the product-ready status brings huge development burden and hinders our prototype process. Instead, we present the above design to our third-party vendors and seek for a satisfying product to act as DxPU_PROXY.

As a result, we build two customization-level implementation systems of DxPU, whose \(RTT_{DxPU}s\) are 6.8 us and 4.9 us, respectively. According to our performance model, such \(RTT_{DxPU}s\) keep the performance of AI workloads in GPUs above 90%. Our systems implement a proprietary protocol for the conversion between PCIe TLP and network packets, which is neither PCIe over TCP nor RoCE (RDMA over Converged Network). From the host server and GPU side, DxPU_PROXYs are inserted into the PCIe slot of the host server and GPU Box. And DxPU_PROXYs are interconnected via the protocol. The PCIe interface for a GPU is PCIe Gen 3 \(\times\) 16, while the two 100GbE network cables are used for each DxPU_PROXY to get interconnected. Note that the interconnection between DxPU_PROXYs is not limited to the network cable in design. For example, another different implementation could be building the network fabric based on Infiniband RDMA technology. Furthermore, The implementation systems demonstrate the practicality of DxPU and verify the accuracy of our performance model.

We set up a disaggregated GPU pool consisting of 32 NVIDIA GPUs (16 Tesla V100 GPUs in the first type GPU boxes and 16 RTX 6000 GPUs in the second type GPU boxes), coupled with four host servers. In addition, the \(RTT_{DxPU}s\) is 6.8 us. And the detailed configuration can be seen from Table 5.

To measure the components of \(RTT_{DxPU}\) , we request a single Tesla V100 GPU from the pool and perform repeated read-write operations in the GPU memory space. During these operations, we calculate the time length of different parts in \(RTT_{DxPU}\) . Generally, \(RTT_{DxPU}\) can be split into three parts: original time latency, network transmission, and packet conversion. As can be seen from Table 6, they make up 17.7%, 27.9%, and 54.4% of \(RTT_{DxPU}\) , respectively. The original time latency pertains to the duration taken to transmit PCIe packets without DxPU, where GPUs are PCIe devices on the host server. The network transmission time denotes the time required to transmit network packets between DxPU_PROXYs. The packet conversion time indicates the duration required to convert PCIe packets to network packets and any additional time spent due to the design of the network protocol (such as error control and redundancy check).

To demonstrate the impact of \(RTT_{DxPU}\) on the bandwidth between the host server and GPUs, we record their values in Table 7. The bandwidth from the host server to GPUs experiences a rapid drop to only 24.1% due to non-posted transactions, whereas the bandwidth from GPUs to the host server remains largely unaffected.

From the GPU pool, We request a single RTX 6000 GPU for basic workloads, eight Tesla V100 GPUs (supporting NVLINKs) for AI workloads, and a single RTX 6000 GPU for graphics rendering workloads. We compare the evaluation results in DxPU with those in the native environment, whose architecture is server-centric. And other configurations of DxPU and the native environment are the same. It is to be noted that, here, the performance denotes GPU compute capability in DxPU compared with the native one, which is different from the model prediction accuracy in AI scenarios.

We aim at measuring the performance impact brought by DxPU on the typical user cases in the cloud. To this end, we adopt the following standard benchmarks:

As is mentioned above, in DxPU, the command latency is longer, compared with the native environment. Hence, DxPU is not appropriate for GPU workloads consisting of too many short-duration kernels and memory operations. Reducing the command latency is also considered as our future work.

For users of DxPU, the proportions of short-duration kernels and memory operations should be reduced in applications. And in multi-GPU scenarios, the number of DxPU_PROXY s should be considered based on the communication bandwidth.

Command Latency Overhead Mitigation. The effect caused by command latency could be mitigated with two strategies: interaction reduction and latency hiding. For the former, it aims to reduce the interaction between the host server and GPUs. For example, kernel fusion technique could be adopted to reduce the number of kernels executed. For the latter, we could leverage concurrent and asynchronous techniques (such as streaming in CUDA) to perform multiple operations simultaneously and in pipeline.

Adoption of PCIe Gen 4. Currently, DxPU is built on PCIe Gen 3. As PCIe Gen 4 supports the double throughput of PCIe Gen 3, the number of in-flight #tags in GPUs would increase. Correspondingly, the throughput of PCIe memory read transaction and memcpy(HtoD) would increase, too. Meanwhile, communication efficiency and performance of DxPU in multi-GPU scenarios would also be better. In the future, we will deploy DxPU on PCIe Gen 4.

PCIe Read Transaction Avoidance. As memcpy(HtoD) is built on PCIe Memory Read Transaction, the throughput of memcpy(HtoD) in DxPU decreases dramatically (refer to Equation (1)). To avoid this effect, an alternative memcpy(HtoD) implementation strategy can be adopted: replacing PCIe Memory Read Transaction with PCIe Memory Write Transaction. It can be achieved in two ways by leveraging hardware features of CPUs. First, CPUs execute data move instructions to transfer data from host memory to GPU memory directly, which can be accelerated by leveraging the SIMD feature. Second, we can leverage the DMA engine to finish the data transfer, which would issue PCIe Write Memory Transaction. We implement a prototype for the first one and the result shows that \(RdTP_{DxPU}\) increases from 2.7 GB/s to 9.44 GB/s.

Although GPU disaggregation is beneficial to cloud providers in resource utilization and elasticity, the distribution scheme should also be well designed. For example, GPUs in the same pool can be grouped and indexed in advance, so user requirements can be satisfied quickly. In addition, there should be spare GPUs in accordance with the equipment failure rate. So, broken GPUs can be replaced quickly to ensure that users have a good experience.

As more and more users choose to store their private data in the cloud, the importance of data privacy in the cloud is highlighted. Thus, there have been many researches towards data privacy protection [23, 51]. In AI training and inference, the model and data are considered as user privacy, which is stored in TLPs during transmission. To prevent the privacy leakage during network communication, end-to-end encryption can be applied in our future work.

Resource disaggregation has attracted extensive attention from academia and industry due to its efficiency in resource management, such as Intel Rack Scale Design [3], IBM Composable System [33], dReDBox [31], and so on. However, although there are some studies trying to provide software support for them, it is still challenging to adopt them in practice due to the compatibility or performance issue.

With increasing demands for GPUs in the cloud, GPU virtualization is proposed to split compute capability of a single GPU server and allocate it in a more flexible way. VOCL [49], rCUDA [34], vCUDA [45], and DS-CUDA [44] realize GPU virtualization via API remoting without significant performance penalty. To deal with software maintenance in high-level libraries, LoGV [37], GPUvm [46], and gScale [50] implement Para and full virtualization at the GPU driver level. Among the hardware virtualization methods [47], Intel VT-d [28] and AMD-Vi [6] fail to support sharing a single GPU between multiple VMs. And NVIDIA GRID [20] overcomes the difficulties and supports multiplexing a single GPU between VMs. However, the mechanism of GPU virtualization means that it cannot provide stronger compute capability than a single GPU server.

As the era of big data comes, the scale of Machine Learning system is becoming larger in training data and model parameters. Since increasing the compute capability of a single server costs massive systematic efforts [32, 48], distributed Machine Learning is proposed to overcome these challenges. However, such a distributed system requires more knowledge in parallel programming and multi-machine communication, forming bottlenecks for researchers. Hence, programmers need to pay more attention to reducing the network latency and optimizing the whole system [40, 41].

Network processors, like Intel IXP Series [29], are specialized integrated circuits (ICs) designed to efficiently handle and process network-related tasks, such as data packet routing, switching, security, and quality of service (QoS) management. They play a crucial role in enabling the high-speed data communication that underpins modern computer networks. In contrast, DxPU_PROXY serves the purpose of converting between PCIe and network packets, as described in Section 3.3.