PhD Dissertation Defense: Trinayan Baruah

PhD Dissertation Defense: Improving the Virtual Memory Efficiency of GPUs

Trinayan Baruah

Location: Zoom Link

Abstract: GPUs have been adopted widely based their ability to exploit data-level parallelism found in modern-day applications, ranging from high performance computing to machine learning. This widespread adoption has, in part, been accelerated by the development of more intuitive high-level programming languages, efficient runtimes and drivers, and easier mechanisms to manage data movement. Modern day GPUs and multi-GPU systems utilize virtual memory systems, enabling programmers to access large address spaces that are beyond the physical memory limits a GPU. There mechanisms have built in mechanisms for memory translation, sparing the programmer from having to reason about complex data-movement operations. Virtual memory support on a GPU includes both hardware and software support. At the hardware level, translation lookaside buffers~(TLBs) are used to cache translations close to the compute units. At the software level, the programming model supports a unified memory model which automates the movement of pages across multiple devices in a a system. Despite the improvements in programmability, due to inefficiencies existing virtual memory management mechanisms, including TLB management and page migration policies, the performance obtained on today’s GPUs is sub-optimal.
In this dissertation, we first identify the key challenges in virtual memory support for GPUs today. We then propose mechanisms to reduce the bottlenecks arising from virtual memory support at both a hardware level and at the runtime level. This allows GPUs to fully enjoy the benefits of virtual memory, while ensuring high performance. We also develop simulation tools that enable researchers to explore new and novel virtual memory features in future single GPU and multi-GPU systems.
To enhance hardware support for virtual memory on a GPU, we explore a mechanism that enables prefetching of page-table entries into the GPUs TLBs, thereby reducing the number of TLB misses and improving performance. We also leverage the fact that many page-table entries can be shared across different GPU cores. We design a low-cost interconnect that enables sharing of page-table entries across the GPU cores. To improve the performance of unified memory on multi-GPU systems, we propose a hardware/software mechanism that monitors accesses to each page, and uses this information when making page-migration decisions. We also propose mechanisms to reduce the cost of TLB shootdowns on the GPU during page-migration in NUMA multi-GPU systems.