Wang Receives $500K NSF Grant to Improve VR Streaming Performance and Bandwidth
ECE Assistant Professor Yanzhi Wang (PI) has been awarded a $500K award from the National Science Foundation, titled “CNS Core: Small: Collaborative: Content-Based Viewport Prediction Framework for Live Virtual Reality Streaming.” In collaboration with Rutgers University and Texas State University, the effort will address bandwidth and performance challenges of live streaming due to the delivery of 360-degree views. As part of the project, Wang will develop a new content-based viewport prediction framework to improve the bandwidth and performance in live VR streaming, which predicts the user’s viewport through a fusion of tracking the moving objects in the video, extracting the video semantics, and modeling the user’s viewport of interest. The project will benefit several VR-related fields of study with significant bandwidth savings and performance improvements, such as VR-based live broadcast, healthcare, and scientific visualization.
Bringing Virtual Reality Closer to Actual Reality
Assistant Professor Yanzhi Wang, electrical and computer engineering, focuses his research on improving the efficiency and performance of modern computing technology. The next piece of tech he’s set his sights on: virtual reality. His research project in collaboration with Rutgers University and Texas State University, “CNS Core: Small: Collaborative: Content-Based Viewport Prediction Framework for Live Virtual Reality Streaming,” for which he is the principal investigator of a $500K National Science Foundation award, aims to address the performance and bandwidth issues currently plaguing VR livestreaming.
Since the ultimate goal of virtual reality is to simulate the experience of existing in another space or world, it’s necessary for the user to feel like what they’re looking at is right before their eyes. At current processing speeds and bandwidth, VR livestreams aren’t quite able to keep up with the speed and unpredictability of humans’ head movements and focus, so they have a long way to go before they feel like a representation of actual reality.
“VR systems need to predict accurately, and they need to predict in real time,” Wang says. “Currently, neither is possible.”
Wang’s project tackles VR livestreaming’s current challenges through three research foci: first, what he calls a “content-based viewport prediction framework,” which tracks the objects within the VR system’s view and uses the context to predict where the user’s focus will move next; second, hardware and software techniques that allow the prediction framework to be executed in real time by a large number of users; and finally, evaluative frameworks to gauge the performance of the approach so it can be adjusted as necessary.
According to Wang, once perfected, VR livestreaming could make a huge difference for fields of study in science, healthcare, and technology. For example, clinicians have already used lower-quality versions of the technology to broadcast a live surgery and give medical students a taste of the chaos in a trauma center. The higher the stream quality and the lower the bandwidth waste, the more immersive the VR experience is—and Wang’s project will take virtual reality that next step closer to actual reality.
Abstract Source: NSF
Virtual reality (VR) video streaming has been gaining popularity recently with the rapid adoption of mobile head mounted display (HMD) devices in the consumer video market. As the cost for the immersive experience drops, VR video streaming introduces new bandwidth and performance challenges, especially in live streaming, due to the delivery of 360-degree views. This project develops a new content-based viewport prediction framework to improve the bandwidth and performance in live VR streaming, which predicts the user’s viewport through a fusion of tracking the moving objects in the video, extracting the video semantics, and modeling the user’s viewport of interest.
This project consists of three research thrusts. First, it develops a content-based viewport prediction framework for live VR streaming by tracking the motions and semantics of the objects. Second, it employs hardware and software techniques to facilitate real-time execution and scale the viewport prediction mechanism to a large number of users. Third, it develops evaluation frameworks to verify the functionality, performance, and scalability of the approach. The project uniquely considers the correlation between video content and user behavior, which leverages the deterministic nature of the former to conquer the randomness of the latter.
With the rapidly increasing popularity of VR systems in domain-specific immersive environments, the project will benefit several VR-related fields of studies with significant bandwidth savings and performance improvements, such as VR-based live broadcast, healthcare, and scientific visualization. Moreover, the interdisciplinary nature of the project will enhance the education and recruitment of underrepresented minorities in several science, technology, engineering, and mathematics (STEM) fields.
The project repository will be stored on a publicly accessible server (https://github.com/hwsel). All the project data will be maintained for at least five years following the end of the grant period.
This award reflects NSF’s statutory mission and has been deemed worthy of support through evaluation using the Foundation’s intellectual merit and broader impacts review criteria.