Zhiyong Zhang PhD Proposal

Title: Towards Indoor Mapping and Navigation with Perceptual Aliasing using Visual Semantic SLAM

Committee Members:
Prof. Hanumant Singh (Advisor)
Prof. Huaizu Jiang
Prof. David Rosen

Abstract:
Modern SLAM (Simultaneous Localization And Mapping) techniques allow us to create accurate 3D maps of the environment primarily using visual sensors in GPS-denied regions. In this context, numerous deep learning-based approaches have emerged, enabling the extraction of rich semantic information from images, including shapes, objects, and text.

Leveraging these technologies, our aim is to construct comprehensive 3D maps of indoor environments, which could be utilized by robots for path planning and navigation. Additionally, the solution can be integrated with a large language model, enabling the robot to interact intuitively with people.

This research comprises four main components: Semantic Feature Extraction and Tracking with SLAM: Given that the same semantic features can appear in multiple frames, some of which may not be conducive to feature detection and recognition (such as blurry images or distant views), we are developing a pipeline to ensure the optimal detection and recognition of semantic features within the most suitable frame. The pipeline also involves tracking the same feature across frames while maintaining its 3D location in the global map.

Resolving Perceptual Aliasing: Many indoor places can exhibit high visual similarity, which confuses the robot when powered up with a prior map in its memory. Semantic features can be used to localize the robot in the map, determining its specific floor or room. This capability can also aid SLAM in performing loop closure with high-level information.

Cross-Floor Constraints for SLAM Optimization: Most buildings contain a symmetric layout across floors, which can be exploited to establish constraints between them. For instance, vertically aligned rooms like 425 and 525, as well as elevators, offer opportunities for vertical constraint. Such constraints can enhance SLAM optimization, resulting in improved map accuracy.

Indoor Path Planning and Navigation: Once we have a comprehensive 3D map of the indoor environment, path planning becomes an intuitive way to utilize this map. With semantic features integrated into the map, the robot can associate 3D point clouds with high-level information, such as door numbers or office names. Large language models are available to provide a more human-like way to interact with the robot. For example, a command like “Navigate to Professor Hanumant’s office and locate the book ‘The Hitchhiker’s Guide to the Galaxy'” can be executed by the robot.