Tarik Kelestemur’s PhD Dissertation Defense

Location: ISEC 532

“Combining Classical and Learning-based Methods for Visual and Tactile Manipulation”

Abstract:

Robots that operate in dynamic and ever-changing environments need to make sense of their surroundings and act in them safely and efficiently. This requires the integration of multiple sensory modalities such as visual and tactile. Humans can naturally fuse different feedbacks from the environment or substitute them with one another to perform everyday tasks. For example, to use a computer mouse, we first locate the mouse using vision and then use touch feedback from our fingers to precisely localize the buttons. Ideally, we would like robots to have human-level perception and control of the environment to achieve various tasks. This dissertation address two significant problems toward this overarching goal.

The first problem we consider in this dissertation is figuring out how to use tactile information in conjunction with visual feedback. Robotic manipulators that interact with objects and environments are often equipped with visual sensors such as RGB and depth cameras. They estimate the state of their environment using these sensors and act upon the estimated state. While a large body of previous work has shown that we can achieve impressive results only with visual sensors, more precise and delicate tasks require touch information which gives direct feedback from the environment. To this end, we propose methods for efficiently combining the tactile and visual information to leverage the advantages of these modalities.
The second problem we investigate is how to build visual and tactile manipulation methods that can generalize over the different novel environments and objects. The rise of deep learning has enabled robots to solve challenging perception and control problems using visual and tactile observations while generalizing to novel objects and environments. However, a common issue among deep learning-based methods is that these methods usually work only within the distribution of the training data and do not perform well when they are presented with unseen examples. Furthermore, they cannot distinguish whether they are dealing with in or out-of-distribution data. We propose to address this issue by combining well-established and principled algorithmic priors with the generalization capabilities of deep learning.

In the first part of this dissertation, we investigate the problem of pose estimation of the robotic grippers with respect to the environment and objects. The proposed framework introduces a learnable Bayes filter that can estimate the position of a gripper in a single image of the environment. We learn the observation and motion models of the Bayes filter using modern neural network architectures and use recursive belief updates for tracking the position of the gripper over time. Later, the belief estimation is used as an input to policies where the aim is to solve manipulation tasks using tactile feedback. In the second part, we look at the problem of estimating shapes from partial observations. We propose a method called DeepGPIS that combines a powerful deep learning-based implicit shape representation with a non-parametric inference approach model for implicit surfaces (GPIS) which allows us to generate complete shapes of novel objects and estimate their predictive uncertainties.

Committee:

Prof. Taskin Padir (Advisor)

Prof. Robert Platt (Advisor)

Prof. David Rosen (Advisor)