ECE PhD Dissertation Defense: Mohammadreza Sharif

PhD Dissertation Defense: Human-in-the-loop Prosthetic Robot Hand Control through Contextual Adaptation

Mohammadreza Sharif

Location: ISEC136 or Zoom Link

Abstract: Despite fifty years of research on prosthetic robot hands, this technology is yet to be fully acknowledged by amputees as well as manufacturers, due to lack of robustness and intuitive control. Our goal is to enhance the control robustness and intuitiveness through integrating the context, i.e. knowledge of environment and task, with human bio-signals, e.g. hand trajectory. Although this solution has already been studied in the literature, no unified framework is proposed for multi-modal information fusion. This research is aimed at introducing a novel framework for human-in-the-loop prosthetic robot hand control. We propose our solution in two parts, (1) grasp inference and (2) end-to-end actuator control. For (1), we propose a model-based and a model-free grasp inference framework. Our model-based method is based on particle filters method. With knowledge of context hard-coded into the solution per se, our particle-filter-based framework can incorporate any input signal using a proper weight function. In our model-free method we use hidden Markov mode to learn the grasp-inference task directly from human hand transport trajectories. For (2), we propose a reinforcement learning (RL) framework which learns to control robot actuator directly from the context with less information hard-coded into the solution. We leverage imitation learning (IL) besides RL to overcome challenging exploration in the problem. To provide invariance to the human hand transport trajectories, we first provide a solution based on synthesized trajectories based on a human motion model. Later, we adopt an over-sampling technique for real human hand transport trajectories, to serve as a means of data augmentation. This research provides a step forward in more rigorous frameworks for multi-modal information fusion for prosthetic robot hand control and grasp inference through model- /data-based methods.