Northeastern University College of Engineering

Mock Interview: CommLab Drop-In Workshops
2024-08-01 16:00 - 17:00, 2024-07-25 16:00 - 17:00, 2024-07-18 16:00 - 17:00 (America/New_York)
Join the CommLab any Thursday from 4-5pm ET, we’ll delve into the intricacies of interviews, unveiling effective preparation strategies for any interview scenario. Engage in an interactive setting as we dissect the overall interview experience, discuss common interview scenarios, and share insights on what to do during critical moments. Join this hybrid workshop series through Zoom.

LinkedIn, CV, Resume: CommLab Drop-In Workshops
2024-07-31 16:00 - 17:00, 2024-07-24 16:00 - 17:00, 2024-07-17 16:00 - 17:00
Join the CommLab’s empowering LinkedIn, CV, and Resume Drop-In Workshops any Wednesday from 4 pm to 5 pm ET. This collaborative space offers valuable advice and peer feedback to enhance your online profile and professional presence. Join this workshop series through Zoom.

Graduate Student Fellowship Writing Club-Hosted by NU CommLab and NetSI
2024-07-31 15:00 - 16:00, 2024-07-24 15:00 - 16:00, 2024-07-17 15:00 - 16:00
Join the NU CommLab and NetSI sponsored weekly graduate student fellowship writing club for support in writing your fellowship application! The fellowship writing club meets virtually on Wednesdays from 3-4pm from July 10- August 21. We will offer you an opportunity to ask questions to faculty, staff and students who have reviewed, mentored or applied and received fellowships. We will provide fellowship writing tips and guidance as well as offer writing and draft review sessions. Register to join our Zoom Sessions.

LeetCode Mock Interviews – CommLab Drop-In Workshops
2024-07-30 16:00 - 17:00, 2024-07-23 16:00 - 17:00, 2024-07-16 16:00 - 17:00
Join the CommLab any Tuesday from 4-5 PM for our weekly LeetCode Mock Interview Workshop via Zoom. This workshop is tailored towards programming jobs and prior coding knowledge is expected. Boost your LeetCode problem-solving confidence for interviews by building your speaking skills while solving programming problems.

Kyle Lockwood PhD Dissertation Defense
2024-07-30 13:30 - 14:30
Name:
Kyle Lockwood
Title:
Leveraging Submovements for Prediction and Trajectory Planning in Human-Robot Handover
Date:
7/30/2024
Time:
1:30:00 PM
Location:
ISEC 532 –
Committee Members:
Prof. Deniz Erdogmus (Advisor)
Prof. Eugene Tunik (Co-Advisor)
Prof. Mathew Yarossi
Prof. Tales Imbiriba
Abstract:
Collaborative physical interactions between humans and robots pose difficult modeling challenges. To create natural interactions, engineers must consider human inference of intent, anticipation of action, and coordination of movement. Humans can handle these challenges effortlessly when interacting with one another, but they are very difficult to overcome in robot implementations. Although human-human handover is a seemingly simple task, it requires a complex perception-action coupling to determine when and where the handover will happen, as well as choosing an appropriate trajectory to receive the object. Critically, modeling human-robot handover requires incorporating knowledge about human inference and trajectory planning to obtain seamless interactions. Despite recent advancements in sensing and control, human-robot handovers are far from approaching the fluidity and flexibility of human-human collaboration. Existing predictive models applied to human-robot handover often utilize classification methods and other approaches that suffer in accuracy when encountering noisy human trajectories that are not captured during their training. To address these challenges, this work presents two models that act as robotic surrogates for human inference and trajectory planning in a handover task. This approach delivers promising results while remaining grounded in a physiologically meaningful feature of human motion: Gaussian-shaped submovements in velocity profiles. This thesis analyzes human-human handover kinematics to establish a baseline for model evaluation and investigate the influence of handover role, it presents models for human inference and trajectory planning, and it applies the inference model in human-robot handover experiments.

Yunus Bicer PhD Dissertation Defense
2024-07-29 15:00 - 17:00
Name:
Yunus Bicer
Title:
Novel Methods for Electromyographic Hand Gesture Recognition: Expressive Gestures Sets with Minimal Calibration
Date:
7/29/2024
Time:
3:00:00 PM
Location:
ISEC 632 –
Committee Members:
Prof. Deniz Erdogmus (Advisor)
Prof. Mathew Yarossi (Co-Advisor)
Prof. Eugene Tunik
Prof. Tales Imbiriba
Abstract:
Gesture recognition, the process of interpreting hand gestures through computational algorithms and devices, is essenatial for enhancing human-computer interaction(HCI). This thesis focuses on surface electromyography (sEMG)-based gesture recognition, where the signals generated by muscles are analyzed to identify hand gestures. sEMG systems provides more natural and intuitive interactions compared to traditional input methods and hold significant potential in assistive technology, prosthetics, and immersive environments such as virtual and augmented reality. Despite these advantages, sEMG-based methods face challenges including user-specific variability in signals, limited gesture expressivity, and the need for extensive calibration time. This research aims to address these issues by proposing novel methods for minimizing calibration time and expanding expressivity of gesture recognition capabilities. Key innovations include a real-time probability feedback mechanism to facilitate user adaptation and techniques to recognize a wider range of gestures with minimal training data. This work seeks to enhance the usability and versatility of sEMG-based systems, making them more accessible and effective for various applications.

Shijie Yan PhD Proposal Review
2024-07-29 12:00 - 13:30
Name:
Shijie Yan
Title:
Efficient Monte Carlo light transport algorithms in complex scattering media
Date:
7/29/2024
Time:
12:00:00 PM
Committee Members:
Prof. Qianqian Fang (Advisor)
Prof. Steven Jacques
Prof. David Kaeli
Prof. Edwin Marengo
Abstract:
Modeling light-tissue interactions is crucial for many optical imaging modalities, for which the Monte Carlo (MC) method has been widely recognized as the gold-standard. Despite dramatic speed improvements gained via the use of graphics processing units (GPUs), MC simulations remain computationally intensive. Efficient and accurate MC algorithms are needed to further consider physiologically realistic tissue models, especially for emerging optical imaging techniques. Voxel-based MC (VMC) and mesh-based MC (MMC) are two major MC methods for modeling complex tissues with their respective strengths and weaknesses. While VMC offers higher computational efficiency due to the simple data structure, its accuracy suffers from the terraced boundary shape especially in low-scattering medium; on the other side, MMC offers improved boundary fidelity but can be slow and memory-intensive, particularly at high mesh density. Furthermore, emerging wide-field diffuse optical imaging systems using structured light require more efficient modeling to handle numerous illumination patterns. Additionally, niche applications such as polarized light imaging could also benefit from many of the recent advances from modern MC simulations such as GPU acceleration and handling of complex heterogeneous media.
This proposal is aimed to push the frontiers of modern MC simulation algorithms to fundamentally enhance their utilities in diverse applications. To reduce the staircase effect in VMC, we have developed a hybrid MC algorithm, named split-voxel MC (SVMC), where sub-voxel oblique surfaces are extracted using a marching-cubes algorithm and are incorporated into a memory-efficient voxelated data structure. SVMC allows VMC to handle curved surfaces while remaining computationally efficient. A GPU-accelerated marching-cubes algorithm was also developed to further accelerate SVMC domain preprocessing. On the other hand, to further improve MMC computational efficiency, a dual-grid MMC (DMMC) algorithm was developed to perform fast ray-tracing inside a coarse tetrahedral mesh while saving fluence data over a dense voxelated grid, simultaneously achieving improved speed and output accuracy. To accommodate increasing needs of modeling wide-field pattern based sources, we have developed a “photon sharing’’ MC algorithm that performs simulations of all illumination and detection patterns in parallel, improving computational speed by an order of magnitude. Additionally, we have developed a GPU-accelerated massively-parallel algorithm capable of modeling Mie scattering of sphere particles in three-dimensional media for polarized light imaging, achieving nearly 1000$times$ speed acceleration compared to sequential implementation.
Lastly, we have also investigated a hardware-accelerated MMC algorithm using the NVIDIA OptiX ray-tracing framework, leveraging modern GPU ray-tracing (RT) cores extensively optimized for graphics rendering. Preliminary results demonstrate comparable accuracy and significantly improved simulation speed compared to conventional tetrahedral MMC.

CommLab Drop-In Writing Hours
2024-07-29 10:00 - 12:00, 2024-07-22 10:00 - 12:00, 2024-07-15 10:00 - 12:00
Graduate students, are you looking for a place for focused research writing time? Join the CommLab drop-in writing hours any Mondays from 10 am-12 pm ET. Drop in any Monday and stay for a short time or the whole two hours. CommLab Fellows will be available to provide feedback on your writing. We will be meeting in 13 International Village.

Ruyi Ding PhD Proposal Review
2024-07-29 09:00 - 10:30
Name:
Ruyi Ding
Title:
Towards Robust and Secure Deep Learning: From Training through Deployment to Inference
Date:
7/29/2024
Time:
9:00:00 AM
Committee Members:
Prof. Yunsi Fei (Advisor)
Prof. Aidong Ding
Prof. Lili Su
Abstract:
In recent years, deep learning has experienced rapid advancement, leading to the development of numerous commercial deep neural network (DNN) models across diverse fields such as autonomous driving, healthcare, and recommendation systems. However, this wide adoption has intensified concerns about AI security throughout a neural network’s lifecycle — from training to deployment, and inference. Various vulnerabilities have emerged, threatening confidentiality, privacy, and intellectual property (IP) rights: poisoned training datasets facilitate privacy leakage and backdoor injection; after deployment, models may be misused through unauthorized transfer learning, a new form of IP infringement, and weights and parameters are subject to side-channel assisted model extraction attacks; during inference, adversarial attacks may compromise DNN functionality, causing misclassifications.
This dissertation addresses new security challenges across the neural network lifecycle through several novel contributions. We identify a new poisoning vulnerability in graph neural networks, where injecting poisoned nodes exacerbates link privacy leakage, allowing attackers to steal adjacent information from private training data, highlighting the necessity of robust AI training. To prevent model misuse after deployment, we introduce EncoderLock and Non-transferable Pruning, employing innovative training schemes and pruning methods to restrict the malicious use of pre-trained models through transfer learning, effectively implementing applicability authorization. Towards secure deep learning implementations, we adopt a software-hardware co-design approach to address DNN vulnerabilities. Specifically, we leverage the electromagnetic emanations from DNN accelerators in a new approach called EMShepherd, which detects adversarial examples (AE) on edge devices in a ‘black-box’ manner. To protect deployed DNNs against side-channel-based weight-stealing attacks, we develop PixelMask, which leverages the characteristics of DNN for side-channel defense by masking out unimportant inputs and dropping related operations to obfuscate side-channel signals. Lastly, we explore the use of Trusted Execution Environments (TEE) to safeguard model weights and data privacy against model stealing and membership inference attacks.
This proposal identifies key challenges of robust and secure deep learning, tackles vulnerabilities at various stages of the AI lifecycle, and provides comprehensive protection mechanisms, from securing the training process to safeguarding deployed models, paving the way for more resilient and reliable AI technologies in real-world applications.

Graduate School of Engineering Campus Tour - In Person
2024-07-27 13:00 - 14:00, 2024-07-20 13:00 - 14:00
Want to learn more about Northeastern’s Boston campus? Then we welcome you to sign up for a Graduate School of Engineering campus tour! Led by one of our expert Graduate Student Ambassadors, we’ll show you key locations on campus, in addition to resources specific to Engineering, and answer your questions about Boston. Please complete the registration form linked below to select the date and time that works best for you. Tours are open to all students interested in learning more about the Graduate School of Engineering. We can’t wait to meet you!

Poster Design and Presentation: CommLab Drop-In Workshops
2024-07-25 17:00 - 18:00, 2024-07-18 17:00 - 18:00, 2024-07-11 17:00 - 18:00
The CommLab will host drop-in workshops for poster design and presentation to focus on crafting the best visual communication of your research and telling your research story! We will discuss techniques and implement communication strategies to successfully showcase your work. No matter where you are in the process, whether it is just in the idea phase or you are trying to polish your final poster, we are happy to help you. Join us any Thursday from 5-6pm, on Zoom.

Rui Lou PhD Dissertation Defense
2024-07-25 14:00 - 15:30
Announcing:
PhD Dissertation Defense
Name:
Rui Luo
Title:
Shared Assistance Methods for Human-in-the-loop Robot Systems
Date:
7/25/2024
Time:
2:00:00 PM
Location:
EXP 701A.
Committee Members:
Prof. Taskin Padir (Advisor)
Prof. John Peter Whitney
Prof. Yanzhi Wang
Dr. Mark Zolotas
Abstract:
Fully autonomous robot systems, though highly desired, face substantial theoretical and practical challenges when being deployed into a dynamic environment where human co-exists. To tackle this challenge, this thesis investigates the concept of human-in-the-loop (HITL) systems, which incorporate human input to enhance robot functionality. HITL systems offer a pragmatic alternative, combining human versatility with robotic precision.
This research aims to address critical questions in one specific HITL system which prioritizes the dominant role of human within the system, positioning the robot primarily in an assistive capacity that adheres to human commands to facilitate the achievement of a shared goal. It explores two primary paradigms of shared assistance methods—Shared Control (SC) and Shared Autonomy (SA)—and discuss the system designs as well as specific algorithms to implement the three critical components in a HITL systems: human intention estimation, modulation of human inputs and robot autonomy, and the human-robot communication channel.
Due to the variety of use cases and their specific challenges, four distinct HITL systems are developed and analyzed to exemplify how shared assistance methods could be incorporated to assist human operators: an assistive wheelchair for indoor navigation, a human-centered robot system design for industrial tasks, a mobile bi-manual robot for tele-manipulation, and a VR-based customizable shared control system for fine teleopeartion. Although each system represents a comprehensive robotic solution, the research contributions for each work vary.
In the assistive wheelchair navigation system, the focus was on human intent estimation via low-throughput interface utilizing a recursive Bayesian filter, with significant efforts dedicated to developing a real-time user interface serving as the communication channel. In the human-robot collaboration system for industrial setting, the emphasis was on human state estimation through camera-based posture tracking and exploring the interplay between robot behavior and human ergonomics. For the two teleoperation systems, the primary focus was on the real-time modulation of human inputs and robot autonomy to aid in achieving dexterous manipulation tasks. A novel VR-based user interface was developed to enable users to customize the level of robotic autonomous assistance. Each system was validated through a pilot study involving 10-20 human subjects, accompanied by extensive data analysis to provide insights into designing HITL systems for various applications.
In conclusion, this thesis contributes to a deeper understanding of HITL systems, highlighting their potential to enhance human productivity, ergonomics, and quality of life in various applications through concrete examples. The integration of human intent estimation and real-time shared control methods into robotic systems demonstrates the feasibility and benefits of HITL approaches. Our extensive experimental analysis underscores the critical role of human feedback in designing practical HITL systems that can be deployed in real-world scenarios.

Zhenglun Kong PhD Dissertation Defense
2024-07-23 21:00 - 22:00
Name:
Zhenglun Kong
Title:
Towards Efficient Deep Learning for Vision and Language Applications
Date:
7/23/2024
Time:
9:00:00 PM
Committee Members:
Prof. Yanzhi Wang (Advisor)
Prof. David Kaeli
Prof. Dakuo Wang
Prof. Weiyan Shi
Abstract:
Machine learning and AI have been advancing rapidly in recent years, leading to numerous applications across diverse fields such as autonomous vehicles, entertainment, science, healthcare, and assistive technologies—significantly enhancing daily life. However, this advancement has been accompanied by a significant increase in the size of deep neural network (DNN) models, which poses considerable economic challenges. The substantial costs associated with the training, inference, and deployment of large vision and language models require extensive computational resources and time, proving especially taxing for smaller entities and individuals. This also complicates deployment on resource-constrained devices and in areas with limited infrastructure.
A major challenge is deploying AI models on devices with limited capacity, such as wearables, sensors, and mobile phones. These edge devices, often operating offline and requiring real-time processing, are critical for many applications but struggle to support large models. My dissertation research addresses these pressing issues with the aim of enabling the practical implementation of AI. We ensure the effectiveness of AI models while adapting them for use in constrained environments by tackling fundamental AI challenges from four angles:
1. Managing Massive Computation: We introduce a novel token pruning framework that reduces the latency of Vision Transformers (ViT) by up to 41% compared to existing works on mobile devices. Additionally, we propose a quantization framework for large language models (LLMs), achieving an on-device speedup of up to 2.55x compared to FP16 counterparts across multiple edge devices.
2. Mitigating Training Costs: We develop fast, accurate, and memory-efficient training methods by utilizing a hierarchical data redundancy reduction scheme, which achieves up to a 40% speedup in ViT pre-training with minimal accuracy loss.
3. Merging Multiple Models: We propose an efficient way to merge multiple LLMS, yielding a more advanced and robust LLM while maintaining the model size, as well as reducing knowledge interference.
4. Co-designing Speed-aware Deep Neural Networks: We consider memory access cost, the degree of parallelism, and practical latency in the design of 2D and 3D object detection models for practical deployment.  By addressing these areas, my research aims to enable the effective and efficient use of AI models in constrained environments, ensuring their practical implementation across various applications.

Andrea Lacava PhD Proposal Review on 7/23
2024-07-23 11:30 - 12:30
Name:
Andrea Lacava
Title:
Enabling Intelligent nextG Cellular Networks through the Open RAN Architecture
Date:
7/23/2024
Time:
11:30:00 AM
Location:
EXP 501
Committee Members:
Prof. Tommaso Melodia (Advisor)
Prof. Francesca Cuomo (Advisor)
Prof. Stefano Basagni
Prof. Ioannis Chatzigiannakis
Abstract:
The 5th generation (5G) and beyond of cellular networks will support heterogeneous use cases at an unprecedented scale, thus demanding automated control and optimization of network functionalities, customized to the needs of individual users. However, achieving such fine-grained control over the Radio Access Network (RAN) is unfeasible with the current cellular architecture.
To bridge this gap, the Open RAN paradigm and its specification introduce an “open” architecture with abstractions that facilitate closed-loop control and enable data-driven, intelligent optimization of the RAN at the user-level. This thesis focuses on the design and development of system-level solutions to enable intelligent control in the next generation of cellular networks through the Open RAN architecture. The main research areas explored in this thesis include (i) the design and evaluation of platforms for the creation, datasets generation and testing of the Open RAN architecture solutions; (ii) the development of Artificial Intelligence (AI)/Machine Learning (ML) models for various deployments and networking scenarios; and (iii) innovative methodologies for agile spectrum, infrastructure, and AI management within Open RAN. Among the significant contributions of this thesis are ns-O-RAN, the first open-source simulation platform that integrates a functional 5G protocol stack in Network Simulator 3 (ns-3) with an O-RAN-compliant E2 interface, and the pioneering architectural design and implementation of the dApps, the real-time controllers for the O-RAN architecture. Furthermore, the solutions proposed in this thesis are leveraged to investigate various network optimization use cases deemed critical in cellular networks. The results demonstrate that our approach outperforms traditional Radio Resource Management (RRM) heuristics, enhancing overall RAN conditions at scale in both simulations and state-of-the-art experimental testbeds.

Miead Tehrani Moayyed PhD Dissertation Defense
2024-07-22 13:00 - 14:30
Name:
Miead Tehrani Moayyed
Title:
RF Channel Models for Static and Mobile Scenarios: From Simulations to Models for Large-scale Emulations and Digital Twins
Date:
7/22/2024
Time:
1:00:00 PM
Location:
Room: EXP-601A
Committee Members:
Prof. Stefano Basagni (Advisor)
Prof. Tommaso Melodia
Prof. Milica Stojanovic
Abstract:
The extremely high data rates provided by communications at higher frequency bands, such as mmWave, can address the unprecedented demands of next-generation wireless networks. However, several impairments limit wireless coverage at higher frequencies, necessitating accurate models of wireless scenarios and large-scale testing to test and realize the potential of these new technologies. Large-scale accurate simulations and wireless network emulators now offer a time- and cost-effective solution for performing these tests in a lab before field deployment. This dissertation focuses on modeling, calibration, and validation of realistic RF scenarios for wireless network emulation at scale. The contributions of this work include: (i) Investigating the characteristics of the wireless channel at higher frequencies (mmWave) and evaluating the performance of mmWave communications on top of the NR standard for 5G cellular networks; (ii) developing a streamlined framework to create realistic RF scenarios with mobility support for Finite Input Response (FIR)-based emulators like Colosseum, starting from rich inputs such as precise ray tracing methods or real-field measurements, and (iii) creating an accurate AI-assisted propagation model that integrates joint measurements and simulations, achieving the desired accuracy and reasonable computational requirements for real-time Digital Twin (DT) wireless networks. Particularly:
(i) We derive channel propagation models via ray tracing simulations for mmWave transmissions with applications to V2X communications. We analyze aspects related to blockage modeling, the effects of antenna beamwidth, beam alignment, and multipath fading in urban scenarios, emphasizing the importance of capturing diffuse scattered rays for improved large-scale and small-scale radio channel propagation models. Furthermore, we compare the performance of mmWave 5G NR with the 4G Long-Term Evolution (LTE) standard in a realistic environment and demonstrate the impact of MIMO technology on improving the performance of 5G NR cellular networks. As transmitted radio signals are received as clusters of multipath rays, identifying these clusters provides better spatial and temporal characteristics of the channel. We address the clustering process and its validation across a wide range of frequencies in the mmWave spectrum below 100 GHz. We analyze how the clustering solution changes with narrower-beam antennas and provide a comparison of the cluster characteristics for different types of antennas.
(ii) Our framework for modeling wireless scenarios for large-scale emulators optimally scales down the large set of channel input to the fewer parameters allowed by the emulator using efficient clustering techniques and Channel-Impulse Response (CIR) re-sampling. We demonstrate the effectiveness of the proposed framework by modeling realistic scenarios for Colosseum, starting with rich input from commercial-grade ray tracing software, Wireless InSite (WI) by Remcom. To support mobility, we implement a mobile channel simulator on top of the WI ray-tracer, consisting of two steps: (a) spatially sampling the mobile channels using the ray-tracer, and (b) parsing the ray tracing outputs to extract the channels for each time instant of emulation. We also develop a Software-Defined Radio (SDR)-based channel sounder to precisely characterize emulated RF channels. The sounder framework is fully containerized, scalable, and automated to capture the gains and delays of the channel CIR taps.
(iii) We extend these efforts to develop the first Digital Twins for Mobile Networks (DTMN) on Colosseum, using the RF testbed Arena as a use case. This use case demonstrates the scope and capabilities of Colosseum as a DT, providing the research community with a set of tools to replicate real-world environments. We compare key network performance metrics, namely throughput and SINR, of the Arena/Colosseum DTMN to validate the fidelity of our twinning process. Furthermore, we present an AI-assisted propagation model to generate realistic, real-time, and scalable scenarios for DTMNs. This model seamlessly integrates measurements with ray tracing, providing a high-resolution, realistic channel model. We study the computational complexity and configuration trade-offs associated with ray tracing for high-fidelity prediction, generating a large dataset to train this enhanced AI model. Our proof of concept highlights the accuracy and generalization capabilities of our AI model across previously unseen transmitter (TX) locations and unfamiliar environments, outperforming state-of-the-art approaches and achieving significant improvements in accuracy. We analyze the computational complexity of our AI model, comparing it to high-fidelity ray tracing. Profiling reveals a three-order-of-magnitude acceleration, enabling real-time propagation prediction with reasonable accuracy. We explore key ray tracing parameters contributing to the discrepancy between measurements and simulations and demonstrate the integration of measurements into channel prediction, thereby calibrating the model.

Mehrshad Zandigohar PhD Proposal Review
2024-07-18 11:30 - 12:30
Announcing:
PhD Proposal Review
Name:
Mehrshad Zandigohar
Title:
Deployable and Multimodal Human Grasp Intent Inference in Prosthetic Hand Control
Date:
7/18/2024
Time:
11:30:00 AM
Location: https://teams.microsoft.com/l/meetup-join/19%3ameeting_N2QyNzc1MWMtOWJmMi00NGNmLThlNzctN2JlNjU2Y2I1MmI1%40thread.v2/0?context=%7b%22Tid%22%3a%22a8eec281-aaa3-4dae-ac9b-9a398b9215e7%22%2c%22Oid%22%3a%22de13c261-ac42-49d7-8950-6dec3adaca4e%22%7d
ISEC 532 –
Committee Members:
Prof. Gunar Schirner (Advisor)
Prof. Deniz Erdogmus
Prof. Mallesham Dasari
Prof. Mariusz P. Furmanek
Abstract:
For transradial amputees, robotic prosthetic hands promise to regain the capability to perform daily living activities. Among robotic control methods for prosthetic hand actuators, coarse-grained grasp types are a common means of effortless yet effective control. However, to advance next-generation prosthetic hand control design, it is crucial to address current shortcomings in robustness to out of lab artifacts, generalizability to new environments and deployment of such compute-intensive grasp estimators.
First and foremost, current control methods based on physiological modality such as electromyography (EMG) are prone to yielding poor inference outcomes due to motion artifacts, muscle fatigue, and many more. Similarly, methods based on visual modality are also susceptible to its own artifacts, most often due to object occlusion, lighting changes, etc. To address such drawbacks of single modality approaches, we present a multimodal evidence fusion framework for grasp intent inference using eye-view video, eye-gaze, and EMG from the forearm processed by neural network models. Given the lack of a synchronized multimodal dataset for evaluating multimodal grasp estimation, we propose our own customized HANDSv2 dataset with the most complete EMG profile and visual data synchronized in time. Our experimental results indicate that fusing both modalities, on average, improves the instantaneous upcoming grasp type classification accuracy while in the reaching phase by 13.66% and 14.8%, relative to EMG (81.64% non-fused) and visual evidence (80.5% non-fused) individually, resulting in an overall fusion accuracy of 95.3%.
Although visual grasp classification has shown promising results, the generalizability to unseen object classes remains a significant challenge within the research community. This limitation arises from the fixed number of grasp types available in existing models, contrasted with the virtually infinite variety of objects encountered in the real world. The poor performance of grasp detection models on unseen objects negatively affects users’ independence and quality of life. To address this, we propose Grasp Vision Language Model (Grasp-VLM). Grasp-VLM takes advantage of the zero-shotness capability of large vision language models and teach them to perform human-like reasoning to infer the suitable grasp type estimate based on the object’s physical characteristics suitable for previously unseen objects, resulting in better generalizability in real-life scenarios. Our initial results show a significant 49% accuracy of Grasp-VLM over unseen object types compared to 15.3% accuracy of the current State-of-the-Art.
Lastly, given the computational intensity of such models, which often contain billions of parameters, deploying them to edge devices poses a serious challenge. To mitigate this, we investigate Hybrid Grasp Network (HGN), a deployment infrastructure that combines an edge-specialized model for low-latency operations with a cloud-based universal model ensuring high generalization, effectively balancing performance and resource constraints.
The holistic approach presented in this dissertation tackles four essential areas of robotic prosthetic hand control design. Handsv2 provides a customized dataset filling the gap for a multimodal synchronized dataset. Our multimodal fusion approach effectively outperforms single modality approaches providing accurate and robust grasp type estimations during the entire grasping timeline. In addition, Grasp-VLM addresses the lack of generalizability to new object types providing a more realistic grasp estimation. Lastly, our HGN design aims at providing a real-time solution investigating both speed and accuracy objectives.

Jagatpreet Nir PhD Dissertation Defense
2024-07-18 11:00 - 12:00
Announcing:
PhD Dissertation Defense
Name:
Jagatpreet Nir
Title:
Low Contrast Visual Sensing and Inertial-Aided Navigation in GPS-Denied Environments
Date:
7/18/2024
Time:
11:00:00 AM
Committee Members:
Prof. Hanumant Singh (Advisor)
Prof. Martin Ludvigsen
Prof. Michael Everett
Prof. Pau Closas
Abstract:
Field robots perform complex tasks, necessitating high autonomy and reliable navigation capabilities. Integrating complementary sensors at the hardware level is crucial to maintaining navigation estimates even during sensor failure. This work is motivated by the need for robust and accurate navigation systems for robotic field applications, particularly in diverse and challenging environments. The development of such systems involves balancing design requirements with constraints such as size, weight, power, computational capacity, and cost. Underwater navigation exemplifies navigation in Visually Degraded Environments (VDEs), where Autonomous Underwater Vehicles (AUVs) and Remote Operated Vehicles ( ROVs) navigate in challenging conditions. This thesis focuses on exploring methods to enhance the robustness of visual-inertial odometry systems in VDEs.
The current state-of-the-art Visual Inertial Odometery (VIO) techniques provide high-accuracy navigation estimates in texture-rich scenes. However, robots operating in harsh and unpredictable environments, such as underwater, often encounter VDEs due to low texture, uneven illumination, or backscatter. During prolonged visual degradation, the Inertial Measurement Units (IMUs) become the primary sensor as visual measurements are unreliable. In this reserach, we address the problem of designing an underwater VIO navigation system and algorithmic pipelines to ensure reliable navigation estimates during several seconds of visual degradation, emphasizing the importance of selecting better Micro Electro Mechanical Systems (MEMS) IMUs for dependable performance within a cost budget.
A robust VIO system designed for underwater settings is introduced. Our contributions include a general system design approach for underwater VIO, an algorithmic formulation for fusing deep learning-based Visual Odometry (VO) with IMUs data. The underwater datasets depict visual degradation in real-world settings with a time-synchronized 8-bit grayscale camera and IMU. Our hybrid VIO pipeline integrates IMU measurements with VO estimates from a deep-learning VO engine, combining deep learning with classical sensor fusion techniques to achieve accurate metric and gravity-aligned trajectory estimates even in visually degraded conditions. The proposed system outperforms traditional VIO methods, demonstrating robustness with consistent trajectory estimates and minimal drift during complete visual outages. The extensible design allows for the incorporation of new sensors, addressing various underwater navigation challenges.
To conclude, this thesis focuses on environments where exteroceptive sensing, like cameras, is compromised for extended periods, relying on proprioceptive sensors such as IMUs to navigate. The aim is to quantify navigation accuracy in harsh environments and improve system design at both hardware and software levels. Specifically, underwater visual-inertial navigation for small vehicles is used to demonstrate the principles and algorithms developed. The outlined methodology showcases sensor selection, sensor-fusion algorithms, and individual improvements to build enhanced visual-inertial systems for VDEs and the applicability of the proposed approach from controlled settings to field tests.
