northeastern university seal

Fu Awarded ARO Grant to Advance Technology for Visual Intelligent Systems

raymond fu

Advances in artificial intelligence (AI) hold great promise for improving public safety and national defense, and Electrical and Computer Engineering Professor Yun Raymond Fu and his team are working hard to make that vision a reality.

Fu, who holds a joint appointment in Northeastern’s College of Engineering and the Khoury College of Computer Sciences, has earned a three-year, $390,000 grant from the Army Research Office for “Images Assisted Video Recognition by Heterogeneous Knowledge Transfer.” The project focuses on developing a knowledge transfer network using image-based object recognition to enhance capabilities for video-based object recognition. The results could have far-reaching implications for everything from high-performance drones to surveillance systems and self-driving cars.

Existing video-based object recognition techniques are often inadequate for addressing real-world challenges. “A practical system should be able to rapidly recognize objects from streaming video captured in unconstrained scenarios,” says Fu whose research looks at how data is collected in real-world situations where there are uncertainties and variations unlike a controlled or “constrained” lab environment.

Since large-scale images can be easily obtained from the web, Fu proposes transferring knowledge learned from these images to assist in video recognition, specifically by creating a framework to transfer knowledge across heterogeneous domains, a potentially groundbreaking application that would improve the accuracy of tools to understand image and video content and speed up analysis of a large amount of data collected every day.

Solving the challenge of heterogeneous knowledge transfer

The knowledge transfer problem that Fu and his team will address is similar, he says, to challenges encountered when an individual who learns how to use a dictionary to look up a word in French, must then perform a similar task for a different language, such as Japanese.  “You have gained the skill to use a dictionary even though you don’t know Japanese. Transfer learning means you gain knowledge and apply it to similar situations,” explains Fu. “The difference between the dictionaries is the problem we’re basically trying to solve. In effect, you’re learning from another book, applying what you’ve learned to a different situation.”

“Understanding video data is at the core of many AI applications, including Army Next Gen-Autonomous vehicles, where threats and hazards need to be detected in real-time,” says Dr. Hamid Krim, program manager, Information Processing and Fusion at the Army Research Office, an element of the U.S. Army Combat Capabilities Development Command’s Army Research Laboratory. “To get a machine to learn to detect threats in real-time is very complex, not just because of the cost of training the machine, but also for the machine to be able to operate in unpredictable environments. Dr. Fu’s research is working to simplify the training by teaching AI using plain images collected in various places. While the goal is ambitious, the outlook is promising.”

Fu believes the project opens up new research possibilities that will help society at large. Among the potential applications are: enhancing robot intelligence; increasing the reliability and robustness of driverless cars; improving social media and content search and retrieval; and upgrading surveillance and security systems for airports.

“The Army mission inspired me to invent tools from the knowledge/AI domain to facilitate their needs,” says Fu. “The Army’s support allows us to extend our knowledge set, explore new technologies in AI—and bring Northeastern to the forefront in this field globally.”

Related Faculty: Yun Raymond Fu