7/10/2015 Doug Peterson
Rakesh Nagi leads team that fuses information from human reports and sensors
Written by Doug Peterson
For the past five years, University of Illinois Professor of Industrial and Enterprise Systems Engineering Rakesh Nagi has been the principal investigator for a $6¼ million, multi-university project developing innovative ways to fuse information from “soft sources,” such as human contacts, and “hard sources,” such as cameras and physics-based sensors.
By running this and other intelligence analysis scenarios through their system, the research team demonstrated that it is possible to fuse various streams of hard and soft information, track the source of a threat, and determine if a risk exists. The team has made crucial advances in basic research on information fusion, and they produced software that is already being put to use in government laboratories, says Nagi, head of the U of I Department of Industrial and Enterprise Systems Engineering. Funding for the five-year project has come from a Multidisciplinary University Research Initiative (MURI) grant from the Army Research Office.
“With this research, we’re trying to save lives,” he says. “We’re trying to keep our country and all countries safe, because every nation is subject to terrorism.”
He points out that since the 9/11 attack in New York, in which nearly 3,000 people died, the number of deaths annually due to terrorism has increased almost five-fold.
Nagi came to Illinois in 2013 from the University of Buffalo, which is prime of the MURI research team working on the information fusion problem, along with Penn State, Iona College, and Tennessee State. He has extensive experience in information fusion, military/applied operations research, facilities design, and manufacturing systems. For example, he worked on information coming from sensors that detected changes in vibrations within helicopters—a way to catch early signs that cracks had formed on the helicopter’s gear teeth.
Then in 2001, the attack on the Twin Towers on 9/11 changed the direction of his research.
The daughter of a good friend was scheduled to be working in the Twin Towers on that day, but she just happened to stay home with a sick daughter. This brought terrorism close to home for Nagi, who then began to focus his work on fusing intelligence information.
Intelligence analysts face a classic Big Data problem: How do you pluck critical information out of the ocean of information pouring in and make the connections that ultimately trigger an alert that a terrorist attack might be developing?
Hard data comes from sensors, such as unattended ground sensors, or UGS, which pick up various types of signals and images to detect the movement of people and vehicles. Hard data can be quantified, but the challenge, Nagi says, is how to fuse it with soft information coming from human observers. A major problem is converting the natural language in observer reports into something that can be associated with hard data.
“We call it soft information because its accuracy depends on how well a person can make an estimate,” he says. “For instance, how well can a person judge height or age?”
According to Nagi, accuracy depends on characteristics of the observer, characteristics of the target, and environmental conditions, such as low lighting.
The MURI team ran extensive studies on student observers trying to estimate such characteristics as height and weight; building on these findings, the system takes into consideration the uncertainty that comes from human observations.
“Machines are very good at certain things, and people are good at certain things, such as judging emotion,” he says. “The question is how do we take advantage of both strengths?”
Their system processes information in the form of graphs, which include “nodes” and “edges.” Nodes represent entities such as people, vehicles, locations, or organizations, while edges represent the relationships among them. Using these graphs, their information fusion system can look at the tangled relationships among various entities and connect the dots.
The system makes associations between multiple graphs and then matches the information with templates that model certain threat scenarios. If a situation comes close to matching one of the templates, it will score high and raise warning flags for an intelligence analyst.
For example, one obvious situation might be the convergence of several factors—a person connected to a suspect organization attends flight school and purchases a one-way airplane ticket. The combination of these details could alert intelligence analysts to a potential 9/11-style attack.
The MURI team also worked on ways to fuse hard data from very different sensors. It found ways to connect the acoustic data on vehicles with the visual images of vehicles to confirm, for instance, that a van moving toward the courthouse is the same white van owned by the suspected terrorist. They also worked on ways to fuse three-dimensional images with two-dimensional pictures to determine if a suspect showing up in both images is actually the same person.
“We’re finding ways to assist the intelligence analysts in making sense of this large amount of information,” Nagi says.
He says he is especially pleased that the basic research advances they made didn’t wind up lost in the “Valley of Death,” where research is forgotten and never becomes applied. They used their basic research to create prototype software, which is being transitioned to Army research laboratories.
As he puts it, “We managed to get this research over the hump and into use.”