Catalyze Innovation that Advances Health

“Human-in-the-Loop” Approach to Eye-Tracking Data Analysis


This novel algorithmic approach to annotating mobile eye-tracking data incorporates human insight using a semi-automatic decision-making process.


Allan Fong

Daniel Hoffman

Raj Ratwani, PhD

Zach Hettinger, MD

What is it? What does it do?

Eye tracking is proven technology that consumer marketers and researchers use regularly, but it has not been widely used in health care because identification of specific objects in highly dynamic environments is a significant data analysis challenge. These challenges result in gaps in understanding how clinicians see information, process this information, and interact in their complex environments.

MedStar Health’s novel system for analyzing eye-tracking data leverages human insight with machine learning and results in a streamlined, more robust analysis process. Combining a computer’s ability to quickly process large quantities of data and a human's ability to decipher and make decisions for complex images and situations allows researchers to apply the science of eye tracking to study how clinicians can more safely and efficiently interact with their environment to deliver the best care possible. This includes the design of computer interfaces, like the electronic health record (EHR), as well as complex environments, like the operating room.

Why is it better?

  • The blended approach of this “human-in-the-loop system” improves accuracy and reduces the time spent on object identification and annotation in mobile eye tracking studies

  • No training data or predefined models of the environment are necessary to implement this system

  • The system is platform agnostic and can be used with various software systems and eye tracker devices



What is its current status?

The prototype was developed and validated outside a controlled laboratory setting. One test involved a participant looking at three different objects from various angles and the other involved a physician performing an intubation on a mannequin. In these scenarios, the novel system was tested using two different comparison techniques: spatial pyramid histogram and optical flow. Overall accuracy was 82.3% compared to manually annotated data for ground truth.

U.S. patent 10,037,479Human-assisted learning in eye tracking applications”.  Companies interested in collaborating or licensing this technology may contact MedStar Inventor Services at [email protected].

Click to enlarge image

Click image to enlarge