What Are You Gazing At? An Approach to Use Eye-Tracking for Robotic Process Automation

User Interface (UI) logs are crucial in capturing and analyzing user behavior, enabling a comprehensive understanding of business processes and eventual process automation with Robotic Process Automation (RPA). However, extracting meaningful insights from UI logs becomes challenging, especially when dealing with complex and information-dense graphical user interfaces. This paper presents a novel approach that leverages eye-tracking technology to address this challenge. The proposed solution incorporates gaze fixation (i.e., where the user pays attention to the user interfaces) into the UI log, which is then used to filter irrelevant information from it. Two gaze-based filtering methods are presented and evaluated using synthetic and real-life screenshots. Preliminary results demonstrate that the method effectively reduces the irrelevant UI elements by an average of 76% while keeping meaningful information on the screen.