Search Results Viewing Behavior vis-à-vis Relevance Criteria

Abstract: We conducted a lab-based experiment to investigate relationship between multiple criteria used in information relevance judgments and eye fixation behavior on search engine results. We collected eye-tracking data and conducted gaze-cued retrospective think-aloud (RTA). Data from RTA was coded with criteria used by participants in judging search results as relevant. The criteria were analyzed in relation to search engine result page (SERP) sequence and result rank on SERPs. The results of our study aligned with previous research, showing the effect of result rank on SERPs. Our results newly showed that specific source and topicality were the two most often used criteria for relevance judgments. Specific source was the most often used criteria initially but was then surpassed by topicality on subsequent SERPs and on lower result ranks. On first SERPs, fixation duration was significantly longer on results judged on topicality than on specific source. Pupils dilated significantly on the top ranked result on most SERP pages.