Abstract: Multimodal machine learning is a core research area spanning the language, visual and acoustic modalities. The central challenge in multimodal learning involves learning representations that can process and relate information from multiple modalities. In this paper, we propose two methods for unsupervised learning of joint multimodal representations using sequence to sequence (Seq2Seq) methods: a […]

Abstract: Digital video advertising is growing exponentially. It is expected that digital video ad spending of the US will see double-digit growth annually through 2020 (eMarketer, 2016). Moreover, advertisers are spending on average more than $10 million annually on Digital Video, representing an 85% increase from 2 years (iab, 2016). This huge increase is mediated by […]

Abstract: Personal context-specific experience can affect how a designer evaluates a design problem and proposes solutions. However, this effect was seldom discovered in a quantitative manner in problem-solving design tasks. This paper uses empirical evidence and quantitative methods to show the effects of novice designers’ contextual experience on design tasks, particularly as it relates to […]

Abstract: Metacomprehension is key to successful learning of complex topics when using multimedia materials. The goal of this study was to determine if eye-movement dyads could be: (1) identified by sequence mining techniques, and (2) aligned with self-reported metacognitive judgments during learning with multimedia materials that contain conceptual discrepancies designed to interfere with participants’ metacomprehension. Thirty-two […]

Abstract: Ensuring the safety of mixed traffic environments, in which human drivers interact with autonomous vehicles, is an impending challenge. A virtual traffic environment provides a risk-free opportunity to let human drivers interact with autonomous vehicles, indicating how variability in traffic environments and human responses compromises safety. Analyzing the section of road preceding an intersection […]

Abstract: In this paper, we present a solution for computer assisted emotional analysis of game session. The proposed approach combines eye movements and facial expressions to annotate the perceived game objects with the expressed dominate emotions. Moreover, our system EMOGRAPH (Emotional Graph) gives easy access to information about user experience and predicts player’s emotions. The […]

Abstract: We investigated how college students’ (n = 40) different levels of action unit 4 (AU4: brow lowerer), metacognitive monitoring process use and pre-test score were associated with metacognitive monitoring accuracy during learning with a hypermedia-based ITS. Results revealed that participants with high pre-test scores had the highest accuracy scores with low levels of AU4 and use […]

Abstract: The goal of this study was to examine 38 undergraduate and graduate students’ note taking and summarizing, and the relationship between emotions, the accuracy of those notes and summaries, and proportional learning gain, during learning with MetaTutor, an ITS that fosters self-regulated learning while learning complex science topics. Results revealed that students expressed both […]

Abstract: In this paper we present Leverage Score Ordering, a novel technique for determining the ordering of data in the training of deep neural networks. Our technique is based on the distributed computation of leverage scores using random projections. These computed leverage scores provide a flexible and efficient method to determine the optimal ordering of training […]

Abstract: Affect and metacognition play a central role in learning. We examine the relationships between students’ affective state dynamics, metacognitive judgments, and performance during learning with MetaTutorIVH, an advanced learning technology for human biology education. Student emotions were tracked using facial expression recognition embedded within MetaTutorIVH and transitions between emotions theorized to be important to […]