PPIG 2006 - 18th Annual Workshop
Program Visualization: Comparing Eye-Tracking Patterns with Comprehension Summaries and Performance
Roman Bednarik, Niko Myller, Erkki Sutinen, Markku Tukiainen
Abstract: We present preliminary results of an experiment in computer program comprehension that was conducted to find out whether visual strategies can characterize low- and high-comprehenders. In addition, we investigated whether the type and quality of externalized mental models can be associated with the visual strategies.
Participants of various levels of experience used a program visualization tool, Jeliot, to comprehend short Java programs, while their eye-movements were recorded. Comprehension summaries were evaluated for correctness as a measure of performance and also analyzed using Good’s information-types scheme. Times spent on viewing certain structures of the program visualization were analyzed and correlated with the information types found in comprehension summaries. Depending on comprehension performance and target program, some information types were found to be correlated with eye-data patterns.
Comprehension performance did not significantly correlate with information types. When the visual strategies of low-comprehenders were similar to those of highcomprehenders, the comprehension outcome of the low-comprehenders was poor. When the strategies diverged, the mental models of low-comprehenders tend to match those of high-comprehenders. Based on the results, we propose that eyetracking can help to partially predict the mental model that is built during comprehension. We discuss limitations and future directions of this research.