Rajeev K. Pandey & Margaret M. Burnett
Department of Computer Science, Oregon State University, Corvallis, Oregon 97331-3202, USA
Empirical studies comparing the effectiveness of visual programming languages versus textual programming languages have rarely been attempted. But now that research in visual programming languages is maturing, it is becoming critical that visual language researchers begin to substantiate claims that visual languages make programming easier. As a step in this direction, we performed an empirical study to determine whether solutions to vector and matrix manipulation tasks were more easily constructed in a visual or a textual programming language.
Selected for this study was the language Forms/3 , a form-based, declarative visual programming language; Pascal, a "normal" textual language typical of those most widely used; and a simplified version of APL, dubbed OSU-APL, representing a textual language especially well-suited for the problem domain. We were interested in finding out whether Forms/3 was "better" than the textual languages for vector and matrix manipulation tasks.
An in-class experiment was conducted in which the participants first were given a lecture on the application of Pascal, OSU- APL, and Forms/3 to various matrix manipulation tasks. All subjects were then given a timed test in which they were asked to compose solutions to two matrix manipulation problems in all three languages. Our hypothesis was that the visual programming language Forms/3 is better suited for constructing matrix manipulation programs than the textual languages Pascal and OSU- APL.
The results of our study, involving 60 participants and using within-subject experimental design, supported this hypothesis. A 40 minute lecture was administered to the subjects that described in an example-driven manner the constructs required to compose solutions to the same problems in all three languages. The subjects (junior and senior Computer Science undergraduates) were then asked to construct solutions to two problems. We used an ordinal scale, "categories of correctness", to grade the student's on-paper solutions, and employed Chi-square analysis to determine statistical relevance.
The first problem was to append two appropriately shaped matrices. Analysis indicated a statistically significant difference in the Forms/3 versus Pascal results, but not in the Forms/3 versus OSU-APL performance. The second problem, more difficult to probe subject understanding, was to produce the first N Fibonacci numbers. On this problem, Forms/3 outperformed OSU-APL. There was no statistically significant difference in the Forms/3 versus Pascal performance.
The above statistical analyses support the claim that constructing matrix manipulation programs in Forms/3 is at least as easy as in the textual languages Pascal and OSU-APL. Statistical analysis of the whole on a "categories of correctness" basis supports the claim that matrix manipulation programs can be more easily constructed in the visual language Forms/3 than in the textual languages Pascal and OSU-APL. To our knowledge, this is the first empirical study that examines the constructibility aspect of visual versus textual languages.
This work was presented at the 1993 IEEE Symposium on Visual Languages, held in Bergen, Norway. An 8-page description of the experiment, along with subject demographics, sample problems, difficulties encountered and results, appeared in the Proceedings of the 1993 IEEE Symposium on Visual Languages .