Number 21, December `97
Edited by Paul Mulholland P.Mulholland@open.ac.uk
Contents
Report on ESP 7 Workshop by Susan Wiedenbeck and Jean Scholtz
PPIGers and Software Process Improvement by Judith Segal
HTML Authoring as Programming? by Chris Roast
Spotlight on PPIGers: Rafael O. Fontao, Kenji Kaijiri, Babak Khazaei, Slim Masmoudi, John Pane, Pablo Romero, Jim Spohrer, Jon Whittle and Andrew Walenstein
Call for Papers: The Forum for Negative Results (FNR)
PPIG ‘98 Workshop: Call for participation
About PPIG
The Psychology of Programming Interest Group (PPIG) was established in 1987 in order to bring together people from diverse communities to explore common interests in the psychological aspects of programming and/or in computational aspects of psychology. ‘Programming’, here, is interpreted in the broadest sense to include any aspect of software development. The group, which at present numbers approximately 300 world-wide, includes cognitive scientists, psychologists, computer scientists, software engineers, software developers, HCI people et al., in both Universities and industry.
PPIG aims to provide a forum for the rapid dissemination of results, ideas, and language or paradigm tool development, circumventing the long time-lag of conferences and journals. It does this by maintaining two electronic mailing lists - one for announcements and one for discussion - by publishing two newsletters a year, maintaining pages on the World Wide Web, and organising a workshop annually, together with other workshops as and when required.
The annual workshops, which always attract a high percentage of attendees from outside the United Kingdom, consist of keynote addresses by eminent practitioners in the relevant fields, discussion panels, software demonstrations and seminar-like presentations. Invited speakers have included Professors Jack Carroll, Bill Curtis, Laura Leventhal, Clayton Lewis, Gary Olson, Peter Polson, Elliot Soloway and Willemien Visser. Venues have included the Universities of Warwick (1989), Wolverhampton (1990), Huddersfield (1991), Loughborough (Jan 1992), INRIA (Paris) (Dec. 1992), the Open University (Jan. 1994), University of Edinburgh (1995), University of KaHo Sint Lieven, Ghent (1996) and Sheffield Hallam University (1997).
In 1996, for the first time, a workshop was held specifically to allow post-graduate students in the relevant disciplines to come together, give presentations and exchange ideas. It is hoped that this will be the first of a series.
There is no subscription. Financial help in the past has come from Xerox EuroPARC, the DTI and EPSRC. Further information is available from the organiser, Judith Segal.
The PPIG website can be found at http://www.ppig.org/
Report on ESP 7 Workshop
Susan Wiedenbeck and Jean Scholtz
The Empirical Studies of Programmers: Seventh Workshop (ESP 7) was held in Alexandria, VA. October 24-26, 1997. Dr. Susan Wiedenbeck and Dr. Jean Scholtz were the co-chairs of the event. The first ESP was held in 1986, so this conference marks the second decade of ESP workshops. There were just under 60 attendees.
On Friday, prior to the actual start of the workshop, Charles Kann, a graduate student from George Washington University, and Dr. Laura Leventhal from Bowling Green University organized a student conference. Fourteen graduate students doing research in empirical studies of programming met to discuss their work. The students and other attendees jointly participated in the opening reception and poster session for ESP. About 20 posters were presented.
The ESP program ran from Saturday morning until noon on Sunday. There were two invited speakers, two panel presentations and eleven papers. A short summary of these events follows.
Invited speakers:
Watts Humphrey, Software Engineering Institute, “What do we Know about Programming?”
Watts described his PSP - Personal Software Process. This process is designed to teach programmers how to program efficiently. He, as well as other instructors, have taught this course and collected data to determine how well the process works. The data collected during these classes shows that programmers using the PSP method find more code defects prior to compiling their code than programmers not following this process.
Dr. Iris Vessey, Indiana University School of Business, “Problems versus Solutions: The Role of the Application Domain in Software”
Dr. Vessey pointed out that although most computer programs arise due to a problem in a particular domain, we have historically treated program development as application independent. She called attention to strong and weak problem solving methods. Strong problem solving methods focus on specific types of problems, while weak methods are generic and can be applied to many types of problems, but not optimally. Dr. Vessey classified the software communities as product, process, problem and others (expert systems and languages) and discussed the trend toward application specific solutions in each. Dr. Vessey concluded saying that the field needs to distinguish between the problem and the solution and identify ways to characterize each in order to determine the “best” solution for a problem type.
Paper sessions:
The first session was entitled “Program Design and Maintenance.” J. Kim, J. Hahn and J. Lerch presented “How is the Designer different from the User? Focusing on a software development methodology.” The authors reported on a study in which designers of an object-oriented methodology and students (users) in the object-oriented methodology class were asked to develop a solution to a specified problem. The time each group spent in the four different problem spaces (rule, representation, instance, and paradigm) were measured. They found that the designers used the paradigm space as a major supporting space and that they actively searched the representation space and quickly arrived at solutions which were not extremely detailed. The users, on the other hand, used the instance space as a major supporting space, searched the rule and representation space and arrived at rather detailed solutions after considerable time. The authors note that further research needs to be conducted, but they feel that the representation in the object-oriented methodology may not have been rich enough to convey the design model to the users.
M. Petre and A. F. Blackwell presented “A Glimpse of Expert Programmers’ Mental Imagery.” They conducted studies of expert programmers doing program design and discussing these designs with other programmers. They found many instances of mental imagery being used by software designers, suggesting a similarity to other design activities. This implies that further research should investigate types of representations that could be used to support these images.
“An Experimental Evaluation of Quality Guidelines on the Maintainability of Object-oriented Design Documents” by L. Briand, C. Bunse, and J. Daly was the third paper in this session. Two object-oriented systems were developed; one was developed in accordance with the Coad and Yourdon design principles. Subjects looked at both systems and were asked questions about their overall understanding of the designs. They were then asked to do an impact analysis of changes that would have to be made for 1) a change in customer requirements and 2) enhanced functionality. The authors found that the system designed according to the Coad and Yourdon principles was easy to understand and to do impact analysis on. The authors have made available the experimental materials to anyone who would like to try replicating and improving upon the experiment.
The second session, “Software Visualization and Visual Programming”, consisted of three papers: P. Mulholland, “Using a fine-grained comparative evaluation technique to understand and design software visualization tools”, K. Whitley and A. F. Blackweel, “Visual programming: the outlook from academia and industry”, and C. Cook, M. Burnett, and D. Boom, " A bug’s eye view of immediate visual feedback in direct manipulation programming systems."
P. Mulholland used protocol analysis to identify user behavior using visualization tools and to give us an understanding of the effectiveness of different visualizations. He classified verbalizations of: information access from the display, the use of comprehension strategies, and misunderstandings of the visualization and execution. He used the analysis from the protocols of users using four visualization techniques to develop a new visualization tool. Testing the new tool using the same methodology showed that it was considerably better than the previous tool.
Whitley and Blackwell surveyed three groups of people as to their opinions about visual programming. They surveyed the existing literature to gather opinions that researchers had about visual programming. They conducted surveys of professional programmers at a trade show and they surveyed a group of Labview programmers. Researchers in visual programming were optimistic about the benefits of visual programming. Professional programmers were skeptical about benefits of visual programming. Labview users, however, were confident that Labview was beneficial to their programming efforts.
C. Cook presented the third paper which looked at the effect of adding immediate visual feedback to a direct manipulation programming language. Although immediate visual feedback did not help with all debugging, it did help in certain instances. The authors found that the type of bug and the position of the buggy code on the screen were influencing factors in how long subjects took to locate and correct the bug.
Two papers were presented in the “Program Comprehension” session. One by V. Ramalingam and S. Wiedenbeck and the second by A. von Mayrhauser and A. M Vans. V. Rmalingam presented " An empirical study of novice program comprehension in the imperative and object-oriented styles." In this study novices were given both object-oriented programs and imperative programs written in C++ and asked questions about different information categories. Overall, novices performed better at comprehending the imperative programs. However, they were able to obtain more domain-level knowledge from the object-oriented programs.
A. von Mayrhauser reported on “Program Understanding Behavior during Debugging of Large Scale Software.” They conducted protocol analyses of professional maintenance programmers debugging large computer programs. They classified three levels of abstraction that programmers work at: the program level which is a control flow abstraction of the code, a situation model which uses the program model and creates a data-flow or functional abstraction and the top-down or domain model which is an abstraction of the application domain. They looked at the levels of abstraction that the programmers worked at and concluded that programmers with little domain experience or little knowledge of the software worked at lower levels of abstraction until they accumulated enough knowledge to make higher level connections. However, programmers who had some domain experience could more easily make connections from the lower levels. Programmers who knew both the code and the domain could easily make connections between all three levels of abstraction.
Three papers presented at ESP were studies of programmers using visual programming languages. M. G. Williams and J. N. Blehler presented “A Study of program entry time predictions for application-specific visual and textual languages.” In this study, the authors looked at the differences in time that it took programmers to enter the program code written in a visual language versus the time it took programmers to enter the program code written in a textual language. A keystroke level model for the entry times was created and validated by an empirical study. These models can be used to approximate the code entry and editing tasks in studies comparing programming tasks in different languages.
H. J. Gottried and M. M. Burnett presented “Programming complex objects in spreadsheets: an empirical study comparing textual formula entry with direct manipulation and gestures.” An empirical study was conducted to compare techniques such as direct manipulation and gestures to typing for entering textual formula into spreadsheets. They found that subjects were faster and more correct when using the graphical definition techniques. Moreover, subjects preferred using the graphical techniques to define formulae.
C. Traynor and M. Williams presented “A study of end-user programming for geographical information systems.” The authors conducted an empirical study of a programming by demonstration system. End users of a GIS system were given program comprehension and program editing tasks. Subjects were able to comprehend and edit simple programs without error, but had difficulties on more complex programs, implying that some of the more complex constructs in the visual language need to be redesigned. However, subjects preference ratings implied that programming by demonstration is a promising technique.
Two panels were also presented at the workshop. Robert Rist organized a panel on past, present and future research issues in ESP. Panel members were Francoise Detienne, Marian Petre and MaryBeth Rosson. Rob presented an overview of research on program design. Francoise discussed the literature in program comprehension. Marian talked about the work that has been done in notations, including visual languages. MaryBeth gave an overview of the research in computer science education in programming languages and the work in programming environments.
Jim Herbslab was the organizer of a panel “Empirical Studies of Software Production: Methods, Substance, and Impact.” Panelists were Beki Grinter, Marian Petre, and Adam Porter. Jean Scholtz was the discussant. The panelists were asked to address the question: “how can empirical researchers of software production have the greatest impact on understanding the essential problems and on improving practice?” Marian made the point that this was the wrong question. A major impact can be made collectively - through many small impacts. In order for theory to ultimately have impact it has to be accessible to practitioners. Adam discussed how to have leverage on the development process. In his studies of software inspections, he tries to minimize the intrusion but he emphasizes the cost-benefit of being able to improve the process through studies. Beki advocated looking at the collaborative aspects of programming and using theory to explain practice. Qualitative studies open up new research questions but they take time to carryout and finding access to groups to study is difficult. Jean called for the ESP community to study software developers, not just programmers. She noted that new methods will need to be developed to study the activities of software developers, such as programming, communication, collaboration.
The next ESP workshop will be held in June of 1999. Co-Chairs of the event are Laura Leventhal, Irv Katz, and Marian Petre. Proceedings of ESP 7 are available from ACM Press.
PPIGers and Software Process Improvement
Judith Segal j.segal@mcs.surrey.ac.uk
You might recall that I sent the following to announce@ppig.org:-
“Is anybody out there doing any work on the organisational (or related) issues of software process improvement?”
As usual, I was overwhelmed by the generosity of PPIGers in taking the time to reply to me and describe their work. As promised, here is a precis of the response to the call:-
Khaled El Emam, of the Fraunhofer Institute for Experimental Software Engineering (IESE), elemam@iese.fhg.de, directed me to the following web site:- http://www.iese.fhg.de/ISERN/pub/isern_biblio_tech.html
Dennis Goldenson, of CMU, dg@sei.cmu.edu, described some work he’s doing with Jim Herbsleb and Larry Constantine, in which they plan to survey organisations with a view to considering Larry’s organisational paradigms, organisational communication patterns and how CMM based improvement fits in with other approaches. He’s also working with colleagues from the CMU business school on a study of the determinants of success in software measurement programs.
Sabine Sonnentag, ao_sonnentag@macmail.psy.uva.nl, is working on organisational aspects of software development in general, and included the following references:
Heinbokel, T., Sonnentag, S. , Frese, M., Stolte, W., & Brodbeck F.C. (1996).Don’t underestimate the problems of user centredness in software development projects - there are many! Behaviour & Information Technology, 15, 226-236.
Sonnentag, S., Frese, M., Stolte, W., Heinbokel, T., & Brodbeck, F.C. (1994). Goal orientation of team leaders: Its effects on performance and group interaction in software development projects. The European Work and Organizational Psychologist, 4, 153-168.
Sonnentag, S., Frese, M., Brodbeck, F.C., & Heinbokel, T. (in press). Use of design methods, team leaders’ goal orientation, and team effectiveness: A follow-up study in software development projects. International Journal of Human-Computer Interaction.
Brad Appleton drew my attention to a paper he wrote for PLoP ‘97, in which he discusses 10 successful patterns of software improvement. The paper is at http://www.enteract.com/~bradapp/docs/
Charles Knutson has written a paper in which he considers the different system test approaches used in a single company (Novell, where he was the System Test Manager) over a ten-year period. More details are available from him.
Lutz Prechelt (http://www.ipd.ira.uka.de/~prechelt/) is working on the ‘personal software process’ (PSP) methodology of Watts Humphrey. In particular, he is considering whether error protocoling and protocol analysis can be introduced without a PSP course, and also comparing errors made by students who have, and have not, followed a PSP course.
Ray Dawson (R.J.Dawson@Lboro.ac.UK) is researching into the general area of software metrics collection, including some of the psychological and organisational issues.
Willemien Visser , Francoise Detienne and Pierre Robillard are involved in a study of the cognitive activities in collaborative software development, in which the processes actually used by software engineers are observed and analysed by means of videotaping technical review meetings. In addition, Pierre, together with industrial partners, is involved in defining measurements for software process activities, and then analysing them, in order to formulate recommendations.
Bill Curtis sent me the following: “My entire company, TeraQuest Metrics, works with organizations to implement software process improvement. We now have loads of assessment questionnaire data, risk data for improvement programs, and a trove of practical experiences.” He mentioned that Prof. Lars Mattaissen at the U. of Aalborg in Denmark is also performing research on SPI and has a large research contract from the Danish government.
The last response was from Stan Rifkin of Master Systems Inc. who said: “There is little software process improvement work that does NOT deal with organizational issues.” Stan is interested in validating Weick’s conjecture in SENSEMAKING IN ORGANIZATIONS that there are only two kinds of problems, those of uncertainty and those of ambiguity. He uses Talcott Parsons’ functional prerequisites (from his General Theory of Action) to diagnose organizations and to guide implementation of SPI, along with some seminal work in a PhD dissertation by Cynthia Lopata.
Again, many many thanks to the people above for their generous responses: I hope I have not mis- (or under-) represented anybody.
HTML Authoring as Programming?
Chris Roast c.r.roast@shu.ac.uk
One area of languages in computing that has not been looked at from a PPIG perspective, as far as I know, is the ubiquitous Hyper-Text Markup Language (HTML). (HTML is the language on which the World Wide Web feeds!) The language is of particular interest because it is being extensively used by novices and non-experts.
The original motivation behind HTML was for it to be a mark-up language in which a document’s logical structure could be clearly defined, thus enabling different viewers to present the document appropriately. However the language also allows markup tags that have a physical interpretation for any viewer. For example, the logical tag <STRONG> marks text as strong, depending upon the browser this may be achieved by an alternative typeface or type style. By contrast the physical tag <B> forces the use of a bold typeface by a browser. The use of physical tags provides more control over what appears on a display, but conflicts with the original purpose of HTML since it forces documents to become more and more browser and device specific. The recent commercial interest in HTML, and its popular take-up, has resulted in many more language extensions almost all of which are physical in nature. For example, <BODY BGCOLOR="#FFFFFF"> sets a pages background.
Although, HTML is designed for web authors, this task is in some ways analogous to some programming tasks, since the notation used to control appearance (HTML) is once removed from the actual appearances that different users and browsers may encounter. The difference between physical and logic perspectives in HTML is analogous to that of low and high level languages. Physical tags are more specific to device details and also offer fine-level control, by contrast logical tags offer a generic perspective lacking in detailed control.
Pilot Study
A final year project at Sheffield Hallam University was to perform a pilot study with the aim of examining the factors HTML authors take into account when authoring. In particular, whether authors view the HTML they produce to be a logical or physical representation of a document. Twelve authors were studied, being given a small task to design an HTML page which conformed to a sample with a clear logical structure and also some specific physical elements (such as a logo). In addition, subjects filled in a simple questionnaire.
Results are not significant but suggest that the more experienced and technically competent the HTML author the more likely they are to work at a physical level (and avoid logical tags).
What does this tell us? The physical tags do require considerable expertise to use, hence it is unlikely that novice authors will use them. In addition, it suggests that the model of authoring adopted is a physical one which focuses upon the concrete details without regard for their device and browser specific nature.
Suggestions, ideas and volunteers for further studies on a postcard to Chris Roast, Computing Research Centre, Sheffield Hallam University, Sheffield. (Email. c.r.roast@shu.ac.uk).
Spotlight on PPIGers
Rafael O. Fontao
I am PPIG member since 1995 however I have not yet e-mailed a short resume to share with other fellow PPIGers in the next issue of PPIG Newsletter. Born, on April 22, 1943. I got an Electrical Engineering degree ( UNS, Bahia Blanca ,Argentina, 1967 ), Master of Science ( Computer Science oriented, at Stanford University, California, U.S.A. 1973) and a specialization on Systems Simulation, at Sevilla , Spain, 1983. I am currently Professor at Universidad Nacional del Sur, Bahia Blanca, Argentina in Systems Programming. My research interest in connection with PPIGers is on programming methodologies. In particular I am working on a programming methodology which I devived about 20 years ago. This methodology is focused on the programming process. There are steps to conceive a program from the description of the problem in natural language to a final step in which the original problem is written in a formal language. At each step we conceive a finite state automata to represent the process of refinement rather than the state of variables as the process run.
A copy of this paper may be requested my e-mail. All comments are welcome.
Kenji Kaijiri
My research (ppig-related) interest is:
- Program recognition techniques
- Program slicing techniques
My current research activity is
- Plan matching system to be used as program style checker
- Program fine grained repository to be used as reverse engineering data base
Shinshu University, Faculty of Engineering, Information Engineering
Kenji Kaijiri
(http://sukyou10.cs.shinshu-u.ac.jp/~kaijiri/home_kaijiri.html)
Babak Khazaei
Babak Khazaei who has over the last couple of years been working closely with the sheffield hallam empirical studies of programming group and who was co-editor of PPIG97, has moved from wolverhampton to take up a senior lecturer post at sheffield hallam.
Slim Masmoudi
I’m a Ph.D student in Cognitive Psychology, working on the cognitive modeling of the expertise of debugging complex programs. The aim of the study is to bring out the sub-activities and processes involved in the debugging activity, such as comprehension and hypothesis construction-testing. The study is conducted according to four points:
- the stimulus structures and their mental representations
- the dynamics of the problem-solving process
- the solving-oriented knowledge structures
- the domain-oriented knowledge structures
Moreover, I’m very interested in the development of new data-collection and analysis methods, especially, the content analysis techniques. In the context of my study, I conceived a new content-collection and analysis method adapted to the study of a debugging activity made by experts. Also, I used some experimental techniques to study the comprehension process and the conceptual organisation of expert programmers.
John Pane
pane+@cs.cmu.edu
A Programming System for Children that is Designed for Usability
Most children are not programmers but would like to create their own software. Typically they would like to create programs that are similar to their favorite commercial applications, such as games and educational software. These are highly-interactive and graphical, and contain animations, simulations and multimedia. Current programming tools are either too difficult for beginners to use, or are not effective for producing this class of software.
In my research, I will address this problem by developing a new programming language and environment for children that is based on a card game metaphor. This new system will be called HANDS, for Human-centered Advances for Novice Development of Software. The new card game metaphor for computation will make it easier for beginners to learn and understand how to program, because it will provide concrete real-world analogs for many of the abstract concepts of programming. While HANDS will be capable of producing a full range of programs, a fundamental focus of the design will be usability by children creating the highly-interactive software mentioned above. Throughout the design and refinement of HANDS, I will apply prior research about the psychology of programming and human-computer interaction, my own new empirical studies about the ways that non-programmers naturally express solutions to programming tasks, and usability testing. While the language will be text-based, HANDS will exploit modern technologies to simplify the programming process, including direct manipulation and demonstrational techniques, programming environment technologies like structure editors, and mechanisms for extending programs by harnessing and reusing existing components.
My thesis is that HANDS will be easier for children to learn and use than existing systems for beginners, and that users will be able to use HANDS to create sophisticated programs that rival commercial software such as games and educational simulations. I will evaluate this thesis through user studies that compare the HANDS system to other programming systems for beginners.
Pablo Romero
juanr@cogs.susx.ac.uk
School of Cognitive and Computing Sciences, University of Sussex, Falmer, Brighton BN1 9QH. UK.
I’m a PhD. student at Sussex University working with Ben du Boulay. I’m interested in program debugging and comprehension. My PhD. project is about the strategic and structural knowledge needed for comprehension when debugging in Prolog. Specifically, I want to find out whether there are such things as focal structures for Prolog programmers, what is the nature of these focal structures and how are they used.
Jim Spohrer
Jim Spohrer at Apple Computer is encouraging PPIG-ers to look at the Educational Object Economy site (http://trp.research.apple.com/EdEconomy) and use material in it as the basis for empirical studies in software reuse. The EOE site is a directory to almost 1000 Java-based educational objects produced at universities and companies the world over. To promote software reuse of the applets, the library of applets can be searched in a variety of ways, including looking for applets with sources available (about 25%) of the applets. The EOE is trying to develop an “intellectual capital appreciation” license that promotes sharing and reuse of source code for educational objects. For example, when someone is creating educational software that needs a grapher, rather than build it from scratch, they can try to reuse an existing grapher. Recently, Raquel Navarro-Preito, a PPIG-er from Spain, was a visiting scholar at Apple, looking at social dimensions that complement the cognitive dimensions that influence software code reuse. An educational object economy is like an authoring community, with educators, researchers, businesses, and government agencies also involved. Authoring tools cannot answer the question of what has already been built by other using the tool, but an educational object economy or authoring community, can help someone not only find out what has been built, but who built it, and their willingness to share the code for others to reuse. If other groups would like to start their own educational object economy or similar member-generated content web sites, the Filemaker 4 databases for the EOE are being made freely available soon.
Jon Whittle
I am currently in my final year of a PhD. I am interested in providing tools that support more efficient learning of programming languages. I have developed and implemented a novel editor for ML. Programs are constructed by applying a sequence of high-level editing commands to old programs. The editor also carries out certain analyses of the programs as they are built, so that programs produced in the editor (called CYNTHIA) are guaranteed to be syntactically correct, well-typed, well-defined and terminating.
I am currently undergoing an evaluation of CYNTHIA using 45 ML students. I am particularly interested in different evaluation techniques. I am currently using informal techniques but would like to hear about successful formal evaluations.
Andrew Walenstein
walenste@cs.sfu.ca
School of Computing Science
Simon Fraser University
Greetings PPIGers (or is it PPIGies?)
I am a PhD student and am presently struggling to breathe life into a research proposal. My main focus occasionally appears to be cognitive aspects of program maintenance, but I think of myself primarily as a maintenance tool researcher searching for a theoretical basis for explaining the utility of software comprehension tools. As a result, I’ve been interested in better understanding the relationships between tool characteristics and cognitive aspects of comprehension in well-motivated, individual, experts maintainers. For example, the interaction techniques supported by a tool for program restructuring might affect the strategies pursued in restructuring tasks.
I started my postgraduate work with the intention of attempting to generate software visualization for maintainers. While familiarizing myself with some of the more technical aspects of the prospect (source code analysis, graph layout, etc.), I found that the type of questions which intrigued me the most were the ones I had trouble answering to any degree of satisfaction. These have tended to be quite topical questions concerning humans interacting with external artifacts. For instance: “how, specifically, can visualization facilitate program comprehension?”, and (this week’s favorite) “(how) can cognitive comprehension models be used in tool rationales?”