| Preprint Copies | Copyright 1998 IEEE. Published in the 13th Conference on Automated Software Engineering (ASE'98) October 13-16, 1998 in Honolulu, Hawaii. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works, must be obtained from the IEEE. Contact: Manager, Copyrights and Permissions / IEEE Service Center / 445 Hoes Lane / P.O. Box 1331 / Piscataway, NJ 08855-1331, USA. / Telephone: + Intl. 908-562-3966. |
Abstract
Cognitive models of software comprehension are potential sources of theoretical knowledge for tool designers. Although their use in analysis of existing tools is fairly well-established, the literature has shown only limited use of such models for directly developing design ideas. This paper suggests a way of utilizing existing cognitive models of software comprehension to generate design goals and suggest design strategies early in the development cycle. A crucial part of our method is a scheme for explaining the value of tool features by describing the mechanisms that are presumed to underly the expected improvements in task performance.
Cognitive models of software comprehension have been introduced and studied, in part, out of the desire to improve the tools, languages, and environments available to software developers and maintainers (vMV). Cognitive models of comprehension might very well help us understand comprehenders better, but how well do they help explain the value of existing tools? Moreover, how well do they help when it comes to designing tools?
We will briefly argue that the application of cognitive models early in the design cycle has been limited due to the relatively restricted forms of reasoning that they have most commonly supported. Then a means of potentially extending their usefulness by reasoning forward from such models will be proposed. The trick is to introduce a method of accounting for improvements in task performance. We propose that this be done by explicit appeals to how the tool features modify cognition. Due to space limitations, only the basic method of utilizing the comprehension models is described. Our focus will be on using existing comprehension models for designing industrial-scale maintenance and reverse-engineering tools.
One of the most significant uses of cognitive models of software comprehension for designing and re-designing tools has been to facilitate usability testing and analysis. In these circumstances, designers will reason backwards (CR) from existing (or partly designed) tools to cognitive models of how the user performs the tasks of interest. This type of backward reasoning has frequently been used to reveal potential difficulties that can be identified by the model. For example Soloway et. al. (SPLLL) traced the causes of maintenance errors from the properties of a displayed program to the comprehension strategy employed by their subjects. This sort of method of reasoning backwards from artifacts to cognitive models has proven to be very useful. The ``cognitive dimensions'' framework (Green 89) (Green-Petre) (GDG) is a prominent attempt at codifying general design principles from this type of analysis. It serves as an instructive lesson in creating widely applicable advice for designers using cognitive models of task performance.
The cognitive dimensions framework defines a set of abstract features (the ``dimensions'') of programmer artifacts such as software development environments. The dimensions provide a vocabulary for discussing possible drawbacks and problems with notations and the interfaces that support them. For instance, the dimension of ``hidden dependencies'' suggests that if important dependencies are not made explicit by the system then they can be missed, creating errors during modification. The concept of hidden dependencies precisely describes the problem that Soloway et. al. had found using field study techniques as a type of task analysis (SPLLL) The advantage of the cognitive dimensions is that the analysis can often be ``quick and cheap'' (Green-Petre) and the analysis framework is not situation-specific (GDG). That is, the dimensions are abstract and codify problem categories in a framework that can be applied to any similar design problem.
The cognitive dimensions can point out potential problems because they relate features of the artifacts to a model of how the user performs their tasks (specifically, the user is modeled as solving their problems in an opportunistic and exploratory fashion (Green 89)). It is worthwhile dwelling for a moment on the source of their descriptive power: it is the causal linking of the properties of the artifact and the user model that can be seen to generate the design issues identified by the cognitive dimensions. All reasoning about the adequacy or relative desirability of an artifact is ultimately based upon the perceived relation between artifact and model. The causal links are thus a source of explanatory power, but what links do we pursue, and why?
Despite their advantages, the dimensions are relatively poor at directly suggesting how performance can be improved. They are consequently less well-suited for reasoning forward than for issuing warnings about design consequences. Warnings still serve designers by constraining design (SC), and can be invaluable in analytical usability evaluation (e.g., of visual languages (Green-Petre)). However reasoning forward with the dimensions is only indirectly addressed. Green and Petre suggest that it is a matter of using ``standard remedies'' during re-design (Green-Petre). Here we set our sights on more specific advice: in the words of Singley and Carroll, we wish to help designers cross the ``analytic–synthetic gap,'' (CR, pg. 243). The desired point of crossing concerns how to reason about performance improvement.
Reasoning about performance requires a type of causal link between user models and artifacts that is different from the type frequently used in the cognitive dimensions. In particular, the links must be to explanations (i.e. possible mechanisms) of why performance is improved by using the artifact. Such links can be used during both analysis and design. As we shall argue in the next section, reasoning from model towards artifact may be facilitated by describing a design reasoning space.
The process of designing software comprehension tools can be described as a search for ways to create artifacts in order to achieve design goals. The main idea being pursued in this paper is that one way to assist designers in this search is to make knowledge about potential design options accessible, and describe general strategies for achieving the desired design goals. When generated from empirically founded models, this knowledge forms a theory-based resource that augments current design methods. We will outline a step towards using prior cognitive models to develop such a design toolkit.
The key idea used in our approach is that cognitive models allow the description of an appropriate design reasoning space (as compared to a design space (Green-Petre), which also describes an artifact space). Since our interest is in reasoning about performance, the points within the space are explanations of how performance is improved. Links from artifacts to these explanations can be used by a designer to build claims (CR) as to why the artifacts achieve design goals. We shall describe the broad outline of the design reasoning space, including how we envision claim generation by designers. We will then touch on how design goals can be identified by examining the features of the cognitive models.
Most of the common comprehension models (such as von Mayrhauser and Vans' (vMV)) model cognitive activity and products in arguably similar manners, and at roughly the same level of abstractness. Hence we shall not pick any particular model to concentrate on. Instead our focus is on performance explanation development. The notion of cognition redistribution appears to generate an appropriate design reasoning space, given typical comprehension models. Using this notion, the various reorganizations of cognitive structure and activity forms the explanations that build design claims.
We have chosen cognition redistribution because it appears to describe many of the important design claims that lie scattered in the literature. The contribution of this section is not a generative theory which reconstructs these redistribution claims; rather it aims (1) to show one possibility of an integrative framework for previously diffuse or ad-hoc design reasoning, and (2) to try to filter from this theoretical soup a useful lexicon of descriptive terms (GDG). Thus the following should be thought of as the beginnings of an attempt to organize some of the existing design reasoning.
Rigi uses spring layouts to spatially cluster representations of program elements which are likely to be logically clustered within a module. Without Rigi, the module identification process might involve opportunistically searching for potentially related elements with a text editor, reasoning about their relationships, and then evaluating their relative cohesion and coupling. Using Rigi, module construction can utilize cluster perception to focus attention on candidate modules, replacing much of the opportunistic search through the program text. In addition, cluster complexity judgments can also be used to replace reasoning about cohesion and coupling.
Analyzing the potential impact of Rigi on a specific cognitive model of bottom-up comprehension provides the basic mechanism for a type of psychological claims analysis of the tool (cf. Carroll and Rosson (CR)). In the example above, a claim for performance improvement can be created by linking the presence of the appropriately clustered program representation to an instance of perceptual operator replacement. Attendant improvements in performance (e.g. speed, cognitive ease) can be used to claim the achievement of the appropriate design goals.
The simplest forms of external cognition distribution are quite familiar: reasoning can be replaced by automated forms (e.g. inference systems, abstract interpretation), constraints can be enforced by tools after formally coding them (e.g. type-checking compilers), and knowledge can be recorded in knowledge bases, notations, and formalisms (e.g. programming-plan libraries). These options might be termed process, constraint, and knowledge distribution, respectively (cf. Scaife and Rogers (SR)). Moreover, the notion of distributed cognition also can effectively handle more subtle, partial redistributions. The key is not to consider the external processing as a wholesale replacement of a cognitive process, but as a distribution of it between the tool and user. RMTool (MNS) provides an example.
In RMTool, an abstract model of a software system and its mapping to a lower level model of the same system are created by a maintainer. The models and their mapping can be viewed as encoding a hypothesis about the system in question. From the point of view of distributed cognition, writing out the abstract model and its mapping externalizes (part of) the comprehender's hypothesis about how the model can be bound to lower-level entities. The maintainer updates the external representation of the hypothesis as it evolves.
Furthermore, from the externalized hypothesis, RMTool can automatically generate a ``reflexion model'' showing where the model and its map overlap with reality. The reflexion model generation represents a distributed part of the overall hypothesis verification process. In particular, the tool automates a portion of the overall search for confirmatory and conflicting evidence for the hypothesis. This portion might otherwise be performed by a ``manual'' search through the code.
The overall process of using RMTool corresponds closely to a distributed (but non-hierarchical) version of Brooks' hypothesis binding model (Brooks). Only part of the complete process (i.e. the evidence search) is performed externally, and only part of the knowledge (i.e. the relevant hypotheses and argumentation evidence) is distributed onto the tool. Notice that having a model of the entire process is important to arguing the contributions of the tool.
As with Rigi, the above type of analysis of RMTool provides fuel for specific claims about its value. Possible claims include the advantage of memory offloading by externalizing complex hypotheses onto a graph editor, or a reduction in cognitive processing by having the tool search for evidence. Unfortunately these claims only scratch the surface of the cost accounting that would need to be done to convert the design claim into a scientific theory of human–tool interaction (see for example Davies' work on display-based problem solving (Davies)). Regardless of whether a full-fledged theory of distributed cognition in reverse engineering is eventually developed, for the moment we point out that the vocabulary seems to be expressive and therefore hints at being potentially inspirational during early design.
Cognitive comprehension models can help set design goals because they act as a generalized task analysis that can be applied to specific problem instances. For instance when designing a hypothetical tool to recover implicit event sequence abstractions in legacy Ada code, a designer might start with a bottom-up model of comprehension. Effort might then be directed to the question of how to redistribute some of the processes of finding and describing related events, and of comparing their coupling and cohesion. These steps form the basic strategy for reasoning with cognitive models: start with (or build) a model of the activity of interest and then examine the cognitive processes and knowledge used for opportunities for redistribution. Efforts to categorize the basic tasks and design goals by analyzing the models (cf. the work of Storey et. al. (SFM)) can improve the effort-focusing capability of the models.
We argued that existing cognitive models can be used to reason forwards during design, but that in order to do so they must be augmented with a means of reasoning about how and why tool features aid the comprehender. We suggested that the idea of redistributing cognition as it is described by a comprehension model provides a framework for discussing improvements in task performance. A taxonomy of ways of redistributing cognition, in combination with comprehension models, can become part of an intellectual toolkit for designers.
Perhaps the most important advantage of our approach is that the focus on design claims exposes the reasoning underlying the expectations for performance improvement. Although we have angled our discussion towards the issue of design, in reality the explicit form of the design claims help during analysis as well as design. The claims clarify some of the important issues which are at stake. Exposing these claims could enable (or even encourage) the testing of the claims underlying a tool's design. This could make the current practice of tool evaluation more diagnostic: with specific claims, experimenters can perform evaluations that directly test underlying mechanisms rather than simply collecting overall performance comparisons.
Thanks go to Robert D. Cameron and Dave Fracchia for their helpful comments on this work. This work was supported in part by a scholarship and a research grant from the Natural Sciences and Engineering Research Council of Canada.