College of Information Sciences and Technology
The Pennsylvania State University

The Effects of Interface Design on Telephone Dialing Performance 

Andrew R. Freed
arf132@psu.edu

6 May 2009

 
College of IST, University Park, PA  16802

Class Project for IST 331

The Effects of Interface Design on Telephone Dialing Performance 
Andrew R. Freed
arf132@psu.edu
College of Information Sciences and Technology
The Pennsylvania State University
University Park, PA  16802
6 May 2003
Abstract
Interface design has an important impact on user performance.  While some features of telephone interfaces have become relatively standardized (such as the layout of the keypad), there are other design features that can affect the way users interact with these interfaces.  These features include the overall size of the interface and the size and labels of buttons.
We look at several prototype telephone interfaces with a variety of tools for automatic testing.  Several analyses are presented.  The first analysis uses Fitts’ Law to determine the minimum possible usage time.  Fitts’ Law provided a quick analysis of the telephones but did not consider any factors other than target sizes and distances.
The second analysis uses a cognitive model to make predictions about other factors such as visual search. The model used an organized search strategy that produced similar behavior to the experiment subjects but failed for a few extreme interfaces.   Although the model dialing time predictions are too high, we propose that this model is worth paying attention to because the dialing time orderings among phones and number of fixations closely follows the observed data.
A short, 100-200 word description of what we need to know, what we have done, with enough details that a reader will not only be interested, but can use the abstract as a memory aid.  That is, tell him the punch line here!
Table of Contents 
[for longer reports, again to help memory and writing, word will do this for you if you outline your headings.]

1.  Introduction: Expert evaluations of interfaces	1
2.  Expert evaluations of the Situation Awareness Panel (SAP)	2
2.1  Cognitive walk throughs, heuristic evaluation, and semi-structured interviews	3
2.2  The Situation Awareness Panel	4
2.3  The study	6
2.3.1  Participants	6
2.3.2  Material and equipment	6
2.3.3  Procedure	6
2.4   Results and Discussion	10
3.  Conclusions	12
4.  References	14
Appendix A.  Feedback on the initial Situation Awareness Panel	16
Appendix B.  The new Situation Awareness Panel	21
Appendix C.  Feedback on the new Situation Awareness Panel	22

1.  Introduction: Expert evaluations of interfaces
Newell’s (1990) call for a unified theory of cognition has led to a new way of research in cognitive psychology.  During the last two decades many researchers have started examining psychological phenomena through situation-specific theories that they implement as computer programs.  These theories, called cognitive models, remain confined to the constraints imposed by grand theories of cognition, generally known as cognitive architectures.
Cognitive architectures and cognitive models have been used extensively as means for exploring the mechanisms involved in human cognition.  However, cognitive models have been also built with the goal of being used as surrogate users.  Models with human-like behavior can replace humans in many situations ranging from cognitive tutoring (Anderson, Corbett, Koedinger, & Pelletier, 1995) to usability testing of interfaces (Ritter & Young, 2001; St. Amant, 2000). 
The use of cognitive models as surrogate users is especially appealing for situations in which human expertise is either costly or difficult to recruit.  Military training has been one of those fields that typically require a great amount of human resources. Cognitive modeling provides an alternative avenue for supporting military training. Cognitive models as intelligent agents can populate synthetic environments representing some or all of the entities involved in real combats, thus enabling the use of realistic environments for training purposes (Pew & Mavor, 1998; Ritter, Shadbolt, Elliman, Young, Gobet, & Baxter, in press). One such attempt has been the TacAir-Soar system (Tambe, Johnson, Jones, Koss, Laird, Rosenbloom, et al., 1995), which employs cognitive models developed with the Soar architecture (Laird, Newell, & Rosenbloom, 1987; Newell, 1990) to simulate the behavior of military personnel in fixed-wing aircraft missions.  The benefits of using TacAir-Soar are particularly evident in large-scale simulation exercises in which many of the entities involved can be driven by Soar models instead of human users.  For example, in the Stow ’97 exercise, up to 3,700 computer-generated forces were involved as both friendly and enemy entities (Jones, Laird, Nielsen, Coulter, Kenny, & Koss, 1999).
While using cognitive models to either answer psychological questions or to replace human users provides great advantages, serious problems have been identified as well.  One of the problems is the limited reuse of cognitive models.  It seems fair to say that cognitive models are not typically used by researchers other than the ones who developed them.  Part of the problem can be attributed to the lack of graphical-user interfaces for many of the models developed (Ritter, Jones, & Baxter, 1998). Without graphic displays, observing and understanding the behavior of the cognitive models is restricted, which can contribute to limiting their adoption by others.  
2.  Expert evaluations of the Situation Awareness Panel (SAP)
The goal of the project was to understand and improve the SAP as a tool for inspecting the behavior and reasoning of the Soar agents that populate the JSAF simulation environment.  Our attempt was focused on the validity and usability of the SAP.  
Validity refers to whether the types of information displayed on the SAP is truly in the awareness of actual pilots engaged in air combat.  Usability refers to whether people using the SAP can understand the model based on what they can see through the SAP.  
2.1  Cognitive walk throughs, heuristic evaluation, and semi-structured interviews
There are a variety of approaches that we could have used to examine the usability of the SAP interface.  We could have done a task analysis (Schraagen, Chipman, & Shalin, 2000) if we had a list of what tasks users were performing, but we were attempting to create such a list.  We were not trying to optimize performance, per se, so timing users on tasks was not appropriate either.  We were not just looking at learning, as our users would either just be watching it (with very little learning, we would hope), or would be working with it for a while (with quite extensive learning) and did not have a task analysis in hand, so cognitive walk-throughs (Polson, Lewis, Rieman, & Wharton, 1992) seemed not quite appropriate either.  
In the end, we did what could be described as guided heuristic evaluation.  We prepared a subset of tasks that we knew the interface would be used for.  We had potential and existing users and a variety of usability experts broadly defined perform these tasks with the interface.  We observed these users and also had them comment on the problems they had.  After performing these tasks, we debriefed them in order to find out what other tasks they would like to have been able to perform with the interface.  In some cases, they could do these tasks, in others, we were able to add these tasks to our developing task analysis.  In some ways, our approach was similar to semi-structured knowledge acquisition interviews (see, for example, SigArt ACM Special Interest Group on Artificial Intelligence, 1989).
2.2  The Situation Awareness Panel
The Situation Awareness Panel is a graphical tool that enables the user of the JSAF simulation to observe a Soar agents' understanding of a situation, their goals, and their history (Jones, 1999).  It is, in essence, a window into the Soar agent's awareness of the current situation.  Figure 1 shows a screenshot of the SAP of an agent named Shooter2.  
Figure comes after its introduction.  ditto for tables.

Figure 1. A screenshot of the SAP.
(d) 	The Agent Awareness Display occupies the rest of the SAP. This display enables the user to inspect the current state of the model's awareness.  It is basically a view of the model's reasoning about what is going on in its world (which is not necessarily an accurate depiction of what is really going on).  Entities with which the agent had contact (through vision, radar, or radio) are all represented in the display and marked with different colors to indicate whether they were friendly, hostile, unknown, or inactive.  The type of contact is represented by different type styles.  The user can adjust the scale of the Agent Awareness Display by choosing a different number from the "View Scale" drop-down menu.
2.3  The study
2.3.1  Participants
The participants were twelve experts coming from different disciplines. Table 1 lists the area of expertise of our participants.  The first eight participants completed the study in our laboratory, while the last four were did so under the same equipment but on the site of their employment. All participants received monetary reimbursement in exchange for their participation.  Participants were run individually with each experimental session lasting between an hour and two hours.  The last four participants also observed a new and improved version of the SAP that became available to us shortly before the time they were run.  Appendix B presents a screenshot of the new SAP.
2.3.2  Material and equipment
The JSAF simulation environment was presented on 19-inch monitor attached to a Dell Optiplex computer running Red Hat Linux 6.1.  All experimental sessions were videotaped with the use of a SONY TRV-120 Hi-8 digital camcorder.  In addition, the computer desktop activity of the first eight subjects was videotaped on VHS tape.  Participants read and signed an informed consent form prior to the beginning, and they were debriefed upon completion of the study.
2.3.3  Procedure
Each experimental session started by providing the participant with a short description of the SAP taken from Avraamides's manual (Avraamides, 2001) and a description of the scenario within the JSAF environment that would executed by the models.  Soar Technology provided us with three pre-programmed scenarios, from which we selected the Defensive Counter Air (DCA) scenario for our testing purposes.  The description of the DCA scenario that subjects read before the study is presented in Figure 2.

Figure 2. Screenshot of the instructions provided to participants.
The study was divided into two parts.  The goal of the first part was to familiarize our subjects with JSAF.  We therefore asked them to perform a number of actions such as the ones shown in Table 2.  This part lasted longer for the first eight participants because they were completely unfamiliar with JSAF.
Table 2. Example activities for familiarization with JSAF.
o	As soon as the Plan View Display (see Appendix [of the instructions] for a screenshot) becomes visible, zoom into the map and locate the position of the AEW.  Please report the approximate longitude and latitude of the AEW.
o	When the Agent Window’s appear select “1:2,000,000” from the Scale menu of the Plan View Display.  Report the names of the agents you can see.
	
These simple tasks were introduced in order to guide our subjects to explore the various windows of the JSAF simulation and learn some important functions, for example, zooming into the map with either pressing simultaneously the two mouse buttons or using the Scale menu.  We believe that some familiarity with the other windows of JSAF is needed in order to make possible the efficient use of the SAP.  For example, inspecting the Plan View Display a user could determine which of the agents is more likely to have a target in its awareness, and then use the SAP to examine this feature.
The second part of the experiment had tasks that required that subjects observe the four displays of the SAP at the time the agents were engaged in combat as defined by the on-going scenario.  Some example tasks are presented in Table 3.
As can be seen from the list of tasks in Table 3, we asked our subjects a variety of questions that differed in terms of both what they were required to do and what aspect of the SAP was brought into focus.  Some of the questions required that subjects simply observed what was going on and others required some interaction of the subject with the interface. Observation questions were aimed at assessing whether the SAP was successful at conveying the information that it was supposed to convey. We were primarily interested in seeing whether our subjects could easily pick up from each panel of the SAP the information that the SAP was meant to provide.  
Table 3. Example tasks used for exploring and interacting with the SAP.
o	Identify the two aircraft that are dispatched by the AEW to engage the enemy and bring up the SAP of one of them. What do you think that each part of the SAP does?
o	What is the altitude and speed of the agent that you are inspecting? Is the agent currently ascending or descending?
o	What friendly agents are in your agent’s awareness? What is their status? 
o	Are any enemy aircraft present in the Agent Awareness Display?  If not, go back to the map to inspect the positions of the enemy aircraft. Bring up the SAP of the agent that you think is the most likely to be aware of the enemy
o	If the enemy is within awareness, is it also within the Radar range of the agent?
o	Are there any inactive or unknown agents present in the awareness of the agent you are inspecting. If yes, can you identify those agents from the Plan View Display?
o	Go to the Plan View Display, go to the “On order” menu and select “MIG29 FWA Sweep Base”. Find the agent that you think the enemy would appear in its SAP and report its most active goal.
o	What is the most recent history event of the agent you are inspecting?
o	Close the SAP.
	
Difficulties with and sometimes misunderstandings of information were of particular interest since they provide points to consider for future implementations of the SAP.  The feedback we got from our subjects enabled us to determine ways of improving the visual interface of the SAP in future designs.  Questions that required the subjects to interact with the simulation in order to initiate some action (i.e., “MIG29 FWA Sweep Base”) provided a way for our subjects to evaluate the reasoning and the behavior of the agent, while it was engaging in action to achieve or prevent a user-initiated goal.  The primary focus of the study was to determine the success of the SAP interface at revealing to users the reasoning of the agents and pinpoint their limitations, particularly in assessing whether the reasoning of the Soar models was realistic.
2.4  Analysis and Results
All videotapes were reviewed at a later time by the experimenters and a list of potential problems with the SAP interface was generated.  The problems noted along with feedback from our subjects on how to deal with them enabled us to generate a set of suggestions for the improvement of the interface.  Because our study focused on providing feedback for the improvement of the SAP, the present paper does not address any of the positive feedback received from our participants, which was substantial. 
A total of 35 problematic issues for the SAP display were identified by our experts.  The table in Appendix A presents a list of the main problems that were identified and the suggestions we were able to provide for overcoming them.  Suggestions are subjective to the experimenters but they depend wholly on feedback obtained from our subjects.  
Appendix C presents feedback we received about the new version of the SAP.  Eleven different problems were noted by the four participants who observed the new SAP.
Table 4 reports the disposition of these two sets of suggestions.  The totals do not equal the number of suggestions because some suggestions were put into two categories, particularly where the suggestion included multiple aspects.  About half of the suggestions have been (done) or are being considered to be included (possible) into the revised SAP.  Some are unlikely to be done (unlikely), and several of the suggestions represent problems in the old SAP that have been avoided rather than fixed in the revised SAP.  There are also several of the more complex suggestions that remain under discussion. 
Table 4.  Disposition of the usability problems found for the original and revised SAPs. 
	Original SAP	Revised SAP	
Done	5	3	
Possible and/or planned	11	6	
Unlikely to be done	5	0	
No longer applicable	10	0	
Under discussion	6	2	
Table 4.  Set of tasks found in this analysis method.
Perception (Inputs) - What inputs does the model get?
Inputs does the model get from instruments
	Radar and IFF values (if from display), and visual input
	Voice input/communication from other agents
	Other perceptual events
Constants in perception, e.g., due north
Where is our agent's attention (for analyst) -- perhaps with a spotlight metaphor (this was used by Chong in the AMBR project to good reviews)
Actions (Outputs) - What actions has the model done?
What plane/pilot/RIO has said and done
	details of those actions if complex
What milestones are there, and what's the range of types of milestones, i.e., what could have been there but are not, and thus why are they not?
Physical environment features that affect the agent's body
Weather
Terrain, including base location, feet wet/dry, ground threats, places to land for RWA
Unknown but suspected ground threats will be an interesting thing to display
Mental environment -- Current Goals and Active Plans
Active goals, and their current status
Inactive goals, and why inactive (complete list of all possible goals and plans, and their status)
Details of the goals
Remaining steps in a goal/plan with associated physical location
Distance to target or other key events that agent would keep in attention and update, such as time left on CAP (thus timing)
Long term memory contents and active elements
Structure of memory and other mental objects
Contents of short term memory
Contents of perceptual (iconic)  memory
Capacity remaining in each capacity, e.g., working memory, idle (slack) time in central processor.
Pop-up display of changes/targets of goals for turning, climbing,  accelerating,  i.e., when the plane starts to do any of these, a pop-up window appears over or in the SAP indicating what is being attempted
An articulate model that comment on its behavior
	what other operators were available
	why operators were or were not chosen (cf. Lewis Johnson's work)
Social environment
Cultural/political/historical facts that influence behavior (declarative facts)
Rules of engagement (perhaps available but not displayed if they don't change often)
Other social context of team, broadly defined
Mental models of other agents
(actual vs. mental may indeed be different)
x,  y,  z, heading,  roll, yaw, pitch, speed, weapons
dx, dy, dz, d(heading),d(roll, yaw, pitch, speed)
Model of what the other planes are and what they are doing and what they are going to do
	(this list repeated one level down based on what they think you are going to do!)
What other planes have said
What other planes have been told, perhaps from a specific  range of time
What other planes can see (their radar might not be as good, and you might know it)
Physical status of other plane, damaged or not, fuel status, munitions, etc.
Physical status of other plane pilot, healthy, tired, bored
Military environment (task and hardware of own agent)
Written instructions
x, y, z, heading,  roll, yaw, pitch, speed, weapons
dx, dy, dz, d(heading),d(roll, yaw, pitch, speed)
Munitions capabilities if novel (otherwise, assumed or reconstructed), and range
	
Based on the feedback we got from our participants and our experience building interfaces for cognitive models (Ritter et al., 1998; Ritter & Larkin, 1994) we created the task-analysis shown in Table 4.  We believe this task analysis can be used as a guide for designing interfaces for cognitive models of military content.  These tasks include what all users need to know to understand their models, so interfaces that supported these tasks would also be useful more generally.
This task analysis includes tasks that would be expected.  That the Perceptions and Actions of the model are needed by analysts will not be a surprise to many modelers.  Likewise, access to the Mental environment of the model should not be surprising, but this is not fully supported by every modeling environment.  Similarly, because the models are increasingly becoming embodied and subject to their environment, the modelers need to know what aspects of the environment influence a model.
Including the Social environment of the model explicitly is somewhat novel.  These agents clearly have social aspects to their behavior and reasoning.  This appears to be a different type of knowledge and processing, a type that interfaces should make available to modelers.  
The Mental models of other agents is of increasing importance for cognitive models as they become team members, and this is particularly true for models in synthetic environments that need to understand colleagues and advisaries.  Finally, the model and the modeler need to keep in mind aspects of the environment related to their specific task.  In the case of these models, the domain is a military one.  Other models are likely to require additional information related to their domain of performance.
3.  Conclusions
Using cognitive models as surrogate users in military simulations provides the capability of training military personnel even individually.  In a JSAF simulation, thousands of entities can be represented as computer-generated forces providing the feel of a realistic environment without the need to recruit great numbers of humans to participate in the simulation. The success of simulators depends greatly on how realistic the behavior of the cognitive models is.  
Graphical interfaces that make the behavior of these models visible to human users enable the validation and the improvement of the models.  So far, not much emphasis has been placed on the design of graphical interfaces for cognitive models.  Even when graphical interfaces have been supplied along with models, they have been designed based on the “common sense” of their developer. 
We believe that graphical interfaces are very important for cognitive models.  By making their internal state more visible, graphical interfaces allow a better understanding of the reasoning and actions of the model and therefore lead to easier debugging, better validation, and more powerful demonstrations of the models. 
Given the importance of graphical interfaces, we argue that they should undergo rigorous testing to assess their usability.  The present study provides an example of how this can be done.  The provided task-analysis can be used to guide the design of future modeling interfaces.  Keeping and extending this task analysis will help design better interfaces so that they support the user performing their tasks.
Acknowledgments
We need to acknowledge the help of our subject matter experts who provided the feedback that we summarize.  The experts at BMH were particularly helpful at a distance.  Glenn Taylor and others at Soar Technology provided invaluable help in installing and maintaining the SAP for our use.  Comments from Angie Barnhill have improved this report.
4.  References
Anderson, J. R., Corbett, A. T., Koedinger, K. R., & Pelletier, R. (1995). Cognitive tutors: Lessons learned. Journal of the Learning Sciences, 4(2), 167-207.
Avraamides, M. N. (2001). A brief manual for running a JSAF Demo and examining the Situation Awareness Panel (Tech. Note No. 2001-1). Applied Cognitive Science Lab, School of Information Sciences and Technology, Penn State. [available upon request].
Jones, R. M. (1999). Graphic visualization of situation awareness and mental state for intelligent computer-generated forces. In Proceedings of the Eighth Conference on Computer Generated Forces and Behavioral Representations.  219-222. Orlando, FL: Division of Continuing Education, University of Central Florida.
Jones, R. M., Laird, J. E., Nielsen, P. E., Coulter, K. J., Kenny, P., & Koss, F. V. (1999). Automated intelligent pilots for combat flight simulation. AI Magazine, 20(1), 27-41.
Laird, J. E., Newell, A., & Rosenbloom, P. S. (1987). Soar: An architecture for general intelligence. Artificial Intelligence, 33(1), 1-64.
Newell, A. (1990). Unified theories of cognition. Cambridge, MA: Harvard University Press.
Ousterhout, J. K. (1994). Tcl and the Tk toolkit. Reading, MA: Addison-Wesley.
Pew, R. W., & Mavor, A. S. (Eds.). (1998).  Modeling human and organizational behavior: Application to military simulations. Washington, DC: National Academy Press. books.nap.edu/catalog/6173.html.
Polson, P. G., Lewis, C., Rieman, J., & Wharton, C. (1992). Cognitive walkthroughs: A method for theory-based evaluation of user interfaces. International Journal of Man-Machine Studies, 36, 741-773.
Ritter, F. E., Jones, R. M., & Baxter, G. D. (1998). Reusable models and graphical interfaces: Realising the potential of a unified theory of cognition. In U. Schmid, J. Krems, & F. Wysotzki (Eds.), Mind modeling - A cognitive science approach to reasoning, learning and discovery. 83-109. Lengerich, Germany: Pabst Scientific Publishing.
Ritter, F. E., & Larkin, J. H. (1994). Using process models to summarize sequences of human actions. Human-Computer Interaction, 9(3), 345-383.
Ritter, F. E., Shadbolt, N. R., Elliman, D., Young, R., Gobet, F., & Baxter, G. D. (in press). Techniques for modeling human performance in synthetic environments: A supplementary review. Wright-Patterson Air Force Base, OH: Human Systems Information Analysis Center.
Appendix A.  
As needed