• unlimited access with print and download
    $ 37 00
  • read full document, no print or download, expires after 72 hours
    $ 4 99
More info
Unlimited access including download and printing, plus availability for reading and annotating in your in your Udini library.
  • Access to this article in your Udini library for 72 hours from purchase.
  • The article will not be available for download or print.
  • Upgrade to the full version of this document at a reduced price.
  • Your trial access payment is credited when purchasing the full version.
Buy
Continue searching

Comparison of eye movement data to direct measures of situation awareness for development of a novel measurement technique in dynamic, uncontrolled test environments

ProQuest Dissertations and Theses, 2009
Dissertation
Author: Kristin Suzanne Moore
Abstract:
Situation awareness (SA) is a measure of an individual's knowledge and understanding of the current and expected future states of a situation. While there are numerous options for SA measurement, none are currently suitable in dynamic, uncontrolled environments. Direct measures of SA are the most common, but require a large amount of researcher control as well as the ability to stop operators during a task in order to ask questions about their levels of SA. The current research explored the relationship between direct measures of SA and eye tracking measures as a first step in the development of an unobtrusive SA measure to be used in less controllable, dynamic environments. Two studies compared participant eye movements and SA in driving and air traffic control scenarios. Both studies showed that the more individuals fixated on an important, task-relevant event, the higher their SA for that event. The studies also provide evidence that the way operators allocate attention (i.e., distributed widely or narrowly) affects their SA as well as their task performance. In addition, study 2 results showed positive correlations between SA and task performance. The results indicate that eye tracking may be a viable option for measuring SA in environments not conducive to current direct SA measurement techniques. Future research should continue to explore which eye movement variables best predict participant SA, as well as to investigate the relationship between attention allocation and SA.

TABLE OF CONTENTS

Page

TITLE PAGE ............................................................................................................... i

ABSTRACT ................................................................................................................ ii

DEDICATION ........................................................................................................... iii

ACKNOWLEDGMENTS .......................................................................................... iv

LIST OF TABLES .................................................................................................... vii

LIST OF FIGURES .................................................................................................. viii

CHAPTER

I. INTRODUCTION ..................................................................................... 1

Perceptual and cognitive processes and structures in SA ....................... 5 Effects of SA on performance .............................................................12 Situation awareness measurement techniques ......................................13 Goals of the current study ....................................................................17 Use of direct query measures for SA measurement ..............................18 Eye tracking research ..........................................................................20 Eye tracking studies ............................................................................22 Rationale for the current research ........................................................29 Rationale for Study 1 ...........................................................................31

II. STUDY 1 ..................................................................................................33

Method ................................................................................................33 Participants ....................................................................................33 Apparatus ......................................................................................33 Design ...........................................................................................35 Materials and tasks ........................................................................35 Procedure ......................................................................................38 Results and Discussion ........................................................................38 Data collection and preliminary analysis ........................................38 Data analysis plan ..........................................................................40 Generalized estimated equations analysis.......................................42 Percent time fixating on events ......................................................46

vi

Table of Contents (Continued) Page

Number of fixations during an event ..............................................48 Mean fixation duration during an event..........................................50 Study 1 general discussion .............................................................52

III. STUDY 2 ..................................................................................................53

Method ................................................................................................60 Participants ....................................................................................60 Materials .......................................................................................61 Design ...........................................................................................65 Procedure ......................................................................................66 Results and Discussion ........................................................................67 Data collection and scoring of eye movement variables .................67 SA scoring .....................................................................................71 Determining final eye movement predictor variables .....................72 Scoring of ATC performance variables ..........................................73 Overall descriptive statistics ..........................................................74 Effects of eye movements on SA ...................................................77 Effects of eye movements on performance .....................................87 Effects of SA on performance ........................................................93 Study 2 general discussion .............................................................95 Separation Conflict Case Studies ....................................................... 100

IV. GENERAL DISCUSSION ...................................................................... 117

APPENDICES ......................................................................................................... 127

A: Situation Awareness Queries ................................................................... 128 B: Demographic Questionnaire .................................................................... 130

REFERENCES......................................................................................................... 131

vii

LIST OF TABLES

Table Page

1.1 Frequency and percentage of errors leading to accidents between 1989 and 1992 .......................................................12

1.2 Common eye tracking metrics and their relationship to performance ....................................................................................21

2.1 Name and description of each event ..........................................................36

2.2 Percent correct on SA queries ...................................................................43

2.3 Percent time fixating events for correct and incorrect SA responses .......................................................................................46

2.4 Number of fixations during an event for correct and incorrect SA responses ........................................................................49

2.5 Mean fixation duration during an event for accurate and inaccurate SA ...............................................................................51

3.1 Descriptive statistics for SA, performance, and eye Movement variables for each participant .............................................75

3.2 Predictor and dependent variable descriptive statistics for aircraft and scenario for analysis of effect of eye movement variables on SA ...................................................................80

3.3 Mixed model results for eye movement predictor variables and SA dependent variables.................................................................82

3.4 Eye movement predictor variables and performance dependent variables descriptive statistics ............................................88

3.5 Mixed model results for eye movement predictor variables and performance dependent variables .................................................89

3.6 Mixed model results for SA predictor variables and performance dependent variables ........................................................94

viii

LIST OF FIGURES

Figure Page

1.1 Endsley’s model of situation awareness...................................................... 7

2.1 The Tobii 1750 Eye Tracker ......................................................................34

2.2 Screenshot of the monitor for a scenario containing four cars ..............................................................................................35

2.3 Screenshot of a question and response map presented after the completion of a scenario ........................................................37

2.4 The percent correct for SA accuracy by group and number of cars.....................................................................................44

3.1 Screenshot of the TRACON II ATC Simulator ..........................................62

3.2 Screenshots illustrating how AOIs were defined ........................................69

3.3 Lowest and highest participant NNI values ...............................................76

3.4 Standard deviation between percent fixations upon individual aircraft by SA future queries percent correct .......................86

3.5a Percent time fixating on aircraft AOIs by number of actions remaining ............................................................................90

3.5b Mean duration of fixations on aircraft AOIs by number of actions remaining ...............................................................90

3.6a Total number of fixations before the query break by sum of errors in the scenario .......................................................................91

3.6b NNI smallest rectangle value calculated using fixations up to break by sum of errors in the scenario .........................................91

3.7 Percent of time fixating on airports and relevant fixes by Number of actions remaining at the end of the scenario .......................92

3.8 Current SA percent correct by sum of errors in the scenario ......................95

ix

List of Figures (Continued)

Figure Page

3.9 Preventive planning Example 1 ............................................................... 103

3.10 Preventive planning Example 2 ............................................................... 104

3.11 Conflict recognition Example 1 ............................................................... 105

3.12 Conflict recognition Example 2 ............................................................... 106

3.13 Separation conflict Example 1 ................................................................. 107

3.14 Separation conflict Example 2 ................................................................. 108

1

CHAPTER ONE

INTRODUCTION

Whenever a task is performed, no matter how small, a person must coordinate a myriad of cognitive and physical processes. Consider, for example, a person simply cleaning his kitchen. He must know how to clean, what type of cleaning products to use, where he should clean, which areas have already been cleaned, and what is left to be cleaned. The cognitive processes involved in the act of cleaning alone involve long term memory for what types of products to use, short term memory for what surfaces have already been cleaned and are yet to be cleaned, attention to continue the cleaning process, and so on. Other, more multifaceted tasks require a more complex set of mental processes with higher consequences for errors. For example, a pilot of a commercial aircraft must use short and long term memory, attention and decision making to safely navigate the aircraft from take-off to landing. While all of these constructs are important, they are not the only processes involved when completing dynamic tasks. A mistake during the flight could cause injuries or deaths, therefore it is important for researchers to have an intricate understanding of what processes are involved and how errors occur. In addition to the other processes involved, situation awareness (SA) is one construct that has consistently been correlated with performance on a variety of tasks in various domains. The current study explores the construct of SA and its measurement in two task domains. The introduction will survey the research on SA and its components, as well as current measurement methods. Physiological measures are rarely used to measure SA;

2

the current research examines the relationship between eye tracking and direct measures of SA to determine if eye tracking is a viable measurement option when other options are not. Two studies will compare different eye movement measures and direct SA measures in both driving and air traffic control scenarios. Research examining both the construct of SA and methods of measurement follows. SA has been a topic of interest since World War I (Press, 1986; as cited by Endsley, 1995c), but only in the past three decades has it been extensively researched. Many researchers have operationally defined and measured SA, but further discussion is needed to better understand its meaning. From a global perspective, SA is attending to and understanding what is occurring in the environment immediately surrounding an individual during a dynamic (i.e., changing) situation. Clearly, this is ambiguous and in need of further clarification. Although the construct had been implied previously, a widely accepted, formal definition of SA was not introduced until 1987. Endsley defines SA as, “The perception of the elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future” (Endsley, 1987, 1988, 1995c). While this is the most cited definition of SA, there are several other viable interpretations (e.g., Adams, Tenney, & Pew, 1995; Smith & Hancock, 1995). The definition is still not without disagreement, but it is important to first understand why the construct of SA is even a relevant component of performance. One way to illustrate the importance of SA is to describe situations where a loss of SA had negative consequences. A simple example of loss of SA is when an outfielder catches a fly ball but fails to throw a runner out because he does not realize it is not the

3

last out of the inning (Tenney & Pew, 2006). Another, more severe example is the death of almost 5000 people between 1978 and 1992 from airplane accidents due to controlled flight into terrain. A lack of SA was determined to be the cause of 74% of those accidents (Woodhouse & Woodhouse, 1995; from Durso & Gronlund, 1999). In general, having an understanding of the past, present, and future components of a situation should lead to better performance, with a loss of this understanding potentially resulting in devastating consequences in high risk tasks. Even if the outcome is not catastrophic, costly errors may result from a loss of SA. Researchers continue to operationally define and measure the SA construct with the ultimate goal of designing interfaces and implementing training procedures that will increase operator SA and reduce human error. Situation awareness research has been conducted in a variety of real-time, dynamic domains including air traffic control (ATC) (e.g., Endsley & Smolensky, 1998; Durso, Truitt, Hackworth, Crutchfield, & Manning, 1998b), aviation (e.g., Kaber, Endsley, Wright, & Warren, 2002), anesthesiology (e.g., Gaba, Howard, & Small, 1995), nuclear power plants (e.g., Hogg, Folleso, Strand-Volden, & Torralba, 1995), driving (e.g., Gugerty, 1997), military command and control (e.g., Gorman, Cooke, & Winner, 2006; Salmon et al., 2007; Stanton et al., 2006), and even football (e.g., Walker & Fisk, 1995). Endsley (1987, 1988, 1995c) distinguishes between three levels of SA: Level 1 – perception, Level 2 – comprehension, and Level 3 – projection. An example of the three levels of SA from ATC would be a controller perceiving the number of aircraft in a particular airspace on the radar screen (Level 1), integrating information about an aircraft’s heading, altitude and airspeed in order to comprehend that it is beginning its

4

arrival approach (Level 2), and projecting how long it will take the aircraft to reach its destination (Level 3). Though Endsley’s definition may be the most widely accepted, it is by no means the only definition of SA. Additionally, her definition is not complete, as it is difficult to define in detail a complex construct. In that sense, SA is similar to mental constructs such as attention, memory, and consciousness. All are complex, difficult to define wholly, not directly observable and not without disagreement among experts. Some researchers question whether SA should even be considered a psychological construct, separate from other clearly defined constructs (Crane, 1992; Dekker & Hollnagel, 2004). To those that criticize, the continued use and application of SA is a testament to its importance beyond already existing constructs (Wickens, 2008). Though most agree SA is a construct separate from others, researchers continue to debate the definition, and in turn, the processes which affect development and maintenance of SA. Two frameworks, the information processing approach and the ecological view, are typically the basis of theories of SA. Endsley’s definition of SA is based on the information processing approach, where SA is viewed as a product of a number of cognitive processes (Durso & Gronlund, 1999). Flach (1995) and others (Smith & Hancock, 1995; Adams, Tenney & Pew, 1995) advocate a more holistic, ecological approach to situation awareness, one that is based upon the perception-action cycle (Neisser, 1976). The ecological view defines SA as both a product and a process of the perception-action cycle (Durso & Gronlund, 1999). In Smith and Hancock’s (1995) ecological view, SA is defined as “adaptive, externally directed consciousness” and is “directly related to stress, mental workload, and other energetic constructs that are facets

5

of consciousness” (1995, pg. 138). Even though the theoretical framework of SA continues to be debated, specifically what components and processes should or should not be included in the definition, the processes which make up the information processing approach have been studied in a variety of task domains and add to the understanding of SA. Perceptual and cognitive processes and structures in SA The information-processing approach describes behavior and cognition underlying behavior in terms of processes (such as attention, comprehension, or memory retrieval) and the states of knowledge produced by these processes (such as a consciously recognized object or a retrieved memory). Applying this general approach to the dynamic situations addressed by SA, SA is viewed as knowledge of the current and expected future states of a situation (SA as knowledge or product) and is comprised of set of attentional and comprehension processes that gather, interpret and update this knowledge. Previous experiences and training, among other things, will affect knowledge of the current situation as well as what is expected to occur in the future. This view of SA as both processes and knowledge produced by these processes is exemplified in the following description: “By defining SA as a generative process of knowledge creation and informed action taking, we expressly deny that SA is merely a snapshot of the agent’s current mental model. Rather, SA guides the process of modifying knowledge – that is, of constructing a representation of current and likely events” (Smith & Hancock, 1995, pg. 142).

6

However, Endsley (1995c) and others are careful to point out that SA does not involve processes underlying decision making and response execution; and although it may be influenced by constructs such as workload, working memory and attention, it is independent from them. Endsley (1995c) explains that if these constructs become a part of the definition of SA, its independence will be lost. One study found no relationship between mental workload and SA in a review of 23 experiments (Vidulich, 2000). After dividing the studies by interface manipulation type, varied results were found. When researchers added information to an interface to improve SA, the resulting mental workload scores were mixed. When researchers simply rearranged the available information, a majority of the studies found an increase in SA and a decrease in mental workload. Thus, in certain circumstances mental workload and SA may co-vary, but little consistency between the two constructs has been found. Endsley’s (1987, 1988, 1995c) high-level model of how SA fits into the stages of information processing is illustrated in Figure 1.1.

7

Figure 1.1. Endsley’s model of situation awareness (adapted from Endsley, 1995c).

The model illustrates that multiple components are involved in the development and maintenance of SA. Even when people experience the same situation in the same environmental conditions, individual differences will likely lead to varying levels of SA due to variations in ability, experience and training. In addition, each of their specific goals and expectations will affect their perceptions. System factors also affect SA; if the system does not provide all of the necessary information for complete understanding of the environment, an individual is not going to be able to achieve higher levels of SA, regardless of other factors. Finally, environmental factors, such as varying levels of stress, will affect SA in different ways (Endsley, 1995c). Recall that there are three levels of SA as described by Endsley. Level 1 SA is defined as the perception of elements in the environment and can be thought of as

8

analogous to “word-level information prior to combining the words into phrases” (Durso & Gronlund, 1999, pg. 291). Level 2 SA is defined as the comprehension of the current situation. Comprehension occurs through the synthesis of the elements perceived in Level 1. Level 2 reflects the idea that the outcome of many perceptual processes is the recognition or comprehension of a meaningful object or event. Level 3 SA is defined as the projection of future status, and reflects the fact that the meaning of many dynamic events cannot be comprehended without anticipating how these events will play out in the near future. The three levels of SA in Endsley’s definition are very broad in describing the high-level processes of perception, comprehension, and projection underlying SA. Several researchers have studied these processes in more detail in a variety of research domains. Perception and comprehension can be examined by considering what leads one to perceive and comprehend. Both the SEEV model of attention allocation and the Construction Integration model explore the components of perception and comprehension in more detail. The SEEV model of how focal attention is allocated in real-time tasks is made up of four elements comprising the acronym SEEV – Salient events, Effort, Expectancy, and the Value of events (Wickens et al., 2005). The SEEV model includes both bottom-up and top-down processes. The salience of events in the environment is determined by their ability to capture attention in a bottom-up fashion. Effort (E), Expectancy (E) and Value (V) are top-down processes determined by operator understanding of the situation and previous experience among other factors. Effort refers to the physical difficulty of

9

shifting attention to an object, e.g., the length of a saccade or head movement. Expectancy is proportional to how frequently information about an object is changing. As the frequency of information change increases, an operator will sample the object more often to attempt to avoid missing relevant information. Value refers to the priority or importance of an object. As the value increases, again the sampling should increase due to the higher importance level. The SEEV model predicts that people will allocate more attention to salient, high-value objects that are changing rapidly and that are easy to attend to. The model has been partially validated by empirical studies of driving whose results show that as the value and the rate of information change of objects increases, people allocate more attention to those objects (Horrey, Wickens & Consalus, 2006). The construction-integration model was developed to better understand discourse comprehension (Kintsch, 1988, 2005). While the model has been primarily applied to how discourse is comprehended, it is also applicable to how information in dynamic environments is comprehended (Durso, Rawson & Girotto, 2007). In Kintsch’s (1988) view, comprehension of words and sentences begins as a bottom-up process; the context is not considered until later stages. In the first stage, the sense-selection stage, when a word is read a network of propositions and connections are formed without consideration of the context. In this stage, understanding begins by rapidly reducing the number of potential word meanings to a manageable number; the potential meanings are initially selected based on a context-free approach to the meaning of the particular sentence component. In the second stage, which involves top-down processing, associations with the context (e.g., nearby words) helps reduce the number of potential meanings further.

10

This is the sense-elaboration stage. In the final stage, further understanding occurs based upon the long-term memory knowledge-base of the operator (further top-down processing). Someone with robust knowledge will likely obtain a quicker and more sound understanding as a situation progresses. The construction-integration model is aptly named because comprehension is made up of the integration of an understanding of word meaning constructed from what is in the environment (bottom-up) as well as what the operator already knows (top-down) (Kintsch, 1988). It is important to understand that this process is cyclical due to the limited cognitive capacity of humans. In terms of text comprehension, cycles are typically at the sentence level; integration occurs when nodes from one cycle are carried over and integrated into the next (Durso et al., 2007; Kintsch, 1988) Durso et al. (2007) point out that the construction-integration model is analogous to the way operators develops SA over time through the bottom-up process of perception of information in the environment as well as the top-down processes of developing a situational model using environmental context and their own knowledge base. Operators must develop an eventbase in order to construct a representation of the environment around them (Durso et al., 2007). If SA develops in the same way as discourse comprehension, eventbase development begins through a strictly bottom-up process, similar to the salience component of the SEEV model. The integration of elements obtained from Level 1 perception would be initially context-free, with operator knowledge aiding in the winnowing out and eventual selection of an event meaning (i.e., comprehension). The need for context to guide comprehension may partially explain

11

why SA must be built up over time and is not instantly obtained. Top-down processing is needed to suppress irrelevant information and only allow the appropriate meaning in a specific context to appear. By looking at the process of developing SA from the perspective of the SEEV model and the Construction-Integration model, the following view of SA emerges. SA will be improved to the extent that operators use cues like task-priority and rate of information change to guide their attention allocation to dynamic events as these events change over space and time. Then once a high-priority event is focused on, SA will be improved to the extent that operators’ comprehension process allows quick and accurate comprehension of this event. Turning from the processes used to maintain SA to the cognitive structures underlying SA, the product of comprehension is commonly thought to be stored and updated in a situation model residing in working memory. It is easiest to understand situation models in the context of text comprehension, which is made up of both situation models and textbase. The textbase consists of the elements that allow an individual to have a word-level understanding of the text, or understanding simply the words without any additional inputs. The situation model of an individual is necessary to interpret and have a higher understanding of the words and their relationship to one another to form meaning and make the text coherent. The components involved in a situation model include an understanding of the language, knowledge of the world, and past experiences of the individual (Kintsch, 1998). As pointed out by Durso et al. (2007), the construct of

12

a situation model should translate well from comprehending text to comprehending real- time situations. Effects of SA on Performance While it is important to understand the theoretical underpinnings of SA, it is equally important to understand how not obtaining higher SA or losing it once obtained can affect performance. Endsley (1995a) reviewed 24 accident reports from the National Transportation Safety Board (NTSB) from 1989 – 1992. Of the 24 accidents, it was determined that 17 were the result of human error with 15 of those related to SA. A further analysis of the accidents involving SA revealed that there were 32 SA errors (several accidents involved more than one error). From these reports, a taxonomy of errors was developed, with the number of recorded errors for each failure listed in Table 1.1 below (From Endsley, 1995a). Table 1.1 Frequency and percentage of errors leading to accidents between 1989 and 1992

Frequency Percentage Level 1: Failure to correctly perceive information

23

71.9

• Data not available • Data difficult to detect or perceive • Failure to monitor or observe data • Misperception of data •

Memory failure

3

5 10 4 1

9.4

15.6 31.3 12.5 3.1 Level 2: Failure to comprehend situation

7

21.9

Lack of or poor mental model

• Use of incorrect mental model • Over-reliance on default values in mental model • Other 1

2 1 3 3.1

6.3 3.1 9.4 Level 3: Failure to project situation into the future

2

6.3

• Lack of or poor mental model •

Overprojection of current trends

1

1 3.1

3.1

13

One of the key indicators of SA while driving involves hazard perception (Horswill & McKenna, 2004). In driving tasks, hazard perception is the only skill that has correlated with performance across numerous studies. A review of the literature on hazard perception and performance revealed that hazard perception ability is a good predictor of on-road crashes. In a large scale study of 100,000 drivers that measured the predictability of a hazard perception test, Hull and Christie (1992) found that drivers who scored low on the test were twice as likely as those who scored high to be involved in a fatal accident within one year (Horswill & McKenna, 2004). In driving research, SA (measured by hazard perception) has continually been positively correlated with good driving performance. Even though one might assume that high levels of SA would be equated with higher performance levels, this is not always the case. Instead, SA should be viewed as a factor that affects performance, with high SA typically, though not always, leading to high levels of performance (Endsley, 1995c). A high level of SA can occur during low levels of performance and vice versa. For example, a novice system operator may be aware of a problem but may not have the expertise to solve it before an error occurs. Also, with high levels of automation, system performance may be high even if an operator experiences a loss of SA. Situation awareness measurement techniques There are a large variety of measurement techniques that have been employed to determine an individual’s level of SA. Three types of methods are typically discussed: subjective measures, implicit measures, and explicit (direct) measures (Sarter & Woods,

14

1995). A fourth method of measurement that has received considerably less attention, but warrants further investigation, is the use of physiological measures. Each method has several advantages and disadvantages; currently no method is clearly superior to the others. Subjective measures (including self- and observer-rating techniques) simply determine an operator’s SA by asking the operator after the task is completed or by having a subject matter expert (SME) observe the operator and rate his SA. The most common subjective SA measure is the Situation Awareness Rating Technique (SART) (Taylor, 1990). Subjective measures, such as the SART, are favorable because they are relatively easy to implement and do not require a large amount of preparation beforehand. In addition, they can be used in dynamic, field-based research. There are several drawbacks to subjective measures; the main one being that studies have shown that SART neither correlates with performance or other measures of SA (Endsley, 1995b; Salmon et al. 2008a). Other issues with subjective ratings include the possibility that participants’ task performance may affect SA ratings afterward. Participants may, for example, take the result of the task (i.e., pass or fail) and rate their SA based on their performance. Additionally, participants may not have an understanding of what their true SA is, believing that they were very aware when in fact they missed pertinent information in the environment. Observer ratings (typically from SMEs) are also not ideal because SA is an internal construct, making it inherently difficult to observe in others (Endsley, 1995b; Salmon, Stanton, Walker, & Green, 2006; Sarter & Woods, 1995).

Full document contains 147 pages
Abstract: Situation awareness (SA) is a measure of an individual's knowledge and understanding of the current and expected future states of a situation. While there are numerous options for SA measurement, none are currently suitable in dynamic, uncontrolled environments. Direct measures of SA are the most common, but require a large amount of researcher control as well as the ability to stop operators during a task in order to ask questions about their levels of SA. The current research explored the relationship between direct measures of SA and eye tracking measures as a first step in the development of an unobtrusive SA measure to be used in less controllable, dynamic environments. Two studies compared participant eye movements and SA in driving and air traffic control scenarios. Both studies showed that the more individuals fixated on an important, task-relevant event, the higher their SA for that event. The studies also provide evidence that the way operators allocate attention (i.e., distributed widely or narrowly) affects their SA as well as their task performance. In addition, study 2 results showed positive correlations between SA and task performance. The results indicate that eye tracking may be a viable option for measuring SA in environments not conducive to current direct SA measurement techniques. Future research should continue to explore which eye movement variables best predict participant SA, as well as to investigate the relationship between attention allocation and SA.