Impact of system development methodology use on employee role ambiguity and role conflict
v Table of Contents Acknowledgments iv List of Tables viii List of Figures ix CHAPTER 1. INTRODUCTION 1 Introduction to the Problem 1 Background of the Study 3 Statement of the Problem 3 Purpose of the Study 4 Rationale 5 Research Questions 5 Significance of the Study 6 Definition of Terms 8 Assumptions and Limitations 9 Nature of the Study 10 Organization of the Remainder of the Study 11 CHAPTER 2. LITERATURE REVIEW 12 Role Ambiguity and Role Conflict 13 Introduction to Role Ambiguity and Role Conflict 13 Impact of RA and RC 15 Survey Instruments for Measuring Role Ambiguity and Role Conflict 21 System Development Methodologies 27 Debate over SDM Usefulness 29
vi Extending the Research on SDM Efficacy 32 Connecting Role Ambiguity, Role Conflict, and SDM Use 36 Development Methodologies as a Source of Structure 36 Importance of RA and RC for Information System Professionals 38 Conclusion 40 CHAPTER 3. METHODOLOGY 42 Research Design 43 Sample 44 Instrumentation and Measures 45 Data Collection 48 Data Analysis 48 Validity and Reliability 50 Ethical Considerations 51 Conclusion 52 CHAPTER 4. RESULTS 54 Data Collection and Demographics 54 Survey Reliability 56 Analysis of Hypotheses 57 Conclusion 68 CHAPTER 5. DISCUSSION, IMPLICATIONS, RECOMMENDATIONS 70 Research Synopsis 70 Discussion of Results 72 Study Limitations 76
vii Recommendation for Future Research 77 Conclusion 79 REFERENCES 82 APPENDIX. MODIFIED RHL SURVEY INSTRUMENT 89
viii List of Tables Table 1. Hypotheses Effect Size and Power 55 Table 2. Demographic Information 55 Table 3: RA/RC One-Sample Kolmogorov-Smirnov Test 58 Table 4. Independent-Samples t-test 59 Table 5. SDM Duration One-Sample Kolmogorov-Smirnov Test 61 Table 6. SDM Duration and RA Test for Correlations 63 Table 7. SDM Duration and RC Test for Correlations 63 Table 8. Employee Experience One-Sample Kolmogorov-Smirnov Test 64 Table 9. Employee Experience and RA Test for Correlations 66 Table 10. Employee Experience and RC Test for Correlations 66 Table 11. Tests of Between-Subjects Effects for RA and SDM Use 67 Table 12. Tests of Between-Subjects Effects for RA and SDM Use 68
ix List of Figures Figure 1. Research model with two dependent variables 43 Figure 2. Boxplot of RA levels and SDM use 58 Figure 3. Boxplot of RC levels and SDM use 60 Figure 4. Scatter diagram of RA and SDM Duration 62 Figure 5. Scatter diagram of RC and SDM Duration 62 Figure 6. Scatter diagram of RA and Employee Experience 65 Figure 7. Scatter diagram of RC and Employee Experience 65
CHAPTER 1. INTRODUCTION
Introduction to the Problem During the last twelve years, several surveys have been conducted on the success rate of information system (IS) development projects, and the results have been alarming. In general, these reports show that between 15% and 30% of projects were canceled, and of those that were completed, 50% to 70% were drastically over budget or failed to meet end-user expectations (Ewusi-Mensah, 1997; Jiang, Klein, Hwang, Huang, & Hung, 2004; Kappelman, McKeeman, & Zhang, 2006; Nidumolu, 1995). Research in the IS field has progressed rapidly over the last decade, but the high failure rate of IS projects appears to be unaffected. Perhaps one reason for this dichotomy was that, as Kautz, Madsen, and Nørbjerg (2007) suggested, IS researchers tend to “concentrate only on the unique features and apparent newness of certain IS phenomena, such as all Web-related matters, while disregarding fundamental concerns and challenges (still) at the core of the field” (p. 218). It was for this reason, and the apparent dearth of research in this particular field of study, that it was worthwhile to return to a concept that has long been taken for granted: the benefits of employing a system development methodology in IS projects. Textbooks on systems analysis, design, and development often include discussions of the importance of using system development methodologies (SDM) but often fail to provide any empirical evidence to support this assertion (Hoffer, George, &
2 Valacich, 2005; Laudon & Laudon, 2004; Whitten, Bentley, & Dittman, 2004). Peer reviewed journal articles were not exempt from this assumption as evidenced by the introduction to the subject given by Kautz et al. (2007). Despite the nearly ubiquitous nature of this assumption, the growing body of research on SDM effectiveness indicated that the benefits of employing an SDM were mixed at best. To date, many of the studies on SDM use have focused on the benefits to the system being built such as the systems quality, performance, or maintainability aspects (Ashrafi, 2003; Dekleva, 1992; Jiang et al., 2004). While this research was valuable, it overlooked a more fundamental aspect of SDM use: how it affects those employees that have adopted it. In his article, Fitzgerald (1996) highlighted a significant problem with the IS research on SDMs when he stated that “rational motives are the ones assumed in many methodologies, and consequently they do not cope well with social and human factors” (p. 14). This was also evident in the focus of SDM research noted above, where the outcome of the system was of primary importance and the actual developers were all but ignored. In contrast, this study examined how SDM use influenced the developers. In particular, this research focused on what effect, if any, SDM use had on a system developer’s perceived role conflict and ambiguity; two factors long researched in organizational behavior literature, but conspicuously lacking in IS research. A problem central to IS research, according to Kautz et al. (2007), was that core concepts were not fully understood before the research focus shifted to a new territory. A specific instance of this could be seen in the abundant research on the adoption of SDMs within organizations (Fitzgerald, 1998; Hardgrave, Davis, & Riemenschneider, 2003; Moore & Benbasat, 1991; Mustonen-Ollila & Lyytinen, 2003; Sauer & Lau, 1997; Woo,
3 Mikusauskas, Bartlett, & Law, 2006), while at the same time very little was known about how the implementation of SDMs influenced people within the organization. In other words, a great deal of information exists in regards to the best practices for employing an SDM, but the impact this may have on employees remained unclear. In an attempt to overcome this problem area in SDM research, this study focused on providing insight into the relationship between SDM use and an employee’s sense of role conflict and ambiguity.
Background of the Study Information System (IS) development projects were found in virtually all industries in the United States. Furthermore, these projects were not confined to specific geographic areas. Considering the varied nature of IS projects, both in location and purpose; it was important that this study was as representative of IS projects as possible and not limited to any industry or geographic area within the U.S.A. Following in the footsteps of other researchers such as Jiang et al. (2004), this study gathered information from current and previous members of IS project teams based on their enrollment in the IEEE Computer Society. The assumption was that members of this organization were representative of the national community of IS professionals.
Statement of the Problem In research that evaluates SDM effectiveness, the focus was invariably on how SDM use affected the quality of the finished system or the success of the project. The results of this research often called into question the efficacy of SDM use, as evidenced
4 by Fitzgerald’s (1996) critique in which he suggested that formalized development methodologies have consistently failed to provide the benefits they were assumed to bestow on IS projects. The problem, however, was that there has not been enough research on the effect of SDM use on the development team. Avison, Fitzgerald, and Powell (2001) astutely noted that “research should be used to re-examine common ‘wisdom’ as well as make new discoveries” (p. 9). A current problem in IS research was the apparent assumption that the benefits of a formalized methodology should be evaluated in the context of its direct benefit to the project or system. But this assumption excludes the most important component of any project: the team members. In order to rectify this problem, researchers need to consider SDM use from a human perspective rather than an explicitly system oriented perspective.
Purpose of the Study The purpose of this research was to examine the relationship, if it existed, between SDM use and perceived RA and RC among employees working on IS projects. Rather than focusing on how SDMs affected project success or system quality, this study examined how they affected project team members. This was an overly broad topic for any single study, and required focusing in on a specific aspect of the developers’ employment experience. For this study, the focus was the employees’ perception of role ambiguity (RA) and role complexity (RC) within the context of their IS project team. Correlation analysis of survey data was conducted in order to ascertain whether or not a relationship existed between either RA or RC and the use of an SDM. In their article, Jiang et al. (2004) pointed out that there appeared to be a correlation between the
5 duration of SDM use and its benefit to the project team. This suggested that duration should also be considered when evaluating the correlation between SDM and RA or RC.
Rationale A significant reason for engaging in this research was to return to a defining theme of IS development and to examine it from an alternate perspective. Various researchers in this field of study have noted the importance of re-evaluating current assumptions and seeking a deeper knowledge of the core aspects of the IS profession that have been forsaken in the pursuit of new fields of study (Avison et al., 2001; Kautz et al., 2007). By examining the use of SDMs from a behavioral perspective, it was expected that new light would be shown upon a topic that had been taken for granted. While there was certainly room for debate as to the usefulness of SDMs on project success, there was very little actual debate as to how implementing an SDM affected the project team members’ sense of job satisfaction, stress levels, or other characteristics. While this lack of knowledge left a significant range of characteristics to choose from, the focus of this research was on assessing the impact SDM use had on two specific employee related factors: role ambiguity and role conflict.
Research Questions For this study, there were four primary research questions. First, was there a correlation between role conflict and SDM use? Second, was there a correlation between role ambiguity and SDM use? Third, did the effect of SDM use on role conflict and role ambiguity vary by the duration that the SDM has been used within the project team?
6 Fourth, did the effect of SDM use on role conflict and role ambiguity vary by the respondent’s level of experience working on IS projects? To answer these questions, the following alternative hypotheses were tested during the course of this research: H1: There is a difference in employee perceptions of his level of role ambiguity (RA) between SDM users and non-SDM users.
H2: There is a difference in employee perceptions of his level of role conflict (RC) between SDM users and non-SDM users.
H3: The length of time an SDM has been in use within the organization is correlated with perceived levels of RA and RC
H4: Employee experience in the field of IS development is correlated with RA and RC
H5: When employee experience is controlled for, the correlation between SDM use and RA/RC will be more pronounced than if this factor was not included in the analysis
Additionally, the results of this study were compared to a survey conducted by Fitzgerald (1998). Fitzgerald’s study focused on “the extent to which systems development methodologies are used in practice” (p. 317) and found that SDM use was surprisingly low. Comparing the results of the current study to those gathered by Fitzgerald over ten years ago have made it possible to identify if there has been a cumulative trend to engage in SDM use or not.
Significance of the Study The need for exploring the human dimension of SDM use was explicated in some detail, but the reason for choosing RA and RC for this purpose requires further elaboration. Both role ambiguity and role conflict were discussed in great detail in the
7 literature review, but a brief overview of these two variables served to illustrate the significance of this study. Over thirty years ago, role theory was being discussed by researchers, especially in regards to its impact on organizations. In particular, Rizzo, House, and Lirtzman (1970) speculated on two key components of role theory and how they could lead to changes in an employee’s job satisfaction and effectiveness. These components, RA and RC, were seen as important factors in enhancing or decreasing an employee’s perception of her job and, subsequently, how well she performed it. In their meta-analysis, Tubre and Collins (2000) reevaluated the research conducted on RA and RC in order to determine the effect these variables had on job performance. The primary finding of this study, a “negative relationship… between role ambiguity and job performance” (p. 155), was also consistently reported in other studies. In contrast, role conflict was found to have a significantly weaker effect on job performance, but the effect was slightly increased “for professional, technical, and managerial jobs” (p. 164) which was particularly relevant to the current study. Although SDM use was not consistently linked to increased project completion rates or system quality, it was worthwhile to know if they could lead to increased performance from employees by reducing role ambiguity and, to a lesser degree, role conflict. Another reason for considering RA and RC in this study was that both variables were routinely included in research on burnout and turnover rates among employees. According to a critical appraisal conducted by Maudgalya, Wallace, Daraiseh, and Salem (2006), a significant proportion of research conducted on burnout among information technology (IT) professionals identified both RA and RC as key factors. The conclusion presented by the authors was that both RA and RC should be kept to a minimum in
8 organizations in order to reduce employee turnover rates. This was especially important in the IT industry where burnout rates (and the cost of replacing employees) could be quite high (Maudgalya et al., 2006). To date, the focus of SDM research was on how SDM use directly affected IS projects, but determining if SDMs provided indirect benefits would be very beneficial. If a relationship did exist between SDM use and RA or RC, then it could be worthwhile to employ an SDM as a proactive step in reducing RA and RC. Conversely, if SDM use was shown to increase either RA or RC, then it would be one more reason to reconsider their use.
Definition of Terms Following is a list of terms commonly encountered in IS and Role Theory research. A brief definition of each was given for each term. Job Role. Behaviors and activities expected from an employee occupying a specific position within an organization were said to be within that employee’s job role (Tubre & Collins, 2000). This term was often shortened to “role” in this study. Role Ambiguity (RA). This was the degree to which an employee feels he/she had enough information about the job to perform it correctly (Rizzo et al., 1970). Role Conflict (RC). This was the degree to which an employee’s understanding of her job role differed from the employers expectations. The greater the gap between an employee’s understanding and the employer’s expectations, the greater the conflict between the two would be (Tubre & Collins, 2000). Software Development Process. This process consists of tasks that were engaged in during the course of developing software (Deephouse, Mukhopadhyay, Goldenson, &
9 Kellner, 1995). These tasks tend to be arranged in an order that was established within the organization. Systems Development Methodology. A plan for decomposing the development effort into manageable tasks and activities arranged in a prescribed order with the intent of making it possible to consistently develop information systems (Duggan, 2004; Fitzgerald, 1996). This term was often used interchangeably with Software Development Methodology depending on the project being described. When the project focused exclusively on developing software, the latter term was used. In contrast, if the project involved hardware or human components as well as software components, then the former term was used. Throughout this study, the abbreviation “SDM” was used explicitly to denote systems development methodologies.
Assumptions and Limitations In choosing a sample frame based on a professional organization such as the IEEE Computer Society, an assumption was made that the members accurately represented the larger community of IS/IT professionals. Although this approach was not ideal, it was fairly common in IS research, such as that conducted by Jiang et al. (2004), due to the fact that finding participants for this type of research was notoriously difficult. Other sampling techniques, such as snowballing, were often used in IS research, but these tended to come with significant limitations of their own. In essence, using membership in a professional organization as a criterion for participating in the study provides a mechanism for reaching the appropriate type of people for the study. It was assumed that
10 this sample frame provided access to a broad enough range of IS professionals in order to allow the results to be generalizable to the population. One limitation of this study was due to the fact that it employed a quasi- experimental design where the participants were segregated into two different groups based on their use of an SDM (Cooper & Schindler, 2007). There may be a fundamental difference between organizations that employed an SDM and those that did not, and this could be a confounding variable in the proposed research. It was also important to keep in mind that this study asked participants to consider a previous experience working on an IS project and report on it. As Russ-Eft and Hoover (2005) pointed out, “[t]his design depend[ed] on the accuracy of participants’ recall, as well as their willingness to provide ‘truthful’ data” (p. 83). Bias could have been introduced if respondents tended to exaggerate, either positively or negatively, their previous experiences with IS development.
Nature of the Study This study was an empirical analysis of the relationship between role ambiguity, role conflict, and SDM use. In addition, the length of time an SDM was used within the organization, as well as the respondents’ experience level, were considered as possible mitigating factors. Primary data for this study was gathered through the use of a traditional mail survey with the sample frame consisting of IS workers associated with the IEEE Computer Society. As for the survey instrument, a review of the available literature indicated that there was significant debate about the validity and reliability of several of the more prominent RA/RC surveys. However, due to its ubiquity and recent
11 assessments of this instruments usefulness, the Rizzo, House, and Lirtzman (RHL) survey was employed in this study.
Organization of the Remainder of the Study In the following chapter, many of the topics discussed here were considered in greater detail as part of the literature review. The literature review established the foundation for this research, highlighted its benefits, and further elaborated on the potential connection between SDM use and RA/RC. In chapter 3, the research methodology was outlined with a particular emphasis on the survey instrument and sampling technique. Chapter 4 considered the actual results of the survey and whether or not the proposed hypotheses were supported. Furthermore, this chapter provided information on the reliability of the survey instrument as it was used in this study as well as the efficacy of the sample size. Lastly, in chapter 5 the research questions outlined in this study were considered in light of the results of the analysis discussed in chapter 4. In addition to discussing the implications of this research, future research ideas and the study’s limitations were discussed in the final chapter.
CHAPTER 2. LITERATURE REVIEW
The focus of this research was on answering four questions related to the use of system development methodologies (SDM) within the context of information system (IS) development projects. The first two questions were about the correlation, if any, between SDM use and employee perceptions of role ambiguity (RA) and role conflict (RC). The third and fourth questions were about potential moderating or confounding variables: the length of time the SDM has been in use within the organization and the years of experience the employee has in the field of IS development. In this chapter, the available literature was examined in order to justify this research, identify any potentially confounding variables that need to be taken into account when developing the research methodology, and provide a detailed understanding of the phenomenon being researched. This chapter begins with a thorough review of research on RA and RC, their impact on organizations, and the limitations of the most commonly used RA/RC survey instruments. Next, the literature on SDM use was examined with the intent of highlighting both the need for further research in this area and the ongoing debate as to the usefulness of employing SDMs. Finally, this chapter concludes by explicating why SDM use should be considered in the context of its influence on RA/RC.
13 Role Ambiguity and Role Conflict Introduction to Role Ambiguity and Role Conflict Originally identified by Kahn, Wolfe, Quinn, Snoek, and Rosenthal in 1964 (Breaugh & Colihan, 1994), role ambiguity has become a prominent factor in organizational research, especially as it pertains to work stress (Fried, Ben-David, Tiegs, Avital, & Yeverechyahu, 1998). Traditionally, the term “role ambiguity” was used to denote the level of knowledge employees had in regards to their job, how to perform it, and what was likely to happen if their performance was not adequate (Tremblay & Roger, 2004). While other researchers have attempted to alter this definition to focus less on the knowledge aspect and more on the unpredictable nature of some careers, the more traditional definition was the one most commonly used in RA research (Pearce, 1981). In either case, RA still referred to the extent that an employee was able to comprehend his job and evaluate the likely outcome of actions that he may take. As RA increased, an employee was likely to find it more difficult to identify the best course of action needed to meet his job goals. In fact, it may become difficult to even identify what the specific goals are. On the other hand, when RA was at a minimum level, the employee could easily identify what tasks needed to be accomplished and in what order. Role conflict, according to Tubre and Collins (2000), “occur[ed] when, due to conflicting information, the individual [was] unable to do everything that [was] expected” (p. 157). Another definition of RC was “an incompatibility between job tasks, resources, rules or policies, and other people” (Nicholson & Goh, 1983, p. 149). Put another way, RC was experienced by an employee when something in his work environment prohibited him from performing his duties. One example of this could be seen when an
14 employee receives direction from two different sources. Consider an IS developer who was asked by the customer to add a feature to the system, which was in direct conflict with the manager’s request to freeze the system design. In a situation like this, the level of RC experienced by the employee increased, and the result was that there was a decrease in employee performance after a prolonged level of high RC (Fried et al., 1998; Onyemah, 2008; Tubre & Collins, 2000). Although RA and RC were often assumed to be distinct constructs, they were frequently found together in the research literature. As Smith, Tisak, and Schmieder (1993) put it; “the traditional conceptualization of role ambiguity and role conflict [is] as two related, although separate, constructs” (p. 38). This conceptualization has been supported in various studies. For example, Schaubroeck, Ganster, Sime, and Ditman (1993) found in their experiment that a change in an employee’s perceived level of RA could lead to a similar change in the level of experienced RC. Similarly, research engaged in by Fried et al. (1998) demonstrated that RA and RC tend to have a cumulative effect on an employee’s job performance. This apparent relationship between RA and RC was one explanation as to why so much RA research also included RC as a variable. It was also possible that the assumption that RA and RC were separate variables was incorrect (Fried et al., 1998). Disentangling the relationship between RA and RC exceeded the scope of the current study, but given the apparent relationship shared by RA and RC, it would be premature to exclude one or the other from this research.
15 Impact of RA and RC Between 1964 and 1979, no less than nine studies were conducted that demonstrated that “high levels of role conflict and role ambiguity result in unfavorable outcomes for both the individual and the organizations” (Nicholson & Goh, 1983, p. 148). A representative example of these types of research was found in an article written by Miles (1976) where RA and RC were shown to be correlated with various job-related aspects such as anxiety, satisfaction, and performance. The unfavorable side-effects of RA and RC have been well documented, especially in the field of work stress research.
Role ambiguity and role conflict as a source of employee stress. Even a cursory review of the literature on work stress supports the assertion presented by Fried et al. (1998): RA and RC were “two of the most frequently examined sources of work stress” (p. 19). In her article, Pearce (1981) indicated that between 1964 and 1976, nine studies were conducted on the relationship between RA and work stress, and only one of those studies did not find a significant relationship between these variables. Not only was this relationship a source of frequent research, but it continued to be consistently demonstrated over four decades after the seminal research was conducted (King & Sethi, 1998; Raghavan, Sakaguchi, & Mahaney, 2008). A logical explanation for the link between RA, RC, and work stress was presented by Schaubroeck et al. (1993) with an emphasis on the feelings of frustration, anxiety, and tension engendered by RA and RC. The assumption was that as ambiguity increased, it became more difficult for the employee to define meaningful objectives that led towards achieving career goals. In the case of RC, “the self-perception of competence and
16 effectiveness” (p. 3) deteriorated as the amount of conflict increased. In both cases, the result was a sense of tension, anxiety, and stress experienced by the employee. Stress experienced by employees was likely to have negative side effects, one of which could be a decrease in job performance. The argument for this, as presented by Fried et al. (1998), was that humans are limited in their capacity of cognitive resources and that RA and RC could place a heavy burden on these fixed resources. In the work environment, an employee’s cognitive resources were likely to be engaged as the employee negotiated her daily work load. Increased levels of RA and RC further tax these cognitive resources. If the employee’s resources were drained, the result would be a decrease in performance. As intuitive as the theory presented by Fried et al. was, they went on to note that there was a paucity of research empirically linking RA, RC, and job performance. Interestingly, this sentiment appeared to be at odds with opinions held by other researchers (Breaugh & Colihan, 1994), including Tubre and Collins (2000) who conducted a meta-analysis over 10 years prior to the article by Fried et al. Those researchers suggested a strong body of literature on the relationship between RA, RC, and job performance. Working on filling the research gap they felt existed, Fried et al. (1998) engaged in a survey study designed to evaluate the effect of both RA and RC on employee job performance. This study was unique in that it considered the combined effects of RA and RC rather than as independent factors, and the result was empirical evidence that these factors had a more significant impact on job performance when they coexisted. A meta- analysis suggested that the extant research demonstrated an exceptionally weak link between RC and job performance (Tubre & Collins, 2000). This suggestion may not be