Development of Knowledge, Skills, and Abilities as Core Competencies for Technology Protection Professionals
Table of Contents List of Tables vii Chapter 1: Introduction 1 Background 3 Problem Statement 5 Purpose 6 Theoretical Framework 8 Research Questions 11 Nature of the Study 13 Significance of the Study 14 Definitions 15 Summary 17 Chapter 2: Literature Review 19 Current landscape necessitating RTP 21 RTP as a Formal Profession 33 Previous KSA Research for Professions 36 Historical foundations for RTP KSA identification 38 Summary 50 Chapter 3: Research Method 52 Research Methods and Design(s) 53 Participants 56 Materials/Instruments 58 Data Collection, Processing, and Analysis 60 Methodological Assumptions, Limitations, and Delimitations 62 Ethical Assurances 65 Summary 66 Chapter 4: Findings 68 Description of Participants 69 Findings for Round One 71 Findings for Round Two 85 Findings for Round Three 89 Evaluation of Findings 104 Summary 113 Chapter 5: Implications, Recommendations, and Conclusions 117 Implications 119 Recommendations 126 Conclusions 129 References 133
Appendix A: Development of Knowledge, Skills, and Abilities as Core Competencies for Technology Protection Professionals Consent Form 140 Appendix B: First Round Survey and Instructions 141 Appendix C: Second Round Survey and Instructions 154 Appendix D: Third Round Survey and Instructions 166 VI
List of Tables Table 1: Sample Instrument KSA Items 60 Table 2: Specific Training Completed 70 Table 3: Round One Survey KSA Items 73 Table 4: Round One Mean and Standard Deviation Results 83 Table 5: Round Two Survey KSA Items 86 Table 6: Round Two Mean and Standard Deviation Results 88 Table 7: Round Three Survey KSA Items 90 Table 8: Round Two Mean and Standard Deviation Results 101 Table 9: Final RTP KSA Items 108 vii
1 Chapter 1: Introduction United States military commanders depend on advances in research and technologies to maintain superiority on the modern battlefield. Research and technologies initially created for the military often lead to research spin-offs applied to the commercial sector and vice versa (Luchsinger & Blois, 2009). Whether by selling advanced military equipment directly to allied nations or by creating better products for commercial use, research and technology information plays a large role in national economic competitiveness (Leheny, 2009). Historically, protecting technology has become a concern when a nation is at war. New technologies can provide a battlefield advantage, and preserving this advantage can save soldiers' lives. Substantial effort was devoted to protecting research and technology during World War II. However, over the decades, protection received less and less emphasis. The extensive use of technology during the Gulf War in many ways reinvigorated the need to protect United States research efforts (Chant, 2001). In 1997, when the United States Under Secretary of Defense for Intelligence (USDI) published the Security, Intelligence, and Counterintelligence Support to Acquisition Program Protection Directive (Department of Defense Instruction, 5200.39, 1997), research and technology protection (RTP) professionals re-emerged to protect vital research and technologies providing military and economic competitiveness. This directive mandated program managers to address the protection of critical technology during each major phase of a defense acquisition program.
2 The RTP profession continued to grow as other directors of various federal agencies such as NASA mandated the protection of research and emerging technologies. As noted by Erwin (2008), "the technology protection program has become the fastest growing arm of the Defense Criminal Investigative Service. Investigations under this program involve the transfer of weapons technology and other sensitive information to foreign organizations without proper State Department export licenses" (para 2). Program managers at the DoD are required to conduct RTP as outlined in the DoD 5000 regulation series. As described in the DoD Instruction Security, Intelligence, and Counterintelligence Support to Acquisition Program Protection, 5200.39R (2002), RTP is mandated for defense acquisition programs and is defined as the protecting and safeguarding of research and technology anywhere in the acquisition process. Research and technology protection requires integration of the various security disciplines, counterintelligence, and systems security engineering. As discussed by Kanter and Van Atta this process (1993): Includes a large investment in human capital, not only involving trained individuals with long experience, but also involving teams that take years to assemble. Such teams are expensive and difficult to assemble, but they are also difficult to hold together if they do not have serious work. (p. 56) Conducting research and developing new technologies to maintain military superiority and economic competitiveness require a significant investment and the protection of this national investment drives the requirements for RTP. As estimated by the National Science Board (2008), the United States government,
3 academia, and industry invest approximately $340 billion annually for research and development activities. At the same time the United States government and commercial industries are investing in research and technology development, over 100 countries are actively engaged in attempting to steal the fruits of these investment activities (DSS, 2006). Emphasis on RTP has continued to grow, but studies and research on the subject have not. A paucity of research exists regarding the research and technology protection profession. The remainder of the current chapter includes a discussion of the problem addressed and the purpose of the study. After outlining the problem and purpose of the study, the chapter includes an overview of the theoretical framework and research questions guiding the study. The chapter then concludes with a discussion of the significance of the study for the RTP profession. Background With threats to United States research and technologies growing exponentially each year, the United States Under Secretary of Defense for Acquisition, Logistics, and Technology (USDAT&L) outlined criteria to identify research and technology requiring protection. When discussing the effects associated with losing valuable research data, the USDI noted, "protecting critical technologies preserves the United States Government's resources in research and development as an investment, rather than as an expense, and enhances the United States industrial base competitiveness in the international marketplace" (DoD, 1997, p. 23).
4 With so many nations engaged in stealing United States research and technology information, the National Counterintelligence Executive (NCIX) (2003) published the Annual Report to Congress on Foreign Economic Collection and Industrial Espionage, estimating the "combined costs of foreign and domestic economic espionage, including the theft of intellectual property, is as high as $300 billion per year and rising" (p. vii). The NCIX findings are consistent with the American Society for Industrial Security (ASIS) (2007) Trends in Proprietary Information Loss report. Leading members of ASIS surveyed Fortune 500 companies and concluded the theft of research and technology costs United States companies billions of dollars annually. The consequences of loss were further outlined by Nunez (2005) who noted the Department of Justice estimate that intellectual property losses cost United States companies as much as $250 billion last year and the Chamber of Commerce estimates losses of 750,000 jobs annually because of intellectual property theft. While much of the targeted technology is related to defense, Kanter and Van Atta (1993) highlighted the "DoD needs to emphasize that while it depends upon the nation's technology and industrial base, it cannot be unilaterally responsible for its health and well-being" (P- 13). With the consequences of loss so severe, many government agencies have revamped technology protection efforts. To highlight the importance placed on protecting research and technologies, the United States Army recently re- awarded a 35.8 million dollar contract to Qinetiq North America to operate the United States Army Research and Technology Protection Center to provide RTP
5 "expertise and support to research and engineering centers and acquisition programs throughout the Army" (Qinetiq, 2007, para. 2). Recognizing the similar value of space-related research and technology, NASA recently awarded a multi-million dollar contract to United Space Alliance to "create an effective and efficient Research and Technology Protection program. The RTP program would utilize the lessons learned from the DoD to identify and protect various forms of NASA research and technology information from unauthorized disclosure or exploitation" (J. Day, personal communication, June 4, 2008). Problem Statement The problem addressed in the Delphi study was government agencies such as the DoD, NASA, and the Federal Bureau of Investigation (FBI) had not developed or established a set of knowledge, skills, and abilities (KSAs) for individuals responsible for protecting technologies that provide a national advantage. While a multitude of KSA research exists on a wide variety of professions, previous research for the RTP profession is nonexistent (Kay & Moncarz, 2004; Michalik, 2007; Rivkin, Lewis, & Cox, 2001). The lack of an established set of core competencies for technology protection personnel may negatively affect the ability of government agencies to protect national investments in research and technology development. A lack of KSAs for technology protection practitioners may result in agencies inappropriately releasing technology to other nations or organizations to the detriment of national security (Aerospace Daily, 1999). Kanter and Van Atta
6 (1993) noted technology rapidly spreads throughout the globe, and developing a national security technology policy is a vital necessity. Without KSAs, developing job descriptions and requirements to hire and train RTP personnel is extremely difficult (G. Gaddy, personal communication, June 4, 2008). The difficulties associated with hiring and training RTP personnel can have significant negative effects on the agency's efforts to protect innovative technology. Technology protection requires personnel with multi-disciplined experience to perform a wide variety of activities. Previous KSA research pertaining to RTP professionals is an unexplored research area. The lack of previous research has hindered developing consistent RTP job descriptions and duties and creating training specific to the RTP field. Randeree (2006) noted "the current literature has examined how, why, when, and where to leverage knowledge assets; they have ignored the question - how to secure knowledge assets" (p. 149). The KSAs developed at the conclusion of the current study may provide an initial foundation to address these challenges within the RTP profession. Purpose The purpose of the Delphi study was to identify and evaluate KSAs that subject matter experts perceived as requisite for RTP professionals in the DoD and other federal agencies. The methodology included an exhaustive literature review, and the use of 10 subject matter experts participating in 3 structured survey rounds to gain consensus on specific KSAs they perceived as vital for the
7 RTP profession. Identified KSAs for the RTP profession will enable specific employment qualification development and courses of instruction for the RTP field. Using the KSA model to define core competencies for specific fields and professions is widely accepted within government and industry. The United States Office of Personnel Management (2007) outlined the use of the KSA model to define federal position qualifications in the Delegated Examining Operations Handbook. Various professional societies also utilize the KSA model to establish core competencies for members (Kay & Moncarz, 2004; Michalik, 2007; Stowe, 2007). Program managers within the DoD, NASA, and the FBI have established RTP programs, but a recognized set of core competencies for those charged with implementing protection initiatives for multi-billion dollar research and technology efforts has not been established. Without identified KSAs, the education and experience level of current RTP practitioners vary greatly within both the DoD and NASA. The variance in education and experience among RTP personnel may be a contributing factor to the escalating costs associated with the loss of research and technology information. For the Delphi study, 10 technology protection subject matter experts identified and evaluated the KSAs they perceived as required for RTP professionals. The participation of experienced RTP practitioners currently working within the field provided insight regarding which specific KSAs they perceived as critical to the profession. Experts selected for participation were those who had published in the research and technology protection field, had
8 presented at research and technology conferences, symposia, and training events, and were recognized by their organizations and the RTP community as experts within the field. The use of stringent criteria to select Delphi panelists mitigated the potential effect of having self-proclaimed experts on the panel. The choice of panelist selection considered robust participation from the various government and commercial organizations currently conducting RTP to reflect the general constituency of the research and technology protection community. The RTP field currently contains several position titles. The primary government RTP positions which panelists held included technology protection officers, technology protection engineers, and security specialists. Commercial RTP efforts contain positions entitled program protection architects and technology transfer specialists, and study panelists were representative of both. Based on research of previous Delphi studies, the optimal population size of participants is between 8 and 10 persons. Fewer than 8 participants may introduce potential internal validity issues, and greater than 10 participants would provide little, if any, effect on the consensus of results achieved by the panelists. Theoretical Framework The theoretical foundation for the study was the theory of professions and human resource development theory. The theory of professions centers around the relationships between occupational groups, and the exclusive application of specific knowledge practitioners possess to perform their work functions (Sundin & Hedman, 2005). The theory of professions holds a strong interest in distinguishing between professions based upon comparing the difference in traits
9 of occupational groups (MacDonald, 1995). Within the social sciences, a tradition of theory of professions has evolved to address the increasing specialization and institutional expertise required of practitioners from a plethora of fields (Evetts, 2003; Foumier, 1999). Olson (2002) discussed two primary principles defining a profession: first, a layperson cannot evaluate the effectiveness of the practice adequately; and second, a significant effect to the public would occur if incompetent people completed the actions required. Dean (1995) specifically highlighted the requirement of people within an accepted profession to have special skills and knowledge as well as extensive education setting them apart from the layperson or personnel from similar professions. Kennedy (2000) asserted a profession is composed of people with not just any sort of specialized knowledge, but accumulated knowledge acquired over time gained by the insight of peers and by predecessors' experience and analysis. McCarthy (1996) noted professionals master a set of practical and intellectual skills and become members of communities that possesses specific values, norms, and expectations regarding their personal conduct. Using the theory of professions as a foundation, previous researchers completed studies to identify core competencies or KSAs for a wide range of professions, but previous research did not include the RTP profession. The research and technology protection profession is a multi-million dollar industry charged with protecting billions of dollars of research information critical for American national military and economic competitiveness (Qinetiq, 2007).
10 Barker (2004) explored attributes to characterize a specific profession and outlined several criteria distinguishing a profession. While meeting the criteria for RTP to be distinguished as a formal profession, practitioners have suffered from a lack of research to develop or codify a set of core competencies or KSAs that can aid in the continued growth and professionalization of the occupation. Bucher and Strauss (1961) demonstrated professions are in a continual process of segmenting based upon different interests and fields of activity. In line with the findings of Bucher and Strauss (1961), the emerging and expanding RTP profession may be a logical outgrowth of the increasing pace of technological innovation, the need to protect the innovation from competitors, and the segmentation of the security and intelligence professions. The segmentation may stem from the requirement to apply specific methods and mechanisms to identify and protect the unique aspects of technological innovation from inappropriate disclosure to competitors. Human resource development theory encompasses a whole-system approach to professions, which utilizes organizational development, training and development, and career development to improve organizational, group, and individual effectiveness (DeSimone & Harris, 1998). As noted by Spencer and Spencer (1993), employees within professions operate at varying levels of proficiency. Specific guidelines, which outline the knowledge, skills, and abilities of the profession, can enable higher levels of proficiency among practitioners (Spencer & Spencer, 1993). Without established KSAs "companies can only teach employees basic skills and leave it to them to develop competency"
11 (Spencer & Spencer, as cited in Hale, 2008, p. 2). By developing and utilizing KSAs, professions can train new practitioners to a higher standard of performance (Spencer & Spencer, 1993). Establishing an initial set of core competencies or KSAs for the RTP field may promote the continued development and professionalism of the RTP field in concert with the findings of previous researchers (Spencer & Spencer, 1993; Hale, 2008). Stevens and Campion (1994) demonstrated the use of the KSA model to define position requirements increased the likelihood of candidates being able to perform work-related activities successfully. To provide protection for research and technology in an effective and efficient manner requires skilled and experienced RTP professionals. Developing and using RTP-specific KSAs can assist in growing and training research and technology protection professionals. Kay and Moncarz (2004) articulated the use of the KSA model to define professional competencies as a building block to develop career-specific training, job descriptions, and employee evaluations. Michalik (2007) highlighted the benefits gained by utilizing KSA surveys to serve as a blueprint to design and build career-specific training and education programs. For the current study, initial KSAs for the RTP profession were identified that could further the development of detailed RTP training curriculums, position descriptions, and employee evaluation systems. Research Questions As commercial and government investments in research and technology continue to grow, so do the associated costs to companies and to the nation if
12 thieves illegally obtain the research and technology information. The costs of research and technology theft have reached the unprecedented level of hundreds of billions of dollars annually. In light of these factors, there has been an increased emphasis on hiring RTP professionals over the past decade. While there has been substantial growth of both government and contractor RTP positions, the qualifications and competencies RTP professionals currently possess are inconsistent among organizations. The lack of and need for established criteria to define an RTP professional was the basis of the current study. The lack of previous research for the RTP field created a gap in research that may hinder the continued growth of the RTP profession. As discussed by Desouza (2006) "research into knowledge protection has been lacking. If knowledge is to be a truly competitive resource, its security deserves attention. Failure to secure knowledge can cost an organization dearly in terms of loss of competitive advantages" (p. 1). The current study contributes to filling this research gap and expanding the limited information available on RTP. Specifically, this Delphi study utilized survey instruments administered to 10 RTP subject matter experts to investigate the following 3 research questions: 1. What specific education and training do RTP subject matter experts perceive as necessary for RTP professionals? 2. What knowledge, skills, and abilities do RTP subject matter experts perceive as critical for RTP professionals? 3. Why do the RTP subject matter expert panelists perceive the identified KSAs as essential for the RTP professional?
13 Nature of the Study The Delphi study utilized a panel of RTP subject matter experts to identify and evaluate a set of skills, abilities and, knowledge they perceived as necessary for research and technology protection professionals. To gain consensus, the Delphi panelists completed three rounds of surveys containing Likert-style scaled questions. The results of each survey round were computed; when 80% of the panelists reached agreement on either KSA inclusion or exclusion, consensus was deemed to be achieved. The Delphi methodology was chosen as the most appropriate technique to elicit the greatest range of responses from the research and technology protection subject matter experts. The Delphi questionnaires contained Likert-style scaled item questions, open-ended questions, and ranking or prioritization questions for participant responses. The initial Delphi survey gathered respondent demographic data and KSA statements from various professions including security, intelligence, and engineering, compiled from existing sources such as the Occupational Information Network (ONET) database and Federal position requirements. From reviewing sources such as ONET, a database containing KSA items for a multitude of professions, as well as federal KSAs for intelligence and security personnel who currently staff RTP positions, an initial list of KSAs was created for the Delphi panelists to review. During each Delphi round of surveys, the subject matter experts had opportunities to add additional KSA statements. The surveys included the individual panelist responses to generate a list of potential RTP KSAs. Through repeated Delphi questionnaire rounds, the KSA
14 list was systematically refined until a core competency skill set was developed based upon respondent consensus. By conducting the Delphi questionnaires in rounds and by providing the respondents with the ability to rank order and rate previous round group responses, the study validity was improved further. Whereas the study was qualitative in nature and did not specifically call for an in-depth review of variables, some quantitative data was gathered and analyzed to enable researchers to duplicate the study or expand upon the results in the future. Consistent with other researchers who identified KSAs for professions (Stevens & Campion, 1994; Kay & Moncarz, 2004; and Stowe, 2007), this Delphi study included basic demographic variables (e.g., age, gender, and level of education). Significance of the Study The paucity of RTP research and the lack of KSA or core competency- related research for the RTP field in particular have hindered developing consistent RTP job descriptions and duties, and the creation of training specific to the RTP field. The research findings presented in the current study contribute to and expand the overall understanding of core competencies RTP subject matter experts perceived as vital for RTP practitioners. The lack of specific research for the RTP field served as a catalyst to establish an agreed upon set of core competencies, or KSAs, for current and future technology protection practitioners. Based upon the study findings, federal agencies will now have a basis for developing RTP professional qualifications for employment and the development of RTP specific courses of instruction, enabling organizations such
15 as the DoD and NASA to hire, train, and retain research and technology protection professionals more effectively. Researchers have conducted KSA studies for a vast variety of professions. For example, the ONET database currently contains thousands of KSAs for various professions. While KSA research is a well-established approach to identify core competencies for professions, the research was absent for the growing research and technology protection profession. The lack of KSAs contributed to government agencies staffing RTP positions in a haphazard manner. Results of the haphazard RTP staffing include wide discrepancies between the level of experience, competence, and abilities of those hired to perform protection activities in a thorough and efficient manner. The lack of RTP-specific KSAs is a hidden vulnerability putting United States investments in advanced research and technology in increasing danger of unauthorized disclosure or theft. Definitions Some of the terms utilized for the current study have a common understanding, and the following section defines the terms as they pertain to this research effort. Abilities. In KSA research studies, 'abilities' refers to the demonstrated competence to perform an activity resulting in an observable work product required for job performance. The activities leading to a work product are directly linked to the job or task under review (White, 2003).
16 Knowledge. In KSA research studies, 'knowledge' refers to what a person knows based upon his or her experience or association that aids in job performance. The experience usually concerns familiarity with procedures and processes that make adequate performance on the job possible (CDC, 2008). KSA. For the purposes of the current study and to remain consistent with the government application process, a 'KSA' is defined as a specific special qualification, attribute, or ability a candidate possesses for a particular position or job. The KSAs are unique requirements a hiring agency is searching for to fill a selected position. The primary purpose of KSAs is to measure the qualities of applicants setting them apart from others to perform required duties, activities, and responsibilities successfully (CDD, 2008). Research and technology protection (RTP). Originally coined by the DoD and updated in draft DoD regulation 5200.39R (2002), for purposes of the current study, 'RTP' is defined as identifying and protecting selected research and technology that provides a military or economic advantage anywhere within the organization or within the acquisition process. Protection involves a multi- disciplined approach that relies on integrating all of the security disciplines to protect the information from unauthorized disclosure. Skills. For KSA research studies, the term 'skills' refers to how a person uses knowledge in an effective and efficient manner to execute tasks in the performance of a job. Skills often concern the ability to manipulate data or things and can be measured by performance tests (CDC, 2008).
17 Summary Chapter 1 included a discussion of the problem addressed and the purpose of the study. After outlining the problem and purpose of the study, the chapter included an overview of the theoretical framework and research questions guiding the study. The chapter then concluded with a discussion of the significance of the study for the RTP profession. Research and technology protection professionals help to preserve a national military advantage and the economic and industrial base (Leheny, 2009 & DoDI 5200.39, 1997). Government and corporate entities invest billions of dollars annually to develop new technologies (National Science Board, 2008). A vital aspect of ensuring the maximum benefit of technology investments is safeguarding the information with research and technology protection activities (NCIX, 2003; ASIS, 2007 & Nunez, 2005). Established RTP programs exist across the government, but a defined set of skills, abilities, and knowledge for those charged with creating and implementing research and technology protection is lacking. To establish a defined set of RTP core competencies it was necessary to utilize a panel of RTP subject matter experts to identify the KSAs they perceived as critical for research and technology protection professionals. The methodology included an exhaustive literature review, the compilation of a list of existing KSAs for potential application for RTP professionals, and the use of subject matter experts utilizing the Delphi methodology to gain consensus on specific KSAs for the RTP profession. Based upon the study findings, federal