• unlimited access with print and download
    $ 37 00
  • read full document, no print or download, expires after 72 hours
    $ 4 99
More info
Unlimited access including download and printing, plus availability for reading and annotating in your in your Udini library.
  • Access to this article in your Udini library for 72 hours from purchase.
  • The article will not be available for download or print.
  • Upgrade to the full version of this document at a reduced price.
  • Your trial access payment is credited when purchasing the full version.
Buy
Continue searching

Testing the Initial Validity and Reliability of a Biopsychosocial Evaluation Instrument for the Purpose of Appraising Medical Students' Competency

Dissertation
Author: Melissa Arthur
Abstract:
The Medical Education literature discusses the importance of training competent medical professionals. The public expects physicians to be sensitive and aware of multiple factors, not just medical ailments. The biopsychosocial approach is a guide used to assist doctors in the exploration of a patient's narrative and in diagnosing a patient's difficulty based on multiple aspects of a patient's life including biological, psychological and sociological. During the 1980s, a pedagogical shift occurred in medical education toward an emphasis on the importance of teaching communication skills and taking a biopsychosocial approach to medicine. However, presently there is not a known, validated biopsychosocial assessment instrument in the medical education literature. This study attempts to provide psychometric validity for a biopsychosocial evaluation instrument. This evaluation tool is currently utilized to assess third and fourth year medical students during a complex Integrated Standardized Patient Examination (ISPE). The ISPE format provides an opportunity for students to demonstrate competency of skills in an integrated fashion. A sample of 149 digital recordings of medical student ISPE encounters was used to test the validity and reliability of the biopsychosocial D-D ISPE (Diabetes, Depression, Integrated Standardized Patient Examination) scoring instrument. The instrument's psychometric properties were determined using a variety of statistical analyses. The analyses included a card sort and an exploratory factor analysis to test the instrument's dimensionality. Discriminant and convergent validity methods were conducted in order to compare the ISPE scoring instrument's constructs to the constructs of two other previously validated instruments. The D-D ISPE communication and shared decision making, indicate weaker convergent validity than hypothesized. The D-D-ISPE biological scale evidenced discriminant validity (i.e., non significant correlations) with the measures of communication and shared decision making. The coder's scores of the two comparison instruments demonstrated good inter-rater reliability. The initial validity test also determined the instrument is reliable and valid. Cronbach's alpha tests determined the instrument as a whole and its subscales demonstrate high internal consistency. The instrument dimensionality analyzed by the card sort and the factor analysis, demonstrated the instrument possess three distinctive biological, psychological and sociological/communication factors. The psychological, biological and social Communication subscales are all moderately to strongly correlate with the D-D ISPE total scores. These findings suggest that the instrument can assess biopsychosocial competency of medical students during an ISPE examine focused on diabetes and depression. Finally, strengths and limitations of the study are examined and its implications for future use in medical education and research are explored.

D-D ISPE INSTRUMENT vi Table of Contents Abstract Page i List of Tables Page ix Acknowledgments Page x Chapter I: Introduction Page 1 I. Communication and Counseling Skills Page 3 II. Medical Education Evaluation Page 4 III. Observed Structured Clinical Examination Page 5 IV. Problem Statement Page 6 V. Study Purpose Page 8 VI. Significance of This Research Page 10

Chapter II: Review of the Literature Page 12 I. Test Validity Page 12 II. Doctor-Patient Relationship Page 14 a. Historical Context Page 15 III. The Biopsychosocial Model Page 18 a. Medical Education and the Biopsychosocial Model Page 20 IV. Medical Education and Communication Skills Page 23 a. Communication Skills Studies Page 26 V. Observed Structured Clinical Examination Page 28 a. History of Assessment Standards in Medical Education Page 28 b. OSCE Performance Studies Setting Clinical Standards Page 30 c. Medical Student Perceptions of the OSCE Page 34 d. OSCE and Communication Page 35 e. Communication Education Intervention Using Standardized Patients Page 38 g. OSCE Validity Page 43 h. OSCE Reliability Page 43 i. Assessment Tools Page 49 j. The Integrated Standardized Patient Examination Page 50 k. Conclusion Page 54

Chapter III: Methodology Page 69 I. Overview Page 69 II. Context Page 70 III. Biopsychosocial Case Development Page 71 IV. Standardized Patient Training Page 72 V. Physical Set Up for the ISPE Page 74

D-D ISPE INSTRUMENT vii VI. Feedback Information from Pilot Data Page 75 VII. Grading Page 76 VIII. Participants Page 77 a. Demographics Page 78 IX. Procedures Page 78 X. Instrument Dimensionality Page 79 a. Card Sort Second Iteration Page 80 b. The Theory Behind the Factor Analysis Methodology Page 80 c. Test Assumptions Page 81 d. Reliability Page 84 e. Group Differences Page 85 XI. Instruments Page 85 a. ISPE Page 85 b. Common Ground Page 86 c. Options Scale Page 90 Chapter IV: Results Page 97 I. Descriptive Statistics Page 97 a. Sample Data Page 97 II. Evidence of Validity Page 98 III. Factor Analysis Page 99 IV. Card Sort Analysis Page 108 a. Second Iteration Page 110 V. Evidence of Reliability Page 113 a. Internal Consistency Page 113 VI. Construct Validity Page 113 a. Convergent Validity Page 113 b. Discriminant Validity Page 115 VII. Inter-Rater Reliability Page 115

Chapter V: Discussion of the Results Page 117 I. Findings and Card Sort Page 120 a. Unique Finding Page 124 II. Internal Consistency Results Page 125 III. Construct Validity Page 127 a. Implications of the Findings for Future Training of Physicians Page 128 IV. Limitations Page 133 V. Strengths Page 133 VI. Future Research Page 134 VII. Conclusions Page 135

References Page 137

D-D ISPE INSTRUMENT viii

Appendices Page 154 A. Definitions Page 154 B. ISPE Instrument Page 158 C. Upstate Internal Review Board Approval Page 163 D. Syracuse University Internal Review Board Approval Page 164 E. Student Demographic Form Page 165 F. Common Ground Instrument Page 167 G. Option Scale Instrument Page 168 H. Scree Plot Page 171 I. Items That Did Not Load on the Three Identified Factors Page 172

Curriculum Vitae Page 173

D-D ISPE INSTRUMENT ix List of Tables

Table 1 Biopsychosocial Studies Page 56

Table 2 Sample Demographics Page 97

Table 3 Students ISPE Scores Page 98

Table 4 Rotated Factor Matrix and Final Communalities Page 100

Table 5 D-D ISPE Total and Factor Index Score Descriptive Data Statistics Page 103

Table 6 D-D ISPE Total and Factor Correlations Scores Page 104

Table 7 MANOVA Results Page 106

Table 8 Expert Panel Demographics Page 107

Table 9 First Iteration: Expert Panel Card Sort Page 108

Table 10 Expert Panel, Dimension Results Page 110

Table 11 Correlation Matrix for Convergent and Discriminate Validity Page 113

D-D ISPE INSTRUMENT x Acknowledgements

I am fortunate to have been provided the opportunity and support at work and home to pursue a doctorate degree. I appreciate the many people who assisted me throughout the entire process and contributed to my learning. First my committee: Chair, James Bellini, Ph.D., who never wavered in his confidence in me. His time, expertise and dedication to the process were invaluable; Janine Bernard Ph.D., who always offered clear feedback and expertise; and Wanda Fremont, MD, who added her expertise and knowledge of medical education. SUNY Upstate Medical University Andrea Manyon, MD, Chair of the Department of Family Medicine, Christopher Morley, Ph.D. and John Epling, MD, who supported my time by providing a quiet space and research time through the Department of Family Medicine. Grant: HRSA grant D54HP05462, "Administrative Academic Units - Research Infrastructure for a Center for Excellence in Primary Care." Andrea T. Manyon, PI) Gene Bailey, MD, the Family Medicine Clerkship Director, and his assistant, Carin McAbee. The Upstate Medical University class members who participated in the Family Medicine clerkship 2008-2009. Steven Harris, MA, Director of the standardized patient program and Amber Hansel, the coordinator of the standardized patient program. The standardized patients who played the case role of Charlotte Wilkins. Administrative support: Kathleen Barzee, MA and William Grant, M.Ed., Ed.D.

D-D ISPE INSTRUMENT xi St. Joseph s Hospital Health Center James Tucker, MD, Family Medicine Residency Director, who also granted permission to work on my dissertation on Mondays and permitted the residents to double up on their behavioral science rotation. Anne Louise, who assisted with various administrative task and constant encouragement. Julie Nedell provided administrative assistance and support throughout my doctoral work. Sandra Zajac, MLIS, St. Joseph s Hospital Librarian and Anthony Lasinski provided administrative support. The St Joseph Family Medicine residency faculty including ten of the card sort participants: Dawn Brink-Cymerman, MD, DeAnn Cummings, MD, Gerry Edwards, MD, Stephen Hoag, MD, Pamela Horst, MD, Lynne Humphrey, MD, Michael Kernan, MD, David Kolva, MD, Jennifer McCaul, MD, Kristen McNamara, MD, Elizabeth McNany, MD, Larry Novak, MD, John O Brien, MD, Matthew Picone, MD, Sandy Sulik, MD, James Tucker, MD Syracuse University Counseling and Human Service Faculty: Drs, James Bellini, Janine Bernard, Dick Hackney, Dennis Gilbride, Richard Shin, Melissa Luke, Derek Seward. Administrative assistant: Sindy Pitts Classmates: John Kiweewa, Ph.D., Tikana Truitt , MA, Kris Goodrich, Ph.D., Sean Finnerty, MA , Gena Nelson, MA, Juleen Buser, Ph.D., Trevor Buser Ph.D., James Raper, Ph.D., Harue Ishii, Ph.D., Kathy Castle, Kyoung Mi Choi, MA, Melissa Jennings Carman, Ph.D., Lance Smith, Ph.D, Melissa Luke, Ph.D

D-D ISPE INSTRUMENT xii Coders: Kris Goodrich, Ph.D., Jennifer Rogers, MA Amalia Stanciu, MA, Sarah Spiegelhoff, MA Emotional support: Melissa Luke Ph.D, Robbie Saletsky, Ph. D., Ron Saletsky Ph.D., Sue Maar, MA, Al Mortiarty, Donna Moriarty, Naomi Schayes, Dolph Schayes, Carrie Goettsch, DC, life time friend Laura Wolf, MSW. Mentors: Mac Baird, MD and Eleanor Macklin, Ph.D. Both taught me the importance of taking a biopsychosocial approach. Family: My husband, David Schayes, MSW, who supported me in every way imaginable! I appreciate his patience, love and emotional support through out this entire process. My children, Hannah and Julia, who are now educated in the meaning of a dissertation and what is involved to receive a doctorate. I appreciated their many hugs, kisses and smiles throughout the process. And lastly, my mother, Rita Arthur (August 14, 1933 - April 10, 2010).

D-D ISPE INSTRUMENT 1 Chapter One Introduction Overview The changing population of the United States creates challenges for practicing physicians and medical students. The population is becoming increasingly diverse and aging with various health concerns. An individualized approach to health care is essential to account for unique differences presented by various patients (Cohen, Iton, Davis & Rodriguez, 2009). Individualized attention is also essential for understanding different cultural values and beliefs, while sensitivity is a necessity to care for the elderly and to manage the impact that individual lifestyles have on health. Morbidity and mortality in the United States are associated with health behaviors and lifestyles; the ten leading causes of death are due to lifestyle behaviors that include tobacco use, alcohol misuse, illicit drug use, risky sexual behaviors, poor diet and inactivity. These lifestyle behaviors account for up to 50% of the United States mortality (Danaei et al., 2009; McGinnis & Foege, 1993; Mokdad, Marks, Stroup & Gerberding, 2004). Lifestyle and health behaviors related to chronic diseases including heart disease, diabetes, hypertension and stroke have been on the rise over the last 50 years. Some characterize the health status of the United States citizens as escalating to a national crisis (Cohen et al., 2009). Mental health factors also affect health and well-being. In a national 12-month survey of mental health services, primary care physicians were found to be the sole providers treating the majority of patients with mental health difficulties (Wang et al., 2005). Often, patients who receive mental health treatment from primary care physicians are under-treated or poorly treated (Roy-Byrne, Joesch, Wang & Kessler, 2009; Wang et

D-D ISPE INSTRUMENT 2 al., 2005). It is estimated that 5 to 10 percent of patients in primary practice have major depression (Ballenger et al., 1999; Luber et al., 2000; www.hhs.gov). Despite these estimates, routine screenings for depression (Coyne & Schwenk, 1995; Penn, Boland, McCatney, Kohn & Mulvey, 1997) and domestic violence (Caralis & Musialowski, 1997; Elliott, Nerney, Jones & Friedmann, 2002; Hamberger, Saunder & Hovey, 1993) are not done. Moreover, twelve to twenty-one percent of children and adolescents have psychiatric disorders with at least mild functional impairment (Roberts, Attkission & Rosenblatt, 1998). Physicians, particularly primary care physicians, are expected to diagnose, counsel and treat individuals and families that have a gamut of mental health and emotional concerns. Pediatricians and family medicine physicians prescribe 85% of psychotropic medications taken by children (Goodwin, Gould, Blanco & Olfson, 2001). These percentages indicate that primary care physicians are delivering a large portion of child psychiatry. Primary care physicians often do not feel their training adequately prepares them for this professional role and responsibility (Fremont, Nastasi, Newman & Roizen, 2008). Proper identification, diagnoses and treatment cannot be accomplished by taking a purely biomedical approach (Solomon & Kington, 2002). In order for a physician to provide holistic care, effective communication, a therapeutic relationship and a biopsychosocial approach are all essential. The biopsychosocial approach simultaneously guides the physician to be cognizant of the biological, psychological, and psychosocial influences of the patient s life. The approach assists the physician in understanding the complexity of each unique patient s life circumstances and provides a framework to offer health prevention counseling that

D-D ISPE INSTRUMENT 3 include smoking cessation, weight loss and alcohol reduction strategies (Kreuter, Chheda & Bull, 2000). Taking a biopsychosocial approach enables the physician to identify psychosocial, lifestyle or psychological concerns that may be negatively influencing the patient s health and wellness. Communication and Counseling Skills Literature on communication, doctor-patient relationship skills and patient- centered care exemplify features of counseling such as open-ended questions, active listening, having a non-judgmental stance, empathetic understanding, reflective listening and being collaborative (Association of American Medical Colleges, Report III Contemporary Issue in Medicine: Communication in Medicine, 1999; Makoul, 2001; Duffy et al., 2004). The Medical Education literature does not label these skills as counseling, yet these skills are in fact the core concepts related to person-centered counseling (Rogers, 1961) and are essential to, and the basis of, counseling proficiency. Therefore, in order to be competent, physicians must demonstrate positive communication competency that includes such proficiency. Practicing physicians who have strong communication skills tend to develop positive doctor-patient relationships that are characterized by patient-centered care. Under a patient-centered model, physicians use a collaborative approach with their patients and encourage them during the consultation to express

their concerns and ideas regarding diagnosis and treatment. The patients of physicians who use the collaborative approach are more satisfied with the doctor-patient relationship and are more likely to adhere to their physician s treatment recommendations (McKinstry, 2000; Hall, Roter & Katz, 1987; Stewart, 1995; Stewart et al., 2000; Stiles, Putman, Wolf & James, 1979) and

D-D ISPE INSTRUMENT 4 file fewer law suits against their physicians (Stelfox, Gandhi, Orav & Gustafson, 2005). Patient-centered visits lower health care costs and improve patient health outcomes compared to visits in which physicians employ solely diagnostic and biomedical approaches (Stewart et al., 2000). It is imperative for physicians and medical students to communicate effectively, to ask the right questions and to determine the patients medical history in order to derive the correct diagnosis. Medical education professionals strongly concur that practicing from a biopsychosocial perspective is essential for good quality of practice and comprehensive patient care (Doherty & Baird, 1983; Stuart & Lieberman, 1986; McDaniel, Hepworth & Doherty, 1992; Frankel, Quill & McDaniel, 2003). Physicians are faced with complex, emotionally-charged circumstances that require sensitive communication skills. Examples of these circumstances include delivering bad news to a patient, such as a new diagnosis of a chronic or a life-threatening disease or the news that a loved one has recently died (Ury, Berkman & Weber, 2003). It is the educational institutions responsibility to train physicians not merely to be competent diagnosticians but also to be able to understand and cope with the emotional intensity often presented by their patients. Medical Education Evaluation Medical Student assessments historically included authentic assessment as part of the evaluation of the student s clinical proficiency. Authentic assessment principles evaluate the learners ability to apply academic learning to practical situations (Collins, Brown & Newman, 1989) and involve the complex task of assessing the students ability to integrate knowledge beyond the recall of facts (Collins et al., 1989; Granello & Hazler,

D-D ISPE INSTRUMENT 5 1998). Support for authentic assessment dates back to as early as 1916 when John Dewey argued that schools should emphasize meaningful and purposeful activities directly related to everyday life. The first use of authentic assessment in medical education occurred in the hospital setting when the evaluation of students was expanded beyond paper and pencil written exams to include evaluating their ability to demonstrate that they can integrate medical information and communicate this information to real, hospitalized patients. More current authentic assessment methods include videotaped encounters and live observation in the outpatient setting. Over the last thirty years, clinical skills have been assessed using a standardized patient during an Observed Structured Clinical Examination (See Appendix A for definitions). The profession of medical education has placed renewed emphasis on clinical competence, with particular focus placed on communication skills. In 1999, the National Board of Medical Education began testing medical students clinical medical knowledge with an 8-hour multiple-choice test and, in 2004, a clinical skills portion that included communication was incorporated as part of the Step Two Medical Licensing Examination. This examination includes 12 standardized patients and 15-minute, multi- station Observed Structured Clinical Exams (USLME Bulletin, 2009). Observed Structured Clinical Examination The Observed Structured Clinical Examination (OSCE) is a method utilized to impartially evaluate medical students on a variety of clinical skills and was first developed by R.M. Harden in Scotland in 1975. Harden explains that the OSCE evaluates students on what they do versus what they know. This authentic assessment is distinctly different from written or oral examinations in terms of what it evaluates (Harden,

D-D ISPE INSTRUMENT 6 Stevenson, Downie & Wilson, 1975). The OSCE was designed to evaluate medical students on various clinical skill competencies and, in current implementation of the OSCE model, separate stations are used to evaluate individual skills. A typical exam consists of 20 individual stations, each of which is five-minutes in duration. All of the stations test a specific skill and use a simulated patient; students demonstrate their skills and competency in each procedure or diagnosis at each station. The competencies tested consist of physical examination, diagnostic procedures and interpretation of the findings, prescription writing, communication, history taking, providing explanations to patients, advising patients and providing patient education (Harden et al., 1975). OSCE evaluators observe and grade students at a station using a checklist specific to each skill set (Kahn, Merrill, Anderson & Szerlip, 2001). The OSCE examination also evaluates the final product of the students skills, such as the student s write-up of the findings of the standardized patients

diagnoses and illnesses (Kahn et al., 2001). A recently developed standardized exam, the Integrated Standardized Patient Examination (ISPE), evaluates multiple student skills concurrently rather than one skill at a time and permits students to integrate medical information while simultaneously demonstrating communication skills. This approach is more authentic to an outpatient setting since the 20-minute exam mirrors the integrated approach to patient care and the timing of a genuine outpatient office encounter (Feeley, Manyon, Servoss & Panzarella, 2003). This research project uses the ISPE process as the basis of the study. Problem Statement It is established and documented that practicing physicians, residents and students must communicate effectively in order to ask the right questions, in the right way, and to

D-D ISPE INSTRUMENT 7 determine the patient s medical story as a method to identify the correct diagnosis. Physicians confronted with complex situations must possess sensitive communication skills in order to tell a patient, for instance, the bad news about their life-threatening diagnosis, to share with them a new diagnosis of a chronic disease, or to explain that their loved one has recently died (Ury et al., 2003). Despite the positive efforts in medical education that have been put forth in communication skills training, as well as efforts to include authentic assessments, many practicing physicians, residents and medical students feel unprepared to face challenges presented to them during doctor-patient relationship encounters (Ury et al., 2003; Park, Wolfe, Gokhale, Winickoff & Rigotti, 2005; Rogers et al., 2002). Starting in the 1980 s the public placed stronger demands on the medical education system by stressing the importance of well-trained, competent medical experts. This demand has influenced the culture of medical education, causing it to place an emphasis on competency-based education including communication skills and reject the assumption that students only learned these skills through observation and modeling. Instead, the Accreditation Council for Graduate Medical Education (ACGME) now categorizes medical competence into six related domains (medical knowledge, patient care, professionalism, communication and interpersonal skills, practice-based learning and improvement, and systems-based practice) and requires medical students to demonstrate their competency in each of the domains (ACGME, retrieved, October, 2009). Competency-based education, now an essential part of the culture of medical education, is shaping both teaching and assessment of clinical skills and has become a

D-D ISPE INSTRUMENT 8 method of ensuring professional quality. However, medical education presently lacks quality assessment methods for evaluating student competencies that are both valid and reliable and that can be generalized beyond individual medical education institutions. Evidence of this void in appropriate assessment approaches to competency-based student evaluation and outcome measurement is found in the literature, where there exists only vague acknowledgement of the subject. This void in assessment methods is especially deficient when compared to the evident progress that has resulted from pedagogical shifts to competency-based training. Research that focuses on clinical skills competencies and valid and reliable methods of assessments is therefore still needed. Study Purpose This dissertation research conducts the preliminary testing to validate the scoring instrument used to grade the ISPE, which is a summative evaluation held at the completion of the Upstate Medical University Family Medicine Departments clerkship. The study subjects were 149 third and fourth year Upstate Medical University students. The ISPE for the Family Medicine clerkship focuses on the students ability to integrate clinical skills and medical knowledge. The ISPE uses a clinical scenario that incorporates biological, psychological, social and medical issues in a single vignette that is presented to the student by a standardized patient. The case vignette derives from an actual clinical case and emulates a patient with diabetes and co-morbid depression who is from a low socioeconomic bracket. The ISPE involves standardized patients who are trained actors used to portray

the simulated case s important aspects that highlight the biopsychosocial perspective.

D-D ISPE INSTRUMENT 9 Each student s Family Medicine clerkship grade is reliant upon their ISPE score. This is a real-world consequence and it is important that the instrument used in the ISPE is a validated instrument. Therefore, a primary goal of this study was to provide basic reliability and validity data of the ISPE scoring instrument. An expert, multi-discipline panel drew upon the literature and their professional experience to construct the (D-D ISPE) Diabetes, Depression Integrated Standardized Patient Examination, biopsychosocial-scoring instrument and develop the case. While this process contributes to strong content validity, further psychometric validity investigation is warranted to determine if the ISPE measures the student skills it is intended to measure. This study is an investigation into the instrument s psychometric validity and reliability. Since the D-D ISPE scoring measure is a new instrument, inferences drawn from its scores need to be validated. According to Messik, validity refers to the utility of the inferences made from the measurement score (Messick, 1995), while the American Psychological Association says validity proposes validation strategies that include content, criterion, and construct validity (1999). It is well accepted in the psychological literature that measures must be validated for each individualized use of a measure and testing (The Standards for Educational and Psychological Testing (AERA, APA, NCME, 1999). Therefore, there are four research questions for this study: • Question 1. Is the D-D ISPE scoring instrument a reliable and valid scoring tool to assess biopsychosocial competency? • Question 2. What are the different dimensions of the D-D ISPE scoring instrument?

D-D ISPE INSTRUMENT 10 • Question 3. Does the D-D ISPE demonstrate convergent and discriminant validity? • Question 4. Do the D-D ISPE scores vary based on group differences? To determine if the D-D ISPE is a reliable and valid scoring instrument to detect students abilities to perform a biopsychosocial patient assessment, the validation process examined the D-D ISPE instrument s dimensionality using two separate methods - a factor analysis and a card sort. To further assess student performance, digital recordings of the students D-D ISPE encounter were scored by four independent coders using two separate validated instruments: Common Ground, which is an instrument to assess communication skills, and the Options scale, which is designed to assess shared decision- making skills. The subscale scores of these two different validated instruments were used to compare similar and dissimilar constructs from the D-D ISPE scoring instrument. The comparison process tested for convergent and discriminant validity of D-D ISPE scores. Testing of inter-rater reliability was conducted using four independent coders scores. Cronbach s alpha reliability was calculated for all of the instruments and the D-D ISPE sub-scales. Significance of This Research This study is significant since it was a first attempt at validating a biopsychosocial-oriented instrument that is used for grading medical students during an ISPE. There are no known studies evaluating simultaneously the multiple dimensions of the biopsychosocial assessment, which include the biological (medical factors), psychological and sociological factors. The literature discusses the specific use of checklists for global skill measurement with standardized patient examinations. However,

D-D ISPE INSTRUMENT 11 the D-D ISPE scoring instrument lacks formal assessment for psychometric validation. This omission leads to ambiguous conclusions regarding student competency. Providing validation of the instrument is important since students grades are dependent upon the scores obtained from the instrument. This initial psychometric validation study allows future educators to utilize the D-D ISPE instrument to measure medical student competency as they conduct a biopsychosocial patient encounter. This study further refines the ISPE biopsychosocial scoring instrument and attempts to provide adequate psychometric data that strengthen the conclusions drawn regarding student performance and that provide a valid and reliable instrument for use in future research. The study research questions are addressed in the chapters that follow. The chapters include background information, the details of the research and its design, along with results and discussion sections. Chapter Two provides background information and a foundation for research study and reviews the existing and pertinent literature. Chapter Three contains a description of the methods used to implement and conduct the research study. Chapter Four focus on the data analysis and the results. Chapter Five is the discussion section and includes the result implications of findings for medical education programs, a critique of the research study s strengths and limitations, and suggestions for future research.

D-D ISPE INSTRUMENT 12 Chapter II: Review of the Literature The review of the literature will explore fundamental aspects of the study. First, a brief description of test validity is explored, followed by an overview of the medical education literature related to the doctor patient relationship. Next the biopsychosocial medical education literature and communication skills and communication skills assessment will be discussed. Next, the literature related to validity and reliability of the Observed Structured Clinical Exams (OSCE) as a method of assessing medical students skills will be discussed. Finally, the reliability and validity of the OSCE scoring instruments will be reviewed. Test Validity Test validity research is considered the examination of truth (Grim & Yarnold, 2006). In other words, does the instrument test accurately the inferences or conclusions that it is intending to make? Messick (1995) developed the concept of validity. Messick stated, Validity is an overall evaluative judgment of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of interpretation and actions basis of test scores (p. 742). He emphasized the importance of multiple measures to assure validity as a unified concept. The American Psychological Association s technical recommendations for Psychological Test and Diagnostic Techniques (1999) utilize a taxonomy of validation methods. Consistent with this early work, Grim and Yarnold (2006) explained the importance of establishing three forms of evidence of interrelated validity. The first form of is content-related, the second is criterion-related and the third is construct-related evidence.

D-D ISPE INSTRUMENT 13 Content-related evidence demonstrates if the items on the instrument include the content material of the attributes of interest. Content-related validity is assessed by a detailed examination of the actual content materials of the test items to establish their appropriateness for the test. Content-related decisions for tests are often based upon expert opinion. Criterion-related evidence focuses on the suitability of the instrument s scores to make for making selection decisions. Criterion related validity is a selection process test and is assessed by comparing the test scores with a non-test criterion. The third, and most comprehensive, type of validity is construct validity. Construct validity refers to data supporting the instrument s ability to measure the attributes of interest. Since constructs are often abstract and unobservable, multiple studies are needed over time in order to achieve sufficient evidence of construct-related validity evidence. Two sources of validity that further establish an instrument's construct validity of an instrument are convergent and discriminant validity (Messick, 1995). Campbell and Fiske (1959), and later Messick (1995), describe convergent and discriminant validity; convergent validity refers to the confirmation by independent measures that produce high correlation coefficients between instruments designed to measure the same construct. The goal of discriminant validity is to confirm validity by demonstrating a lack of correlations between the unrelated constructs of two independent instruments. Campbell and Fiske (1959) developed a multi-trait, multi-method matrix to assist in assessment of validity. The matrix consists of a table representing the significance of the relationships between constructs and their measures.

Full document contains 194 pages
Abstract: The Medical Education literature discusses the importance of training competent medical professionals. The public expects physicians to be sensitive and aware of multiple factors, not just medical ailments. The biopsychosocial approach is a guide used to assist doctors in the exploration of a patient's narrative and in diagnosing a patient's difficulty based on multiple aspects of a patient's life including biological, psychological and sociological. During the 1980s, a pedagogical shift occurred in medical education toward an emphasis on the importance of teaching communication skills and taking a biopsychosocial approach to medicine. However, presently there is not a known, validated biopsychosocial assessment instrument in the medical education literature. This study attempts to provide psychometric validity for a biopsychosocial evaluation instrument. This evaluation tool is currently utilized to assess third and fourth year medical students during a complex Integrated Standardized Patient Examination (ISPE). The ISPE format provides an opportunity for students to demonstrate competency of skills in an integrated fashion. A sample of 149 digital recordings of medical student ISPE encounters was used to test the validity and reliability of the biopsychosocial D-D ISPE (Diabetes, Depression, Integrated Standardized Patient Examination) scoring instrument. The instrument's psychometric properties were determined using a variety of statistical analyses. The analyses included a card sort and an exploratory factor analysis to test the instrument's dimensionality. Discriminant and convergent validity methods were conducted in order to compare the ISPE scoring instrument's constructs to the constructs of two other previously validated instruments. The D-D ISPE communication and shared decision making, indicate weaker convergent validity than hypothesized. The D-D-ISPE biological scale evidenced discriminant validity (i.e., non significant correlations) with the measures of communication and shared decision making. The coder's scores of the two comparison instruments demonstrated good inter-rater reliability. The initial validity test also determined the instrument is reliable and valid. Cronbach's alpha tests determined the instrument as a whole and its subscales demonstrate high internal consistency. The instrument dimensionality analyzed by the card sort and the factor analysis, demonstrated the instrument possess three distinctive biological, psychological and sociological/communication factors. The psychological, biological and social Communication subscales are all moderately to strongly correlate with the D-D ISPE total scores. These findings suggest that the instrument can assess biopsychosocial competency of medical students during an ISPE examine focused on diabetes and depression. Finally, strengths and limitations of the study are examined and its implications for future use in medical education and research are explored.