Advertisement

Can a structured, behavior-based interview predict future resident success?

Published:April 01, 2011DOI:https://doi.org/10.1016/j.ajog.2011.02.019

      Objective

      To determine whether a structured, behavior-based applicant interview predicts future success in an obstetrics and gynecology residency program.

      Study Design

      Using a modified pre-post study design, we compared behavior-based interview scores of our residency applicants to a postmatch evaluation completed by the applicant's current residency program director. Applicants were evaluated on the following areas: academic record, professionalism, leadership, trainability/suitability for the specialty, and fit for the program.

      Results

      Information was obtained for 45 (63%) applicants. The overall interview score did not correlate with overall resident performance. Applicant leadership subscore was predictive of leadership performance as a resident (P = .042). Academic record was associated with patient care performance as a resident (P = .014), but only for graduates of US medical schools. Five residents changed programs; these residents had significantly lower scores for trainability/suitability for the specialty (P = .020).

      Conclusion

      Behavioral interviewing can provide predictive information regarding success in an obstetrics and gynecology training program.

      Key words

      Each academic year, residency programs and applicants expend significant time, energy, and financial resources interviewing applicants for future residency positions. In the 2009 National Residency Matching Program (NRMP) main residency match, a total of 51,882 applicants pursued the 25,185 training positions available in the United States. This included 1796 applicants for the 1185 residency positions in the specialty of obstetrics and gynecology.
      National Residency Matching Program resultsdata
      2009 main residency match.
      For Editors' Commentary, see Table of Contents
      See related editorial, page 369
      Despite these investments, little is known about the use of the residency interview in predicting an applicant's future performance as a resident. For instance, Metro et al
      • Metro D.G.
      • Talarico J.F.
      • Patel R.M.
      • Wetmore A.L.
      The resident application process and its correlation to future performance as a resident.
      reviewed interview scores for their applicants to determine whether the scores correlated with any measures used to evaluate the residents during their training. Interview scores did not correlate with any of the measures, including knowledge, judgment, motor skills, or intrapersonal attitudes.
      • Metro D.G.
      • Talarico J.F.
      • Patel R.M.
      • Wetmore A.L.
      The resident application process and its correlation to future performance as a resident.
      Performance on the United States Medical Licensing Examination (USMLE) has been shown to be positively correlated with intraining service examinations, but not with faculty evaluation of resident performance.
      • Bell J.G.
      • Kanellitsas I.
      • Shaffer L.
      Selection of obstetrics and gynecology residents on the basis of medical school performance.
      • Brothers T.E.
      • Wetherholt S.
      Importance of the faculty interview during the resident application process.
      Other studies have shown interview scores did not predict physicians at risk of later impairment.
      • Dubovsky S.L.
      • Gendel M.
      • Dubovsky A.N.
      • Rosse J.
      • Levin R.
      • House R.
      Do data obtained from admissions interviews and resident evaluations predict later personal and practice problems?.
      Furthermore, resident attrition is a significant problem for many programs. After a cohort of 1055 residents started their obstetrics and gynecology residency training in 2001, McAlister et al
      • McAlister R.P.
      • Andriole D.A.
      • Brotherton S.E.
      • Jeffe D.B.
      Attrition in residents entering US obstetrics and gynecology residencies: analysis of national GME census data.
      found that 21.6% of residents either: switched programs, changed specialties, completed training on an atypical academic cycle, or left graduate medical education all together.
      In 2006, the St. Vincent Hospital Department of Obstetrics/Gynecoloy embarked on a program of behavior-based interviewing for all applicants to its residency program, with the hope of identifying applicants who would be successful both in the chosen specialty and in the St. Vincent program. This study reports the results of this interview process to predict future resident performance.

      Materials and Methods

      The St. Vincent Hospital Obstetrics/Gynecology residency is a community-based program consisting of 4 residents per academic year (approved for expansion to 5 positions in June 2007). The hospital is 1 of 4 main teaching sites for the required third year obstetrics/gynecology clerkship for medical students from the Indiana University School of Medicine. Approximately 200-250 students apply for the available residency positions, with the majority of the US graduates matriculating from programs in the Midwest region.
      Beginning in November 2006, the St. Vincent Hospital Department of Obstetrics/Gynecology embarked on a structured behavior-based interview for applicants to its residency program. Applicants were scored on the following areas: academic record, professionalism, leadership, trainability/suitability for the specialty, and fit for the program. Academic record was scored by the program director through a review of the information available in the written application. The other 4 aspects were measured through a series of behavior-based questions at individual interview stations. Interviews for professionalism, leadership, and trainability/suitability were conducted simultaneously by 2 faculty members. When scheduling prevented the presence of both members, 1 faculty member conducted the interview. Fit for the program was assessed by a panel of 3 residents. Through literature review and discussions with subject matter experts, behavior-based questions pertinent to each subject area were developed, as were scoring sheets with specific examples of scores for particular responses (measured aspects of each subject area, sample scoring sheets, and sample questions, SUPPLEMENTARY FIGURE 1, SUPPLEMENTARY FIGURE 2, SUPPLEMENTARY FIGURE 3, SUPPLEMENTARY FIGURE 4). Faculty and residents were educated regarding the new interview protocol during a series of formal and informal meetings. During the interviews, faculty and residents could ask any of the potential questions available for their session, allowing some flexibility from interview to interview. At the end of each interview encounter, every interviewing faculty or resident completed their score sheet separately and independently from their partner(s). Scores for all 5 categories were determined by the average of the interviewer's scores. With each category having a potential value of 36, the maximum interview score for each applicant was 180. These scores provided the initial ranking list of applicants on which all future discussions were based. A database was maintained with all interview data, as well as medical school location (US-based program or international-based program).
      For all applicants from November 2006 through January 2008, eventual match sites were identified through a combination of (1) the NRMP Match Results by Matched Applicant provided to the St. Vincent program, (2) the Council on Resident Education in Obstetrics and Gynecology (CREOG) resident database, and (3) individual program web site reviews. The program director for each program was identified through the listing of accredited programs at the Accreditation Council for Graduate Medical Education website (www.acgme.org).
      An electronic survey was developed and sent to each director of a program into which a St. Vincent applicant had matched. Program director e-mails were identified through the AGCME database. E-mails with links to the electronic survey were sent to the appropriate program directors in September 2009, at which time the former applicants would have completed either 1 or 2 years of residency training. The survey was designed to measure the following resident attributes:
      • Patient care
      • Medical knowledge
      • Surgical skills
      • Communication
      • Professionalism
      • Clinical documentation
      • Leadership
      • Teamwork
      • Overall impression
      In addition, questions regarding awards, disciplinary actions for academic or professionalism-related issues, and the resident's continued status in the program were included. If the resident was no longer with the training program, program directors were asked to explain the change in training status. Information on type of program (community or university) was also collected.
      For residents matching into the St. Vincent training program, it was thought the program director's completion of the survey would lead to potential bias, as he is the principal investigator of this study (E.S.). Therefore, the same survey for St. Vincent residents was completed by the consensus opinion of a core group of 3 faculty members with extensive clinical experience with the residents.
      To maintain applicant confidentiality, once data were collected from the electronic survey and matched to the original interview score, all data were deidentified. When means between groups were compared, Student t tests were performed to determine statistical significance. Linear regression was used to determine relationships between pretest (interview) and posttest (electronic survey) scores. A P value of < .05 was considered significant.
      The project was submitted to the St. Vincent Hospital Institutional Review Board (IRB) and was granted exempt status as an educational project.

      Results

      From November 2006 through January 2008, a total of 80 applicants interviewed for residency positions in obstetrics/gynecology at St. Vincent Hospital. Of these applicants, 65 (81%) were women, 15 (19%) were men, and 74 (92.5%) were from US allopathic training programs. Of these, 72 matched into obstetrics/gynecology training programs, and 8 either did not match or matched into other specialty programs (4 family medicine, 1 pediatrics, 1 general surgery, 1 anesthesia, and 1 unmatched).
      The 72 applicants matched into a total of 42 obstetrics/gynecology residency training programs. Of the 42 electronic surveys sent to these program directors, a total of 24 surveys (57%) were completed, providing follow-up data for 36 of the applicants. Including the 9 residents matching into the St. Vincent program, this provided follow-up data for 45 (63%) of the applicants matching into obstetrics/gynecology training programs.
      In evaluating the assessment of overall resident performance by their program director, there was not a significant relationship between the overall interview score and the overall performance score (P = .784, Figure 1). This remained the case when separating results based by type of training program (university based, n = 25, P = .337; community based, n = 20, P = .952).
      Figure thumbnail gr1
      FIGURE 1Overall interview score vs overall performance as assessed by program director
      P = .784.
      Strand. Predicting resident success. Am J Obstet Gynecol 2011.
      Scores for the applicant's academic record did not predict resident scores for patient care (P = .147) or medical knowledge (P = .125). However, when analyzing results for only graduates of US medical schools (43/45 applicants with available data), a significant relationship did exist between academic record and patient care (P = .014, Figure 2). Scores for medical knowledge, though, remained nonsignificant (P = .075).
      Figure thumbnail gr2
      FIGURE 2Interview score of academic record vs patient care performance as a resident, US medical school graduates only
      P = .014.
      Strand. Predicting resident success. Am J Obstet Gynecol 2011.
      Applicant scores for professionalism did not predict resident scores for professionalism (P = .685, Figure 3). However, a relationship did exist between applicant scores in leadership and leadership scores as a resident (P = .042, Figure 4).
      Figure thumbnail gr3
      FIGURE 3Interview professionalism score vs professionalism rating as a resident
      P = .685.
      Strand. Predicting resident success. Am J Obstet Gynecol 2011.
      Figure thumbnail gr4
      FIGURE 4Interview leadership score vs leadership rating as a resident
      P = .042.
      Strand. Predicting resident success. Am J Obstet Gynecol 2011.
      Of the 45 applicants with available data, 5 (11%) changed programs during the time of the study (3 changed to different obstetrics/gynecology programs, 1 to anesthesia, and 1 to emergency medicine). Based on univariate logistic regression of interview scores to program status (still in program or changed programs), only trainability/suitability was a significant predictor of remaining in the training program (P = .020, Table 1). Fit for the program was not predictive of resident attrition (P = .898, Table 1).
      TABLEUnivariate logistic regression of interview scores vs continued status in the program
      Interview itemPrediction of remaining in program (P value)
      Academic record.930
      Professionalism.916
      Leadership.430
      Suitability/trainability.020
      Fit for program.898
      Overall score.575
      Strand. Predicting resident success. Am J Obstet Gynecol 2011.

      Comment

      Developed by industrial psychologists, behavioral interviewing focuses on the premise that past performance is the best indicator of future performance. Instead of using vague questions such as “What are your strengths and weaknesses?” behavioral interviews involve questions regarding specific behaviors that often represent decision-making skills, critical thinking, and interpersonal communication styles. In one of the first studies to evaluate the use of behavioral interviews, Janz
      • Janz T.
      Initial comparisons of patterned behavior-based interviews versus unstructured interviews.
      compared behavior-based interviews to traditional, unstructured interviews of prospective teaching assistants. Using student ratings of the teaching assistants as the outcome measure, behavioral interviews were significantly more predictive of performance than the traditional interview.
      • Janz T.
      Initial comparisons of patterned behavior-based interviews versus unstructured interviews.
      Given the potential benefit, many businesses have changed to behavior-based interviewing, including 65% of Fortune 500 companies.
      • Byers M.
      Interview Rx: a powerful guide for making your next interview a success.
      Because cognitive factors, such as grades/transcripts, USMLE scores, and class rank have not correlated with clinical success as a resident,
      • Bell J.G.
      • Kanellitsas I.
      • Shaffer L.
      Selection of obstetrics and gynecology residents on the basis of medical school performance.
      • Brothers T.E.
      • Wetherholt S.
      Importance of the faculty interview during the resident application process.
      • Erlandson E.E.
      • Calhoun J.G.
      • Barrack F.M.
      • et al.
      Resident selection: applicant selection criteria compared with performance.
      • Hojat M.
      • Gonnella J.S.
      • Veloski J.J.
      • Erdmann J.B.
      Is the glass half full or half empty? A reexamination of the associations between assessment measures during medical school and clinical competence after graduation.
      programs are turning to behavioral interviewing for candidate assessment.
      • Lyon D.
      • Wiper D.
      Finding our colleagues, finding ourselves: behavioral interviewing and a critical assessment of how we identify and interview residency candidates.
      In this study of a standardized behavior-based interview process, important significant associations were found. Specifically, applicants with higher scores for leadership tended to be rated higher for this characteristic as a resident. In addition, applicants who changed training programs had lower interview scores in trainability/suitability. This may illustrate an important risk factor to be considered in selecting an applicant, given the significant time, expense, and disruption that occurs when a resident leaves a program and a replacement is sought.
      Interestingly, one measure of our applicant assessment, academic record, was not behavior-based and yet did significantly correlate with patient care performance as a resident. This stands in contrast to the cognitive factors assessed in the previous studies. Our finding was significant only when comparing data for graduates of US medical schools. The association lost significance when data for applicants from international medical schools was included. This may indicate that cognitive factors (USMLE scores, transcripts) for international graduates may be particularly poor predictors of future performance, and should be considered in the applicant's overall assessment.
      Our global interview score, though, did not predict future performance as a resident. This stands in contrast to a study by Olawaiye et al,
      • Olawaiye A.
      • Yeh J.
      • Withiam-Leitch M.
      Resident selection process and prediction of clinical performance in an obstetrics and gynecology program.
      where an applicant's position on the program's NRMP rank list did predict future first year clinical performance. There may be several reasons for the different findings. In the study by Olawaiye et al,
      • Olawaiye A.
      • Yeh J.
      • Withiam-Leitch M.
      Resident selection process and prediction of clinical performance in an obstetrics and gynecology program.
      similar but different categories were measured during the interview—communication skills, insight into specialty, motivation, compassion, and “fit” into the program. The nature of the interview questions (behavior-based or otherwise) was not described. Finally, resident performance was not compared with the overall interview score (or subscore), but rather to “rank percentile,” which represented the applicant's position on the final ranking list.
      Our study is the first attempt to use behavioral interviewing to predict future resident performance. In addition, ours is the first to go beyond global scores and investigate subscores such as leadership, professionalism, and trainability/suitability as possible predictors of future success. These findings point to areas of potential future study, as programs look for predictive characteristics of resident success or failure.
      One strength of the study is its assessment of residents in a number of different programs. Program directors completing the survey were unaware of the original applicant's evaluation, which allowed for an unbiased assessment of the applicant's performance as a resident. Removing the St. Vincent program director from the evaluation process limited this bias as well, although a potential for bias remained if the St. Vincent faculty involved remembered the applicant's interview results. Interviews had occurred roughly 2 to 3 years previously, which makes significant bias unlikely.
      An additional strength of the study was the preparation of residents and faculty for the new interview technique, and the creation of standardized grading templates with specific examples of answers corresponding to specific marks. This should have limited variation in evaluations submitted on different interview dates.
      There are several weaknesses to the study. Although the survey provided data for 63% of our applicants, it is unknown if the remaining 37% were more or less successful as residents. Program directors with residents demonstrating poor performance may have been less likely to complete a survey regarding their performance, despite reassurances regarding confidentiality of the subjects. Program expectations of a quality resident may also be substantially different among programs. Although our data did not show significant differences when analyzed comparing results from community-based or university-based programs, variations in programs may create different definitions of “success.” Although we attempted to standardize our survey for the program directors, we certainly must allow that expectations of performance vary from program to program.
      Overall, our study points to the potential usefulness of behavioral interviewing to identify either predictors of success or risk factors for failure during residency. Trainability/suitability, in particular, may be a subject of further interest in identifying applicants at risk for changing programs. Future studies should investigate additional noncognitive traits of our applicants to help better predict future residency success.

      Appendix

      Figure thumbnail gra1
      SUPPLEMENTARY FIGURE 1Factors measured in St. Vincent's behavioral interviews
      Strand. Predicting resident success. Am J Obstet Gynecol 2011.
      Figure thumbnail gra2a
      SUPPLEMENTARY FIGURE 2Evaluation scoring sheets
      Strand. Predicting resident success. Am J Obstet Gynecol 2011.
      Figure thumbnail gra2b
      SUPPLEMENTARY FIGURE 2Evaluation scoring sheets
      Strand. Predicting resident success. Am J Obstet Gynecol 2011.
      Figure thumbnail gra2c
      SUPPLEMENTARY FIGURE 2Evaluation scoring sheets
      Strand. Predicting resident success. Am J Obstet Gynecol 2011.
      Figure thumbnail gra2d
      SUPPLEMENTARY FIGURE 2Evaluation scoring sheets
      Strand. Predicting resident success. Am J Obstet Gynecol 2011.
      Figure thumbnail gra2e
      SUPPLEMENTARY FIGURE 2Evaluation scoring sheets
      Strand. Predicting resident success. Am J Obstet Gynecol 2011.
      Figure thumbnail gra3
      SUPPLEMENTARY FIGURE 3Sample interview questions
      Strand. Predicting resident success. Am J Obstet Gynecol 2011.
      Figure thumbnail gra4
      SUPPLEMENTARY FIGURE 4Resident performance survey
      Strand. Predicting resident success. Am J Obstet Gynecol 2011.

      References

        • National Residency Matching Program resultsdata
        2009 main residency match.
        National Residency Matching Program. 2009;
        • Metro D.G.
        • Talarico J.F.
        • Patel R.M.
        • Wetmore A.L.
        The resident application process and its correlation to future performance as a resident.
        Anesth Analg. 2005; 100: 502-505
        • Bell J.G.
        • Kanellitsas I.
        • Shaffer L.
        Selection of obstetrics and gynecology residents on the basis of medical school performance.
        Am J Obstet Gynecol. 2002; 186: 1091-1094
        • Brothers T.E.
        • Wetherholt S.
        Importance of the faculty interview during the resident application process.
        J Surg Educ. 2007; 64: 378-385
        • Dubovsky S.L.
        • Gendel M.
        • Dubovsky A.N.
        • Rosse J.
        • Levin R.
        • House R.
        Do data obtained from admissions interviews and resident evaluations predict later personal and practice problems?.
        Acad Psychiatry. 2005; 29: 443-447
        • McAlister R.P.
        • Andriole D.A.
        • Brotherton S.E.
        • Jeffe D.B.
        Attrition in residents entering US obstetrics and gynecology residencies: analysis of national GME census data.
        Am J Obstet Gynecol. 2008; 199: 574.e1-574.e6
        • Janz T.
        Initial comparisons of patterned behavior-based interviews versus unstructured interviews.
        J Appl Psychol. 1982; 67: 577-580
        • Byers M.
        Interview Rx: a powerful guide for making your next interview a success.
        3rd ed. Nearline Publishers, Conyers, GA2007
        • Erlandson E.E.
        • Calhoun J.G.
        • Barrack F.M.
        • et al.
        Resident selection: applicant selection criteria compared with performance.
        Surgery. 1982; 92: 270-275
        • Hojat M.
        • Gonnella J.S.
        • Veloski J.J.
        • Erdmann J.B.
        Is the glass half full or half empty?.
        Acad Med. 1993; 68: S69-S76
        • Lyon D.
        • Wiper D.
        Finding our colleagues, finding ourselves: behavioral interviewing and a critical assessment of how we identify and interview residency candidates.
        APGO/CREOG Annual Meeting, San Diego, CA2009
        • Olawaiye A.
        • Yeh J.
        • Withiam-Leitch M.
        Resident selection process and prediction of clinical performance in an obstetrics and gynecology program.
        Teach Learn Med. 2006; 18: 310-315