If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
To determine whether a structured, behavior-based applicant interview predicts future success in an obstetrics and gynecology residency program.
Using a modified pre-post study design, we compared behavior-based interview scores of our residency applicants to a postmatch evaluation completed by the applicant's current residency program director. Applicants were evaluated on the following areas: academic record, professionalism, leadership, trainability/suitability for the specialty, and fit for the program.
Information was obtained for 45 (63%) applicants. The overall interview score did not correlate with overall resident performance. Applicant leadership subscore was predictive of leadership performance as a resident (P = .042). Academic record was associated with patient care performance as a resident (P = .014), but only for graduates of US medical schools. Five residents changed programs; these residents had significantly lower scores for trainability/suitability for the specialty (P = .020).
Behavioral interviewing can provide predictive information regarding success in an obstetrics and gynecology training program.
Each academic year, residency programs and applicants expend significant time, energy, and financial resources interviewing applicants for future residency positions. In the 2009 National Residency Matching Program (NRMP) main residency match, a total of 51,882 applicants pursued the 25,185 training positions available in the United States. This included 1796 applicants for the 1185 residency positions in the specialty of obstetrics and gynecology.
reviewed interview scores for their applicants to determine whether the scores correlated with any measures used to evaluate the residents during their training. Interview scores did not correlate with any of the measures, including knowledge, judgment, motor skills, or intrapersonal attitudes.
Performance on the United States Medical Licensing Examination (USMLE) has been shown to be positively correlated with intraining service examinations, but not with faculty evaluation of resident performance.
found that 21.6% of residents either: switched programs, changed specialties, completed training on an atypical academic cycle, or left graduate medical education all together.
In 2006, the St. Vincent Hospital Department of Obstetrics/Gynecoloy embarked on a program of behavior-based interviewing for all applicants to its residency program, with the hope of identifying applicants who would be successful both in the chosen specialty and in the St. Vincent program. This study reports the results of this interview process to predict future resident performance.
Materials and Methods
The St. Vincent Hospital Obstetrics/Gynecology residency is a community-based program consisting of 4 residents per academic year (approved for expansion to 5 positions in June 2007). The hospital is 1 of 4 main teaching sites for the required third year obstetrics/gynecology clerkship for medical students from the Indiana University School of Medicine. Approximately 200-250 students apply for the available residency positions, with the majority of the US graduates matriculating from programs in the Midwest region.
Beginning in November 2006, the St. Vincent Hospital Department of Obstetrics/Gynecology embarked on a structured behavior-based interview for applicants to its residency program. Applicants were scored on the following areas: academic record, professionalism, leadership, trainability/suitability for the specialty, and fit for the program. Academic record was scored by the program director through a review of the information available in the written application. The other 4 aspects were measured through a series of behavior-based questions at individual interview stations. Interviews for professionalism, leadership, and trainability/suitability were conducted simultaneously by 2 faculty members. When scheduling prevented the presence of both members, 1 faculty member conducted the interview. Fit for the program was assessed by a panel of 3 residents. Through literature review and discussions with subject matter experts, behavior-based questions pertinent to each subject area were developed, as were scoring sheets with specific examples of scores for particular responses (measured aspects of each subject area, sample scoring sheets, and sample questions, SUPPLEMENTARY FIGURE 1, SUPPLEMENTARY FIGURE 2, SUPPLEMENTARY FIGURE 3, SUPPLEMENTARY FIGURE 4). Faculty and residents were educated regarding the new interview protocol during a series of formal and informal meetings. During the interviews, faculty and residents could ask any of the potential questions available for their session, allowing some flexibility from interview to interview. At the end of each interview encounter, every interviewing faculty or resident completed their score sheet separately and independently from their partner(s). Scores for all 5 categories were determined by the average of the interviewer's scores. With each category having a potential value of 36, the maximum interview score for each applicant was 180. These scores provided the initial ranking list of applicants on which all future discussions were based. A database was maintained with all interview data, as well as medical school location (US-based program or international-based program).
For all applicants from November 2006 through January 2008, eventual match sites were identified through a combination of (1) the NRMP Match Results by Matched Applicant provided to the St. Vincent program, (2) the Council on Resident Education in Obstetrics and Gynecology (CREOG) resident database, and (3) individual program web site reviews. The program director for each program was identified through the listing of accredited programs at the Accreditation Council for Graduate Medical Education website (www.acgme.org).
An electronic survey was developed and sent to each director of a program into which a St. Vincent applicant had matched. Program director e-mails were identified through the AGCME database. E-mails with links to the electronic survey were sent to the appropriate program directors in September 2009, at which time the former applicants would have completed either 1 or 2 years of residency training. The survey was designed to measure the following resident attributes:
In addition, questions regarding awards, disciplinary actions for academic or professionalism-related issues, and the resident's continued status in the program were included. If the resident was no longer with the training program, program directors were asked to explain the change in training status. Information on type of program (community or university) was also collected.
For residents matching into the St. Vincent training program, it was thought the program director's completion of the survey would lead to potential bias, as he is the principal investigator of this study (E.S.). Therefore, the same survey for St. Vincent residents was completed by the consensus opinion of a core group of 3 faculty members with extensive clinical experience with the residents.
To maintain applicant confidentiality, once data were collected from the electronic survey and matched to the original interview score, all data were deidentified. When means between groups were compared, Student t tests were performed to determine statistical significance. Linear regression was used to determine relationships between pretest (interview) and posttest (electronic survey) scores. A P value of < .05 was considered significant.
The project was submitted to the St. Vincent Hospital Institutional Review Board (IRB) and was granted exempt status as an educational project.
From November 2006 through January 2008, a total of 80 applicants interviewed for residency positions in obstetrics/gynecology at St. Vincent Hospital. Of these applicants, 65 (81%) were women, 15 (19%) were men, and 74 (92.5%) were from US allopathic training programs. Of these, 72 matched into obstetrics/gynecology training programs, and 8 either did not match or matched into other specialty programs (4 family medicine, 1 pediatrics, 1 general surgery, 1 anesthesia, and 1 unmatched).
The 72 applicants matched into a total of 42 obstetrics/gynecology residency training programs. Of the 42 electronic surveys sent to these program directors, a total of 24 surveys (57%) were completed, providing follow-up data for 36 of the applicants. Including the 9 residents matching into the St. Vincent program, this provided follow-up data for 45 (63%) of the applicants matching into obstetrics/gynecology training programs.
In evaluating the assessment of overall resident performance by their program director, there was not a significant relationship between the overall interview score and the overall performance score (P = .784, Figure 1). This remained the case when separating results based by type of training program (university based, n = 25, P = .337; community based, n = 20, P = .952).
Scores for the applicant's academic record did not predict resident scores for patient care (P = .147) or medical knowledge (P = .125). However, when analyzing results for only graduates of US medical schools (43/45 applicants with available data), a significant relationship did exist between academic record and patient care (P = .014, Figure 2). Scores for medical knowledge, though, remained nonsignificant (P = .075).
Applicant scores for professionalism did not predict resident scores for professionalism (P = .685, Figure 3). However, a relationship did exist between applicant scores in leadership and leadership scores as a resident (P = .042, Figure 4).
Of the 45 applicants with available data, 5 (11%) changed programs during the time of the study (3 changed to different obstetrics/gynecology programs, 1 to anesthesia, and 1 to emergency medicine). Based on univariate logistic regression of interview scores to program status (still in program or changed programs), only trainability/suitability was a significant predictor of remaining in the training program (P = .020, Table 1). Fit for the program was not predictive of resident attrition (P = .898, Table 1).
TABLEUnivariate logistic regression of interview scores vs continued status in the program
Prediction of remaining in program (P value)
Fit for program
Strand. Predicting resident success. Am J Obstet Gynecol 2011.
Developed by industrial psychologists, behavioral interviewing focuses on the premise that past performance is the best indicator of future performance. Instead of using vague questions such as “What are your strengths and weaknesses?” behavioral interviews involve questions regarding specific behaviors that often represent decision-making skills, critical thinking, and interpersonal communication styles. In one of the first studies to evaluate the use of behavioral interviews, Janz
compared behavior-based interviews to traditional, unstructured interviews of prospective teaching assistants. Using student ratings of the teaching assistants as the outcome measure, behavioral interviews were significantly more predictive of performance than the traditional interview.
In this study of a standardized behavior-based interview process, important significant associations were found. Specifically, applicants with higher scores for leadership tended to be rated higher for this characteristic as a resident. In addition, applicants who changed training programs had lower interview scores in trainability/suitability. This may illustrate an important risk factor to be considered in selecting an applicant, given the significant time, expense, and disruption that occurs when a resident leaves a program and a replacement is sought.
Interestingly, one measure of our applicant assessment, academic record, was not behavior-based and yet did significantly correlate with patient care performance as a resident. This stands in contrast to the cognitive factors assessed in the previous studies. Our finding was significant only when comparing data for graduates of US medical schools. The association lost significance when data for applicants from international medical schools was included. This may indicate that cognitive factors (USMLE scores, transcripts) for international graduates may be particularly poor predictors of future performance, and should be considered in the applicant's overall assessment.
Our global interview score, though, did not predict future performance as a resident. This stands in contrast to a study by Olawaiye et al,
where an applicant's position on the program's NRMP rank list did predict future first year clinical performance. There may be several reasons for the different findings. In the study by Olawaiye et al,
similar but different categories were measured during the interview—communication skills, insight into specialty, motivation, compassion, and “fit” into the program. The nature of the interview questions (behavior-based or otherwise) was not described. Finally, resident performance was not compared with the overall interview score (or subscore), but rather to “rank percentile,” which represented the applicant's position on the final ranking list.
Our study is the first attempt to use behavioral interviewing to predict future resident performance. In addition, ours is the first to go beyond global scores and investigate subscores such as leadership, professionalism, and trainability/suitability as possible predictors of future success. These findings point to areas of potential future study, as programs look for predictive characteristics of resident success or failure.
One strength of the study is its assessment of residents in a number of different programs. Program directors completing the survey were unaware of the original applicant's evaluation, which allowed for an unbiased assessment of the applicant's performance as a resident. Removing the St. Vincent program director from the evaluation process limited this bias as well, although a potential for bias remained if the St. Vincent faculty involved remembered the applicant's interview results. Interviews had occurred roughly 2 to 3 years previously, which makes significant bias unlikely.
An additional strength of the study was the preparation of residents and faculty for the new interview technique, and the creation of standardized grading templates with specific examples of answers corresponding to specific marks. This should have limited variation in evaluations submitted on different interview dates.
There are several weaknesses to the study. Although the survey provided data for 63% of our applicants, it is unknown if the remaining 37% were more or less successful as residents. Program directors with residents demonstrating poor performance may have been less likely to complete a survey regarding their performance, despite reassurances regarding confidentiality of the subjects. Program expectations of a quality resident may also be substantially different among programs. Although our data did not show significant differences when analyzed comparing results from community-based or university-based programs, variations in programs may create different definitions of “success.” Although we attempted to standardize our survey for the program directors, we certainly must allow that expectations of performance vary from program to program.
Overall, our study points to the potential usefulness of behavioral interviewing to identify either predictors of success or risk factors for failure during residency. Trainability/suitability, in particular, may be a subject of further interest in identifying applicants at risk for changing programs. Future studies should investigate additional noncognitive traits of our applicants to help better predict future residency success.