Advertisement

The Fundamentals of Vaginal Surgery pilot study: developing, validating, and setting proficiency scores for a vaginal surgical skills simulation system

Published:August 28, 2021DOI:https://doi.org/10.1016/j.ajog.2021.08.037

      Background

      Surgical training in the simulation lab can develop basic skills that translate to the operating room. Standardized, basic skills programs that are supported by validated assessment measures exist for open, laparoscopic, and endoscopic surgery; however, there is yet to be a nationally recognized and widely implemented basic skills program specifically for vaginal surgery.

      Objective

      Develop a vaginal surgical simulation system; evaluate robust validity evidence for the simulation system and its related performance measures; and establish a proficiency score that discriminates between novice and experienced vaginal surgeon performance.

      Study Design

      In this 3-phased study, we developed the Fundamentals of Vaginal Surgery simulation system consisting of (1) the Fundamentals of Vaginal Surgery Trainer, a task trainer; (2) a validated regimen of tasks to be performed on the trainer; and (3) performance measures to determine proficiency. In Phase I, we developed the task trainer and selected surgical tasks by performing a needs assessment and hierarchical task analyses, with review and consensus from an expert panel. In Phase II, we conducted a national survey of vaginal surgeons to collect validity evidence regarding test content, response process, and internal structure relevant to the simulation system. In Phase III, we compared performance of novice (first and second year residents) and experienced (third and fourth year residents, fellows, and faculty) surgeons on the simulation system to evaluate relevant relationships to other variables and consequences. Performance measures were analyzed to set a proficiency score that would discriminate between novice and expert (faculty) vaginal surgical performance.

      Results

      A novel task trainer and 6 basic vaginal surgical skills were developed in Phase I. In Phase II, the survey responses of 48 participants (27 faculty surgeons, 6 fellows, and 14 residents) were evaluated on the dimensions of test content, response process, and internal structure. To support evidence of test content, the participants deemed the task trainer and surgical tasks representative of intended surgical field and supportive of typical surgical actions (mean scores, 3.8–4.4/5). For response process, rater-data analysis revealed high rating variability regarding prototype color. This early evidence confirmed the value of a white prototype. For internal structure, there was high agreement among rater groups (obstetricians and gynecologists generalists vs Female Pelvic Medicine and Reconstructive Surgery specialists: interclass correlation coefficient range, 0.59–0.91; learners vs faculty interclass correlation coefficient range, 0.64–1.0). There were no differences in ratings across institution type, surgeon volume, expertise (P>.14). In Phase III, we analyzed performance from 23 participants (15 [65%] obstetricians and gynecologists residents, 3 [13%] fellows, and 5 [22%] Female Pelvic Medicine and Reconstructive Surgery faculty). Experienced surgeons scored significantly higher than novice surgeons (median, 467.5; interquartile range, [402.5–542.5] vs median, 261.5; interquartile range, [211.5–351.0]; P<.001). Based on these data, setting a proficiency score threshold at 400 results in 0% (0/6) novices attaining the score, with 100% (5/5) experts exceeding it.

      Conclusion

      We present validity evidence relevant to all 5 sources which supports the use of this novel simulation system for basic vaginal surgical skills. To complement the system, a proficiency score of 400 was established to discriminate between novices and experts.

      Key words

      To read this article in full you will need to make a payment

      Subscribe:

      Subscribe to American Journal of Obstetrics & Gynecology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Flum D.R.
        • Koepsell T.
        • Heagerty P.
        • Pellegrini C.A.
        The nationwide frequency of major adverse outcomes in antireflux surgery and the role of surgeon experience, 1992-1997.
        J Am Coll Surg. 2002; 195: 611-618
        • Scott D.J.
        Patient safety, competency, and the future of surgical simulation.
        Simul Healthc. 2006; 1: 164-170
        • Carraccio C.
        • Englander R.
        • Van Melle E.
        • et al.
        Advancing competency-based medical education: a charter for clinician-educators.
        Acad Med. 2016; 91: 645-649
        • de Montbrun S.L.
        • Macrae H.
        Simulation in surgical education.
        Clin Colon Rectal Surg. 2012; 25: 156-165
        • Goova M.T.
        • Hollett L.A.
        • Tesfay S.T.
        • et al.
        Implementation, construct validity, and benefit of a proficiency-based knot-tying and suturing curriculum.
        J Surg Educ. 2008; 65: 309-315
        • Mashaud L.B.
        • Arain N.A.
        • Hogg D.C.
        • Scott D.J.
        Development, validation, and implementation of a cost-effective intermediate-level proficiency-based knot-tying and suturing curriculum for surgery residents.
        J Surg Educ. 2013; 70: 193-199
        • Vassiliou M.C.
        • Dunkin B.J.
        • Marks J.M.
        • Fried G.M.
        FLS and FES: comprehensive models of training and assessment.
        Surg Clin North Am. 2010; 90: 535-558
        • Nugent E.
        • Shirilla N.
        • Hafeez A.
        • et al.
        Development and evaluation of a simulator-based laparoscopic training program for surgical novices.
        Surg Endosc. 2013; 27: 214-221
        • Washburn E.E.
        • Cohen S.L.
        • Manoucheri E.
        • Zurawin R.K.
        • Einarsson J.I.
        Trends in reported resident surgical experience in hysterectomy.
        J Minim Invasive Gynecol. 2014; 21: 1067-1070
      1. Altman KB, Burrell D, Chen G, Chou B, Fashokun T, Vaginal Hysterectomy Simulation. ACOG Simulation Working Group. Available at: https://www.acog.org/education-and-events/simulations/scog008/simulation. Accessed March 1, 2021.

        • Balgobin S.
        • Owens D.M.
        • Florian-Rodriguez M.E.
        • Wai C.Y.
        • McCord E.H.
        • Hamid C.A.
        Vaginal hysterectomy suturing skills training model and curriculum.
        Obstet Gynecol. 2019; 134: 553-558
        • Barrier B.F.
        • Thompson A.B.
        • McCullough M.W.
        • Occhino J.A.
        A novel and inexpensive vaginal hysterectomy simulator.
        Simul Healthc. 2012; 7: 374-379
        • Burkett L.S.
        • Makin J.
        • Ackenbom M.
        • Artsen A.
        • Bradley M.
        Validation of transvaginal hysterectomy surgical model - modification of the flowerpot model to improve vesicovaginal plane simulation.
        J Minim Invasive Gynecol. 2021; 28: 1526-1530
        • Chen C.C.G.
        • Lockrow E.G.
        • DeStephano C.C.
        • et al.
        Establishing validity for a vaginal hysterectomy simulation model for surgical skills assessment.
        Obstet Gynecol. 2020; 136: 942-949
        • Geoffrion R.
        • Suen M.W.
        • Koenig N.A.
        • et al.
        Teaching vaginal surgery to junior residents: initial validation of 3 novel procedure-specific low-fidelity models.
        J Surg Educ. 2016; 73: 157-161
        • Greer J.A.
        • Segal S.
        • Salva C.R.
        • Arya L.A.
        Development and validation of simulation training for vaginal hysterectomy.
        J Minim Invasive Gynecol. 2014; 21: 74-82
        • Miyazaki D.
        • Matthews C.A.
        • Kia M.V.
        • El Haraki A.S.
        • Miyazaki N.
        • Chen C.C.G.
        Validation of an educational simulation model for vaginal hysterectomy training: a pilot study.
        Int Urogynecol J. 2019; 30: 1329-1336
        • Vaughan M.H.
        • Kim-Fine S.
        • Hullfish K.L.
        • Smith T.M.
        • Siddiqui N.Y.
        • Trowbridge E.R.
        Validation of the simulated vaginal hysterectomy trainer.
        J Minim Invasive Gynecol. 2018; 25: 1101-1106
        • Zoorob D.
        • Frenn R.
        • Moffitt M.
        • et al.
        Multi-institutional validation of a vaginal hysterectomy simulation model for resident training.
        J Minim Invasive Gynecol. 2021; 28: 1490-1496.e1
        • Burkett D.
        • Horwitz J.
        • Kennedy V.
        • Murphy D.
        • Graziano S.
        • Kenton K.
        Assessing current trends in resident hysterectomy training.
        Female Pelvic Med Reconstr Surg. 2011; 17: 210-214
        • American Educational Research Association, American Psychological Association, National Council on Measurement in Education
        Standards for educational and psychological testing. 2009.
        (Available at:) (Accessed March 1, 2021)
        • Messick S.
        Validity.
        in: Linn R. Educational measurement. 3rd ed. American Council on Education and Macmillan, New York, NY1989: 13-103
        • Cook D.A.
        • Zendejas B.
        • Hamstra S.J.
        • Hatala R.
        • Brydges R.
        What counts as validity evidence? Examples and prevalence in a systematic review of simulation-based assessment.
        Adv Health Sci Educ Theory Pract. 2014; 19: 233-250
        • Cheng A.
        • Kessler D.
        • Mackinnon R.
        • et al.
        Reporting guidelines for health care simulation research: extensions to the CONSORT and STROBE statements.
        Simul Healthc. 2016; 11: 238-248
        • Barnhart K.T.
        • Izquierdo A.
        • Pretorius E.S.
        • Shera D.M.
        • Shabbout M.
        • Shaunik A.
        Baseline dimensions of the human vagina.
        Hum Reprod. 2006; 21: 1618-1622
        • Kern D.E.
        • Thomas P.A.
        • Hughes M.T.
        Curriculum development for medical education: a six-step approach. Vol Baltimore, Md.
        2nd ed. Johns Hopkins University Press, Baltimore, MD2009
        • Sarker S.K.
        • Chang A.
        • Albrani T.
        • Vincent C.
        Constructing hierarchical task analysis in surgery.
        Surg Endosc. 2008; 22: 107-111
        • DeLancey J.O.L.
        Basic exercises in surgical technique.
        Davis and Geck, Wayne, NJ1987
        • Derossis A.M.
        • Fried G.M.
        • Abrahamowicz M.
        • Sigman H.H.
        • Barkun J.S.
        • Meakins J.L.
        Development of a model for training and evaluation of laparoscopic skills.
        Am J Surg. 1998; 175: 482-487
        • Fraser S.A.
        • Klassen D.R.
        • Feldman L.S.
        • Ghitulescu G.A.
        • Stanbridge D.
        • Fried G.M.
        Evaluating laparoscopic skills: setting the pass/fail score for the MISTELS system.
        Surg Endosc. 2003; 17: 964-967
        • Ritter E.M.
        • Scott D.J.
        Design of a proficiency-based skills training curriculum for the fundamentals of laparoscopic surgery.
        Surg Innov. 2007; 14: 107-112
        • Laubert T.
        • Esnaashari H.
        • Auerswald P.
        • et al.
        Conception of the Lübeck Toolbox Curriculum for basic minimally invasive surgery skills.
        Langenbecks Arch Surg. 2018; 403: 271-278
        • Linacre J.M.
        Many-facet Rasch management.
        MESA Press, Chicago, IL1989
        • Nicholls D.
        • Sweet L.
        • Muller A.
        • Hyett J.
        Teaching psychomotor skills in the twenty-first century: revisiting and reviewing instructional approaches through the lens of contemporary literature.
        Med Teach. 2016; 38: 1056-1063
        • Scott D.J.
        • Goova M.T.
        • Tesfay S.T.
        A cost-effective proficiency-based knot-tying and suturing curriculum for residency programs.
        J Surg Res. 2007; 141: 7-15
        • Munro M.G.
        • Advincula A.P.
        • Banks E.H.
        • et al.
        Essentials in minimally invasive gynecology manual skills construct validation trial.
        Obstet Gynecol. 2020; 136: 83-96
        • Hanczar B.
        • Hua J.
        • Sima C.
        • Weinstein J.
        • Bittner M.
        • Dougherty E.R.
        Small-sample precision of ROC-related estimates.
        Bioinformatics. 2010; 26: 822-830