Malachi G. Sheahan1, Claudie Sheahan1, Jodi Gerdes1, Jean Bismuth3, Jason Lee5, Murray Shames2, Julia Wilkinson1, John Eidt4
1Louisiana State University Health Sciences Center, Marrero, LA; 2University of South Florida, Tampa, FL; 3The Methodist Hospital, Houston, TX; 4Greenville Health System University Medical Center, Greenville, SC; 5Stanford University Medical Center, Stanford, CA.
OBJECTIVES: While a standardized approach has been taken to assess the cognitive skills of vascular trainees in the United States, no formal system exists to test technical ability. Our goals were to develop and validate three vascular skill assessment models, as well as to train assessors to deliver consistent evaluations.
METHODS: Twenty surgical trainees (range: 4th year student-PGY 5 resident) completed three vascular skill assessment models, each under the observation of two experienced assessors blinded to their training level. Two models were designed to simulate an end-to-side anastomosis (ES) and a patch angioplasty (Patch). A third model required suturing around a clock-face design printed on patch material (CF). ACGME log experience was recorded. Secondary evaluations of the finished models were then performed by four blinded assessors.
RESULTS: Inter-rater reliability among the 7 assessors was high (α=0.91). Evaluations acquired by direct observation correlated well with participants' training level/experience for all three models (ES r=0.85/0.83, Patch r=0.71/0.69, CF r=0.82/0.84). Highest correlation with training level/experience was obtained with a combined score for each participant incorporating all observed ratings on each model (r=0.90/0.93). Construct validity was demonstrated by each model's ability to discern junior (Pre-MD to PGY2) from senior (PGY 3-5) trainees (ES p<0.005, Patch p<0.05, CF p<0.001). Internal consistency was confirmed for each participant on all three models (α=0.89). Finished product evaluation demonstrated fair to poor correlation with training level/experience (ES r=0.58/0.51, Patch r=0.53/0.44, CF r=0.24/0.12).
CONCLUSIONS: These results provide construct validity for three vascular skill assessment models. Our data also demonstrates that the most accurate assessments are obtained by direct observation with trained evaluators. Validated models and experienced assessors should be considered obligatory components of a meaningful system for vascular skill testing and improvement.
AUTHOR DISCLOSURES: J. Bismuth: Nothing to disclose; J. Eidt: Nothing to disclose; J. Gerdes: Nothing to disclose; J. Lee: Nothing to disclose; M. Shames: Nothing to disclose; C. Sheahan: Nothing to disclose; M. G. Sheahan: WL Gore Inc., Research grants; J. Wilkinson: Nothing to disclose.
Posted April 2013