Communication failures, poor decision-making or a lack of leadership may result in negative patient outcomes in the operating room but also on the ward/emergency room setting. Assessment tools such as the Non Technical Skills for Surgeons (NOTSS), the revised Non technical Skills (NOTECHS) Scale or the Ottawa Global Rating Scale (GRS), have only been applied to operating room/critical care settings. We assessed the construct validity and inter-rater reliability of these tools in a simulated surgical ward environment.

Twenty surgical residents (PGY1-2) were video-recorded managing a simulated post-operative complication using a high-fidelity full body simulator in a mock surgical ward. Subjects were randomized to one of 3 scenarios (hemorrhagic shock/septic shock/pulmonary embolism). Two independent experts rated subjects' non-technical skills performance using the NOTSS, NOTECHS and Ottawa GRS. For each scale, the intra-class correlation (ICC), (two-way mixed effect, single measures) measured the inter-rater reliability and Mann-Whitney U test was compared scores across resident training level.

Inter-rater reliability was strong for the NOTSS (ICC=0.75, p<0.001), NOTECHS (ICC=0.77, p<0.001) and Ottawa GRS (ICC=0.74, p<0.001) rating scales. PGY 2 residents scored better than PGY1 residents on each of the non-technical rating scales, but reached statistical significance (p<0.05) only on the NOTECHS and Ottawa GRS.

We demonstrated the construct validity and inter-rater reliability of existing non-technical rating scales in a post-operative crisis management domain. Future surgical curricula might incorporate these scales to assess residents' non-technical skills during post-operative crisis management.