Key result
The Regional Anaesthesia Procedural Skills (RAPS) tool demonstrated high reliability, with both the checklist and global rating scale achieving identical intraclass correlation coefficients of 0.80.
Answers are grounded in this paper and cite their sources.
The authors defend the Regional Anaesthesia Procedural Skills (RAPS) tool, emphasizing that both checklists and global rating scales demonstrate high reliability when assessing procedural skills in context.
We thank Ahmed et al. for their interest in our paper and for seeking further information on the educational concepts underlying the design of the Regional Anaesthesia Procedural Skills (RAPS) tool 1. With regards to the descriptors for the checklist items, it is important to not confuse subjectivity with context. Items were designed to encompass the behaviour that was being assessed, while each institution further applies the behaviour to their local context. As an example, ‘sets up equipment properly’ is a necessary step in regional anaesthesia performance. However, this will have different meanings for an institution that uses a specially-designed block room with fixed locations for equipment, using an ultrasound machine from Company A, compared to another institution without a procedure bay using a machine from Company B. One can write a tool with step-by-step descriptors, but being excessively prescriptive limits cross-institutional applicability, as well as creating an unwieldy tool that defeats clinical feasibility. Instead, we (like others 2) encourage users of assessment tools to define the behaviour in context, and this was the approach taken in our previous work in evaluating assessment tools 3, 4. We also point out that, despite no training in the use of RAPS, we found that the checklist was highly reliable amongst the ten experts (representing eight different institutions across five Australian cities) during the test-retest validation phase of our study, suggesting that users can successfully interpret and apply the items included in it. The assertion that Likert-based global rating scales are less reliable than dichotomous checklists is not supported by the evidence. Both types share similar characteristics 5-7. Indeed, we found for the RAPS tool that both the checklist and global rating scale had identical intraclass correlation coefficients (ICC) of 0.80 during the clinical validation phase. A recent systematic review of 45 studies has confirmed this, finding that global rating scales are actually more likely to discriminate expert behaviours, and slightly more reliable than checklists 8. We agree with Bould et al's editorial that ‘the current best evidence for a gold standard for assessment of procedural skills in anaesthesia consists of a combination of previously validated checklists and global rating scales, used prospectively by a trained observer, for a procedure performed in an actual patient’ 9. It is not entirely correct that ICC does not measure agreement; it has six different forms, and it is important to articulate which form is used for analysis 10, 11. In our study methods, we specified the ICC (2A, k) model, which measures both absolute agreement and correlation between assessors' score differences. This type of ICC is actually a very stringent test of reliability. The consultant anaesthetist and anaesthesia assistant were advised not to prepare, comment, or intervene during filming of trainees performing the procedure. They were permitted to intervene if they believed patient safety was about to be compromised. In the videos, this intervention was almost always in the form of a verbal prompt to assess the trainee's cognitive process. If the trainee's response still showed a lack of understanding, the supervising anaesthetist provided a corrective statement. This automatically constituted an unsatisfactory performance in that aspect of the block. Lastly, there is no dispute that different regional anaesthesia techniques activate different cognitive abilities and psychomotor skills. Acknowledgement of these different levels of complexity already exists by defining blocks as basic, intermediate, and advanced. However, we do not believe that each block deserves its own assessment tool. Instead, we have shown in our study that the RAPS tool can encompass these different skillsets, and achieve successful assessment of performance with good reliability, validity, and feasibility.
No takes yet. Share an insight, caveat, or question.
Chuan et al. (2016) conducted a letter in Regional anaesthesia procedural skills assessment. Regional Anaesthesia Procedural Skills (RAPS) tool was evaluated on Intraclass correlation coefficient (ICC) for reliability (ICC 0.80). The Regional Anaesthesia Procedural Skills (RAPS) tool demonstrated high reliability, with both the checklist and global rating scale achieving identical intraclass correlation coefficients of 0.80.
Synapse has enriched 10 related papers. Consider them for comparative context: