The NHS Improvement report on operating theatres: really ‘getting it right first time’?

This is the authors' abstract. We don't add key points for this paper.

With a foreword written by the Chair of the Getting It Right First Time (GIRFT) programme, the NHS Improvement (NHSI) report on theatre productivity, Operating Theatres: Opportunities to Reduce Waiting Lists, would seem to be the final word 1. Yet, despite being based on an extensive dataset of some 1.7 million cases, the report is written in a style that does not acknowledge a large amount of evidence-based literature and therefore offers some worryingly false analyses. NHS Improvement works in partnership with GIRFT; but has this report really ‘got it right first time’? Nobody can dispute the fact that the huge investment in operating theatres could be better used. Waiting lists are too long, and the introduction to the NHSI report summarises some sobering baseline performance data 1. It is commendable that the ‘unit of operating time’ chosen in the report is the whole time from the start of anaesthesia to arrival of patient in recovery (what the report refers to as ‘touchtime’). All too often some previous analyses have focussed only on surgical ‘cutting time’ which leads to flawed conclusions. It is common sense that lists should be planned in advance: the report suggests 6 weeks in advance with reviews at 4 weeks and 2 weeks, although any other cycle (like 7/5/3) might work equally well. Finally, the report makes clear that anaesthetists and surgeons must be directly engaged with list planning and theatre management, and be supported to do so. In the spirit of that last sentiment, this article is a constructive critique from an engaged anaesthetist. For any national report to be persuasive, its narrative must reflect the everyday experiences of its readers. Anaesthetists, and many surgeons, will therefore be struck by the lack of any data at all in the report on over-running lists. It is as if over-runs simply do not happen or are of no concern to NHSI when in most readers’ experiences, they are probably the main cause of operating theatre havoc. As an afterthought, the report does acknowledge, rather grudgingly, that over-runs do arise but claims (unsubstantiated) that they are almost always due to late starts and gaps between cases and not due to overbooking. Although some readers may readily accept that surgeons and managers never overbook lists, others will remain sceptical of this claim. The literature in fact confirms there is no correlation between start and finish time, so we already know that late starts do not cause late finishes 2-4. Although the NHSI report recognises that cancellations might contribute to apparent under-running of lists and makes suitable adjustments to the data to account for these, it estimates a cancellation rate of just 5%, almost a third of that directly measured by the specialty's own recent national sprint audit, SNAP-2 5. This large disparity in estimates warrants explanation. A separate issue from cancellations is ‘downtime’, which is the total time during a list when there is no patient contact by either the anaesthetist or surgeon; that is, the time waiting for the next patient to arrive or when anaesthesia cannot start due to delays in theatre, etc. It would be desirable to reduce or eliminate downtime to maximise anaesthesia/surgical time and hence overall productivity. Reassuringly, it is already known from published literature that downtime rarely constitutes >15% of total list time 2, 6, a statistic in fact confirmed by the report's own data 1. Figure 1 is a replot of Fig. 3 from the NHSI report, this time displaying downtime. The NHSI report in fact incorrectly scales the right-hand y-axis on this key figure. Nevertheless, this confirms that the vast majority of lists has a relatively modest downtime of <15%, as previously published 2, 6. We shall discuss below why it may be difficult or impossible to improve on this. Perhaps the report reviewed existing literature but if so, it is surprising that this is not cited. Operating theatre management is not a nascent science but a well-established field with a huge literature base. There is little reason for anyone – NHSI included – to try to re-invent the wheel. Metrics like ‘efficiency’ are well defined in terms of use of resource 7-9 and in terms of economics 10, 11. To take a specific example, the report suggests that Trusts could consider extending list durations from 4 to 8 h or more. In principle this is quite reasonable. However, the discussion overlooks the reality that if one surgeon (Paul) has a 4 h morning list and another (Peter) a 4 h afternoon list then the only way to extend the morning list is, literally, to rob Peter to pay Paul. It is not then clear when poor Peter is supposed to operate? Moreover, the calculations upon which optimal list durations should be based have already been described in detail by Dexter's group in the USA 12, and readers – and NHSI – are encouraged to access the helpful resource Dexter co-ordinates (see: https://www.franklindexter.net/education.htm). In any proper analysis of operating theatres, it is only ‘time’ that is relevant and not ‘number of operations’. In assessing performance, it does not matter whether a team completes 1, 10 or 100 operations, all that matters for analysis is the time it takes them. The fallacy of counting operations is self evident; some operations take little time, whereas others are much longer, so teams undertaking complex, long surgeries are unfairly penalised by a measure of n of operations. Yet, the report emphasises how ‘44,000 more urology operations’ etc. could have been performed if time had been better spent. What does this mean? Clearly, 44,000 circumcisions have different implications from 44,000 radical nephrectomy cases 13. What, anyway, is an ‘average’ urology case? All we really need to know is how many minutes of potential operating time could have been saved and we are surely intelligent enough to work out the rest. Instead, the report's authors have converted the simple data on ‘time unused’ on a list into a more complex calculation of ‘opportunity lost’ expressed as n of operations. Perhaps the authors knew that, had they kept to ‘time’ as the proper currency for analysis, then they would have been led to an inevitable conclusion which might have been difficult politically to digest. To meet any demand, we must waste time; that is, have spare, unused capacity 14. It is mathematically impossible to have both full utilisation of a resource and no queue or waiting list. Full utilisation implies a saturated system with inadequate capacity, that will always generate a queue or waiting list 15, 16. The only way to have a zero waiting list is to have a considerable amount of unused capacity. This does not mean that intentionally sitting idle will solve the waiting list problem, but it does mean that even when working very hard, ‘wasted’ time remains a mathematical inevitability. Calculations in demand-capacity analysis or queuing theory when applied to healthcare show that the optimum level of ‘waste’ in this sense is generally ~15% 14, 17-19. Or, expressed similarly, optimum utilisation is ~85%; any more than this and the system becomes pressured. In fact, even by its own calculations, the NHSI report estimates an overall waste (opportunity lost) of just ~17% (see Table 1 in the report), which is remarkably close to this optimal estimate. In other words, all the report might have described is a reasonably well-utilised system with only marginal room for increased resource utilisation. A recently published book on operating theatre management suggests, tongue-in-cheek, that readers should march on the Department of Health (DoH) with placards shouting ‘Give us variances! Give us standard deviations (SDs)!’ 20. We can now see why. Like so much of the data emerging from DoH, all variances have been rigorously expunged from the data in this NHSI report. Error bars appear anathema. Yet, the methods described suggest that SDs (or non-parametric variances like interquartile range, IQR) would have been essential for interpretation. For example, the report uses a graphical presentation much beloved of the NHS, which is to stratify each Trust from lowest to highest performing. Then, it offers the argument (also beloved of the NHS) that ‘if the lowest-performer were as good as the best, there would be x% improvement’ in the outcome being measured. The weakness of this argument is that the error bars (variances) are missing from each individual Trust's data. If we assume (as null hypothesis) that there is no difference between Trusts, then the inter-Trust variance is simply reflecting the within-Trust variance and in reality there is no difference across Trusts. Expressed differently, just because one Trust or team appears as a poor performer in a snapshot, it does not mean it is consistently so, and it is only the variances that will tell us the extent to which a snapshot is a true reflection of its performance. Figure 2 uses the data offered by the NHSI report that an open cholecystectomy takes a mean of 110 min with SD ~20 min and randomly generates 92 data-points (representing sampled Trusts) ordered from lowest to highest. The resulting graph (Fig. 2) is remarkably almost identical to Fig. 5 of the NHSI report. Thus, a random variance around the stated mean is likely to yield the given plot, without necessarily implying a true difference between the ‘best’ and ‘worst’. A second reason why variances are essential is in the planning of lists. The NHSI report appears to falls into the trap of ‘booking to the mean’. Figure 1 of the report presents an underbooked ENT list (Table 1). At this point, the report appears to contain several errors. On a 240-min list the report presents the unused theatre time as 85 min. Yet, the sum of the three operation times given is 160 min (65 + 60 + 35 min), which in fact means that 80 min, not 85 min, was unused time. So, the report is incorrect to state this list finished 40 min early. Assuming the late start was truly 30 min and the downtime was 15 min, this list must have finished 35 min early. Finally, it is stated that the ‘average’ operation time for this list was 55 min (based on the three timings ([65 + 60 + 35]/3), but in fact this simple calculation comes to 53 min. Although these errors are small, it is unclear why they have been made. Either the list was not 4 h long as stated, or the operation times are wrong, or the unused times quoted are wrong, or all of the above. Nevertheless, if we disregard all the errors and take at face value the times as presented, the NHSI report concludes that, since a total unused time of 85 min exceeds their calculated average case time of 55 min, an extra case could easily have been accommodated. The longest duration case was 65 min and this seems well within the 85-min opportunity that was apparently lost. This logic is false. The correct way to schedule a list 21, 22 requires the use of the variances (Table 1, third column). Figure 3 shows the probability of the original three-case ENT list finishing at certain times. There is a near-zero chance of it over-running its allotted 240 min, so we are all agreed this was originally grossly under-booked. Suppose, as implied by the NHSI report, we do indeed add another 65-min case: the red line shows the result. There is now a 60% chance of over-running, and indeed a not insignificant (20%) chance of over-running by > 30 min, which is a serious over-run. Why have we snatched defeat from the jaws of victory? The answer is the flaw in ‘scheduling to the mean’ which, because it takes no account of operation variance, will always result in organisational chaos. It is correct that an extra case could be added but very important to specify that its maximum duration should be only ~40 min – much less than the 85 min apparently available – which yields just 12% chance of over-running (see green line in Fig. 2). Note that this would still leave on average 45 min unused time on the list: that is, about 19% of the list time is still ‘wasted’ and nothing can be done about that (see the discussion on ‘waste’, above). If the NHSI report was written for politicians or the public, to make a case for investment in a theatre improvement strategy, the content and style might serve some purpose. However, as Feynman wryly concluded in his minority report on the spaceship Challenger disaster, “for a successful technology, reality must take precedence over public relations, for nature cannot be fooled” 23. Unfortunately the errors and omissions make the NHSI report almost worthless for a professional audience, who will already have ready access to a better, peer-reviewed evidence base. Any report that selectively avoids discussion of the known large proportion of over-running lists – for that reason alone – cannot engage anaesthetists or surgeons at the front line of the theatre environment. Indeed, were this NHSI report a submission to a journal, not only would it be rejected out of hand but this imbalance could also call into question the integrity of the data. There have been, and remain, high hopes for data-driven analyses like the GIRFT programme. The data it is collecting and the backing of the resources of NHSI give this project enormous potential to transform theatre efficiency. However, for that mission to be successful it is essential that the outputs are complete, transparent and presented in a style that facilitates independent verification. The critique in this article might cause readers to reject the NHSI report. But it is an example of front-line staff taking a direct interest in theatre management, and therefore serves not to undermine NHS initiatives, but rather through open debate, to strengthen data-driven analyses. JP is an Editor of Anaesthesia and an elected member of the Council of the Royal College of Anaesthetists and Chair of the Safe Anaesthesia Liaison Group. The views expressed are personal and not the official views of any of those organisations.

Operating theatre scheduling must incorporate statistical variance rather than relying solely on mean operation times to avoid over-runs and optimize efficiency.