A good pilot to navigate troubled waters

The costs involved with the management of patients with a proximal femoral fracture are placing a huge financial stress on public health systems around the globe. In the UK alone, it has been estimated that proximal femoral fractures cost the NHS over £1.1 billion (US$1.5 billion, €1.2 billion) per annum 1. Short of any seismic shifts in the incidence of hip fractures or the cost of providing care, this financial burden is only likely to increase with the ageing population. In the meantime, any measures that can reduce the incidence of complications associated with the peri-operative management of this patient population may have a significant beneficial impact on both the financial and human costs of caring for them. To this end, Canty et al. 2 ought to be congratulated for conducting such a rigorous pilot study on the impact of pre-operative focused cardiac ultrasound on the morbidity and mortality of this condition, the results of which are published in this issue of Anaesthesia. There is variable understanding of the so-called pilot or feasibility studies among researchers, reviewers, editors and readers 3. Although the terms ‘pilot’ and ‘feasibility’ are often used interchangeably, many are calling for more precision with regard to their use in describing the methods of clinical trials 4. A feasibility study, which need not be randomised or powered, aims to determine whether or not selected elements of a study are possible. The sample size is chosen to best investigate areas of methodological and pragmatic uncertainty such as: the standard deviation of the primary outcome; the ease of participant recruitment; the willingness of participants to be randomised; the ease of participant follow-up; the questionnaire response rate; and the time required to collect all the necessary data. The standard deviation of the primary outcome may then be used to power a definitive study, however, an emphasis on descriptive statistics such as effect size, a p value or a multivariate analysis is generally discouraged. Pilot studies are a small-scale version of a definitive randomised controlled trial and are designed to ensure all components of a study protocol work smoothly. They should test the methods of the study and not the effectiveness of the intervention. There are many misconceptions and it is common for small, single-centre, unfunded, opportunistic or student-directed research to be wrongly labelled as a pilot study 5. Unfortunately, many are never published due to their unfamiliar presentation and authors are often tempted to emphasise statistical and clinical significance over and above feasibility. The boundaries between a pilot and feasibility study may therefore appear blurred, nevertheless, Anaesthesia places focus on the appropriate reporting of feasibility outcomes as opposed to the effectiveness or significance of a clinical intervention for such studies. The paper by Canty et al. is therefore an excellent example of a well-designed and well-reported pilot feasibility study 2. The primary outcome for this pilot trial was a 30-day postoperative collapsed composite outcome of all-cause death, acute kidney injury, non-fatal myocardial infarction, stroke, pulmonary embolism or cardiac arrest, and the method of analysis is an any vs. no-event approach 6. In this type of analysis, the presence of any one of the complications in the bundle is considered positive for the primary outcome and this, in theory, increases the power of the study in detecting treatment effect across the composite end-point 7. The investigators included the same variables in the list of secondary outcomes so that the frequency of each event contributing to the primary outcome is easily discerned, and, therefore it can be seen if any one high frequency event is driving positive results. Although the use of composite outcomes does have disadvantages in terms of interpretation, the investigators in the current study have adhered to the good practice guidelines of forming a composite. These include making sure that the composite outcome can well capture the health state of interest, that each component of the outcome should be plausibly affected by the intervention, and that components should be moderately correlated with each other 8. Canty et al. found that a primary outcome event occurred in seven out of the 49 patients treated in the ultrasound group (14.3%) compared with 12 out of the 51 in the control group (23.5%). This gives a relative risk reduction in a primary outcome event of 39% when cardiac ultrasound is undertaken (relative risk reduction is referred to as ‘group separation’ in the current study and is calculated by 1–(14.3%/23.5%) = 39%. This may be more easily understood if these data are expressed as number needed to treat (NNT). The NNT is calculated using the 1/absolute risk reduction; in this case, 1/(23.5–14.3%) = 10.87. This signifies that 11 patients have to be screened using ultrasound to prevent one primary outcome event. The OR for the primary composite outcome was 0.54 (95%CI 0.19–1.52); this is interpreted as 0.54 patients suffering a primary outcome event in the ultrasound group to every one person suffering from a primary outcome event in the control group. The reader is reminded that if the OR is one, there is no difference in outcome between the ultrasound and control group, and since the 95%CI crosses one, the OR was not statistically significant. However, the current study is a feasibility study and is presented as such, with no sample size and power calculations. Using the results generated from this pilot study, the investigators have calculated that 984 patients per arm would be required for an adequately powered randomised controlled trial. This calculation is based on a more conservative effect size of 25% (i.e. assuming the relative risk reduction is 25% as opposed to the 39% as indicated in this pilot study). A relative risk reduction of 25% translates to a NNT of 17; that is, 17 patients will have to be screened with ultrasound to prevent one case of primary outcome. The details of how the sample size calculation was performed are provided in Box 1. P = pooled proportion q1 = 0.5 (the proportion of patients in the ultrasound group) q0 = 0.5 (the proportion of patients in the control group) P1 = 0.176 (the risk of primary outcome in the ultrasound group) P0 = 0.235 (the risk of primary outcome in the control group) α = probability of type I error (here it is 0.05) β = probability of type II error (here it is 0.1) Zα = critical Z value for a given α (here it is 1.960) Zβ = critical Z value for a given β (here it is 1.282). The current data inform the investigators and readers of crucial elements for consideration in a larger trial, including key logistic and feasibility issues, intervention effect size and the appropriateness of the primary outcome as presented. However, one needs to appreciate that the intervention evaluated in this pilot study was ‘diagnostic’ in nature: pre-operative focused cardiac ultrasound only provided a more accurate picture of the cardiovascular status of the patients. Its beneficial treatment effects still required the attending anaesthetist to make rational and consistent management choices based on the availability of this information. It is physiologically conceivable that this may alter patient management, especially with respect to fluid administration as outlined in Table 5 from Canty et al. 2. One the other hand, the relationship of management changes to incidence of cerebrovascular events or pulmonary embolism is more circuitous, although it could be argued that dehydration could predispose one to the development of such complications. In summary, this pilot study used a composite outcome which was both pragmatic and meaningful, to evaluate the value of pre-operative use of focused ultrasound to inform anaesthetists in managing patients with proximal femoral fractures. We look forward to the definitive work that should be three years in the making, if the pilot is right! If you have any questions on methodology, please direct them to Ms Method Matters at msmethodmatters@gmail.com. SC is statistical advisor to Anaesthesia. MC is the Anaesthesia Trainee Fellow. No other competing interests declared.