Apparent versus effective test performance: The importance of understanding and reviewing performance metrics

There are many studies and papers in the scientific literature that address the performance metrics of screening tests.1-5 Often, these studies will present the specificities and sensitivities of the tests, which allows a clinician to think about what those metrics would mean for their patients and clinic. How many patients would this help? How many would be alarmed unnecessarily? How would this influence appointment availability and the workflow in the clinic? A good understanding regarding how a test will perform in a real-world, practical setting is important to ensure that expectations for the patients and clinic are set appropriately. Confusion over the accuracy of the results, how long it will take to receive results, how many patients will receive results and who would best benefit from taking the test are all examples of issues that can arise when data is interpreted without the relevant context. Although this is a challenge even for tests where all samples generate results, it becomes much more so when a significant fraction of samples do not. Here, cell-free DNA based non-invasive prenatal screening (alternatively called cfDNA, NIPT or NIPS) will be used as an illustrative example of such a test. NIPS provides a very good case study since it has excellent reported performance in screening for common trisomies6-9 and a large span of first pass test failures (also termed “no-calls”) ranging from less than 1%–8% or higher, that are commonly removed from the calculations of test performance.10-14 In addition, those excluded samples may be at increased risk of aneuploidy,10, 15, 16 which is why medical societies have published guidelines recommending that these women be offered invasive diagnostic testing.17 Initially, the reported increase in sensitivity and specificity over traditional screening appeared to be very beneficial in routine clinical practice and patient counselling by reducing the number of false positives and minimizing the number of patients needing follow-up care. However, over time, as the clinical experience with the test grew, the scientific community has learned that although NIPS is a great asset to prenatal screening, its performance is not quite as good as the original studies indicated.18, 19 The questions then become “why?” and “what can be done to avoid such discrepancies?” in order to properly counsel patients about the specific tests that they are offered. The answers have to do with how these studies were designed and what the results actually stated as well as how they were interpreted. A deeper look reveals that the common performance metrics of sensitivity and specificity presented by the studies do not typically capture a complete clinical scenario, and thus do not properly prepare the clinicians for how to use the studies' metrics in counseling women for their pregnancies. The reason for this has mainly to do with which samples are included in their performance calculations. In technical evaluation studies, samples that have discordant clinical results from the perspective of the fetus can be excluded or possibly even counted as concordant. In NIPS, this can be represented by confined placental mosaicism, where trisomic tissue might exist in the placenta but not in the fetus itself.20 Also, it is important to acknowledge that gathering complete follow-up data is extremely difficult and a requirement for this would likely be prohibitory for most clinical studies. One of the main challenges to be aware of when considering test performance is that the focus is mainly on sensitivity (detection rate), specificity (sometimes reflected by the false positive rate) as well as positive and negative predictive values (PPV and NPV). The lack of a no-call rate (the fraction of test results that were not reported) as a performance metric together with the sensitivity and specificity often leads to the commonly presented apparent performance being misinterpreted as the effective performance of the test. The “apparent performance” is defined as how well the test performs in the samples that were provided results, while the “effective performance” is how well the test performs in all samples that were tested (regardless of whether a result was given). In the case of NIPS, studies have shown that using apparent over effective metrics will overestimate detection rates,11, 14, 18 with one study showing an overestimation by 1.7%, 1.6% and 7.1% for trisomies 21, 18 and 13, respectively.19 From a clinical perspective, the distinction between “apparent” and “effective” is critical because whenever a lab discards a sample (either calling it a “test failure” or a “no-call result”), it still has clear clinical ramifications, even if the patient opts to get redrawn. It is still a sample from a patient that the clinician has to counsel, even if the test did not supply a positive or negative result. That is why it is important not to disregard the discarded samples. Otherwise, clinicians will not receive a full picture of the clinical impact of the test on their entire patient population. In fact, these results might have a larger clinical impact since they are often harder to counsel due to the necessity of having to first understand the reason for the discard in order to guide the patient through the appropriate clinical options. These challenges can often be met by carefully reviewing the “Methods” section of scientific publications. In this section, all of the exclusion criteria, along with any absent or low-quality follow-up data, should be presented even if it will not be brought up in the “Results” or “Discussion” sections. By critically evaluating these sections in publications or by requesting this information from presenters, one can then be aware of the limitations with the data, and thus will be better able to assess the clinical utility and impact of the study's conclusions. Within NIPS, the NEXT study15 presented follow-up confirmatory results for the 3% (488) of samples that received no-call results, allowing for a more detailed visualization of the aforementioned implications. The study presented their findings as summarized in Table 1. From the apparent test results, the study concluded that there is 100% sensitivity (38/38), 99.9% specificity (15,794/15,803) and an 80.9% PPV (38/47). For a screening test, these are exceptional results and a natural interpretation for many clinicians (beyond “this is a good test for my patients”) is that they would only need to provide further counseling to 47 screen-positive patients. This is significantly better than the approximately 790 patients who would require counseling as a result of the traditional biochemical prenatal screening's 5% screen-positive rate.21, 22 However, it is important to recognize that the 3% no-call rate in this study means that the clinicians have to counsel an additional 488 patients for a total of 535. Since studies have shown that these samples may be at increased risk of additional chromosomal anomalies (fetal or maternal), pregnancy complications and maternal health conditions (including malignancies),17, 20, 23 those additional 488 women may prove even harder to counsel. Literature indicates that 56.5% of women who receive no-call results retake the blood test,24 and 74% of these women receive a result.25 Which, in this scenario, would lead to a resolve rate of 42%. Although these numbers will vary substantially based upon individual regions and clinics and how these women are counseled, using these numbers on the NEXT study population would have led to an additional 205 women who would have received a result, which may have improved the performance rates. Regardless, there would still be an increased clinical burden from the retesting as well as from all 488 women who would have still needed to be counseled and offered diagnostic testing prior to their decision to retake the test.17 Other acceptable interpretations of the data would include viewing the 488 women as “screen-positive”, since current American College of Obstetricians and Gynecologists (ACOG) guidelines state that, for NIPS, these no-call patients be offered invasive testing, similar follow-up to the traditional “screen-positive” patients.17 Alternatively, one could consider that these samples should be regarded as “screen negative” since the test does not state that they are at high risk of Trisomy 21 (columns 2 and 3 in Table 2 respectively). In reality, it would be a mixture of screen-positive and screen-negative interpretations depending upon factors such as the patient's decision-making regarding diagnostic testing (i.e. some samples would be called correctly if the patient opted to pursue diagnostic testing, while some would be missed if the patient forgoes the diagnostic testing). Since there is no “correct” classification of no-call results to capture them in the common metrics, it makes sense that the “apparent performance” (column 1 in Table 2) is presented in most studies. However, the no-call rate should be explicitly stated in the results so that informed readers can draw their own conclusions. Specifically for the NEXT study,15 this effect of no-call samples is considered in the discussion but it is still missing from the test performance summary tables, while in other examples there is no mention of the effect of no-call results at all.24 Regardless of why no-call results are not considered as part of performance calculations, it is important to recognize that they represent patients who need management, counselling and help. In principle, as long as the study states what they are doing in a transparent manner, it should be considered acceptable. However, clinicians have to understand and recognize what the presented results are actually showing and how those results can translate to their real world and thereby affect their practice and their patients. Using the example above, it appears that the NIPS test, compared to the biochemical test, has saved 743 patients from the anxiety of being “screen-positive” and having to consider invasive testing, while also saving their clinicians from the additional duty of scheduling, explaining, managing and helping these patients simply by excluding no-call results. However, in truth, it did not. Since they do not lead to the generation of an inaccurate result, no-calls are often viewed only as a limitation of technology and testing (which they can be) and not something to focus on (which we should). However, test failures and especially no-call results are critical performance metrics that should not be glossed over as they have a critical impact on the clinician and their patients. It is the difference between believing that the test only impacts 47 patients and not 535 it really does. In order to best serve their patients and their own clinic, it is important that the clinician understand whether the studies are providing apparent or effective metrics and what the real-world implications to their patients will be, and what the actual clinical burden is to their practice in terms of additional time, costs and resources. These implications can often be significant due to the potentially large number of patients who are omitted, in particular if the fraction of omitted samples is not considered a performance metric of its own. By excluding samples for which the laboratory is unable or unwilling to provide a result, the performance of the test will appear to be better than what it will be in the clinical experience. It is for these reasons that clinicians must be careful when examining the reported performances of the testing options for their patients. The authors would like to thank Anna Johansson for her input and assistance in making this manuscript possible. Lawrence Prensky is an employee of, and Fredrik Persson is a consultant for, PerkinElmer Inc. PerkinElmer is the maker of the Vanadis NIPT system. Data sharing is not applicable to this article as no new data were created or analyzed in this study.