The steps that characterize data analysis for flow cytometry-based clinical trials can be grouped into quality assessment, compensation, normalization, transformation, cell population identification, cross-sample comparison (population mapping or matching), feature extraction, visualization, and interpretation. Of these steps, cell population identification (i.e., gating), which generates reportables such as cell population counts/percentages and MFI, is the focus of significant efforts to improve the rigor and reproducibility of data analysis. Here, rigor is the application of the scientific method to ensure unbiased and well-controlled analysis, interpretation, and reporting of results. Reproducibility is important as analyses are only validated when they can be duplicated by multiple scientists. It is especially important for clinical studies due to what has been deemed a reproducibility crisis in medicine, as only 11% of a series of preclinical cancer studies could be replicated 1. While manual gating is the gold standard and current practice for cell population identification, assessments of its reproducibility have recognized it as a significant contributor of variation in flow cytometry studies, with interlaboratory C.V.s up to 30% 2. While having a single operator analyze all the data can significantly reduce variability, the chosen individual is still influenced by the same personal biases that result in inconsistencies within and across individuals, centers, and time. Resorting to a single operator to improve reproducibility does not scale for large studies as manual gating can take from 45 to 90 min for one clinical sample 3. With clinical trials now involving thousands of patients assayed with 18 parameters, the challenges associated with the rigor and reproducibility of manual analysis of flow cytometry-based clinical data have only become more apparent and pressing. The first algorithm for automated flow cytometry cell population identification was published in 1985, where it was noted that: “Unfortunately, the use of three or more independent fluorescent parameters complicates the analysis of the resulting data significantly” 4. While data complexity has increased significantly since then, automated analysis approaches for flow cytometry data have also matured. The state of the art is at a stage where results from automated data analysis algorithms have reached a level of maturity that enables them to match, and in many cases exceed, the results produced by human experts. The maturity and acceptance has gone as far as being included as part of the documentation for the FDA approval of a first in class CAR-T therapy 5-7. The overwhelming majority of automated gating methods are unsupervised. Unsupervised algorithms are those approaches that work on data that do not come with predefined labels. With respect to FCM data, these algorithms find by themselves some commonality of the features of cells that can be used to group them together into cell populations. As a class of approaches, these do not require any training nor any or only very limited parameterization making them easy to use. An example parameterization in the context of gating is the process of choosing characteristics that can be used to classify cells into groups, such as predefining how many cell populations exist in the data set. These approaches use internal metrics to make decisions such as: how close events are to each other in multidimensional space based on the choice of the distance measure used, how many cell populations exist (e.g., the “k” in k-means clustering), or when to place a gate between groups of cells (where to make a cut between adjacent events). However, this is also a limitation of unsupervised approaches, as the varying size, shape, and distribution of cell populations within and across samples means global parameterizations need to find the best middle ground. Thus, the factors that make these algorithms easy to use also tend to limit their performance. Unsupervised algorithms also have no way of knowing how to label cell populations. A metric commonly used to assess the performance of clustering approaches is the F1 measure. F1 is the harmonic mean of precision and recall, with a score of 1 indicating that all individual events were placed in the same series cell population by both analysis methods. Here, manual gating is set to be the gold standard, and any deviation from manual results lowers the F1 measure from 1.0 (perfect agreement of every event across all gates measured). The mean F1 measure of the currently best performing unsupervised algorithm is approximately 0.78, with a relatively large distribution in scores across cell populations 6. However, a caveat is having set manual gating as the gold standard, even if an automated gating approach is “better,” any differences will still be counted as mistakes on the part of the algorithm unless such differences are reviewed and the manual analysis is modified. In our experience, detailed review of discordant examples during training is extremely important to both improve the parameterization and not come to false conclusions with respect to a seemingly poor rigor of automated methods. Regardless, the current level of performance may lead to lack of adaptation for clinical studies. However, unsupervised algorithms have been successfully used on some limited use cases 8. Similarly, dimensional reduction techniques and visualization tools such as SPADE, t-SNE, Wanderlust, Citrus, and PhenoGraph are generally useful only for discovery as they do not directly identify well-defined homogenous cell populations in the traditional sense, and cannot generate the required reportables 9. As an alternative, supervised cell population identification methods are parameterized by users based on expert knowledge all the way down to individual cell populations based on their unique characteristics observed in bivariate plots (e.g., their shape and distribution relative to other cell populations). A nonexhaustive overview of the main features of typical supervised and unsupervised analysis tools relative to manual gating was recently conducted by Mair (Fig. 1) 10. Supervised methods tend to outperform unsupervised methods when the goal is to replace manual gating for generating cell population statistics, as evaluated through open peer-reviewed comparative studies 3, 11-14. However, as with other automated methods, there has been limited adoption due to barriers such as the required bioinformatics expertise. Due to patient confidentiality and intellectual property concerns, flow cytometry data from clinical trials have not been published in the peer-reviewed literature. Two recent studies illustrate how automated approaches improved the rigor and reproducibility of the analysis. In “Implementation and Validation of an Automated Flow Cytometry Analysis Pipeline for Human Immune Profiling,” we described the practical implementation of an automated flow cytometry analysis pipeline for human immune profiling 13. Data were generated using two staining panels that identified effector and memory or helper and regulatory T cells. A core panel of 8–10 fluorochromes was used to identify immune subsets of interest to which three fluorochromes can be added to measure 6–15 activation markers per patient per time point, depending on the desired depth of analysis 13 After acquisition, FCS files were opened with the FlowJo software to adjust compensation and the resulting workspaces were read into R for all further data processing 15. While a significant focus has been rightly placed on automated gating, the adoption of automated approaches for cell population identification opens up additional avenues for improving the rigor and reproducibility of clinical trial data analysis within the analysis pipeline once data enter a computational stream. For example, rigor can also be improved through the automated identification of outliers. This starts at the event level, by analyzing various metrics of data acquisition such as flow rate and fluorescent measurement fluctuations within a sample and flagging or removing suspect data points 16, 17. Automated quality checking can also extend to postgating steps. The QUALIFIER algorithm uses the gating template to perform quality checks on gated populations and overall properties of a sample such as cell counts. It divides the data preprocessing from the actual outlier detection process so that the statistics are calculated all at once and the outlier detection and visualization can be done more efficiently and interactively 18. In the Conrad study, once pregating quality checking was competed using such approaches, gates were set by flowDensity. Unlike typical clustering algorithms that tend to identify populations by examining all dimensions simultaneously, flowDensity is based on a sequential bivariate gating approach that generates a set of predefined cell populations using a prespecified approach customized for each cell populations of interest. The algorithm mimics manual gating steps, but chooses the best cutoff for individual markers using characteristics of their density distribution such as the slope of density, or the minimum intersection point between the two peaks of the density (Fig. 2). For activation marker gating, a gate boundary set on the fluorescence minus one control was applied to matching samples with the fluorochrome present. Gates are set independently for each data file based on these rules. Once the parameterization of the gating steps is completed, gating steps run in sequence by a single script, only requiring manual input to specify the directory where the files are located and the desired location for the outputs. The approach tends to be robust, as long as new data files are generally similar to those used to set the gate boundaries. Conrad et al. tested how the analysis pipeline with flowDensity performed compared with manual gating conducted by domain experts on data from five patients. Manual and automated analyses were performed on the same 11 populations of interest over a total of 33 time points (2–3 tests per time point), yielding 1,077 matched populations. Comparison of these populations showed a significant correlation between the manual and automated analysis of the T cell panel (Fig. 3). While the bulk of the populations showed strong agreement, manual gating of populations with few events were not matched as closely by automated analysis. The larger disagreement can be attributed primarily to the small number of events in that population (the impact of small shifts in gate placement is magnified on smaller cell populations) and secondarily to the lack of resolution between populations (i.e., smears). When controls are not used for such rare populations, there is often no objective information available to guide either automated or gating. This makes the evaluation of automated analysis approaches for such cell populations especially challenging. Despite these differences, the observed trends also demonstrated the same trends as those obtained from manual gating, which is an essential criterion for using this method in longitudinal studies (Fig. 3). We similarly demonstrated high reproducibility of a supervised analysis pipeline in peripheral blood from both healthy subjects, patients 10 days after hematopoietic stem-cell transplantation, using different instruments from different vendors and across centers 3. Like the Conrad et al.’s study, we developed an automated analysis workflow based on flowCore and flowDensity. Data were generated using DuraClone's dry reagent technology (Beckman Coulter) preformatted panel antibody cocktails. The performance of automated pipelines was assessed by their ability to match on a per-event basis values obtained by an expert manual analyzer (i.e., the reference manual), currently considered the “gold standard” approach. In addition, we compared the reference manual values with those obtained by two additional manual analyzers who followed an identical gating strategy. Automated analyses produced results that were highly correlated with those obtained in the reference manual analysis. For example, Spearman's rank correlation coefficient [ρ, rs] comparing reference manual with automated analysis for the 14 basic panel cell populations was all >0.8. Most median F1 scores were > 0.9, with an overall F1 average of 0.93. To demonstrate robustness, we obtained an independent data set from the ONE Study, which used the same antibody panel and fluorescence intensity settings 19, and reanalyzed these data manually and with our automated pipelines. When all populations analyzed were combined for the two data sets, correlation values between automated and manual gating were all >0.9, demonstrating that automated gating pipelines developed with one set of data can readily be used to accurately analyze independent data if they are collected using the same standardized methodology. To test the robustness of the analysis pipelines to alternate instrument platforms, we analyzed parallel samples acquired on a Navios (Beckman Coulter, 3 lasers) or a Fortessa X20 (BD Biosciences, 4 lasers), and obtained correlation values >0.99. In general, we found that automated gating agreed slightly less with the reference manual than other manual gating. As in the Conrad study, lower agreement occurred especially for low-abundance, poorly defined populations (e.g., plasmablasts). Indistinct boundaries, such as between CD14+ and CD14++ (particularly in the CD16+ population), as well as between CD16− and CD16+, led to variability in manual gating (Manual 1 and Manual 2 vs. reference manual, both rs = 0.83), as well in automated versus reference manual (rs = 0.83). While rigor and robustness are important drivers for adoption of automated analysis, computational approaches additionally provide significant efficiency gains with their rapid analyses (~1 min of computer time per set of 25 samples from raw data to final spreadsheet versus 10–20 expert hours for the equivalent manual analyses) 3. However, the performance of supervised methods comes at the cost in time for choosing the correct parameterization. This depends on the size and complexity of the panel and the number of cell populations being assayed. It takes on average about two weeks per panel, on par with how long it takes to set up and validate a manual gating hierarchy. Unlike manual gating, once setup is completely automated, gating can be exactly reproducible on a given data set with human effort limited to running the program a second time, with processing taking about 60 s per file 3, 13. However, supervised approaches need to be parameterized based on the heterogeneity that will be observed during the course of the clinical study. While relative shifting of populations is well tolerated (as shown by the high correlations obtained in the studies described above), it is important to have prior knowledge of the diversity of cell populations that occur across all samples of the study. If unexpected populations occur, these samples can be automatically flagged based on quality checking of cell population statistics being out of the expected range (e.g., greater than three standard deviations from the mean percentage of that population for all samples analyzed to date). The approach can then be reparameterized for future samples. This is no different in principle from what would happen with manual analysis. Machine learning holds the promise of reducing the time required to parameterize supervised algorithms 20, 21. However, like unsupervised methods, they have not yet shown the necessary performance. Moving to automated analysis also has the advantage of directly improving the ability to reproduce experiments, an essential part of the scientific process. The Minimum Information About a Flow Cytometry Experiment is an adopted example of one way to provide key pieces of information needed to reproduce results and includes specification of the analysis performed on the resulting data. Documentation of manual gating protocols often includes descriptions such as to set gate boundaries “by eye” with no guidance for rules to follow, or even just images of how gates are set on a single sample with no description of the logic used. Automated cell population identification is fully descriptive of the process to set gate boundaries. The understanding of algorithms available for analysis is enabled through the choice of R/Bioconductor as the programming language for the overwhelming majority (more than 50 to date) of available automated flow cytometry data analysis tools. R is widely used for statistical computing, machine learning, and graphics, and has an extremely large, worldwide community of users, developers, and contributors. R is a free, open-source software environment for statistical computing and graphics that runs across UNIX, Windows, and MacOS. Bioconductor is a repository of R-based tools for the analysis and comprehension of high-throughput genomic data 22, 23. Bioconductor provides a structured platform enforcing compatibility and documentation. A main project goal stated by Bioconductor is to further scientific understanding by producing high-quality documentation and reproducible research. Each Bioconductor package contains one or more vignettes, which are documents that provide a textual, task-oriented description of the package's functionality. These vignettes at minimum demonstrate the algorithm's functionality through worked examples on real data that are also provided alongside the code. Almost all these algorithms have accompanying peer-reviewed publications. In addition, literate programming can be used to make results not only reproducible but easily understandable. It further promotes rigor through improved reporting by changing how software is written from telling a computer what to do, to explaining to humans what we want the computer to do 24. Furthering reproducibility, R Markdown provides a mechanism to generate fully reproducible documents with embedded results generated computationally by the document itself based only on the provided FCS files. Developers can weave together descriptive text and code by using R Markdown and knitr to produce formatted output such as a Word or PDF file that can include dynamically generated figures illustrating gating hierarchies and reportables. Given a set of code and data, exact results should intuitively be reproducible by another scientist. While sharing R with an R Markdown document should reproducibility, many analyses on additional and software as may produce unexpected results or when a different of R or another Reproducibility is only by setup and However, of code so that it with the software to ensure reproducibility can be a As an alternative, it is to provide the environment in its using or software This approach reproducibility by removing of pipeline Thus, the final in reproducible results is the use of which and into up all the of a so that code is reproducible across for all time. on computing and distribution between While is a provides additional features such as reproducible software that can be by a and a of users running of automated methods is not The available methods are generally the result of research. While there is software that provides out of the 11 they are the manual analysis process and not with in However, the of gating software from software vendors do R In addition, through the use of it is to very generate the required for automated This just the specification of the location of the FCS files to be and the desired location of the output files (e.g., files of flagged software The process can even be further by having the a directory and process FCS files as they are there by the analysis pipelines based on supervised gating methods can result in improved reproducibility and and effort in large clinical where the goal is to assess and populations 10. However, adoption of such methods is in its that limit the of computational approaches include lack of and and either to or computational training approaches tend to be published in computational not commonly read by As these become adopted and applied on data to make that in high impact and will to the of increased adoption over time, each new of will also on average be more computationally than the of basic to ensure high quality of raw data is for the of any analysis manual or automated as methods to in their and to in an of manual analysis will to become less than they that supervised gating methods have the which has been shown through peer-reviewed studies and clinical that they can be robust, and than manual analysis for the analysis of clinical data and have performance than unsupervised methods when the goal is to identify cell populations of interest.
No takes yet. Share an insight, caveat, or question.
Ryan R. Brinkman (2019) studied this question.