How Accurate and Reliable Are Exposure Models?

In this issue of the Annals of Work Exposures and Health, several authors report on the validation of exposure assessment models. Since the introduction of the European regulation on Registration, Evaluation, Authorisation and Restriction of Chemicals (REACH; European Parliament, 2006), various risk/exposure assessment tools have been developed and are currently widely used for chemical safety assessments. Between the start of the REACH Registration period in 2008 and September 2014, around 40000 substance dossiers had been submitted to ECHA. As noted by George Box in 1987 ‘essentially, all models are wrong, but some are useful’ (Box et al., 1987), and more recently it has been stated that models cannot and should not replace the collection of good quality exposure measurements (Kromhout, 2016). Nevertheless, the European occupational hygiene community will not be able to collect a sufficient number of exposure measurements to obtain exposure estimates for all relevant existing and new exposure scenarios. The risk assessments under REACH hence follow a tiered approach in which the first tier should provide a conservative (i.e. protective) system that can discriminate between substances in scenarios of some concern and those which are considered safe, and higher tier models should provide a higher degree of accuracy, even if at a cost of less conservative results. Several Tier 1 screening models such as ECETOC TRA, MEASE, EMKGEXPO-TOOL, and Stoffenmanager® are recommended by the European Chemicals Agency (ECHA, 2012) for estimating occupational exposure. Although these screening tools claim to have a broad range of applicability, extensive model validation is lacking. To perform a proper model validation, both the accuracy and reliability should be investigated, but also the intention for which the model estimate is used (worst case, conservative, compliance testing, REACH) is important. Accuracy provides insight into how close model estimates of exposure are to “the truth” [in this case, measurement data are used to represent the true exposure (van Tongeren et al., 2017)], while the reliability is a measure of the consistency of assessments or ability of assessors to reach the same conclusions (Lamb et al., 2017). Various authors have previously highlighted the need for proper validation of exposure assessment tools, which can only be provided by comparison of the tool estimates of exposures against an independent set of measurement data (Kromhout, 2002; Tischer et al., 2003; Marquart et al., 2007; Schinkel et al., 2010; Tielemans et al., 2011; Tischer et al., 2017) ideally covering a wide range of exposure scenarios and agents. A number of small scale validations of the exposure tools used under REACH and their forerunners (e.g. EASE) have been carried out (Bredendiek-Kämper, 2001; Tischer et al., 2003; Creely et al., 2005; Hughson and Cherrie, 2005; Johnston et al., 2005; Jones and Nicas, 2006; Lee et al., 2009; Kindler and Winteler, 2010; Lee et al., 2011; Koppisch et al., 2011; Kupczewska-Dobecka et al., 2011; Schinkel et al., 2011; Hofstetter et al., 2013). In general, these validity studies did not cover a sufficiently broad range of scenarios necessary to provide a complete picture of the applicability of the models. Therefore, the German Federal Institute for Occupational Safety and Health (BAuA) initiated a comprehensive model evaluation in the frame of the ETEAM project [performed by the Institute of Occupational Medicine (IOM) and Fraunhofer Institute for Toxicology and Experimental Medicine (ITEM)], of which the results are reported in several papers within this issue (Lamb et al., 2017; Tischer et al., 2017; van Tongeren et al., 2017). The paper by van Tongeren et al. (2017) in this issue provides a comprehensive overview of the performance of REACH tier one exposure tools (ECETOC TRA, Stoffenmanager, EMKG-Expo-Tool, MEASE) carried out to date to investigate whether different models provide comparable results for the same scenario. To do so, the authors have created a large (N = 2,098) data set of personal exposure measurement data covering a wide range of different exposure scenarios and substances (volatile substances, powders, metals, non-volatile liquids), which itself is a challenging process even for the small number of required Tier 1 tool parameters. Although this seems like a large database, more exposure measurements are needed to monitor exposure from new production processes, new substances and to get insight into long-term trends in exposure levels. The comparisons of measurement results with tool estimates suggest that the tools are generally conservative. However, the tools were more conservative when estimating exposure from powder handling compared to volatile liquids and other exposure categories. In addition, results suggested that tool performance varies between process activities and scenario conditions. van Tongeren et al. (2017) acknowledge that the measurements used in this study are not representative of all exposure situations that the tools are expected to cover under REACH, and therefore more harmonization of data collection and sharing of exposure measurement data in European databases is required. The authors conclude that, although generally conservative, the tools may not always achieve the performance specified in the REACH guidance, i.e. using the 75th or 90th percentile of the exposure distribution for the risk characterisation. Ongoing development, adjustment, and recalibration of the tools with new measurement data are essential to ensure adequate characterisation and control of worker exposure to hazardous substances. When applying exposure assessment tools, users must select from a number of possible input parameters. Hence, results obtained with the tools could be affected by factors such as the professional experience and judgment of the tool user and level of available information. So, in addition to testing the accuracy of exposure assessment models, the inter-rater reliability (i.e. the ability of different assessors to reach the same conclusions about a specific case) of the use of these models should be investigated. User variation in model estimates may occur if a user has a limited understanding of the exposure scenario or when and/or if the exposure model is misused. The impact of user variation could have serious consequences for workers’ health, if an exposure scenario is incorrectly diagnosed as ‘safe’, or for the financial situation of the organization if an exposure scenario is incorrectly diagnosed as ‘unsafe’, which could lead to costly overengineering. Studies of inter-rater reliability have shown substantial variation between assessors (Stewart et al., 2000; Kunac et al., 2006; Friesen et al., 2011; Schinkel et al., 2014; Landberg et al., 2015; Riedmann et al., 2015). The study reported in this issue (Lamb et al., 2017) evaluated the between-user reliability of the tier one REACH exposure assessment tools, i.e. how consistent tool users (i.e. chemical risk assessors, occupational hygienists, product stewardship experts, REACH advisors, exposure scientists, toxicologists) were in comparison with other users when making input parameter choices based on the same information. The authors show that systematic variation associated with individual users was minor compared with random between-user variation. Exposure estimates ranging over several orders of magnitude were generated for the same exposure situation by different tool users. This disturbing finding indicates the need for training and implementation of additional support and quality control systems for all tool users to reduce between-assessor variation. While great emphasis is placed on the training and competence of occupational hygienists carrying out workplace exposure measurements, there is no similar requirement for users when generating and interpreting results from exposure assessment tools. It is important that all tool users receive comprehensive training in tool use and that comprehensive guidance to tools is provided. Alternatively, the implementation of a consensus/team approach could also be helpful in identifying discrepancies or errors in interpretation of determinants, thus increasing reliability (Kunac et al., 2006; Schinkel et al., 2014). The ETEAM project was focused on tier 1 screening tools for REACH and has therefore not included in their evaluation the Advanced REACH Tool (ART; Tielemans et al., 2011), which was developed as a higher tier exposure assessment model. Currently, ART is the only higher tier exposure assessment model recommended by ECHA. Fortunately, the study by Savic et al. (2017), also in this issue, compares ART estimates with exposure measurements (N = 584) collected over many years and many exposure situations (vapors, mists, powders, and abrasive dust) in Switzerland. The ART’s 50th percentile appeared to be accurate and the 90th percentile showed sufficient conservatism for all the types of exposure processed (Savic et al., 2017), but these results are not always consistent with previous studies (Spinazzè et al., 2016; Landberg et al., 2017). This shows the importance of not relying on only a single validation study in a specific applicability domain, but to repeat validation studies over a wide range of different scenarios. The papers in this issue of the Annals of Work Exposures and Health show that the efforts of the occupational hygiene and exposure science communities to develop useful generic exposure assessment approaches and models have given exposure assessors tools to deal with the enormous burden of risk assessments under REACH. However, the results of evaluating these models are worrisome and are considered far from perfect. They need to be interpreted with caution and more knowledge is needed about model functionalities, their applicability domain and the magnitude of uncertainties to be able to apply these generic exposure models in a meaningful way. As inappropriate exposure assessments could have serious consequences for human health or the financial situation of an (industrial) organization, the occupational hygiene and exposure science communities should recognize that occupational exposure assessment is a science and still requires exposure scientists (or occupational hygienists) to use the tools. As exposure measurements form the basis of the exposure assessment approaches, collection of exposure measurements is crucial and these should be used in conjunction with exposure models. Exposure measurement surveys are still needed to increase our insight into exposure variability and the effect of exposure determinants on exposure levels. In addition, exposure measurements are needed to continue the validation and refinement of existing exposure models. The author declares no conflict of interest relating to the material presented in this Article. Its contents, including any opinions and/or conclusions expressed, are solely those of the author.