Evaluation of Tier One Exposure Assessment Models (ETEAM): Project Overview and Methods

Within the European Union, the Registration, Evaluation, Authorisation and restriction of Chemicals (REACH) Regulation (European Parliament, 2006) requires the registration of chemical substances which are manufactured or imported in amounts of 1 tonne or more per year. If the substance being registered is manufactured or imported in quantities of 10 tonnes or more per year, registrants must undertake a chemical safety assessment (CSA) and complete a chemical safety report (CSR). For classified substances, the CSA has to include an exposure assessment for all identified uses. Several computer-based tools are mentioned in the European Chemicals Agency (ECHA) REACH guidance (ECHA, 2012, 2016) as being suitable for assessing worker exposure within the substance registration process. These tools vary in area of application, level of detail and outputs, from simple screening tier 1 tools, which are designed to easily and quickly differentiate those situations that may pose a risk to health from those which do not, to more advanced, higher level tools that should give a more refined and accurate estimate of exposure (for example, Advanced REACH Tool, 2013). Tool predictions are used to identify, iterate, and verify the risk management measures (RMMs) required to control exposure in workplaces, with this information distributed to substance users via the supply chain. Due to the relative simplicity of the tier 1 models and consequently the high level of uncertainty in estimates from these tools, it is important that they provide conservative predictions of exposure, i.e. generally overestimate exposure, while still being efficient screening tools. Although the tier 1 tools claim to have a broad range of applicability, the performance of these tools has not been comprehensively evaluated. The German Federal Institute for Occupational Safety and Health (BAuA) therefore initiated and sponsored a comprehensive validation project (Evaluation of the Tier 1 Exposure Assessment Models—ETEAM) for tools commonly used for screening purposes. Although the Stoffenmanager® is not listed as a first tier tool in the ECHA guidance, it was originally designed as a risk prioritisation tool for small and medium sized enterprises and hence can also be considered as a screening tool. This also applies to the RISKOFDERM model which is listed as a higher tier tool in the ECHA guidance but can be considered as a screening tool due to its clear structure. In total, the following tools were evaluated in the ETEAM project: ECETOC Targeted Risk Assessment (referred to as “ECETOC TRA”) v2 and v3 (ECETOC, 2013); Stoffenmanager v4.5 (referred to as “Stoffenmanager®” henceforth; Stoffenmanager, 2008); the EMKG-EXPO-TOOL (BAuA, 2008); MEASE v1.02.01 (referred to as “MEASE” henceforth; EBRC, 2012a, b) and RISKOFDERM (Warren et al., 2006). Carried out by the Institute of Occupational Medicine (IOM Edinburgh) and the Fraunhofer Institute for Toxicology and Experimental Medicine (ITEM Hannover), the ETEAM project aimed to compare and contrast the REACH exposure assessment models listed above in terms of their conceptual and external validation, scope of application, functionality, reliability, and user-friendliness. This paper outlines the background, methodological approach, and its relationship to the tiered approach and the organisational set up of the ETEAM project. Detailed descriptions of the methods and results of this study are presented in two papers in this issue of the Annals of Work Exposures and Health, (Lamb et al., in press; van Tongeren et al., in press) and in the project reports (Hesse et al., 2015a, b; Lamb et al., 2015c). Risk assessment under REACH follows a tiered approach, whereby the first tier should provide a conservative (i.e. protective) system that can discriminate between substances in scenarios of some concern and those which are considered safe (Schinkel et al., 2010; ECHA, 2016). A tiered approach refers to a process in which the exposure or risk assessment progresses systematically from relatively simple to more complex systems, thereby reducing the uncertainty in the assessment when moving to higher tiers. The aim of such a risk assessment is to describe a final exposure scenario that defines the conditions under which a substance can be used safely. In the first step, relatively simple, easy to use, and inexpensive assessments are performed on all substances and scenarios (tier 1). Due to their screening character, it cannot be expected that all details of the operational conditions and RMMs are sufficiently specified for the subsequent practical implementation of the final exposure scenario. Therefore, it has to be noted that these models are intended to provide an exposure estimate that might be expected in workplaces that handle substances in accordance with the assumption that some of the basic expectations of EU regulations (training, education, controls, etc.) are already in place and properly function. Nevertheless, such a screening approach has a high level of uncertainty which can only be justified as long as the assessment overestimates the exposure (ECHA, 2012). How high this conservatism ought to be is not entirely clear and acceptable level of conservatism probably depends on various circumstances, such as the level of exposure in relation to the DNEL. The corresponding decision is essentially a trade-off between optimising the ability to detect true risk scenarios on one side and system usability on the other (Tielemans et al., 2007). The higher tier exposure estimations (with an accompanying assumption of less inherent uncertainty) are much more specific and/or require more detailed input parameters and exposure determinants: they hence require more time and resources. But even relatively advanced models have uncertainty and hence some level of conservatism should be considered to accommodate at least the scenario uncertainties that arise from differences between the situation as modelled and the actual circumstances of exposure. For some particular situations, a third tier may be necessary, representing a detailed, scenario-specific assessment that is based on measurement data. A schematic presentation of the tiered process, showing the various hypothetical tiers in the context of related model uncertainty and level of conservatism of the exposure assessment and their linkage to the decision cycle, is shown in Fig. 1 (image adapted from EPA, 2001). Central to the concept of a systematic, informed progression is an iterative process of evaluation, deliberation, data collection, work planning, and communication. All of these steps should focus on deciding whether or not the risk assessment, in its current state, is sufficient to support risk management decisions and if the assessment is determined to be insufficient, whether or not progression to a higher tier of complexity would provide a sufficient benefit to warrant the additional effort. Tiered approach in relation to uncertainty and level of conservatism and their linkage to the decision cycle. Within REACH, risk assessment is carried out by calculating the so-called risk characterisation ratio (RCR) which is the quotient of the exposure estimate and the Derived No Effect Level (DNEL) of the substance of concern. Basically, an RCR <1 indicates sufficiently controlled risks so that a higher tier estimate is unnecessary. However, it is important to note that first tier RCRs below but close to 1 may be questionable in cases of high uncertainty. The level of conservatism may not be sufficient, thus a more accurate, less uncertain assessment may be still necessary (see Discussion). Previous experiences with tier 1 models relate in particular to the outcome of compliance checks that ECHA has to perform in collaboration with the national competent authorities. According to ECHA’s “Evaluation under REACH Progress Report 2014” (ECHA, 2014) around 32% of the checked registration dossiers showed compliance issues with regard to exposure assessment and risk characterisation. A frequent observation from compliance checks of CSRs is that the exposure scenarios do not sufficiently reflect the conditions of (safe) use for the user groups in the different markets of a substance. In addition, the automated use of tier 1 tools for mass production of exposure scenarios can lead to unhelpful or misleading risk management advice for these exposure scenarios. Similar observations were highlighted by ECHA (Philips, 2014) and BAuA (Tischer, 2014) at the international ETEAM conference. BAuA is the German Competent Authority for REACH that is responsible for the compliance checks of registration dossiers. Both institutions agreed that almost all exposure estimates provided in the CSR have been developed using general exposure models (especially ECETOC TRA). Other models are rarely used (and then only for refinement). There was also consensus that boundaries of models are not always appreciated by users leading to misinterpretations of the model outcome. BAuA raised concerns about difficulties associated with the use of generic descriptions of exposure scenarios using process categories (PROC; ECHA, 2015) which are intended to give a broad brush coverage of the most common industrial and professional practices. This can mean in practice that the CSR may contain an impressive list of PROCs without any information about the real workplaces and processes. While this on the one hand makes communication down the supply chain straightforward, the resulting information on operational conditions and risk management is of limited use for downstream users. This has two implications: On the one hand it can be hard or even impossible for the Competent Authority evaluating the registration document to understand the actual workplace situation to which the assessment is intended to apply. On the other hand, risk management information generated under REACH can be too generic to be applied within real working environments and has in these cases to be complemented with more specific risk management information from OSH generic scenario descriptions are also an issue from a model validation of Although tier 1 models are intended to a of workplaces, only a small of validation have been carried which to a workplaces In other the validation do not the broad range of scenarios necessary to provide a comprehensive of the tools. In addition, it is whether different models provide results for the scenario. validation is and also differences between the of validation between the tools Tongeren et al., in A comprehensive of the models is therefore not Although the concept of validation has been as an of the of it is in the However, two of model have been as common the external validation that at a of model predictions with data and the which can be as a and uncertainty of the a of the external and are sufficient to describe the performance of a However, it is important and not to note that models are not an by but are used by real Therefore, model should be by the of a so-called et that to describe the include operational of the and for as as the or of the In this it is clear that the conceptual evaluation, the operational and the external validation are For the operational of a model only makes if the the are from a of as In other of one of these in can be misleading the of are in their to other for the ETEAM an approach was that the mentioned above and for more and different This is a complex and the in ETEAM consequently their and the of the project. Within the aim of a comprehensive process, a of work were of which considered different and operational of the tools. The range of work is shown below in 1 with the on the as in this paper a of the Detailed of these work and the are in the accompanying papers of Lamb et van Tongeren et and in the project reports (Hesse et al., 2015a, of ETEAM project work and The complexity of exposure situations at workplaces makes it to describe all all the that their within a conceptual model therefore from a model thus have a level of inherent uncertainty and a model can to some that model is models can only be or to some for a specific et al., the ETEAM project on the of model on terms model or This process is by the following How have been model the model and with and How uncertain are the model and In to these in the ETEAM a comprehensive was performed which in a of for the evaluated information was via and communication with the model On this the conceptual of the models was evaluated in relation to the the model and the tool The and the were also this approach, an of the and conceptual has been while at the time the different used by the model (Hesse et al., is important of the conceptual are inherent in all of the process, thus those that model and their is to information the decision process. In models are generally uncertain in their and the corresponding and This that is uncertainty not only about the model but also about the and of the model that should the exposure situation of a model can of and two different models generally different estimates for the exposure conditions et al., the REACH guidance (ECHA, a of between tools, such differences a for registrants and those responsible for Within the ETEAM a approach has been developed that the uncertainty associated with the models based on the by This approach is in detail by et and a of all of uncertainty. the tools to an exposure users must from input Previous et al., et al., 2014) have that results from exposure assessments of situations using and exposure tools can vary between if the model is used within the of the user in model estimates may if a user has a limited of the exposure scenario conditions and and/or if the model is The ETEAM project therefore aimed to the of the exposure assessment tools, i.e. users were when input about the situation in with other users (Lamb et al., in in this were to and exposure for a set of workplace situations using the tools. The following tool were applied to estimates for for ECETOC v2 and ECETOC v3 and MEASE and Stoffenmanager® and RISKOFDERM EMKG-EXPO-TOOL and RISKOFDERM The between the exposure estimates generated by the different users from these was determined and for differences The of of of in exposure and for out exposure The of was also with the situation and their level of uncertainty in input in the ETEAM study were the usability and of the tools et al., The of this of the process were to experiences with the different tools and the following the tool and of practical the is the of the of the input system with other and system How the user a substance situation the tool input the the with regard to and of the How do the tools the of the user in relation to assessment of workplace these a of with tool users were by a more to of from a the the model to real workplace is to have a simple In this is not a of model results with data. On the other hand such a is an of validation some that model is models can only be or to some for a specific this process of is a of and can the of a model in and of the that the model is not Therefore, the of the external validation was to compare the exposure estimates generated by the exposure assessment tools used under REACH with measurement data for a comprehensive range of tier tools are expected to be the validation process also aimed to describe the of conservatism of the tools and circumstances the tools may not be sufficiently provide as complete a as of the it was that data were the range of of the tools. from a range of data organisational therefore on situations which were under the of the tools (Lamb et al., However, a to a comprehensive exposure measurement for the important measurement results be used for and exposure to and processes. In addition, data were not for the range of mentioned in the REACH guidance or substance In addition, it was to the study to that were by all tools and to some additional only by the of the data related to of with identified for a limited range of substance of the limited of estimates of exposure was hence from the external validation process. For the necessary data in a were to their data on a based data This been developed by the project to the required and tool input parameters to be in the of descriptions and/or were also developed and distributed to data the and process. The data were from the from of the and a with information on the exposure situations in which were or to which data were the related exposure input parameters for all of the and for the various exposure tools to specific exposure situations and the resulting exposure The external validation process required the of tool exposure estimates from the exposure situations with which to compare the workplace measurement data. was therefore necessary to the information in the exposure situation descriptions and then or the required parameters the tools. the of information the tool parameters the outcome of the validation process of the process was was by a of of a of exposure from the the guidance in the ETEAM in relation to of The guidance and in the been agreed with the in of the process. The were particular data to the tools, on the of their of the and/or substances, or with the situations from the data work and were provided to the with about the input process. All assessments were to results of the The control of of all assessments and between input of different In total, the ETEAM results from exposure These and data. The of the were for with higher 10 at The workplace data (with one or more data associated with a particular and data which of to a and comprehensive are a of to the and it is clear that the measurement data used in the ETEAM project are not of all exposure situations that the models are expected to be to under results are only of the performance of the tools in In to the of a comprehensive process groups of were in the project and project management The Federal Institute for Occupational Safety and Health the of an with a in the project from to the comprehensive process required a of work the of work was distributed to two While the Institute of Occupational Medicine (IOM Edinburgh) was in of the operational external validation, and project the conceptual and uncertainty were carried out by the Fraunhofer Institute for Toxicology and Experimental Medicine (ITEM The (IOM and an project management system developed and refined of in the and of The aim of the management system was to support and for the project. was clear from the that a process that to a of models by different European with in the outcome of the project would a international was therefore with a in the project outcome. The of the was to provide details on of the provide measurement of project and In to an of and between the and the and a of were the of the project. The from the that or developed the exposure assessment as as from that provided measurement data for the external validation process. The ECHA and the Institute for Occupational Safety and Health as The also provided a of measurement data for the external validation process. was to the project that the project to a of exposure data. In total, from all and the to the data (see The at the of the project of and two However, one has the in the of the project. The and of the as as the in of were down in a of All decisions were by a simple as more of all a should be noted that in this context that the in the were the outcome of the external validation was not by all of the However, the of the of to the outcome of the of measurement data with model The two not give support to all the ETEAM are essentially related to the external validation was that the of the ETEAM data the and of the data and as as the substance may a of in the was also that not all data have been or tool parameters and exposure these of the the of the external validation paper in the of the is the of the that the raised have been in the final validation paper Tongeren et al., in The and of simple and conservative screening tools was necessary to the of the of exposure assessments required under to their generic character, it has been that and (i.e. may have been for the of simplicity and has therefore been that the tools require and and validation in the of the the registration (ECHA, 2016). This is the first time an of the used REACH worker exposure assessment models has been carried out using an approach that a conceptual evaluation, an external validation and an operational Although the tools are intended to be generally conservative in their of exposure, in the level of conservatism may not have been sufficient to for the uncertainty in the model estimate Tongeren et al., in et al., et al., a model estimate with a level of conservatism is not in sufficient to support risk management decisions to a higher tier of complexity but has to be considered in the context of the tiered risk assessment process. important that has to be is the i.e. the ratio of the exposure level and the RCR <1 is to that risks are sufficiently thus higher tier unnecessary. However, the uncertainty and conservatism of tier 1 exposure estimates may RCRs below but close to 1 In addition, the study results (Lamb et al., in press; Lamb et al., that when presented with descriptions of exposure situations, user in the of input parameters can lead to different Both of exposure and the of user have health might be at if an exposure scenario is as On the other hand, the situation of be if an exposure scenario is as which lead to a of the and the model more in the level of conservatism may be this should be based on provided by the of measurement data for the exposure situation under REACH, for a to such measurement data by of the substance process. The aim of the substance process is to whether the substances and listed in the may pose a risk for health or (ECHA, should be that Competent should be of model predictions that may exposure in with RCRs In the risk assessment is always a trade-off between uncertainty and complexity and the level of required conservatism that should be considered in the substance process (see also Fig. 1). In is always some uncertainty in any exposure Although this uncertainty not arise from the tool and its application, uncertainty should not be situation requires an assessment of whether the parameters have been the tool was used within its scope and the have been This process of exposure assessment is related to the practical implementation of the model which and/or the process of model parameters Therefore, all models used under REACH are and in However, as all of the evaluated tools are classified as screening tools, it cannot be expected that all details of the operational conditions and RMMs are sufficiently specified for their subsequent practical implementation in downstream Therefore, it is still necessary for the downstream user to the provided scenario in a and to perform an additional that implementation of such model parameters on a practical has to be noted that ETEAM not operational related to the practical implementation of the models However, is more one in the implementation process operational difficulties due to and may of assessment of work this is as an important issue et al., should therefore aim to understand and and model implementation or This would to that the of implementation and to more in downstream The ETEAM project was by (BAuA, in The lead for The other of to the presented in this The are to of the have provided their and measurement data to the ETEAM project are also to the European Chemicals Agency that as an and to the that provided measurement data for the external validation process and as an as of the ETEAM responsible for EMKG-EXPO-TOOL The has the in the of the responsible for Stoffenmanager® and RISKOFDERM and all related to and implementation of Stoffenmanager® are carried out by of and responsible for Stoffenmanager® Health and Safety the of the project European for and Toxicology of Chemicals responsible for ECETOC responsible for MEASE Chemicals and Occupational Health for European Chemicals Agency Health Institute for Occupational Safety and Health