Annotations and Reflections

Dear Editor, We have observed an increasing interest in the study of childhood neurodevelopment following in utero exposure to drugs 1-7. We appreciate the efforts to address an important, difficult and poorly studied subject, but we believe there are some major methodological pitfalls to avoid. With specific reference to recent papers, we wish to discuss some general and specific, methodological issues that we believe complicate the interpretation of such data. Of special interest in this perspective is the publication by Brandlistuen et al. 1 that has stirred some controversy and attracted significant public attention 8, 9. In studies associating in utero drug exposure to poor childhood neurodevelopment, this is perhaps the most pivotal area of interest. While pregnancy outcome, such as malformations, miscarriages, small-for-gestational age, etc., is relatively easily quantified, such is not the case for childhood neurodevelopment. Prior to conducting an epidemiological study, one should therefore very carefully assess the data at hand: are these data a valid surrogate for the outcome of interest? While several clinical scoring systems have been developed, the gold standard being the Baylor III score 10, parent-assessed questionnaires are most commonly used. These are substantially more easy to apply in large-scale settings and inherently cost-effective. Quite a few of such questionnaires exist – the most commonly used being the Age and Stages Questionnaires (ASQ), the Motor and Social Development (MSD) scale and the Strength and Difficulties Questionnaire (SDQ) 11-13. Two of the most prominent recent population studies on paracetamol exposure during pregnancy and childhood neurodevelopment from Norwegian and Danish cohorts 1, 2, 14, 15 primarily used questionnaire-based outcome. While several questionnaires were used in the Norwegian study and the analysis in the Danish study was supported by prescription and diagnostic registry-based outcome data, the ASQ and SDQ were important sources of outcome measurements for the assessment of neurodevelopment. Importantly, these and other related questionnaires were originally developed as a cost-effective screening tool to identify children at risk of delayed neurodevelopment for further referral to diagnostic testing and observation 11-13, 16. They were not developed or validated as a means of analysing subtle differences in neurodevelopment within a healthy paediatric population. We thus have serious doubts as to what the scores calculated from such questionnaires 1, 17: Despite the use of what papers typically quote as ‘validated scores’, it is unclear to what extent these itemized questionnaires actually represent measurements of neurodevelopment. To what extent are these questionnaires meaningful in terms of a measurable and reproducible scientific outcome, and, not least, to the mothers observing their children? The content validity of, for example, the Norwegian version of a questionnaire is well documented 18. Unfortunately, the more important construct validity (e.g. to what extent does this proxy actually represent the outcome of interest) of the questionnaire is not sufficiently documented. Thus, clinically meaningful conclusions inferred from slopes derived from a regression analysis cannot be made. Two studies have made a direct assessment of construct validity of the questionnaire-based approach, comparing ASQ and MSD questionnaire-based scoring to the reference standard of clinically assessed Baylor III scores 19, 20. While p-values for comparison document a strong correlation, r2 values are thoroughly unimpressive (0.24–0.31). SDQ has not been validated against the Baylor III score. A recent additional study on the validity of the ASQ specifically studied the validity compared to other tools of intelligence quotient (IQ) assessment in preterm children at the age of 5 years 21. ASQ was a reasonable tool for the detection of severe developmental delay but not suitable for lesser degrees of delayed development in IQ. Alas, while a statistically significant correlation between the outcome measurement applied and the reference may be demonstrated, the degree to which this proxy actually explains the neurodevelopment in children is poor. This severely compromises the clinical inferences that can be allowed from such data. In the case of Brandlistuen et al. 1, the handling of data illustrates some pitfalls that, in our opinion, may lead to potential wrongful interpretations. As discussed above, we do not agree on the handling of the presented Likert-scale questionnaire data as continuous variables in this specific context. However, we do acknowledge that the handling of such data from an ordinate scale by calculation of means and mean differences is subject to disagreement and controversy 22, 23. One may choose to accept arguments put forward by Norman 23 and others: the statistical handling of the ‘numbers’ (i.e. assigned and calculated mean scores) themselves is insensitive to the underlying constructs. However, just therein lies the rub: as long as these values do not reflect the issue at hand, clinically meaningful conclusions cannot readily be made from any inferential statistics applied to the data. To justify inferential conclusions made from handling the available data as a continuous variable, the authors would have to convince us that a meaningful and reproducible, objective estimate of neurodevelopment status can be applied to, say, an average difference of 0.23 score on 4–6 three-point Likert items assessed by the observing mother. While we fully appreciate the brilliant idea of using sibling controls to partially adjust for genetic and environmental factors, we believe that important elements of the authors’ data handling and analysis have been improperly or insufficiently addressed. The authors have included the entire cohort of unexposed (table 4) and the concordant siblings (table 3) in the regression models. We believe that the main analysis should have been confined to discordant siblings, and, additionally, the analysis should take the dependence within these sibling pairs into account, for example using a conditional regression model. By including other sibling pairs or the entire cohort of unexposed in the regression models, some of the sibling adjustment effect is lost, and the study becomes vulnerable to exactly the same confounders that the authors intend to avoid. Additionally, all concordant pairs are used as reference in table 3. This reference population includes a substantial proportion where both siblings are exposed, thus making the coefficient estimates virtually uninterpretable. Lastly, the importance of sibling order as source of significant bias is handled improperly. The authors conclude that there is no such effect, as the confidence intervals of the respective point estimates ‘overlap’. Such inference simply cannot be made 24. The sibling order should have been entered into the regression model. Another highly plausible biological rationale is that the timing of exposure during pregnancy could be of significance. The authors have tested for this by applying qualitatively and quantitatively unspecified ‘numerous tests’ which failed to reach statistical significance. The authors have applied the Bonferroni correction to address the issue of multiple testing. The approach taken by the authors appears counter-intuitive. The concern here is not the risk of a type one error but the opposite: you want to be very sure that this variable is not excluded on the basis of a false-negative inference. Hence, if anything, the level of significance should have been increased rather than adjusted downwards, as it is preferable to accept some noise in the model as opposed to excluding an important parameter. Again, the more obvious and simpler approach is to enter the time variable in the regression model. The attempt by the authors to quantify the extent of adverse neurodevelopment (using the term ‘substantially adverse’) and translate their findings into a relative risk in the general population is unjustified. As we argue above, based on insufficient construct validity of the principal outcome measurement and inadequate statistical handling of the data, there is simply no way to infer a clinically meaningful interpretation of statistically significant β values from the regression analyses performed. The validity of outcome parameters and the translation of questionnaire-based scores of neurodevelopment into a single continuous scale variable should be undertaken with great caution. We believe that a temptation to overstretch the information value that can reasonably be extracted from such questionnaires is imminent. Clinically meaningful interpretation of such data in terms of regression analyses is in our opinion unjustified or, at the very least, subject to underlying assumptions than cannot be verified. Within the highly sensitive field of pregnancy and drugs, diligence is paramount to avoid dissemination of information that otherwise may serve to confuse and worry physicians, pregnant women and the public alike. We have had the experience of having a Letter to the Editor on the paper by Brandlistuen et al. 1 rejected for purely generic (‘..we receive many more papers than we can publish…’) reasons (Personal communication, George Davey Smith, (GDS) Editor-in-Chief, Int J Epidemiol.). This was disturbing as we believe that an open scientific discussion of published data is fundamental to scientific development and essential to dissemination of knowledge. Therefore, we take this opportunity to emphasize our opinion on this subject: Editors should by and large give way for scientific discussions of published papers in their respective journals; in fact, we suggest that this be a prioritized obligation. Sincerely,