The Babel trap: Language inclusion, evidence quality, and limits of multilingual meta-analysis

Systematic reviews and meta-analyses are the foundation of evidence-based medicine, yet their validity depends on an assumption that is rarely examined explicitly: the assembled evidence base is not only comprehensive but also methodologically coherent.[1] The challenge of language inclusion in evidence synthesis has long been framed as a binary choice between restricting searches to English and accepting the risks of language bias, or expanding to non-English literature and managing the attendant uncertainties.[2] In practice, however, the problem has become considerably more specific. An increasingly common pattern in meta-analyses submitted to journals is the dual Chinese-English search, which supplements international databases with a local Chinese database, while excluding literature from other major research ecosystems.[3] This selective expansion does not resolve the language bias problem; it replaces one form of selection with another. More importantly, accumulating evidence from multiple regions and multiple research traditions revealed that the quality differential between locally published and internationally-published research is neither unique to China nor adequately addressed by current methodological frameworks.[4] It is essential to frame research quality concerns across local-language databases as a systemic, internationally documented phenomenon rather than as a characteristic unique to any particular research tradition. Recent evidence from the Gulf Cooperation Council countries illustrated this point clearly. Alraddadi et al. conducted a systematic review of 1990 randomized controlled trials (RCTs) conducted across Saudi Arabia, Bahrain, the United Arab Emirates, Kuwait, Oman, and Qatar, finding that lack of randomization was documented in the majority of reports, and that while the proportion of adequately generated random sequences increased over the study period, allocation concealment and blinding rates showed no parallel improvement.[5] Crucially, journal impact factor was significantly correlated with randomization quality (r = 0.145; P = 0.014), providing direct empirical support for the proposition that quality of methodological conduct is differentially distributed between local and international publication outlets.[5] A comparable pattern has been documented in the Iranian research ecosystem. Kabir et al. examined Iranian RCTs included in Cochrane reviews and found that risk of bias was rated as predominantly high or unclear across the included studies, with systematic errors in reporting and conduct capable of producing both over- and under-estimation of treatment effects.[6] Studies published in Persian-language biomedical databases were known to exhibit substantially lower methodological rigor than counterparts published internationally, a gradient that mirrors the Chinese experience and confirms that the phenomenon is structural rather than culturally specific. Even within the English-language literature, where quality expectations are highest, a large-scale automated assessment of 163,129 RCTs found a 43%–63% probability of questionable research practices, with trial registration and journal impact factor as the strongest predictors of methodological quality.[7] The practical implication is sobering: if English-language RCTs indexed in major databases represent a floor estimate of quality, the additional gradient introduced by incorporating locally-published literature from ecosystems with lower regulatory oversight, limited journal adoption of CONSORT, and negligible trial registration rates is liable to introduce systematic distortions that standard quality appraisal tools are not calibrated to detect. Within this global pattern, the Chinese research ecosystem offers the most extensively documented case study, given the scale of domestic research output, the depth of existing empirical investigation, and the disproportionate reliance on CNKI, Wanfang, and VIP databases in meta-analyses submitted to scientific journals. The foundational concern is that the structural incentives of Chinese academic promotion systematically route higher-quality research to international English-language outlets, which was established and has been repeatedly reconfirmed. Zheng et al. conducted telephone interviews with authors of CNKI-indexed psychiatric trials published between 2013 and 2016 and found that only 9.2% of trials met acceptable standards for both random sequence generation and allocation concealment, that the proportion of studies reporting exclusively positive results was anomalously high, and that not a single trial in the sample had been registered in any trial registry.[8] The investigators noted that methodological quality had shown little improvement in the more than ten years since the initial benchmark assessment. Evidence on the quality of Chinese-language systematic reviews themselves is equally concerning. The most comprehensive recent evaluation, a Campbell systematic review by Guo et al. covering 1,176 systematic reviews published in Chinese social science journals between 2009 and 2022 and retrieved from CNKI, VIP, and Wanfang, found that overall PRISMA reporting completeness was 49.9% and AMSTAR-2 methodological compliance was 45.5%. Only 0.2% of reviews reported protocol registration; only 2.8% assessed reporting bias; and only 1.2% evaluated the certainty of evidence.[9] Sensitivity analyses, the very procedure recommended to assess the robustness of pooled estimates when study quality is variable, were conducted in just 24.8% of reviews. Paradoxically, reviews published in the higher-ranked Chinese Social Sciences Citation Index (CSSCI) journals demonstrated lower reporting completeness and methodological compliance than those published in non-CSSCI journals, suggesting that prestige indicators within the Chinese journal evaluation system do not reliably distinguish evidence quality in the way that the international impact factor does. While the Guo et al. analysis focused on social science disciplines, its findings were broadly consistent with biomedical assessments: Yao et al. applied GRADE assessment to 564 Chinese biomedical meta-analyses and found that 81.8% of outcomes were downgraded for risk of bias alone, and that the proportion of outcomes rated as high or moderate quality was approximately half that observed in contemporaneous Cochrane reviews (32.8% vs. 61.8%; P < 0.001).[10] Although landmark studies, regional consensus statements, and high-impact clinical studies from regional research groups are increasingly disseminated in English-language journals to reach an international audience,[11-13] local-language literature retains irreplaceable value as a repository of epidemiological surveillance data, real-world practice patterns, and community-based evidence that is rarely submitted to indexed international outlets. The epistemic case for non-English literature inclusion remains valid and should not be conflated with uncritical acceptance of locally published research. Restricting evidence synthesis to English-language databases systematically excludes research conducted in populations of direct clinical relevance to non-Western settings, where the traditional exercise interventions and community-based care models for rapidly aging Asian populations generate evidence that is primarily first published in Chinese. The empirical literature on the consequences of language restriction confirmed that exclusion of non-English studies can itself introduce bias, though the magnitude and direction were context-dependent.[11,14] A meta-analysis on the global prevalence of successful aging illustrated this principle well: by searching CNKI, Wanfang, the China Biomedical Database, and Weipu alongside international sources, the review captured Chinese epidemiological data that no English-language database could have provided, rendering the search strategy not merely comprehensive but clinically necessary.[15] Similarly, systematic reviews of frailty interventions addressing Asian populations must engage with Chinese and Korean literature or accept a fundamental representativeness deficit.[16] The critical distinction is between strategic and opportunistic inclusion. Strategic inclusion involves pre-specified, domain-justified decisions to search local-language databases, supported by bilingual methodological expertise, rigorous quality appraisal, and pre-planned language-stratified sensitivity analyses. Opportunistic inclusion, i.e., the addition of Chinese database entries to a search protocol to claim comprehensiveness, without the accompanying methodological infrastructure, is not a strength. It is a source of undisclosed bias that may inflate pooled estimates, amplify heterogeneity, and misrepresent the quality of the underlying evidence base to readers who lack the expertise to independently appraise it. The PRISMA 2020 statement requires transparent documentation of search strategies and language restrictions, but does not provide domain-specific guidance for contexts in which the quality differential between locally and internationally published research is as pronounced as the literature now documents.[17] The Cochrane Collaboration Handbook recommends minimizing language restrictions while underscoring rigorous quality appraisal, but does not specify how reviewers should interpret “unclear” risk-of-bias ratings when the primary literature suggests that structural non-reporting of randomization procedures is the norm rather than the exception in locally published studies.[18] The GRADE approach to evidence quality assessment is appropriately sensitive to risk of bias at the study level, but its application depends on quality appraisers having sufficient methodological and linguistic expertise to distinguish genuine low reporting from genuine low risk. When neither is present, GRADE ratings applied to mixed-language evidence pools may present spurious precision. Against this evidence base, this journal articulates the following expectations for submitted meta-analyses incorporating local-language databases. Pre-specification of the rationale for database inclusion is required: authors must justify the clinical or epidemiological rationale for including CNKI, VIP, Wanfang, SID, Magiran, or analogous local databases, referenced to the specific population or intervention under study. Generic claims of comprehensiveness are insufficient. Bilingual methodological expertise must be documented by name in the Methods section, covering screening, data extraction, and risk-of-bias assessment for all local-language studies; machine translation is not an acceptable substitute for professional reviewer competence. In recognition of the documented pattern of near-universal positive findings and negligible trial registration in locally-published studies, “unclear” risk-of-bias ratings in the domains of random sequence generation and allocation concealment must be treated as functionally equivalent to high risk in primary analyses. Language-stratified subgroup analyses must be pre-specified and reported in the primary manuscript. Sensitivity analyses excluding local-language studies are mandatory when these studies contribute more than 20% of the total sample or of the pooled effect in any primary outcome. Transparent quantification of the contribution of local-language studies to the pooled estimate – in terms of both sample size and effect magnitude – is required in the results section. Finally, claims of reduced language bias should not be made unless the above conditions are demonstrably satisfied. The methodological challenges of local-language database inclusion are neither unique to any single research tradition nor resolvable by search protocol alone. They are structural features of research ecosystems in which quality incentives, regulatory infrastructure, and journal adoption of international reporting standards lag behind international norms. The most recent empirical evidence confirmed that this quality differential is not limited to Chinese-language literature, that it persists in the most recent assessment periods, and that it is sufficiently large to materially distort pooled estimates when locally published studies are incorporated without adequate methodological safeguards. The geriatric medicine community stands to benefit considerably from a broadened multilingual evidence base; the populations we serve are globally distributed, and the research questions of greatest clinical importance are often most prevalent and most studied in non-English-speaking settings. Realizing that benefit, however, requires moving from the performative inclusion of local-language databases to genuinely rigorous multilingual evidence synthesis. The gap between these two practices is not semantic – it is the difference between a meta-analysis that illuminates clinical truth and one that obscures it. Data availability statement Data sharing is not applicable to this article as no datasets were generated or analyzed during the current study. Financial support and sponsorship Nil. Conflicts of interest There are no conflicts of interest.