What does this research mean for the field?

Ordinary, invisible data-processing choices in AI-driven automated text analysis pipelines can manufacture false statistics and significantly distort conclusions without any intentional misinformation. Novelty: ClaimNovelty.NOVEL_FINDING. Consensus alignment: ConsensusAlignment.NEUTRAL.

What question did this study set out to answer?

Examine how routine, invisible data-processing decisions and zero-shot language model selections can inadvertently manufacture false conclusions in automated text analysis pipelines used for official statistics.

September 3, 2026Statistical Journal of the IAOSOpen Access

Misinformation without liars: How invisible data-processing choices manufacture false conclusions in automated text analysis

View Full Paper

Ask AI

Bookmark

Authors

NKNadejda Komendantova PPPetr Pesov

Discussion

Member takes

Overview

Computational analysis reveals that routine text-preprocessing choices severely distort zero-shot language model outputs in automated pipelines, highlighting systemic risks for official statistics.

Key Points

Examine how routine, invisible data-processing decisions and zero-shot language model selections can inadvertently manufacture false conclusions in automated text analysis pipelines used for official statistics.
Tested standard, publicly available zero-shot language model classifiers on web-scraped social media text concerning armed conflict.
Evaluated model sensitivity across routine data-processing variations, including scraped interface strings ('N comments'), alternative prompt phrasings, and classifier substitutions.
Developed a reproducible audit framework designed for statistical agencies monitoring crisis events and producing real-time indicators.
Capturing routine interface text like 'N comments' collapsed classifier confidence on armed-conflict reports from near-certain to below retention thresholds, an effect absent with benign strings of equal length.
Four defensible phrasings of the same analytical query generated nearly disjoint datasets from identical inputs, and substituting classifiers caused unambiguous conflict reports to be discarded.
Distortions were silent and model-specific, producing confident and plausible statistical estimates despite substantial classification errors.

Cite This Study

Komendantova et al. (2026) studied this question.

synapsesocial.com/papers/6a993547636c6408cfa7d677 https://doi.org/10.1177/18747655261480969

View Full Paper

Ask AI

Bookmark

September 3, 2026Statistical Journal of the IAOSOpen Access

Misinformation without liars: How invisible data-processing choices manufacture false conclusions in automated text analysis

View Full Paper

Ask AI

Bookmark

Authors

NKNadejda Komendantova PPPetr Pesov

Discussion

Member takes

Overview

Computational analysis reveals that routine text-preprocessing choices severely distort zero-shot language model outputs in automated pipelines, highlighting systemic risks for official statistics.

Key Points

Examine how routine, invisible data-processing decisions and zero-shot language model selections can inadvertently manufacture false conclusions in automated text analysis pipelines used for official statistics.
Tested standard, publicly available zero-shot language model classifiers on web-scraped social media text concerning armed conflict.
Evaluated model sensitivity across routine data-processing variations, including scraped interface strings ('N comments'), alternative prompt phrasings, and classifier substitutions.
Developed a reproducible audit framework designed for statistical agencies monitoring crisis events and producing real-time indicators.
Capturing routine interface text like 'N comments' collapsed classifier confidence on armed-conflict reports from near-certain to below retention thresholds, an effect absent with benign strings of equal length.
Four defensible phrasings of the same analytical query generated nearly disjoint datasets from identical inputs, and substituting classifiers caused unambiguous conflict reports to be discarded.
Distortions were silent and model-specific, producing confident and plausible statistical estimates despite substantial classification errors.

Cite This Study

Komendantova et al. (2026) studied this question.

synapsesocial.com/papers/6a993547636c6408cfa7d677 https://doi.org/10.1177/18747655261480969

View Full Paper

Ask AI

Bookmark