Development and validation of an automated rule-based tool for assessing TRIPOD + AI reporting adherence in artificial intelligence prediction model studies

Artificial intelligence (AI) and machine-learning prediction models are increasingly used in clinical research, but their trustworthiness depends on transparent, complete, and reproducible reporting. Despite the introduction of the TRIPOD + AI guideline, manual assessment of adherence to its 27 items remains labour-intensive, and no validated automated tool for this purpose has previously been reported. To develop and rigorously validate a transparent, rule-based automated tool (TripodAI) for assessing adherence to the 27-item TRIPOD + AI reporting guideline in AI prediction-model manuscripts. TripodAI is an open-source, rule-based Python package that detects TRIPOD + AI reporting elements and issues a binary (Yes/No) adequacy judgement for each item. The reference standard was a 70-study subsample drawn from a parent meta-epidemiological census of 255 AI-based clinical prediction model studies, previously dual-extracted and adjudicated (inter-rater κ = 0.611, 84.5% raw agreement). Internal validation compared TripodAI against this manual reference standard across 1,890 item-level pairs; a further held-out external validation set drawn from the same parent census set of 70 studies (1,890 item-level pairs) was subsequently assessed. Agreement was quantified using percent agreement and Cohen’s κ; diagnostic performance (accuracy, sensitivity, specificity, precision, F1-score) was calculated using manual review as the reference standard. In internal validation, overall percent agreement was 86.0% and Cohen’s κ was 0.705 (95% CI: 0.639–0.771), indicating substantial agreement. Binary diagnostic performance (n = 1,766) showed accuracy of 0.881, sensitivity of 0.925, specificity of 0.777, precision of 0.906, and F1-score of 0.915. Performance was strongest for standardised methodological items (e.g., algorithm type specified, κ = 1.000; setting and recruitment, κ = 1.000) and weakest for data availability (κ = 0.227) and feature selection method (κ = 0.278). External validation on a held-out set of 70 studies drawn from the same parent census confirmed these findings: percent agreement was 85.1%, Cohen’s κ was 0.691 (95% CI: 0.627–0.756), accuracy was 0.860, sensitivity 0.903, specificity 0.759, precision 0.899, and F1-score 0.901 — demonstrating minimal attenuation from internal validation. TripodAI, a transparent rule-based automated extraction tool, achieved substantial agreement with independent manual review, with high sensitivity and acceptable specificity. While not yet suitable for fully autonomous use, particularly for data-availability statements, the tool performs reliably as a human-in-the-loop decision-support system for large-scale TRIPOD + AI adherence assessment in systematic reviews, meta-research, and journal editorial workflows. Not applicable. To our knowledge, no validated automated tool for scoring adherence to the 27-item TRIPOD + AI checklist existed prior to this study. TripodAI achieved substantial agreement with human manual review in both internal (κ = 0.705) and external (κ = 0.691) validation. The internal validation used independent manual review with adjudication, while the external validation was assessed independently against the available human reference standard. Sensitivity was high (0.90–0.93), with acceptable specificity (0.76–0.78). Performance was strongest for standardised methodological items and weakest for data-availability and feature-selection reporting, which should be flagged for mandatory human review. As a deterministic, fully auditable rule-based system, TripodAI offers a transparent alternative to “black-box” large-language-model tools for large-scale reporting-quality assessment. The tool is best used within a human-in-the-loop framework, supporting rather than replacing expert reviewers in systematic reviews and journal editorial processes.