Generating (Factual? ) Narrative Summaries of RCTs: Experiments with Neural Multi-Document Summarization

We consider the problem of automatically generating a narrative biomedical summary from multiple trial reports. We evaluate modern neural models abstractive summarization of relevant article abstracts from systematic previously conducted by members of the Cochrane collaboration, using authors conclusions section of the review abstract as our target. We enlist professionals to evaluate generated summaries, and we find that modern systems yield consistently fluent and relevant synopses, but that are not always factual. We propose new approaches that capitalize on-specific models to inform summarization, e. g. , by explicitly demarcating of inputs that convey key findings, and emphasizing the reports of and high-quality trials. We find that these strategies modestly improve factual accuracy of generated summaries. Finally, we propose a new method automatically evaluating the factuality of generated narrative evidence using models that infer the directionality of reported findings.