Taming the AI Curator: A Content-Focused Data Description Diagnostic and Assistive Writing Tool

Motivated by short, vague, and incomplete dataset descriptions in the Digital Porous Media Portal (DPMP), we developed and evaluated an LLM-based tool to aid users in writing descriptions that comply with curation and domain science best practices. The tool, named Rocco, has two components: a grader and a writer. The grader assesses the quality of user-submitted descriptions based on their compliance with a rubric of ten guidelines, while the writer uses this feedback, along with additional user input, to iteratively enhance them. The tool integrates different LLM techniques. Few-shot prompting provides examples of expert-evaluated descriptions, and the structured rubric ensures inclusion of essential descriptive content. Retrieval augmented generation (RAG) incorporates relevant information from related publications and user input to improve the completeness and accuracy of the descriptions. Its interactive design preserves the authors’ agency in crafting a data description that represents their research. Rocco’s evaluations of five data descriptions from DPMP were compared to those by domain experts. The results, validated by statistical methods, indicated good agreement between Rocco’s evaluations and those of human experts, but identified a few rubric items that can be refined. This work shows how using prompting and strong evaluation methods, LLM-based tools can effectively and transparently support data curation. Although Rocco was developed for porous media datasets, the tool is readily adaptable to other scientific domains.