A secondary data analysis on hypotheses generated by inexperienced clinical researchers: Cases from a randomized controlled study

Objectives: To obtain insights about inexperienced clinical researchers’ hypothesis quality and associated factors. The findings inform the development of informatics tools to aid the hypothesis generation process. Methods: We analyzed an existing dataset collected through a randomized controlled study, focusing on individual hypotheses and participants. We invited clinical researchers to analyze datasets and develop hypotheses using the think-aloud method. Participants’ screen activity and audio were recorded, transcribed, coded, and analyzed to measure the time and cognitive events (a granular unit of thought processes used by the participants while generating hypotheses). Hypotheses were rated by an expert panel. Here we analyzed (1) the top 5-rated hypotheses, (2) the bottom 5-rated hypotheses, and (3) the participants who generated them. Results: Participants who generated the top 5-rated hypotheses utilized fewer cognitive events and a shorter range of time per hypothesis; their hypotheses presented a higher valid rate, and they were more experienced. Conclusion: Having more experience is positively associated with higher quality and valid rates of the generated hypotheses. The higher-rated hypotheses seem to be positively associated with slightly fewer cognitive events and shorter time. The effect may not be linear. These analyses provide evidence for customized study designs or tool development based on these associated factors.