OPENXRD: A Comprehensive Benchmark Framework for LLM/MLLM XRD Question Answering

Key Points

• The aim is to develop a benchmarking framework for assessing how well LLMs and MLLMs answer questions related to crystallography.
• Developed a comprehensive benchmarking framework called OPENXRD.
• Evaluated performance based on context assimilation in answering questions.
• Included various types of models like LLMs and MLLMs for comparison.
• Demonstrated effective evaluation metrics for model performance.
• Provided insights into models' abilities to assimilate contextual information.
• Showed variability in performance across different types of models.