Analysis and Advancement in Domain-Specific Templated Question Answering

This work addresses the challenge of domain-specific question answering through the intelligent composition of tool sequences using a large language model. We formulate the problem as utilizing a set of tools T to answer a query Q by determining the necessary tools, arguments, and execution sequence. Our approach enhances language model capabilities through prompt engineering, leveraging advanced reasoning, and adopting our custom Chain of Thoughts (CoT) inspired strategy for dynamic, cascaded user engagement. Employing multi-task learning broadens knowledge scope, while transfer learning from domains with richer tooling enhances versatility. Runtime compute costs are optimized through distillation. The evaluation shows our method excels in selecting optimal tool combinations for domainspecific queries, outperforming baseline approaches in accuracy and coverage. This approach provides a reusable framework for constructing proficient and cost-effective domain-specific Question Answering (QA) solutions. Key explorations encompass analysis of prompt engineering for tool interfaces, compositional learning across tools, transfer learning from richer domains, and prompt distillation. These facilitate the practical deployment of LLMs for industrial applications.