Answering real-world clinical questions using large language model, retrieval-augmented generation, and agentic systems

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare

Background:

• Evidence-based medicine faces challenges with data availability and contextualization.
• Large language models (LLMs) offer potential solutions by summarizing literature and generating studies from real-world data.

Purpose of the Study:

• To evaluate the performance of different LLM systems in answering clinical questions.
• To compare general-purpose LLMs with specialized LLMs (RAG-based and agentic) for clinical decision support.

Main Methods:

• Fifty clinical questions were posed to five LLM systems: OpenEvidence (RAG), ChatRWD (agentic), and three general LLMs (ChatGPT-4, Claude 3 Opus, Gemini 1.5 Pro).
• Nine physicians assessed answers for relevance, evidence quality, and clinical actionability.

Main Results:

• General-purpose LLMs provided relevant, evidence-based answers for only 2-10% of questions.
• Specialized LLMs demonstrated superior performance: OpenEvidence (RAG) achieved 24% relevant answers, and ChatRWD (agentic) achieved 58%.
• ChatRWD provided actionable answers for questions lacking existing literature (52%), outperforming others (<10%).

Conclusions:

• Specialized LLM systems substantially outperform general-purpose LLMs for clinical question answering.
• Retrieval-augmented generation (RAG) LLMs excel with existing data, while agentic LLMs are crucial for generating answers when data is scarce.
• Combining RAG and agentic LLMs could enhance evidence availability for patient care.