Applications of ChatGPT and LLMs in medicine

Chatbots and the large language models (LLMs) that underlie the chatbots have the potential to revolutionize the field of medicine. In a previous post, I described how a chatbot assistant can capably respond to patient questions made online with evaluators preferring the ChatGPT responses to physician responses 78.6% of the time. In another study, Google researchers demonstrated that their fine-tuned large language model (LLM) called Med-PaLM 2 was able to perform at an “expert” doctor level answering medical exam (USMLE) questions, scoring 85%. USMLE is a standardized medical licensing exam given to physicians during their training, and ~60% correct is considered a passing score (QH).

In a new short review in Nature Biotechnology, the review authors prognosticate on the near and distant futures of chatbots in three different areas of medicine: clinical, education, and research. They summarize their views in a table in which each column succinctly describes the predicted utility of chatbots for a given area (see Table 1). 

I will focus on the clinical sector column which the authors have subdivided into knowledge and documentation. From a knowledge perspective, ChatGPT makes it easier for doctors to manage various diseases by collecting and organizing evidence from various medical guidelines. In other words, it can serve as a comprehensive knowledge resource that concisely summarizes vast amounts of medical information in a more user-friendly form than a Google search.

But the authors rate this use-case as cautionary for a number of reasons. In particular, the chatbot may not provide the most current, accurate, or relevant information. In the case of ChatGPT, the model's training data stops at November 2021, making its recommendations potentially outdated in today's rapidly evolving medical field. Regular updates are crucial to ensure accuracy.

Perhaps most concerning are issues with accuracy and bias. It is well documented that ChatGPT and LLMs in general can hallucinate or just make up facts which in the field of medicine could have deadly consequences. Solving the hallucination problem will not be easy because of the complexity of the LLMs. Biases can arise from non-representative training data. As a result, the authors recommend careful evaluation by the user of the chatbot-generated information.

Additionally, ChatGPT's responses are often general and may not be directly relevant to specific situations. To get more detailed information, you'll need to provide additional prompts.

Finally, ChatGPT is incapable of producing original concepts, as it relies on pre-existing information. Consequently, it is unable to address inquiries for which data are unavailable. One of the singular advantages of human intelligence is the ability to knit together disparate facts to address totally novel situations that may not be found in the training data.

Thus, the review authors believe that the greatest benefit of ChatGPT in clinical settings might be in automating documentation tasks like emails, progress notes, discharge summaries, radiology reports, insurance letters, and more. As healthcare professionals face increasing pressure to document encounters in real-time, ChatGPT could not only improve the accuracy and completeness of documentation but also lighten their workload significantly. Using ChatGPT to generate standardized notes would help ensure all relevant information is captured consistently and efficiently, freeing up valuable time for patient care. Included in the documentation task category is answering patient questions as mentioned in the first paragraph above.

Perhaps most exciting is the role of chatbots and LLMs in the more distant future of medicine, and the authors envision the following:

"As AI technology continues to evolve, its role in medicine is expected to broaden. One prospective direction is the creation of ChatGPT-integrated decision support tools, offering real-time, patient-specific treatment guidance for clinicians and patients. As machine learning techniques progress, AI-based solutions may assume a predictive role in cancer screening and diagnostics, enhancing the precision of early detection and intervention. ChatGPT is expected to synergize with other technologies, such as imaging, pathology and genomics, to facilitate more rapid and accurate diagnoses to inform treatment decisions. Lastly, ChatGPT is poised to streamline complex administrative tasks that demand intricate coordination, such as scheduling chemotherapy and radiotherapy sessions, ultimately improving the overall efficiency of cancer care management."

In addition to assisting in clinical decision-making and patient management, the AI technology in LLMs could be used for real-time patient-specific treatment guidance, and predicting disease at an earlier stage more accurately. 

Table 1. Current and future applications of ChatGPT in medicine (from Yan et al. Nature Biotechnology, 2023).