Pilot development of an inhaler technique assessment tool using a multimodal large language model

Title : Pilot development of an inhaler technique assessment tool using a multimodal large language model

Abstract:

Background: Improper inhaler technique is common among patients with chronic obstructive pulmonary disease (COPD) and leads to poor disease control. Although repeated in-person education can improve inhaler technique, it is resource-intensive and difficult to sustain. Given that many patients never receive inhaler technique evaluation, multimodal large language model (LLM) tools can offer a practical solution. This study aimed to develop an automated assessment tool for inhalers using a multimodal LLM to support healthcare providers and patients with COPD in improving inhaler technique training. 
Methods: Representative devices were selected for each major inhaler type: Evohaler for pressurized metered-dose inhaler, Ellipta and Breezhaler for dry powder inhaler, and Respimat for soft mist inhaler. For each device, we recorded improper inhaler technique videos that reflected the critical errors reported in previous studies. After iteratively refining prompts using videos demonstrating proper inhaler technique, the performance of the refined prompts was evaluated using videos of improper inhaler technique. Assessment metrics included accuracy, specificity, recall, precision, and F1 score for each inhaler technique step. 
Results: Across all inhaler technique steps for the four representative devices, every computable metric—accuracy, specificity, recall, precision, and F1 score—was 1.0. These values demonstrate that the assessment tool achieved perfect performance for every device and step.  
Conclusion: The multimodal LLM-based assessment tool enables comprehensive assessment of inhaler technique across all types of inhalers. While the tool demonstrated excellent performance under controlled environments, further validation is required in real clinical settings. With continued refinement, this tool has the potential to serve as a practical solution in clinical settings by identifying patients who need repeated inhaler training. It may also reduce the burden of manual evaluations for healthcare providers through automatic, consistent, and standardized assessment.

Biography:

Sola Cho is currently a graduate student in the College of Nursing at Chonnam National University, Republic of Korea. She has clinical experience in the Division of Cardiology at Samsung Medical Center. She is conducting research under the supervision of Professor Ja Yun Choi in the faculty lab at the College of Nursing. Her academic interests include digital health, patient self-management, and respiratory care. She is the first author of a KCI-indexed article titled “Proper Use of Inhalers by Patients with Chronic Obstructive Pulmonary Disease: A Descriptive Literature Review” and is currently working on an AI-based tool to assess inhaler techniques in COPD patients using a multimodal large language model.