Paper Reading Journals #22
Performance Assessment of ChatGPT versus Bard in Detecting Alzheimer’s Dementia
Basic Info
| Item | Content | | —————— | ——————————————————————————- | | Title | Performance Assessment of ChatGPT versus Bard in Detecting Alzheimer’s Dementia | | Journal | Diagnostics | | Author | Balamurali B.T, Jer-Ming Chen | | Year Published | 2024 | | Link | https://doi.org/10.3390/diagnostics14080817 | | Geography of Study | N/A | | Area of Study | Alzheimer’s dementia detection, LLM, ADRD Screening |
Description
This research evaluate LLM performance in screening Alzheimer’s Disease under zero sample conditions. Three LLM are evaluated in this study: ChatGPT-3.5, ChatGPT-4, Bard. The study was conduct under zero sample condition where only spontaneous verbal transcript were given as input. The goal for LLM is to identify sample status of: (1) Alzheimer’s Disease (2) Cognitive Normal (3) Unidentified
Method
Datasource:
Dataset: ADReSSo Challenge - a well-studied dataset of audio recording which “captured interview sessions where participants described the “Cookie Theft Picture” from the Boston Diagnostic Aphasia Examination in English” (Balamurali & Chen, 2024) Sample size: 71 in total. 35 Alzheimer’s Disease, 36 Cognitive Normal Transcription Method: Otter.ai Validation / Benchmark: Mini–Mental State Examination (MMSE)
Model:
LLM Models: (1) ChatGPT-3.5 (2) ChatGPT-4 (3) Bard Prompt: Query 1 and Query 2 Query 1: simple questions Query 2: Read syntax, vocabulary, structure, narration style, grammar, semantic discourse, stylistics, pragmatics… Following a more structured and more concise prompt.
Main Contribution
Innovation Type: Method Innovation
This research conduct pioneer work in adopting LLM under no sample reasoning and prediction to predict ADRD status. It reveals that among three LLM, Bard has the best performance in identify Alzheimer’s Disease with sensitivity of 89%, 0.71 F1 score. While GPT-4 has a specificity of 84%, 70% precision. These result reveals the potential of adopting LLM in ADRD screening since it surpass average guess. However, the authors also indicate it still not ideal choice and need further improvement to be adopt in clinical settings.
Favorite Thing
(1) The research design of this study is valuable. similar design could be adopt to later LLM+ADRD study (2) This study indicate and certify a potential of adopting LLM in ADRD study (3) It indicate one type of data source could be use for LLM ADRD screening. More variety of ADRD screening could be further adopt.