Objective:

The primary goal of this study is to assess the impact of manually reviewed notes on the performance of AI/ML models in cognitive assessment tests for dementia.

Background:

The integration of AI in healthcare is transforming diagnostic processes. This study highlights the crucial role of human intervention in enhancing the diagnostic accuracy of AI/ML models.

Design/Methods:

Dementia patients aged 65 and above were identified in 2018 using ICD-10 diagnosis codes from the Optum de-identified Market Clarity Database. Inclusion criteria required patients to have either two confirmed outpatient diagnoses (at least 30 days apart) or one confirmed inpatient diagnosis. The index event was defined as the first documented diagnosis of dementia. To ensure comprehensive data, 12 months of medical and pharmacy eligibility pre- and post-index was ensured. Cognitive assessment tools (CAT) such as MMSE, MOCA, and SLUMS, along with their respective scores, were identified in both structured and unstructured data. These scores were used to classify the severity of dementia into mild, moderate, and severe categories.

This study involved a comparative analysis of AI/ML models trained on two distinct datasets: one  manually reviewed notes and the other without manual reviewing of notes. The performance of these models was evaluated based on their diagnostic accuracy, sensitivity, and specificity in identifying dementia.

Results:

Among the 101,126 patients, CAT scores were identified in 3% of structured data and 9% of unstructured data. The analysis revealed a significant improvement in the performance of AI/ML models when manually reviewed notes are incorporated. Specifically, models trained on manually reviewed notes demonstrated a higher ( 8 to 10 percentage points higher) diagnostic accuracy, sensitivity, and specificity as compared to model without manual review of notes.

Conclusions:

This study underscores the necessity of collaborative approaches in leveraging AI for better clinical outcomes in dementia assessment.