Objective:

To evaluate whether privacy-preserving local large language models (LLMs) using different architectural and prompting approaches can accurately identify non-arteritic anterior ischemic optic neuropathy (NAION) from longitudinal electronic health records (L-EHR).

Background:

Accurate retrospective identification of acute NAION cases is important for research on risk factors, but ICD-10 coding has limited positive predictive value (PPV), and manual EHR review is time intensive. LLM-based automated review of L-EHR with patient privacy preserved may offer scalable, accurate case identification.

Design/Methods:

This retrospective single-center study included 161 subjects with ≥ 1 ION (ischemic optic neuropathy) ICD-10 code(s) in the L-EHR.  An expert neuro-ophthalmologist classified each subject for presence of acute NAION based on manual L-EHR review.Three locally-deployed (in order to protect privacy) LLM models via llama.cpp were used to  implement four methods of NAION diagnostic classification using unstructured L-EHR data: basic prompting, retrieval-augmented generation (RAG), two-step agentic workflow, and three-step agentic workflow. LLM and expert performance were compared. 

Results:

Expert review confirmed acute NAION in 41/161 subjects (25.5%). Agentic LLM workflows substantially outperformed basic and RAG approaches. For direct NAION diagnosis, agentic methods achieved a mean PPV of 86.7% , compared to 61.2% for basic prompting and 60.5% for RAG. Feature extraction with agentic workflows demonstrated high reliability: afferent pupillary defect detection reached 82.6-100% PPV, optic disc edema 78.1-94.1% PPV, and sudden onset 67.0-88.9% PPV across models. Three-step workflows consistently achieved the most balanced performance across all three models.

Conclusions:

Privacy-preserving agentic LLM workflows achieved high PPV for acute NAION case identification among subjects with ≥1 ICD-10 code for ION, exceeding ICD-10 code PPV and approaching expert PPV of 100%, while maintaining complete local data control and patient confidentiality. These methods offer a scalable approach for case identification in retrospective clinical research.