Development and validation of a novel model for characterizing migraine outcomes within electronic health records utilizing artificial intelligence
Steven Kymes1, Roger Cady1, Nada A Hindiyeh2, Kimberly Alexander3, Daniel Riskin3
1Lundbeck LLC, 2Stanford Headache Clinic at Hoover Pavilion, 3Verantos Inc
Objective:

To define and validate the accuracy of a scalable framework model using electronic health record (EHR) data to measure migraine treatment and prevention outcomes by using artificial intelligence (AI).

Background:
In disease areas such as migraine that report subjective measures (ie. severity, descriptors, etc.) as outcomes, extraction and validation of real-world evidence (RWE) from EHRs can be challenging. Information required to assess migraine endpoints used in clinical trials are not consistently captured in routine care, limiting the utility of these previously defined endpoints in RWE studies. 
Design/Methods:

Headache specialists defined clinical features found in routinely collected data. EHR data were reviewed by two clinical annotators to create a manual reference standard for features included in the migraine outcome model. Data elements were weighted to define a 10-point scale incorporating headache severity (1–7 points) and associated features (0–3 points). Automation (i.e., AI) extracted features from patient encounters and compared to the reference standard. A 70% agreement threshold (within 1 point) between the human annotator and the automated score was considered sufficient extraction accuracy. AI accuracy in identifying features used to construct the outcome model success was defined as reaching an F1 score of 80% for identifying encounters.

Results:

From a total of 2,006 encounters, 11 features were included in the model; average automated extraction F1 scores were 92.0% when applied to unstructured data. Automated extraction model scores matched for 77.2% of encounters exactly and were a within 1 point close match for 82.2%, compared with manual extraction scores—well above the 70% match threshold.

Conclusions:

These data indicate feasibility of AI generated models to generate migraine outcome scores using features commonly captured in real-world settings with high accuracy, providing a scalable approach to EHR-based clinical studies to support migraine prevention and treatment.

10.1212/WNL.0000000000202014