Objective:

Background:

As artificial intelligence (AI) becomes increasingly integrated into scientific research, critical appraisal remains a foundational skill for residents practicing evidence-based medicine. Traditional journal club formats involve pre-session preparation, which can be challenging for resident physicians with limited time. To address these challenges, we implemented a no-prep journal club curriculum that required residents and faculty to critique AI outputs.

Design/Methods:

We conducted four monthly AI-integrated no-prep journal clubs in early 2025, collecting data via focused ethnography and semi-structured interviews. Ethnographic data included session summaries and ChatGPT transcripts. ChatGPT transcripts were graded for prompt quality. Interviews were conducted one-on-one via Zoom. Summaries and interview transcripts were coded descriptively and thematically analyzed using grounded theory.

Results:

Ethnography revealed inconsistent prompt quality and general distrust of outputs. However, residents expressed areas of uncertainty and collaborated in problem-solving while relying on faculty scaffolding to contextualize AI outputs. Based on a grading rubric, prompt quality of ChatGPT interactions varied across sessions (Mean = 70%, SD = 22.5%) with no clear improvement. Five neurology PGY 2-4 residents were interviewed, and several themes were identified. Residents described AI as a helpful scaffold for study design but perceived it as untrustworthy without faculty input. Residents with little prior experience with AI reported increased adoption and confidence using it to support research literacy and clinical decision-making. Residents also requested more instruction on how to use AI tools effectively.

Conclusions: