Prediction of 12-Week Remission in Patients With Depressive Disorder Using Reasoning-Based Large Language Models: Model Development and Validation Study

Overview

This study evaluates the application of reasoning-based large language models to predict 12-week remission outcomes in patients with depressive disorder undergoing antidepressant monotherapy. Given that only 30-40% of patients achieve remission with initial treatment, early identification of likely nonresponders represents a critical unmet need in psychiatric care. The research addresses limitations of prior machine learning approaches, including restricted medication diversity and challenges in clinical interpretability, by testing advanced LLMs enhanced with chain-of-thought reasoning capabilities. The investigation examines three reasoning-enhanced models using multiple prompting strategies on a dataset from the MAKE Biomarker discovery study, encompassing patients treated with 12 different antidepressants across selective serotonin reuptake inhibitors, serotonin and norepinephrine reuptake inhibitors, and other medication classes.

Methods and approach

The study analyzed 390 patients from the MAKE BETTER study who received first-step antidepressant monotherapy between March 2012 and April 2017, after excluding cases with uncommon medications or missing biomarker data. Three reasoning-enhanced large language models were evaluated: ChatGPT o1, ChatGPT o3-mini, and Claude 3.7 Sonnet. Advanced prompting strategies were employed, including zero-shot chain-of-thought, atom-of-thoughts, and a novel referencing of deep research prompt developed by the researchers. Patient data were transformed from numeric coded formats into structured narrative-style reports to enhance model interpretability. Variables included demographic characteristics, psychiatric histories, comorbidities, responses to the Mini-International Neuropsychiatric Interview, adverse childhood experiences, depression subtypes, medication information, suicidality assessments, Hamilton Depression Rating Scale scores, health-related quality of life measures, functional impairment scales, perceived stress, resilience, social support, baseline blood biomarkers, and early treatment response at 2 weeks. For female participants, additional reproductive and hormonal factors were assessed. The primary outcome was 12-week remission, defined as a Hamilton Depression Rating Scale score of 7 or below sustained through the 12-week assessment. Model performance was evaluated using balanced accuracy, sensitivity, specificity, positive predictive value, and negative predictive value. Three psychiatrists independently assessed the clinical validity of model-generated outputs using 5-point Likert scales across dimensions of correctness, consistency, specificity, helpfulness, and human likeness.

Results

Claude 3.7 Sonnet configured with 32,000 reasoning tokens using the referencing of deep research prompt achieved the highest predictive performance, with a balanced accuracy of 0.6697, sensitivity of 0.7183, and specificity of 0.6210. Medication-specific analyses demonstrated negative predictive values of 0.75 or higher across major antidepressants, indicating particular utility in identifying patients unlikely to achieve remission. The clinical evaluation by three independent psychiatrists yielded favorable ratings on 5-point scales for multiple dimensions: correctness (mean 4.3, SD 0.7), consistency (mean 4.2, SD 0.8), specificity (mean 4.2, SD 0.7), and helpfulness (mean 4.2, SD 1.0). The human likeness dimension received a lower rating (mean 3.6, SD 1.7). The performance metrics suggest that reasoning-enhanced models, particularly when augmented with research-informed prompting strategies, can generate clinically interpretable predictions regarding antidepressant treatment outcomes.

Implications

The findings demonstrate that reasoning-based large language models, particularly when enhanced with research-informed prompting methodologies, show potential as interpretable adjunctive tools in treatment planning for depressive disorder. The high negative predictive values across major antidepressant classes suggest particular utility in identifying likely nonresponders, which could inform early treatment modification decisions and reduce the duration of ineffective therapy trials. The favorable clinical validity ratings from psychiatrists across multiple dimensions indicate that model-generated rationales may be sufficiently interpretable and clinically sound for integration into digital mental health workflows. However, the study acknowledges that prospective validation in real-world clinical settings remains essential before implementation in routine practice. The approach addresses key limitations of prior machine learning applications in psychiatry by incorporating diverse medication classes and emphasizing clinical interpretability, though the moderate balanced accuracy indicates room for refinement and the need for continued validation across diverse patient populations and clinical contexts.