Large Language Models for Mental Health Diagnostic Assessments: Exploring The Potential of Large Language Models for Assisting with Mental Health Diagnostic Assessments -- The Depression and Anxiety Case

Basic Information

• Paper ID: 2501.01305
• Title: Large Language Models for Mental Health Diagnostic Assessments: Exploring The Potential of Large Language Models for Assisting with Mental Health Diagnostic Assessments -- The Depression and Anxiety Case
• Authors: Kaushik Roy, Harshul Surana, Darssan Eswaramoorthi, Yuxin Zi, Vedant Palit, Ritvik Garimella, Amit Sheth
• Classification: cs.CL (Computation and Language)
• Publication Date: January 2, 2025 (arXiv preprint)
• Paper Link: https://arxiv.org/abs/2501.01305
• Institutions: University of South Carolina AI Institute, Indian Institute of Research and Science, Indian Institute of Technology

Abstract

Large Language Models (LLMs) are increasingly attracting the attention of medical professionals for assisting diagnostic assessments, with the potential to alleviate healthcare system pressures caused by patient overload and healthcare provider shortages. For LLMs to play an effective role in supporting diagnostic assessments, they must be capable of closely replicating standard diagnostic procedures used by clinicians. This paper specifically investigates the diagnostic assessment process using the Patient Health Questionnaire-9 (PHQ-9) for Major Depressive Disorder (MDD) and the Generalized Anxiety Disorder-7 (GAD-7) questionnaire for Generalized Anxiety Disorder (GAD). The study explores various prompting and fine-tuning techniques to guide both proprietary and open-source LLMs to follow these diagnostic procedures, and evaluates the consistency between LLM-generated diagnostic results and expert-validated ground truth.

Research Background and Motivation

Problem Context

1. Healthcare System Pressures: The current healthcare system faces dual pressures of patient overload and healthcare provider shortages
2. Mental Health Diagnostic Needs: Increasing prevalence of mental health issues necessitates standardized diagnostic assessment tools
3. Potential of LLMs in Healthcare: Large language models demonstrate superior performance in natural language processing tasks and show application potential in medical dialogue scenarios

Research Significance

• Standardized Diagnosis: PHQ-9 and GAD-7 are widely-used standardized assessment tools in clinical practice
• Automation Needs: Automating diagnostic assessments through LLMs can reduce clinician burden
• Consistency Requirements: LLMs must be capable of replicating standard diagnostic procedures used by clinicians for practical application

Limitations of Existing Approaches

1. Scoring Methods: Based solely on text relevance scoring, lacking deep understanding
2. Explainable AI Methods: Using surrogate models such as LIME/SHAP, but with limited clinical interpretability
3. Text Span Identification: Lacking specialized guidance for specific diagnostic criteria

Core Contributions

1. Novel Specialized Model: Proposes DiagnosticLlama, the first fine-tuned model based on Llama architecture specifically designed for diagnostic criteria assessment
2. Comprehensive Evaluation Framework: Establishes a comprehensive evaluation system encompassing both prompting and fine-tuning methods
3. High-Quality Dataset: Constructs an expert-validated synthetic dataset annotated by LLMs to facilitate related research
4. Multi-Model Comparison: Systematically compares performance of proprietary models (GPT-3.5, GPT-4o) and open-source models (Llama-3.1-8b, Mixtral-8x7b)
5. Standardized Methodology: Provides standardized approaches for applying LLMs to PHQ-9 and GAD-7 diagnostic assessments

Methodology Details

Task Definition

Input: Social media post text (as a proxy for patient-clinician interaction) Output: Text span identification and symptom presence judgment for each item in PHQ-9/GAD-7 Constraints: Must strictly adhere to standard diagnostic procedures of PHQ-9 and GAD-7

Model Architecture

1. Prompting Methods

• Naive Prompting: Direct instructional prompts
• Few-Shot Prompting: Prompts providing a small number of examples
• Guided Prompting: Chain-of-Thought prompts with reasoning step guidance

2. Fine-tuning Methods

• Base Model: MentalLlama (trained on 105K mental health instruction data)
• DiagnosticLlama: MentalLlama fine-tuned on the PRIMATE dataset using HuggingFace AutoTrain

Data Processing Pipeline

Ground Truth Dataset Creation

1. Base Data: PRIMATE dataset (social media posts + PHQ-9 annotations)
2. GPT-4o Enhancement: Using GPT-4o to identify text spans corresponding to symptoms
3. Expert Validation: Three clinical experts validate GPT-4o outputs (Cohen's Kappa: 0.74 for PHQ-9, 0.72 for GAD-7)
4. Quality Control: Retaining only annotations consistently approved by experts

Technical Innovations

1. Symptom-Specific Guidance: Specialized prompt templates designed for each symptom in PHQ-9 and GAD-7
2. Multi-Level Evaluation: Dual evaluation system combining hits@k ranking and standard classification metrics
3. Cross-Model Consistency: Validating method effectiveness across multiple LLMs of different scales and types
4. Clinical Validation: Incorporating professional clinicians for quality verification to ensure clinical relevance

Experimental Setup

Datasets

• PRIMATE Dataset: Contains social media posts with PHQ-9-related annotations
• Expert-Validated Subset:
  • PHQ-9: 40 GPT-4o-annotated samples verified by experts
  • GAD-7: 17 GPT-4o-annotated samples verified by experts
• Model-Annotated Data: Multi-model annotation results for a total of 1,034 posts

Evaluation Metrics

1. Hits@k Ranking Metrics:
   • hits@1: Hit rate when the most similar text span ranks first among ground truth
   • hits@5: Hit rate when the most similar text span ranks within top 5 of ground truth
2. Standard Classification Metrics: Accuracy, Precision, Recall, F1-score

Comparison Methods

• Proprietary Models: GPT-3.5-Turbo, GPT-4o-mini
• Open-Source Models: Llama-3.1-8b, Mixtral-8x7b
• Fine-Tuned Models: MentalLlama, DiagnosticLlama
• Traditional Methods: BERT, MentalBERT, MentalRoBERTa
• Machine Learning Methods: Logistic Regression, Random Forest, XGBoost

Implementation Details

• Code-free fine-tuning using HuggingFace AutoTrain
• Identical prompt structure applied across all models to ensure fair comparison
• Random selection of test subsets due to budget and API constraints

Experimental Results

Main Results

PHQ-9 Symptom Annotation Results

Proprietary Model Performance:

Open-Source Model Performance:

Fine-Tuned Model Performance:

GAD-7 Symptom Annotation Results

GAD-7 results demonstrate similar trends to PHQ-9, with both proprietary and open-source models approaching human annotation quality.

Key Findings

1. Model Performance Disparities: Newer-generation LLMs significantly outperform older versions
   • Llama2-7b-chat: F1=0.663
   • Mistral-instruct: F1=0.655
2. Fine-Tuning Challenges: Fine-tuning LLMs for specialized diagnostic tasks is highly challenging
   • MentalLlama directly repeats input, demonstrating the importance of fine-tuning configuration
   • DiagnosticLlama shows improvement but requires further optimization
3. Traditional Method Comparison:
   • BERT: F1=0.69
   • MentalBERT: F1=0.71
   • MentalRoBERTa: F1=0.48
   • Traditional ML methods show poorer performance (highest XGBoost: F1=0.65)

Case Analysis

The paper demonstrates through concrete examples how models identify text spans corresponding to PHQ-9 symptoms, such as identifying "I thought I set myself up for success. Now I believe I was dead wrong for joining" as corresponding to the symptom "feeling like a failure."

Related Work

Main Research Directions

1. Scoring Methods: Text scoring and ranking based on relevance to PHQ-9/GAD-7 symptoms
2. Explainable AI Methods: Using LIME/SHAP techniques to clinically interpret BERT model outputs
3. Text Span Identification: Predicting and summarizing text spans for comparison with manual annotations

Advantages of This Work

• Specialized Guidance: Highly specialized model output guidance targeting specific diagnostic criteria
• Novelty: First diagnostic-specific fine-tuned model based on Llama architecture
• Systematicity: Provides systematic comparison of both prompting and fine-tuning methods

Conclusions and Discussion

Main Conclusions

1. Few-Shot Learning Effectiveness: LLMs in few-shot settings can approach the assessment quality of expert clinicians
2. Reasoning Differences: Despite comparable results, LLM reasoning processes differ significantly from clinical reasoning
3. Fine-Tuning Challenges: Fine-tuning LLMs for mental health diagnostic assistance still faces major technical challenges
4. Practical Potential: The research provides a promising direction for alleviating healthcare system pressures

Limitations

1. Reasoning Consistency: Limited alignment between LLM and clinician reasoning processes
2. Dataset Scale: Expert-validated ground truth dataset is relatively small
3. Budget Constraints: API costs limit large-scale experimental validation
4. Fine-Tuning Complexity: Fine-tuning requires substantial resources and hyperparameter tuning

Future Directions

1. Clinical Applications: Develop applications for clinician use
2. Extended Assessment: Extend DiagnosticLlama to GAD-7 and increase dataset size
3. Complex Questionnaires: Support non-linear structured questionnaires (e.g., CSSRS)
4. Safety Constraints: Integrate terminology restrictions and output rewriting to ensure safety

In-Depth Evaluation

Strengths

1. Strong Clinical Relevance: Directly targets widely-used standardized assessment tools in clinical practice
2. Comprehensive Methodology: Encompasses both prompting and fine-tuning mainstream methods
3. Rigorous Evaluation: Incorporates professional clinician verification to ensure result credibility
4. Open-Source Contribution: Provides models and datasets for community use
5. Sufficient Experimentation: Systematic comparison across multiple models and metrics

Weaknesses

1. Dataset Scale: Expert-validated dataset is relatively small, potentially affecting generalizability of conclusions
2. Domain Limitations: Covers only two conditions (depression and anxiety), with limited scope
3. Reasoning Analysis: Insufficient analysis of differences between LLM and clinician reasoning processes
4. Cost Considerations: Lacks cost-benefit analysis for practical deployment
5. Ethical Discussion: Insufficient discussion of ethical issues in AI-assisted mental health diagnosis

Impact

1. Academic Value: Provides important reference for LLM applications in mental health
2. Practical Value: Provides technical foundation for healthcare institutions to deploy AI-assisted diagnostic systems
3. Social Significance: Promises to alleviate mental health service resource shortages
4. Reproducibility: Open-source code and datasets support research reproduction and extension

Applicable Scenarios

1. Initial Screening: Suitable for large-scale mental health preliminary screening
2. Diagnostic Assistance: Serves as an auxiliary tool rather than replacement for clinicians
3. Telemedicine: Supports remote mental health services
4. Research Tool: Provides automated analysis tools for mental health research

References

The paper cites 29 relevant references covering multiple related fields including LLMs, mental health assessment, prompt engineering, and fine-tuning techniques, providing a solid theoretical foundation for the research.

Overall Assessment: This is an important exploratory work in applying LLMs to mental health diagnostic assessment. The paper employs sound methodology, conducts sufficient experiments, and draws credible conclusions, making valuable contributions to this interdisciplinary field. Despite certain limitations, its pioneering significance and practical value make it an important reference for the field.