Personalized Federated Fine-Tuning of Vision Foundation Models for Healthcare

Basic Information

• Paper ID: 2510.12741
• Title: Personalized Federated Fine-Tuning of Vision Foundation Models for Healthcare
• Authors: Adam Tupper, Christian Gagné (Université Laval, Mila - Quebec AI Institute)
• Classification: cs.CV, cs.DC
• Conference: SMASH 2025
• Paper Link: https://arxiv.org/abs/2510.12741

Abstract

Foundation models open new possibilities for medical AI applications, but require fine-tuning for specific downstream tasks even when pre-trained on health data. Due to patient privacy protection constraints, sharing and aggregating data from different sources remains a challenge. This paper proposes a novel personalized federated fine-tuning method, FedOPAL, which decouples general knowledge and client-specific knowledge by learning orthogonal LoRA adapters, enabling each client to fully leverage both its own data and data from other clients. Preliminary results on real-world federated medical imaging tasks demonstrate that the method is competitive with current federated fine-tuning approaches.

Research Background and Motivation

Core Problems

1. Data Privacy Constraints: Medical data cannot be centrally aggregated for training due to strict privacy regulations
2. Data Scarcity: Annotated datasets in medical imaging are scarce and expensive to obtain
3. Data Heterogeneity: Significant data distribution differences exist across different medical institutions
4. Communication Overhead: Federated learning of large foundation models faces substantial communication costs

Research Motivation

• While federated learning addresses data sharing constraints, it introduces new challenges in foundation model fine-tuning
• Existing methods fail to effectively handle data heterogeneity across clients
• There is a need for a federated fine-tuning approach that leverages global knowledge while maintaining personalization

Core Contributions

1. Proposes FedOPAL Method: A novel personalized federated fine-tuning framework using orthogonal LoRA adapters to separate general and client-specific knowledge
2. Dual Orthogonality Constraint Mechanism: Designs two approaches—weight orthogonalization (FedOPAL-W) and representation orthogonalization (FedOPAL-R)
3. Real Medical Data Validation: Validates the method's effectiveness on two challenging medical imaging datasets: Fed-ISIC 2019 and Camelyon17-WILDS
4. Systematic Comparative Analysis: Comprehensive comparison with multiple state-of-the-art LoRA-based federated learning methods

Methodology Details

Task Definition

In federated medical imaging classification tasks, multiple medical institutions (clients) collaborate to train a shared vision foundation model while protecting their respective data privacy and handling data distribution heterogeneity.

Model Architecture

Dual Adapter Design

FedOPAL employs a dual LoRA adapter architecture:

• Global Adapter: Collaboratively updated by all clients, learning universal features
• Personal Adapter: Private to each client, learning client-specific features

For pre-trained weight matrix $W_0 \in \mathbb{R}^{d \times k}$, the update formula is: $\Delta W = W_0 + B_{global}A_{global} + B_{personal}A_{personal}$

Orthogonality Constraint Mechanism

1. Weight Orthogonalization (FedOPAL-W) Based on the O-LoRA method, orthogonality constraints are imposed on adapter weights: \ell_{orth} = \frac{1}{N} \sum_{i=1}^{N} |(A_i_{global})^T A_i_{personal}|

2. Representation Orthogonalization (FedOPAL-R) Directly constrains learned representations by minimizing cosine similarity: $\ell_{orth} = \frac{1}{NB} \sum_{i=1}^{N} \sum_{j=1}^{B} |\cos(z_{global}^{i,j}, z_{personal}^{i,j})|$

Total loss function: $\ell_{total} = \ell_{task} + \lambda \ell_{orth}$

Technical Innovations

1. Knowledge Decoupling Mechanism: Orthogonality constraints force the two adapters to learn different feature subspaces
2. Dual-Level Orthogonalization: Provides orthogonalization strategies at both weight and representation levels
3. Federated Personalization: Achieves client personalization while maintaining global collaboration

Experimental Setup

Datasets

1. Fed-ISIC 2019

• Scale: 6 clients from medical institutions in Australia, Austria, USA, and Spain
• Task: Skin lesion classification (9 diagnostic categories)
• Characteristics: Significant label shift and uneven sample sizes

2. Camelyon17-WILDS

• Scale: 5 clients (hospitals), each with 7-10 patients
• Task: Patch-level classification of breast cancer metastasis in lymph node slides
• Characteristics: Color variations due to differences in staining procedures

Evaluation Metrics

Balanced accuracy is used to address class imbalance issues

Baseline Methods

• FedIT: Trains A and B matrices of a single global adapter
• FFA-LoRA: Fixes A matrix, trains only B matrix
• FedSA: Shares only A matrix, personalizes B matrix
• FedDPA: Trains independent global and personal adapters
• FedPAL: Baseline method without orthogonality constraints

Implementation Details

• Model: ViT-Tiny (9.7M parameters), pre-trained on ImageNet-21K
• Optimizer: SGD, learning rate η ∈ {1e-4, 5e-4, 1e-3, 5e-3, 1e-2}
• Federated Setting: FedAvg, 100% client participation, 1 epoch local training per round

Experimental Results

Main Results

Fed-ISIC 2019 Task Results:

• FedOPAL-R achieves the best average ranking (3.17) with average accuracy of 0.726
• FedPAL baseline performs well (average ranking 3.50, accuracy 0.726)
• On client 2, FedPAL achieves the highest accuracy of 0.931

Camelyon17-WILDS Task Results:

• FedDPA performs best (average ranking 2.40, accuracy 0.909)
• FedOPAL-R achieves the second-best average accuracy of 0.870
• All federated methods approach centralized training performance (0.905)

Key Findings

1. Method Competitiveness: FedOPAL methods are competitive with existing state-of-the-art approaches
2. Consistency Issues: No single method consistently outperforms across all clients and tasks
3. Orthogonalization Effects: Representation orthogonalization (FedOPAL-R) generally outperforms weight orthogonalization (FedOPAL-W)
4. Room for Improvement: Results indicate further improvement is possible

Related Work

Federated Learning and LoRA Integration

• Parameter Efficiency: LoRA reduces communication overhead and computational requirements
• Personalization Methods: Partial model personalization alleviates data heterogeneity issues
• Recent Advances: Methods like FedDPA and FedSA explore different personalization strategies

Orthogonal Learning Methods

• O-LoRA: Uses orthogonality constraints in continual learning to prevent catastrophic forgetting
• Subspace Learning: Constrains learning of different feature subspaces

Conclusions and Discussion

Main Conclusions

1. FedOPAL successfully decouples general and specific knowledge through orthogonal LoRA adapters
2. The method demonstrates competitiveness with existing approaches on real medical datasets
3. Representation-level orthogonalization is more effective than weight-level orthogonalization

Limitations

1. Performance Consistency: Method performance varies across different clients and tasks
2. Preliminary Results: Current results are preliminary in nature, requiring deeper analysis
3. Hyperparameter Sensitivity: Selection of λ parameter may affect method performance
4. Computational Overhead: Dual adapter design increases computational complexity

Future Directions

1. Adapter Rank Optimization: Explore optimal rank settings for global and local adapters
2. Knowledge Separation Improvement: Refine separation of client-agnostic and client-specific knowledge
3. Extended Evaluation: Evaluate on larger models and more tasks
4. Theoretical Analysis: Provide theoretical guarantees and convergence analysis

In-Depth Evaluation

Strengths

1. Strong Innovation: First to introduce orthogonality constraints in federated LoRA fine-tuning with novel approach
2. High Practical Value: Directly addresses real needs and challenges in medical AI
3. Complete Methodology: Provides two orthogonalization strategies at different levels
4. Comprehensive Experiments: Systematic evaluation on real medical datasets

Weaknesses

1. Weak Theoretical Foundation: Lacks theoretical analysis of orthogonality constraint effectiveness
2. Insufficient Experimental Depth: Missing detailed ablation studies and parameter sensitivity analysis
3. Limited Performance Improvement: Marginal improvements compared to existing methods
4. Limited Evaluation Scope: Evaluation on only two datasets

Impact

1. Academic Contribution: Provides new insights for the intersection of federated learning and medical AI
2. Practical Prospects: Promising for application in real medical federated learning scenarios
3. Scalability: Method framework is extensible to other domains and tasks

Applicable Scenarios

1. Multi-Institution Medical Collaboration: Suitable for federated learning across hospitals
2. Privacy-Sensitive Tasks: Appropriate for applications requiring strict data privacy protection
3. Data Heterogeneity Environments: Particularly suitable for scenarios with large data distribution differences across clients

References

This paper primarily references important works in federated learning (McMahan et al., 2017), LoRA adapters (Hu et al., 2022), and orthogonal learning (Wang et al., 2023), providing a novel solution to federated fine-tuning problems in medical AI.

Summary: This is an innovative work addressing federated learning challenges in medical AI, achieving knowledge decoupling through orthogonal LoRA adapters to improve personalization while protecting privacy. While current results are preliminary, the work provides valuable new insights and methodological frameworks for the field.