SAP: Corrective Machine Unlearning with Scaled Activation Projection for Label Noise Robustness

Basic Information

• Paper ID: 2403.08618
• Title: SAP: Corrective Machine Unlearning with Scaled Activation Projection for Label Noise Robustness
• Authors: Sangamesh Kodge, Deepak Ravikumar, Gobinda Saha, Kaushik Roy (Purdue University)
• Classification: cs.LG cs.AI stat.ML
• Publication Date: January 2, 2025 (arXiv v2)
• Paper Link: https://arxiv.org/abs/2403.08618
• Code Link: https://github.com/sangamesh-kodge/SAP.git

Abstract

Label corruption is a critical problem in deep learning, where incorrectly labeled training samples resulting from non-expert annotation or adversarial attacks significantly degrade model performance. Acquiring large-scale perfectly labeled datasets is costly, and retraining models from scratch incurs substantial computational overhead. To address this, we propose Scaled Activation Projection (SAP), a corrective machine unlearning algorithm based on Singular Value Decomposition (SVD). SAP mitigates label noise by identifying a small set of trustworthy samples using cross-entropy loss and projecting model weights into a clean activation space estimated from these trustworthy samples using SVD. Experiments demonstrate that SAP achieves up to 6% generalization improvement under 25% synthetic corruption on CIFAR datasets, provides an average improvement of approximately 3.2% over noise-robust training methods, and achieves 2.31% generalization improvement on Vision Transformer models on the naturally corrupted Clothing1M dataset.

Research Background and Motivation

Problem Definition

1. Label Noise Problem: Label errors are prevalent in large-scale datasets, originating from:
   • Manual annotation errors
   • Misclassifications by automated labeling systems (e.g., large language models)
   • Malicious data poisoning attacks
2. Limitations of Existing Solutions:
   • Data Cleaning Methods: Require model retraining with high computational costs
   • Noise-Robust Training: Improves robustness but cannot completely eliminate performance gaps
   • Traditional Machine Unlearning: Requires explicit distinction between mislabeled and hard-to-learn samples, difficult in practical applications
3. Research Motivation:
   • Avoid high computational costs of retraining from scratch
   • No need for explicit identification of mislabeled samples
   • Achieve efficient noise mitigation through single-step weight updates

Core Contributions

1. Proposed SAP Algorithm: An SVD-based corrective machine unlearning algorithm that mitigates label noise effects through activation projection
2. Automated Trustworthy Sample Selection: Automatically identifies trustworthy samples using cross-entropy loss, eliminating manual annotation
3. Single-Step Weight Update: Achieves efficient model correction through one SVD computation and weight projection
4. Comprehensive Experimental Validation: Verifies effectiveness in synthetic and real-world noise scenarios, supporting multiple model architectures

Method Details

Task Definition

Given a training dataset $D_{Tr}$ containing label noise, the objective is to correct the parameters $θ^*$ of a trained model such that its generalization performance on the test set approaches that of a model trained on clean data, without requiring retraining.

Model Architecture

1. Weight Update Mechanism

For a linear layer $a_{out} = a_{in}W^T$, SAP projects input activations through an activation alignment matrix $W_p$:

$\hat{a}_{out} = (a_{in}W_p)W^T = a_{in}(WW_p^T)^T = a_{in}\hat{W}^T$

The weight update rule is: $\hat{W} = WW_p^T$

2. Trustworthy Data Estimation

Selects $N_{Trust}$ samples with the lowest cross-entropy loss as the trustworthy set:

$D_{Trust} = \arg\min_S \sum_{(x_i,y_i) \in S} L(θ^*, x_i, y_i)$

where $S = \{S_i \subseteq D_{Tr} | |S_i| = N_{Trust}\}$

3. Representation Matrix Construction

• Linear Layers: $R_{linear} = [(a_i^{in})_{i=1}^{N_{Trust}}]$
• Convolutional Layers: Converts convolution to matrix multiplication via unfold operation, $R_{conv} = [(unfold(a_i^{in})^T)_{i=1}^{N_{Trust}}]$

4. SVD Decomposition and Importance Scaling

Performs SVD decomposition on the representation matrix: $R^l = U^l Σ^l V^{lT}$

Computes importance weights: $λ_i = \frac{α\tilde{σ}_i}{(α-1)\tilde{σ}_i + 1}$

where $\tilde{σ}_i = σ_i^2 / \sum_{j=1}^d σ_j^2$ is the normalized singular value, and $α$ is the scaling coefficient.

5. Activation Alignment Matrix

Constructs the projection matrix: $W_p = UΛU^T$, where $Λ = diag(λ_1, λ_2, ..., λ_d)$

Technical Innovations

1. Automated Processing: No manual identification of erroneous samples required; trustworthy samples are automatically selected via loss function
2. Efficient Updates: Single SVD computation and matrix multiplication complete weight updates, avoiding iterative optimization
3. Activation Space Projection: Suppresses the influence of noisy activations by projecting into clean activation space
4. Architecture Agnostic: Applicable to linear and convolutional layers, supporting diverse network architectures

Experimental Setup

Datasets

1. Synthetic Noise Datasets:
   • CIFAR-10/CIFAR-100
   • Three noise types: symmetric noise, asymmetric noise, hierarchical noise
   • Noise intensities: 10% and 25%
2. Real-World Noise Datasets:
   • Mini-WebVision
   • Clothing1M

Evaluation Metrics

• Test set accuracy
• Performance comparison with baseline methods
• Generalization improvement magnitude

Comparison Methods

• Retrain: Ideal model retrained on clean data
• Vanilla: Baseline model trained on noisy data
• Finetune: Fine-tuned on limited clean data
• SSD: Unlearning algorithm based on selective synaptic suppression
• SCRUB: State-of-the-art machine unlearning algorithm

Implementation Details

• Number of trustworthy samples: 1,000
• Scaling coefficient α search range: 2000, 300000
• Model architectures: VGG11, ResNet18, ResNet50, ViT-B/16
• Optimizer: SGD, learning rate 0.01, weight decay 5×10^-4

Experimental Results

Main Results

Synthetic Noise Experiments

Results on CIFAR-10 and CIFAR-100 datasets demonstrate:

SAP outperforms other unlearning methods across all noise settings, with average improvements of 1.36% (CIFAR-10) and 0.39% (CIFAR-100).

Noise-Robust Training Enhancement

SAP further improves the performance of existing noise-robust methods:

Real-World Noise

Results on real-world noise datasets:

Ablation Studies

Impact of Trustworthy Sample Count

Experiments show diminishing returns when increasing trustworthy samples beyond 1,000, thus 1,000 samples were selected to balance performance and computational efficiency.

Impact of Scaling Coefficient α

α=30000 demonstrates optimal performance across various synthetic noise settings; both larger and smaller α values degrade performance.

Experimental Findings

1. Computational Efficiency: SAP requires only 16 hyperparameter searches, while SCRUB requires 675
2. Robustness: Demonstrates stable performance across different noise types and intensities
3. Scalability: Successfully applied to large-scale datasets and Transformer models
4. Decision Boundary Optimization: Visualization experiments show SAP smooths decision boundaries, reducing overfitting

Related Work

Classification of Label Noise Handling Methods

1. Data Cleaning Methods:
   • Data filtering: Removing mislabeled samples
   • Sample selection: Dynamically selecting training samples
   • Label correction: Correcting incorrect labels
2. Noise-Robust Training:
   • Regularization techniques: Dropout, label smoothing
   • Robust loss functions: Symmetric cross-entropy, MAE
   • Data augmentation: MixUp, MentorMix
3. Corrective Machine Unlearning:
   • Traditional unlearning focuses on privacy protection
   • Corrective unlearning focuses on improving generalization performance

Advantages of This Work

Compared to existing methods, SAP offers:

• No need for explicit identification of erroneous samples
• Single update avoids instability of iterative optimization
• Simple hyperparameter tuning and high computational efficiency

Conclusions and Discussion

Main Conclusions

1. Effectiveness Verification: SAP significantly improves model generalization performance in both synthetic and real-world noise scenarios
2. Efficiency Advantages: Single-step weight updates and simple hyperparameter tuning provide significant computational advantages
3. Broad Applicability: Supports multiple network architectures and dataset scales
4. Practical Value: Can be combined with existing noise-robust methods for further performance improvement

Limitations

1. Trustworthy Sample Assumption: Relies on the assumption that low-loss samples are indeed correctly labeled
2. Hyperparameter Sensitivity: The choice of scaling coefficient α significantly impacts performance
3. Noise Type Constraints: Primarily targets label noise; limited effectiveness on other noise types
4. Insufficient Theoretical Analysis: Lacks theoretical guarantees for method effectiveness

Future Directions

1. Theoretical Analysis: Establish theoretical foundations for SAP effectiveness
2. Adaptive Parameter Selection: Develop methods for automatically selecting optimal α
3. Extended Applications: Explore applications to other noise types and tasks
4. Integration with Other Techniques: Investigate combinations with data augmentation, adversarial training, etc.

In-Depth Evaluation

Strengths

1. Method Innovation:
   • First application of SVD to corrective machine unlearning
   • Novel and effective activation projection concept
   • Automated trustworthy sample selection eliminates manual intervention
2. Experimental Comprehensiveness:
   • Covers multiple noise types and datasets
   • Comparison with multiple baseline methods
   • Includes ablation studies and parameter sensitivity analysis
3. Practical Value:
   • High computational efficiency, easy to deploy
   • Can be combined with existing methods
   • Supports multiple network architectures
4. Result Convincingness:
   • Consistent performance improvements
   • Statistical significance verification
   • Visualization analysis enhances understanding

Weaknesses

1. Weak Theoretical Foundation:
   • Lacks theoretical analysis of method effectiveness
   • Does not explain why SVD projection effectively suppresses noise
2. Assumption Limitations:
   • Assumption that low-loss samples are correctly labeled may not always hold
   • Strong assumptions about noise distribution
3. Parameter Adjustment:
   • Lack of theoretical guidance for α selection
   • Different datasets may require different α values
4. Limited Comparisons:
   • Insufficient comparison with latest noise-robust methods
   • Lacks direct comparison with data cleaning methods

Impact

1. Academic Contribution:
   • Provides new research direction for machine unlearning field
   • Activation projection concept may inspire other applications
2. Practical Application:
   • Provides practical tools for handling real-world label noise
   • Can be integrated into existing training pipelines
3. Reproducibility:
   • Provides complete code implementation
   • Detailed experimental setup description

Applicable Scenarios

1. Scenarios with low-quality dataset labels
2. Situations where data re-annotation is impossible
3. Applications requiring rapid correction of trained models
4. Environments with limited computational resources

References

The paper cites important works in related fields, including:

• Machine Unlearning: SCRUB, SSD, and other methods
• Label Noise Handling: MixUp, MentorMix, SAM, etc.
• Data Cleaning: Confident Learning, etc.
• Foundational Theory: SVD decomposition, activation analysis, etc.

Overall Assessment: The proposed SAP method has significant value in label noise handling. Through clever activation projection design, it achieves efficient model correction. While theoretical analysis is somewhat lacking, experimental validation is comprehensive and practical value is substantial, providing valuable contributions to the field.