A Unified Approach Towards Active Learning and Out-of-Distribution Detection

Basic Information

• Paper ID: 2405.11337
• Title: A Unified Approach Towards Active Learning and Out-of-Distribution Detection
• Authors: Sebastian Schmidt, Leonard Schenk, Leo Schwinn, Stephan Günnemann
• Category: cs.CV
• Publication Date: May 2024 (arXiv v3: October 12, 2025)
• Paper Link: https://arxiv.org/abs/2405.11337

Abstract

When applying deep learning models in open-world scenarios, active learning (AL) strategies are crucial for identifying labeling candidates from large quantities of unlabeled data. In this context, robust out-of-distribution (OOD) detection mechanisms are necessary for handling data outside the application's target distribution. However, current work addresses these two problems separately. This paper introduces SISOM as the first unified solution for both AL and OOD detection. By leveraging feature space distance metrics, SISOM combines the advantages of currently independent tasks to effectively address both problems. On the OpenOOD benchmark, SISOM achieves one first-place and two top-three rankings on near-OOD data, while also achieving top-tier performance on AL tasks.

Research Background and Motivation

Problem Definition

1. Active Learning Challenge: Large-scale deep learning models require extensive annotated data, but annotation costs are high, necessitating intelligent selection of the most valuable samples for annotation
2. Out-of-Distribution Detection Need: Models encounter data outside the training distribution during practical deployment, requiring reliable OOD detection mechanisms
3. Limitations of Independent Processing: Existing methods treat AL and OOD detection as independent tasks, resulting in conflicting design objectives and system complexity

Research Motivation

• Practical Application Requirements: Real-world applications such as mobile robotics and autonomous driving require simultaneous handling of label efficiency and out-of-distribution data issues
• Task Correlation Discovery: The authors observe that near-OOD data and unlabeled data exhibit overlap and ambiguity in the latent space
• Advantages of Unified Approaches: Unified processing eliminates overhead from independent design phases and avoids conflicting design objectives

Core Insights

Through UMAP visualization, the paper reveals that near-OOD data and unlabeled AL candidate data are positioned similarly in feature space, both tending to locate between existing clusters or near decision boundaries, providing theoretical foundation for a unified approach.

Core Contributions

1. First Unified Method: Proposes SISOM as the first unified method specifically designed to simultaneously address AL and OOD detection
2. Feature Space Analysis Technique: Introduces latent space analysis enabling optimization loops for post-training feature space refinement
3. Self-Balancing Fusion Mechanism: Proposes self-balancing fusion of uncertainty and diversity metrics
4. Superior Experimental Performance: Achieves excellent results against highly specialized state-of-the-art methods on common image AL and OOD benchmarks

Method Details

Task Definition

Active Learning: Starting from initial labeled pool L and unlabeled pool U, model f selects new samples A for annotation through query strategy Q(x,f), with the goal of achieving optimal performance with minimal annotation.

OOD Detection: Given a model f trained on distribution Ω, determine whether test sample x̃ comes from in-distribution (InD) or out-of-distribution (OOD):

$G(x, f) = \begin{cases} \text{InD} & \text{if } S(x; f) \geq \lambda \\ \text{OOD} & \text{if } S(x; f) < \lambda \end{cases}$

SISOM Method Architecture

SISOM comprises five core components:

1. Coverage

Constructs informative feature space representations by concatenating multiple network layers: $z = h_1(x) \oplus \cdots \oplus h_j(x) \oplus \cdots \oplus h_n(x)$

2. Feature Enhancement

Uses KL divergence gradient-weighted features: $g = \frac{\partial D_{KL}(u||f(x))}{\partial z}$$\tilde{z} = \sigma(z \odot g)$

where u is a uniform distribution and ⊙ denotes Hadamard product.

3. Distance Ratio

Defines intra-class and inter-class distances: $d_{in}(\tilde{z}) = \min_{z' \in Z_L(c'=c)} ||\tilde{z} - \tilde{z}'||_2$$d_{out}(\tilde{z}) = \min_{z' \in Z_L(c' \neq c)} ||\tilde{z} - \tilde{z}'||_2$

SISOM score: $r(x) = \frac{d_{in}}{d_{out}}$

4. Feature Space Analysis

Computes average distance ratio as a proxy for feature space separability: $r_{avg} = \frac{1}{|L|} \sum_{z \in L} \frac{d_{in}(\sigma(z \odot g))}{d_{out}(\sigma(z \odot g))}$

Adaptively fuses uncertainty and diversity: $\hat{r}_i = \min(r_{avg}, 1) \cdot E_i + \max(1 - r_{avg}, 0) \cdot r_i$

5. Sigmoid Steepness

Optimizes sigmoid function steepness parameter α for each layer to improve feature space separability: $\alpha_{opt} = \arg\min_\alpha r_{avg}(\alpha)$

Technical Innovations

1. Unified Metric Design: Distance ratio simultaneously captures AL and OOD detection requirements
2. Adaptive Fusion: Automatically balances uncertainty and diversity based on feature space quality
3. Multi-layer Feature Integration: Leverages information from multiple network layers to enhance representational capacity
4. Gradient Weighting: Highlights important neurons through KL divergence gradient weighting

Experimental Setup

Datasets

Active Learning:

• CIFAR-10/100: Query size q=1000/2000
• SVHN: Query size q=500
• Uses ResNet18/34 architecture

OOD Detection:

• Follows OpenOOD benchmark settings
• CIFAR-10: Near-OOD (CIFAR-100, Tiny ImageNet), Far-OOD (MNIST, SVHN, etc.)
• CIFAR-100: Near-OOD (CIFAR-10, Tiny ImageNet), Far-OOD (MNIST, SVHN, etc.)
• ImageNet-1k: Near-OOD (SSB-hard, NINCO), Far-OOD (iNaturalist, etc.)

Evaluation Metrics

• Active Learning: Classification accuracy variation with percentage of annotated data
• OOD Detection: AUROC (Area Under ROC Curve)

Comparison Methods

Active Learning Baselines:

• CoreSet, Badge, Learning Loss, CoreGCN
• Semi-supervised settings: TypiClust, ProbCover, PT4AL

OOD Detection Baselines:

• NAC, KNN, MSP, Energy, ReAct, SCALE and 17 other methods

Experimental Results

Main Results

Active Learning Performance

• CIFAR-10: SISOM and SISOMe outperform other methods across all selection cycles
• CIFAR-100: Achieves highest performance in later selection steps
• Semi-supervised Settings: SISOM significantly benefits from pre-training, surpassing PT4AL

OOD Detection Performance

Overall Ranking: SISOMe achieves top-three rankings across all three benchmarks, with first-place overall ranking.

Ablation Studies

1. Optimal Sigmoid Steepness: Improves performance on CIFAR-100 and ImageNet
2. Reduced Subset Selection: Significantly increases inference speed with slight performance improvement
3. Feature Enhancement Effect: KL divergence gradient weighting significantly improves feature space separability

Runtime Analysis

With 5% subset selection, SISOMe runtime reduces to 266 seconds with slight performance improvement.

Full Lifecycle Experiments

Using checkpoints from AL training for OOD detection, SISOMe achieves best performance on both near-OOD and far-OOD, validating the effectiveness of the unified approach.

Related Work

Active Learning

• Uncertainty Methods: Monte Carlo Dropout, ensemble methods
• Diversity Methods: CoreSet, Badge and other gradient-based methods
• Hybrid Methods: Methods combining uncertainty and diversity

OOD Detection

• Preprocessing Methods: Data augmentation, mixture training
• Post-processing Methods: Neuron filtering, weight operations
• Logits-based Methods: Temperature scaling, energy scores
• Feature Space Methods: Mahalanobis distance, k-nearest neighbors

Open-Set Active Learning

Existing methods primarily use independent modules for AL and OOD; this paper is the first to explore the intrinsic connections between the two tasks.

Conclusions and Discussion

Main Conclusions

1. Unified Feasibility: First demonstration that AL and OOD detection can be effectively addressed through a single method
2. Superior Performance: Achieves or approaches state-of-the-art performance on both tasks
3. Practical Value: Simplifies deployment complexity for real-world applications

Limitations

1. Computational Complexity: Distance computation introduces additional overhead, though mitigated through subset selection
2. Feature Space Dependency: Method performance depends on feature space separability
3. Hyperparameter Sensitivity: Sigmoid steepness parameter requires tuning for different datasets

Future Directions

1. Open-Set Active Learning: Extend unified method to open-set AL scenarios
2. Complex Tasks: Extend to more complex tasks such as object detection and semantic segmentation
3. Batch Diversification: Investigate batch diversification techniques' impact on dual-task methods

In-Depth Evaluation

Strengths

1. Strong Innovation: First unified method for AL and OOD detection with significant theoretical and practical value
2. Clever Method Design: Distance ratio design is both simple and effective; adaptive fusion mechanism reflects deep insights
3. Comprehensive Experiments: Covers multiple datasets and baselines, including ablation studies and runtime analysis
4. Solid Theoretical Foundation: Visualization analysis reveals intrinsic connections between the two tasks

Weaknesses

1. Insufficient Theoretical Analysis: Lacks theoretical guarantees for convergence and generalization ability
2. Limited Applicability Scope: Primarily validated on image classification tasks; applicability to other domains remains unknown
3. Complex Parameter Tuning: Multiple hyperparameters require dataset-specific tuning, potentially affecting practical usability

Impact

1. Academic Contribution: Opens new research direction for unified study of AL and OOD detection
2. Practical Value: Significant importance for real-world applications such as mobile robotics and autonomous driving
3. Reproducibility: Provides detailed implementation details and code for easy reproduction and extension

Applicable Scenarios

1. Resource-Constrained Environments: Applications requiring simultaneous consideration of annotation efficiency and OOD detection
2. Real-time Systems: Through subset selection, maintains performance while improving efficiency
3. Open-World Applications: Autonomous driving, robot navigation and other scenarios facing distribution shifts

References

The paper cites extensive related work, primarily including:

• Active Learning: Settles (2010), Sener & Savarese (2018), Ash et al. (2020)
• OOD Detection: Yang et al. (2022), Liu et al. (2020), Sun et al. (2022)
• OpenOOD Benchmark: Yang et al. (2022), Zhang et al. (2023)

Overall Assessment: This is a high-quality research paper proposing an innovative unified method addressing two important problems with sufficient experimental validation, significantly advancing the related field. Despite some theoretical and practical limitations, its pioneering contributions and superior performance make it an important work in the field.