Target Speaker Anonymization in Multi-Speaker Recordings

Basic Information

• Paper ID: 2510.09307
• Title: Target Speaker Anonymization in Multi-Speaker Recordings
• Authors: Natalia Tomashenko¹, Junichi Yamagishi², Xin Wang², Yun Liu², Emmanuel Vincent¹
• Institutions: ¹Université de Lorraine, CNRS, Inria, Loria, France; ²National Institute of Informatics, Tokyo, Japan
• Classification: eess.AS (Audio and Speech Processing), cs.CL (Computational Linguistics), cs.CR (Cryptography and Security)
• Publication Date: October 10, 2025 (arXiv preprint)
• Paper Link: https://arxiv.org/abs/2510.09307

Abstract

Existing speaker anonymization research has primarily focused on single-speaker audio, resulting in techniques and evaluation metrics optimized for such conditions. This work addresses the significant challenge of speaker anonymization in multi-speaker conversational audio, particularly in scenarios requiring anonymization of only a single target speaker. Such scenarios are highly relevant in environments like call centers, where customer privacy must be protected through anonymization of customer voice while preserving operator information. Traditional anonymization methods are often inadequate for this task. Furthermore, current evaluation methodologies cannot accurately assess privacy protection and utility in such complex multi-speaker scenarios. This work aims to bridge these gaps by exploring effective target speaker anonymization strategies for conversational audio, highlighting potential issues in their development and proposing improved evaluation methods.

Research Background and Motivation

Problem Definition

The core problem addressed in this research is selective anonymization of specific target speakers in multi-speaker conversational recordings—a novel and challenging task. Traditional speaker anonymization techniques are designed primarily for single-speaker audio and cannot effectively handle selective anonymization requirements in multi-speaker scenarios.

Significance and Application Value

1. Legal Compliance Requirements: With the implementation of privacy protection regulations such as GDPR, voice data privacy protection has become critical
2. Practical Application Scenarios: In call centers, medical consultations, and similar environments, there is a need to protect customer privacy while preserving service personnel information
3. Technical Challenges: Voice data contains rich personal information (age, gender, health status, emotional state, etc.), requiring privacy protection while maintaining linguistic content

Limitations of Existing Methods

1. Technical Limitations: Existing anonymization methods cannot selectively target specific speakers in mixed audio
2. Insufficient Evaluation: Lack of privacy protection and utility evaluation metrics tailored for multi-speaker scenarios
3. Limited Applicability: Traditional methods perform poorly with overlapping speech and complex conversational scenarios

Core Contributions

1. Proposes Target Speaker Anonymization (TSA) Framework: First systematic approach to selective anonymization in multi-speaker conversations
2. Develops Comprehensive Evaluation Methodology: Establishes privacy protection and utility evaluation systems for multi-speaker anonymization scenarios
3. Experimental Validation and Analysis: Comprehensive experimental evaluation based on two state-of-the-art target speaker extraction methods
4. Identifies Key Challenges: In-depth analysis of inherent limitations and technical challenges, providing guidance for future research

Methodology Details

Task Definition

Input: Mixed audio signals containing multiple speakers
Output: Mixed audio with anonymization applied only to the target speaker
Constraints: Preserve original speech of non-target speakers and maintain overall conversational intelligibility and utility

Model Architecture

TSA Framework Design

TSA employs a three-step pipeline approach:

1. Target Speaker Extraction (TSE):
   • Uses pre-trained speaker embedding vectors to identify the target speaker
   • Estimates complex-valued soft masks to separate target speaker time-frequency spectra
   • Extracts target speaker speech segments from mixed audio
2. Speaker Anonymization:
   • Applies anonymization processing only to extracted target speaker speech
   • Utilizes anonymization systems based on vector quantization bottleneck (VQ-BN) features
   • Synthesizes anonymized speech through HiFi-GAN networks
3. Speech Recombination:
   • Combines anonymized target speaker speech with original non-target speaker speech
   • Generates final partially anonymized mixed audio

Target Speaker Extraction Models

Conformer-based TSE:

• Combines convolutional layers and self-attention mechanisms to process STFT spectrograms
• Reconstructs real and imaginary components of target speaker STFT spectra
• Integrates speaker embeddings to identify and focus on target speakers

WeSep BSRNN TSE:

• Explicitly partitions audio spectra into multiple frequency bands
• Performs fine-grained modeling of unique spectral features for each band
• Based on band-split recurrent neural network architecture

Technical Innovations

1. Novel Framework: First complete solution for target speaker anonymization in multi-speaker scenarios
2. Modular Design: Decoupled TSE and anonymization modules enabling optimization and replacement
3. Innovative Evaluation System: Introduces new metrics such as tcpWER for comprehensive assessment of privacy protection and utility
4. Attacker Modeling: Considers semi-informed attacker scenarios for more realistic privacy evaluation

Experimental Setup

Dataset

• SparseLibri2Mix: Multi-speaker dataset constructed from LibriSpeech test-clean subset
• Overlap Conditions: Five different overlap levels (20%, 40%, 60%, 80%, 100%)
• Data Scale: 500 mixed files per condition, totaling 2,500 files (~5 hours of speech)
• Speaker Count: 40 speakers, with the first speaker designated as target speaker

Evaluation Metrics

Privacy Protection Assessment

• Equal Error Rate (EER): Evaluates anonymization effectiveness using automatic speaker verification (ASV) systems
• Attacker Model: Semi-informed attacker with access to anonymization system and training data

Utility Assessment

• Primary Metric: Time-constrained minimum permutation word error rate (tcpWER)
• Auxiliary Metrics:
  • Diarization error rate (DER)
  • Word error rate (WER) for target speaker ASR
  • Scale-invariant signal-to-distortion ratio (SI-SDR)

Comparison Methods

• Anonymization System: B5 baseline system from VoicePrivacy 2024 Challenge
• TSE Models: Conformer-based TSE vs. WeSep BSRNN TSE
• Evaluation Models: ECAPA-TDNN ASV system, DiCoW ASR system

Experimental Results

Main Results

TSE Model Performance Comparison

Privacy Protection Effectiveness

• Single-speaker Scenario: EER improves from 3.0% to 32.4% after anonymization
• Multi-speaker Scenario:
  • Conformer TSE: Average EER 36.4%
  • WeSep BSRNN TSE: Average EER 36.9%
• Privacy Improvement: 12-14% improvement compared to single-speaker scenarios

Utility Preservation

• tcpWER Results:
  • Conformer TSE: Average 17.8%
  • WeSep BSRNN TSE: Average 14.6% (superior)
• DER Results: WeSep BSRNN outperforms Conformer across all overlap conditions

Ablation Studies

TSE Quality Impact

1. Original Signal Extraction: TSE process causes significant relative decline in EER and WER compared to original mixed signals
2. Anonymization Impact: WER further increases after anonymization, primarily due to insertion errors from residual non-target speaker signals
3. Overlap Degree Impact: TSE performance degrades with increasing overlap, but privacy protection effectiveness remains relatively stable

Attacker Strategy Analysis

• Reference Signal Selection: Attacks using original reference signals outperform those using anonymized reference signals
• TSE Model Consistency: Attacks are most effective when attackers use the same TSE model as users

Experimental Findings

1. TSE is Critical Bottleneck: TSE quality directly impacts final privacy protection and utility
2. Overlapping Speech Challenge: TSE performance significantly degrades under high overlap conditions
3. Insertion Error Problem: Residual non-target speaker signals increase ASR insertion errors
4. Privacy-Utility Trade-off: Inherent trade-off exists between privacy protection and speech utility

Related Work

Speaker Anonymization Research

1. Signal Processing Methods: Simple transformation methods such as McAdams coefficients and pitch shifting
2. Neural Speech Conversion Methods: Anonymization techniques based on decoupled representation learning
3. VoicePrivacy Challenge: Advanced single-speaker anonymization technology development

Target Speaker Extraction

1. Deep Learning Methods: Speech separation techniques based on deep neural networks
2. Attention Mechanisms: Attention mechanisms guided by speaker embeddings
3. Band-Split Techniques: Advanced frequency-domain processing methods such as BSRNN

Multi-Speaker Scenario Research

Existing multi-speaker anonymization research is extremely limited; this paper represents pioneering work in this field.

Conclusions and Discussion

Main Conclusions

1. Technical Feasibility: TSA framework enables selective anonymization of target speakers in multi-speaker scenarios
2. Performance Trade-offs: Trade-offs exist among privacy protection, speech quality, and computational complexity
3. Evaluation Importance: New evaluation metrics are critical for accurately assessing multi-speaker anonymization effectiveness
4. Improvement Potential: Current methods have significant room for improvement in utility preservation

Limitations

1. TSE Dependency: Method performance heavily depends on TSE module quality
2. Computational Complexity: Three-step pipeline increases system complexity and computational overhead
3. Utility Degradation: tcpWER shows significant decline compared to original audio
4. Dataset Limitations: Experiments conducted only on simulated datasets, lacking validation on real conversational data

Future Directions

1. End-to-End Training: Joint training of TSE and anonymization modules to optimize overall performance
2. Improved TSE: Development of TSE models specifically optimized for anonymization tasks
3. Real-Time Processing: Exploration of real-time or near-real-time TSA solutions
4. Multimodal Anonymization: Integration of visual information for multimodal privacy protection

In-Depth Evaluation

Strengths

1. Strong Innovation: First systematic approach to multi-speaker target anonymization, filling an important research gap
2. Complete Methodology: Provides comprehensive solution from technical framework to evaluation methods
3. Sufficient Experiments: Comprehensive comparative experiments across multiple TSE models and overlap conditions
4. In-Depth Analysis: Detailed analysis of module contributions and system limitations
5. Practical Significance: Addresses urgent needs in practical applications such as call centers

Weaknesses

1. Performance Limitations: tcpWER shows significant decline compared to original audio, with utility requiring improvement
2. Computational Efficiency: Three-step pipeline has high computational complexity, limiting real-time applications
3. Data Limitations: Lacks validation on real conversational data
4. Attacker Model: Relatively simple attacker model, not considering more sophisticated attack strategies
5. Privacy Evaluation: EER results of 36-37% indicate remaining privacy leakage risks

Impact

1. Academic Contribution: Pioneering work opening new research direction in multi-speaker target anonymization
2. Practical Value: Provides privacy protection solutions for call centers, medical, and related industries
3. Technology Advancement: Promotes convergence of TSE and speech anonymization technologies
4. Standard Setting: Provides reference for related evaluation standards and benchmark development

Applicable Scenarios

1. Call Centers: Protecting customer privacy while preserving service quality analysis capabilities
2. Medical Consultation: Anonymizing patient speech for medical research and training purposes
3. Legal Recordings: Processing court recordings to protect party privacy
4. Educational Training: Anonymizing student speech for teaching and research purposes

References

This paper cites 31 relevant references covering multiple related fields including voice privacy protection, speaker anonymization, target speaker extraction, and automatic speech recognition, providing a solid theoretical foundation for the research.

Overall Assessment: This is a high-quality research paper making pioneering contributions to the important and challenging problem of multi-speaker voice privacy protection. While there remains room for improvement in technical performance, its innovative framework design, comprehensive evaluation methodology, and in-depth analysis establish an important foundation for subsequent research in this field.