Confidence Calibration in Large Language Model-Based Entity Matching

Basic Information

• Paper ID: 2509.19557
• Title: Confidence Calibration in Large Language Model-Based Entity Matching
• Authors: Iris Kamsteeg, Juan Cardenas-Cartagena, Floris van Beers, Gineke ten Holt, Tsegaye Misikir Tashu, Matias Valdenegro-Toro
• Classification: cs.CL cs.LG
• Publication Date: October 15, 2025 (arXiv v2)
• Institutions: Bernoulli Institute, University of Groningen, The Netherlands; Independent Researcher
• Paper Link: https://arxiv.org/abs/2509.19557

Abstract

This study explores the intersection of large language models and confidence calibration in entity matching. Through empirical investigation, it compares baseline confidence scores from RoBERTa in entity matching tasks with calibrated confidence scores using temperature scaling, Monte Carlo Dropout, and ensemble methods. Experiments are conducted on the Abt-Buy, DBLP-ACM, iTunes-Amazon, and Company datasets. Results demonstrate that the improved RoBERTa model exhibits slight overconfidence, with Expected Calibration Error (ECE) ranging from 0.0043 to 0.0552 across different datasets. The study finds that temperature scaling mitigates this overconfidence, reducing ECE scores by up to 23.83%.

Research Background and Motivation

Problem Definition

Entity Matching (EM) is a critical subtask of entity resolution, aiming to determine whether data entries from different sources refer to the same real-world entity. This is a binary classification problem requiring judgment of whether entity pairs are "matches" or "non-matches."

Significance

1. Multi-domain Application Value: Improves patient care in healthcare, connects birth, marriage, and death records in historical population reconstruction, and is critical for investigation and crime prevention in law enforcement
2. Transparency Requirements: Models must provide reliable confidence scores alongside predictions to enable users to understand model reliability
3. Downstream Task Guidance: Accurate confidence scores can guide decision-making in subsequent tasks

Limitations of Existing Methods

1. Overconfidence Problem: Modern large language models exhibit overconfidence in other NLP tasks, struggling to accurately express prediction uncertainty
2. Research Gap: While LLMs have been studied for confidence calibration, their application in entity matching remains insufficiently explored
3. Lack of Systematic Evaluation: Absence of systematic comparative studies of confidence calibration methods for entity matching tasks

Research Motivation

Provide model prediction transparency, help understand model internal mechanisms, identify model weaknesses, and improve performance. When explicitly knowing where models are uncertain, improvement directions become clearer.

Core Contributions

1. First Systematic Study: First systematic investigation of confidence calibration for LLMs in entity matching
2. Multi-method Comparison: Comprehensive comparison of temperature scaling, Monte Carlo Dropout, and ensemble methods for confidence calibration in entity matching
3. Multi-dataset Validation: Validates method effectiveness and generalization across six datasets from different domains and structures
4. Practical Guidance: Provides best practice recommendations for confidence calibration in real applications, particularly highlighting advantages of temperature scaling

Methodology Details

Task Definition

• Input: Entity pairs from different data sources
• Output: Binary classification labels ("match"/"non-match") and corresponding confidence scores
• Objective: Ensure confidence scores accurately reflect the true probability of correct predictions

Model Architecture

Base Architecture

1. Pre-trained RoBERTa: Uses HuggingFace's RoBERTa-base model as encoder
2. Fully Connected Layer: Single-layer fully connected network added after RoBERTa
3. Sigmoid Output Layer: Produces confidence scores between 0-1
4. Data Serialization: Converts structured data to text sequences using the method from Li et al. (2020)

Confidence Calibration Methods

1. Temperature Scaling

• Applies temperature parameter T to scale logits after Sigmoid output
• Optimizes temperature parameter through grid search on validation set: T ∈ {0.1, 0.2, ..., 10.0}
• Selects temperature value minimizing ECE
• Advantages: lightweight, easy to implement, does not alter F1 scores

2. Monte Carlo Dropout

• Applies dropout (probability p) to fully connected layer during inference
• Performs 10 forward passes and averages outputs
• Grid searches optimal dropout probability: p ∈ {0.05, 0.10, ..., 0.95}
• Selects p with minimum ECE while maintaining F1 scores

3. Ensemble Method

• Trains 5 fully connected layers with different random initializations
• Averages outputs from 5 models as final prediction
• Applies ensemble only to fully connected and Sigmoid layers to minimize computational cost

Technical Innovations

1. Lightweight Implementation: Monte Carlo Dropout and ensemble methods applied only to fully connected layers, minimizing computational overhead
2. Multi-metric Optimization: Can optimize ECE, MCE, or RMSCE depending on application requirements
3. Statistical Significance Verification: Uses paired t-tests (temperature scaling, Monte Carlo Dropout) and unpaired t-tests (ensemble method) to assess improvement significance

Experimental Setup

Datasets

Uses six entity matching datasets from different domains:

Note: S/D denotes structured/dirty data versions; percentages in parentheses indicate positive sample ratios

Evaluation Metrics

1. Expected Calibration Error (ECE): Primary metric measuring average difference between predicted and empirical probabilities
2. Maximum Calibration Error (MCE): Measures worst-case deviation, suitable for high-risk applications
3. Root Mean Square Calibration Error (RMSCE): Emphasizes larger errors more strongly
4. F1 Score: Ensures calibration improvements do not compromise classification performance
5. Visual Analysis: Confidence histograms and reliability diagrams

Baseline Methods

• Baseline: Uncalibrated RoBERTa Sigmoid output
• Calibration Methods: Temperature scaling, Monte Carlo Dropout, ensemble method

Implementation Details

• Training Epochs: 40 (following Li et al. 2020 settings)
• Model Selection: Selects checkpoint with highest F1 on validation set
• Experiment Repetition: Each experiment repeated 5 times with mean and standard deviation reported
• Bin Count: √|D| (where D is dataset size)

Experimental Results

Main Results

Baseline Performance Analysis

RoBERTa exhibits slight overconfidence across all datasets:

• ECE Range: 0.0043-0.0552, lowest on DBLP-ACM, highest on Company dataset
• Confidence Distribution: Model tends to produce extremely high or low prediction probabilities
• F1 Performance: Exceeds 98% on DBLP-ACM, approximately 82% on Company dataset

Calibration Method Comparison

Temperature Scaling Performs Best:

• Significantly reduces ECE by 23.83% on Abt-Buy dataset
• Achieves significant improvements on 4 datasets
• Does not affect F1 score performance

Ablation Studies

Temperature Parameter Analysis

• Optimal Temperature Values: Typically greater than 1.0 (average 1.72±0.51), confirming baseline model overconfidence
• Parameter Stability: Clear optimal temperature value exists for each dataset and run

Dropout Probability Analysis

• Optimal Probability Range: Between 0.5-1.0, exceeding 0.8 for some datasets
• Generalization Issues: Optimal dropout probabilities vary significantly across datasets, lacking consistency

Case Analysis

Confidence histograms reveal:

• Correct Predictions: Primarily concentrated in high confidence intervals
• Incorrect Predictions: More dispersed distribution, but substantial proportion still shows high confidence
• Overlap Problem: Significant overlap between correct and incorrect prediction confidence distributions, indicating insufficient calibration

Experimental Findings

1. Widespread Overconfidence: RoBERTa exhibits varying degrees of overconfidence across all datasets
2. Temperature Scaling Most Effective: Outperforms other methods in improving ECE
3. Computational Efficiency Advantage: Temperature scaling has minimal computational overhead and easy deployment
4. Performance Preservation: Calibration methods essentially do not affect classification performance

Related Work

LLMs in Entity Matching

• BERT Series Models: Brunner and Stockinger (2020) found BERT, RoBERTa and similar models achieve 35.9% F1 improvement over traditional methods
• DITTO System: Li et al. (2020) combined LLMs with optimization techniques for entity matching
• Decoder Models: Applications of GPT-3, ChatGPT, GPT-4 in entity matching research

LLM Confidence Calibration

• Early Findings: Guo et al. (2017) discovered widespread calibration issues in modern neural networks
• BERT/RoBERTa Research: Calibration studies by Desai and Durrett (2020), Xiao et al. (2022) across multiple NLP tasks
• Calibration Methods: Development of temperature scaling, Monte Carlo Dropout, and ensemble methods

Novel Contributions of This Work

• Domain First: First systematic application of confidence calibration to entity matching tasks
• Method Comparison: Comprehensive comparison of multiple calibration methods
• Practical Guidance: Provides best practice recommendations for real applications

Conclusions and Discussion

Main Conclusions

1. Overconfidence Confirmed: RoBERTa exhibits overconfidence in entity matching tasks with ECE scores of 0.0043-0.0552
2. Temperature Scaling Optimal: Temperature scaling is the most effective calibration method, reducing ECE by up to 23.83%
3. Performance Preservation: Confidence calibration does not compromise classification performance
4. Strong Practicality: Temperature scaling is simple to implement and suitable for real deployment

Limitations

1. Model Scale Constraints: Research focuses on relatively small RoBERTa models, not addressing larger modern LLMs
2. Evaluation Metric Limitations: ECE, MCE, RMSCE may not accurately reflect calibration quality in certain cases
3. Computational Constraints: Ensemble method experiments incomplete on Company dataset due to computational limitations
4. Method Singularity: Does not explore combined use of multiple calibration methods

Future Directions

1. Large Model Extension: Extend research to larger-scale language models like GPT-4
2. Method Combination: Explore combinations of temperature scaling with other methods, such as Ensembles+Temperature Scaling
3. Variance Utilization: Leverage variance information from Monte Carlo Dropout and ensemble methods to improve calibration
4. New Evaluation Metrics: Develop evaluation metrics more accurately reflecting calibration quality

In-Depth Evaluation

Strengths

1. High Research Value: Fills research gap in confidence calibration for entity matching
2. Rigorous Experimental Design: Comprehensive comparison across multiple datasets, methods, and metrics
3. Statistical Rigor: Uses appropriate statistical tests to verify result significance
4. Strong Practicality: Provides directly applicable methods and parameter selection guidance
5. Clear Writing: Well-structured paper with accurate technical details

Weaknesses

1. Limited Model Coverage: Studies only RoBERTa architecture
2. Insufficient Theoretical Analysis: Lacks in-depth explanation of why temperature scaling performs best
3. Dataset Scale: Some datasets (e.g., iTunes-Amazon) are relatively small, potentially affecting generalization
4. Computational Resource Constraints: Affects completeness of certain experiments

Impact

1. Academic Contribution: Introduces important confidence calibration research direction to entity matching field
2. Practical Value: Temperature scaling method is simple, effective, and easily deployable in real systems
3. Reproducibility: Detailed experimental settings facilitate reproduction and extension
4. Inspirational Value: Provides important foundation and direction guidance for subsequent research

Applicable Scenarios

1. High-Risk Applications: Medical record matching and similar scenarios requiring reliable confidence estimates
2. Human-Machine Collaboration: Applications needing model uncertainty information to assist human decision-making
3. Quality Control: Using confidence scores to identify difficult samples requiring manual review
4. Model Optimization: Leveraging confidence information to improve model training and data collection strategies

References

1. Guo, C., et al. (2017). On Calibration of Modern Neural Networks. ICML.
2. Li, Y., et al. (2020). Deep Entity Matching with Pre-Trained Language Models. VLDB.
3. Desai, S., & Durrett, G. (2020). Calibration of Pre-trained Transformers. EMNLP.
4. Brunner, U., & Stockinger, K. (2020). Entity Matching with Transformer Architectures. EDBT.
5. Peeters, R., & Bizer, C. (2024). Entity Matching using Large Language Models. arXiv.

Summary: This paper makes important contributions to confidence calibration research in entity matching, providing systematic method comparison and practical solutions. The excellent performance of temperature scaling offers valuable guidance for practical applications. Despite certain limitations, this research establishes a solid foundation for subsequent work and possesses significant academic and practical value.