The Initial Screening Order Problem

Basic Information

• Paper ID: 2307.15398
• Title: The Initial Screening Order Problem
• Authors: Jose M. Alvarez (KU Leuven), Antonio Mastropietro (University of Pisa), Salvatore Ruggieri (University of Pisa)
• Classification: cs.LG cs.CY
• Publication Date: July 2023 (arXiv preprint, updated January 2025)
• Paper Link: https://arxiv.org/abs/2307.15398

Abstract

This paper investigates the role of Initial Screening Order (ISO) in candidate screening. ISO refers to the order in which screeners search through a candidate pool when selecting k candidates. Currently, ISO is typically generated by information access systems (such as online platforms or database queries). Although ISO has significant implications for the optimality and fairness of the selected k candidates (particularly in human screener scenarios), this aspect has been largely overlooked in the literature. The authors define two problem formulations to characterize screener search behavior under a given ISO: the best-k problem (selecting the k best candidates) and the good-k problem (selecting k sufficiently good candidates). To investigate ISO's impact, the authors introduce a human-like screener model and compare it with algorithmic screeners, where human-like screeners are designed to be inconsistent over time. The analysis demonstrates that ISO impedes individual fairness (while maintaining group fairness) and damages the optimality of selected k candidates when human-like screeners solve the good-k problem. This is caused by position bias, wherein candidate evaluation is influenced by their position in the ISO.

Research Background and Motivation

Problem Context

Candidate screening is a complex, labor-intensive task prevalent in scenarios such as resume screening and university admissions. With the advancement of machine learning, information access systems (IAS) such as LinkedIn and Taleo platforms play a central role in candidate screening, typically presenting candidates based on estimated relevance or specific attributes.

Research Motivation

1. Practical Requirements: Based on collaboration experience with a Fortune Global 500 company G in Europe, the authors identified five key practical patterns:
   • G1: Screeners select different ISOs
   • G2: Both complete and partial search methods exist
   • G3: Focus on candidates meeting minimum basic requirements
   • G4: Fairness objectives with diversity representation quotas
   • G5: Approximately one minute per candidate evaluation
2. Theoretical Gap: Existing literature primarily focuses on ISO creation (as fair set selection or ranking problems), but rarely investigates how screeners utilize ISO, particularly human screener behavior.
3. Fairness Concerns: Position bias may cause similar candidates to be treated differently based on their position in the ISO, violating individual fairness principles.

Core Contributions

1. First Formalization of ISO Problem: Establishes ISO as a critical parameter in set selection problems, defining problem formulations for best-k and good-k search behaviors.
2. Introduction of Human-like Screener Model: Proposes a human-like screener model considering fatigue effects and provides theoretical and experimental comparisons with algorithmic screeners.
3. Provision of Flexible Simulation Tools: Develops a simulation framework for studying ISO problems, enabling practitioners to gain insights without running actual screening scenarios.
4. Revelation of Position Bias's Fairness Impact: Demonstrates that ISO leads to individual fairness violations under human-like screeners while maintaining group fairness constraints.

Methodology Details

Task Definition

Given a candidate pool C containing n candidates, each candidate c is described by a feature vector X_c ∈ R^d and protected attribute w_c. Screener h must select a set S^k of k candidates based on ISO θ while satisfying representation quota q (minimum proportion of protected groups in S^k).

Two Problem Formulations

Best-k Problem

Screeners must evaluate the complete candidate pool C and select the k candidates with the highest scores:

argmax_{S^k ∈ [C]^k} U^k_add(S^k, θ) s.t. f(S^k) ≥ q

where the utility function is defined as:

U^k_add(S^k, θ) = Σ_{c∈S^k} s(X_c)

Good-k Problem

Screeners seek the first k candidates meeting minimum requirement ψ, allowing partial search:

argmax_{S^k ∈ [C]^k} U^k_ψ(S^k, θ) s.t. f(S^k) ≥ q

where the utility function is defined as:

U^k_ψ(S^k, θ) = {
  k - Σ_{c∈S^k} p(c, S^k, θ)  if ∀c ∈ S^k, s(X_c) ≥ ψ
  0                            otherwise
}

The penalty function p(c, S^k, θ) measures the "wasted effort" in selecting candidate c.

Screener Models

Algorithmic Screener h_a

Consistently evaluates candidates without fatigue effects, executing according to deterministic algorithms.

Human-like Screener h_h

Introduces fatigue component Φ(t) = λ·t, affecting candidate evaluation:

s_h_h(X_c) + ε

where ε is a random variable dependent on accumulated fatigue, considering two modeling choices:

• ε₁ ~ N(0, v(Φ(t-1))): Variance increases with fatigue
• ε₂ ~ N(μ(Φ(t-1)), v(Φ(t-1))): Mean decreases with fatigue

Search Algorithms

• ExaminationSearch (Algorithm 1): Solves best-k problem, searching in descending score order
• CascadeSearch (Algorithm 2): Solves good-k problem, searching in ISO order
• Corresponding human-like versions (Algorithms 3-4): Incorporate fatigue effects

Experimental Setup

Simulation Framework

Monte Carlo simulation framework developed in R, running 10,000 experiments per parameter set.

Candidate Score Distribution

Three truncated normal distributions t_N(μ, σ) are considered:

• Symmetric Distribution: μ=0.5, σ=0.02 (low probability of top candidates)
• Asymmetric Distribution: μ=0.8, σ=0.05 (moderate probability of top candidates)
• Increasing Distribution: μ=1, σ=0.05 (high probability of top candidates)

ISO Settings

• θ ⊥⊥ s: ISO independent of individual scores (random or alphabetical order)
• θ ⊥̸⊥ s: ISO correlated with scores, correlation coefficient ρ ∈ {-1, -0.8, -0.5}

Experimental Parameters

• Candidate pool size: n = 120, 400, 30
• Selection quantity: k = 6, 20
• Quota: q = 0.5
• Protected group proportion: pr = 0.2
• Minimum requirement: ψ ∈ 0.3, 0.8

Evaluation Metrics

• Ratio to Baseline (RtB): Utility ratio relative to baseline solution
• Jaccard Similarity (JdS): Proportion of candidate overlap

Experimental Results

Without Fatigue (Algorithmic Screeners)

Good-k vs Best-k Performance

1. Score Distribution Impact:
   • Under symmetric distribution, good-k gradually approaches best-k as ψ increases
   • Under asymmetric and increasing distributions, good-k struggles to reach best-k performance even at large ψ
2. ISO Correlation Impact:
   • When ρ = -1 (complete negative correlation), good-k and best-k performance are identical
   • When ρ = -0.5, good-k already approximates best-k well
3. Scale Effects:
   • Larger k/n ratios enable good-k to better approximate best-k
   • ISO impact weakens as k/n increases

With Fatigue (Human-like Screeners)

Fatigue Impact on Good-k

1. Variance-type Fatigue (ε₁):
   • Symmetric distribution shows significant performance degradation at high ψ values
   • Asymmetric and increasing distributions remain relatively stable
2. Bias-type Fatigue (ε₂):
   • Smaller standard deviation results in less impact on symmetric distribution
   • Bias impact is minimal when candidate quality is low

Best-k vs Good-k with Fatigue

• Under fatigue conditions, good-k solutions sometimes outperform best-k solutions
• Indicates that prolonged search is harmful for human-like screeners
• Good-k strategy should be preferred in certain settings

Protective Effect of ISO Correlation

• Providing ISO containing candidate quality information reduces fatigue impact
• Emphasizes the importance of IAS in the ISO problem

Related Work

Fair Set Selection

This paper belongs to the fair set selection literature, related to the Secretary Problem, but focuses on the screening process rather than the interview stage, employing an offline set selection setting.

Position Bias Research

• Click Models: Early formalization and testing of how users search ISO
• Empirical Studies: Echterhoff et al. and Wang et al. provide empirical evidence of position bias in candidate screening
• This Paper's Contribution: Defines ISO as a parameter in problem formulations, uses simulation to capture diverse screening scenarios

Information Access Systems

Existing work primarily focuses on ISO creation (fair ranking), while this paper investigates how screeners utilize ISO, particularly human user behavior.

Conclusions and Discussion

Main Conclusions

1. Fairness Impact of Position Bias: ISO leads to individual fairness violations under human-like screeners, even while maintaining group fairness constraints.
2. Importance of Problem Formulation: Correctly defining problem formulations is crucial for understanding ISO's impact on candidate selection.
3. Complexity of Screener Behavior: Complex relationships exist between best-k and good-k problems, depending on candidate quality distribution, ISO correlation, and fatigue effects.

Limitations

1. Functional Assumption Constraints: Functional assumptions in utility models and fatigue terms may be oversimplified
2. Fatigue Modeling: Linear fatigue accumulation may not be realistic
3. Simulation vs. Reality: Simulation results require validation in real scenarios

Future Directions

1. Alternative Utility Models: Explore other utility models such as exposure discounting
2. Advanced Fatigue Models: Consider cyclical survival models incorporating screener rest periods
3. Human Decision Theory: Integrate more human decision-making theory
4. Parameter Optimization: Use simulation framework to test optimal parameters

In-Depth Evaluation

Strengths

1. Problem Importance: First systematic investigation of ISO, a neglected yet important problem
2. Theoretical Contribution: Provides rigorous mathematical framework and algorithmic implementation
3. Practical Relevance: Insights based on real enterprise collaboration with practical application value
4. Methodological Innovation: Introduces human-like screener model considering fatigue effects
5. Experimental Comprehensiveness: Explores diverse parameter combinations through large-scale simulation
6. Tool Contribution: Provides open-source simulation framework

Weaknesses

1. Model Simplification: Fatigue model is relatively simple, potentially unable to capture complex human behavior
2. Insufficient Validation: Primarily simulation-based, lacking large-scale real data validation
3. Assumption Limitations: Independence assumptions (A1, A2) may not hold in reality
4. Limited Evaluation Metrics: Primarily focuses on utility and overlap, lacking other fairness metrics

Impact

1. Academic Value: Provides new perspective for fair machine learning and human-computer interaction fields
2. Practical Value: Provides guidance for recruitment platform and HR system design
3. Methodological Contribution: Simulation framework is extensible to other decision scenarios
4. Policy Significance: Provides tools for algorithm auditing and fairness regulation

Applicable Scenarios

1. Recruitment Screening: Resume screening, candidate evaluation
2. Educational Assessment: Application review, assignment grading
3. Recommendation Systems: Fair recommendations considering user search behavior
4. Decision Support: Any scenario involving sequential evaluation

Technical Innovation Points

Core Innovations

1. ISO Parameterization: Establishes initial screening order as an explicit problem parameter
2. Dual Problem Formulation: Distinguishes between best-k (global optimum) and good-k (satisficing) objectives
3. Fatigue Modeling: Introduces time-dependent evaluation error model
4. Fairness Analysis: Differentiates distinct manifestations of group and individual fairness

Algorithm Design

ExaminationSearch and CascadeSearch algorithms are ingeniously designed:

• Sequential processing ensures ISO impact
• Quota management ensures fairness constraints
• Human-like versions naturally integrate fatigue effects

Experimental Design

Simulation framework design is comprehensive:

• Multiple score distributions simulate diverse candidate quality scenarios
• ISO correlation parameters control information system quality
• Large-scale repeated experiments ensure result stability

This paper provides important theoretical foundations and practical tools for understanding and improving candidate screening systems, with significant implications for constructing fairer and more effective AI-assisted decision-making systems.