BUSTED at AraGenEval Shared Task: A Comparative Study of Transformer-Based Models for Arabic AI-Generated Text Detection

Basic Information

• Paper ID: 2510.20610
• Title: BUSTED at AraGenEval Shared Task: A Comparative Study of Transformer-Based Models for Arabic AI-Generated Text Detection
• Authors: Ali Zain, Sareem Farooqui, Muhammad Rafi (National University of Computer and Emerging Sciences, FAST, Karachi, Pakistan)
• Classification: cs.CL (Computational Linguistics), cs.AI (Artificial Intelligence)
• Publication Date: October 25, 2025 (arXiv version)
• Paper Link: https://arxiv.org/abs/2510.20610v2

Abstract

This paper presents a detailed account of the BUSTED team's submission to the AraGenEval shared task on Arabic AI-generated text detection, achieving 5th place. The researchers compared the effectiveness of three pre-trained Transformer models: AraELECTRA, CAMeLBERT, and XLM-RoBERTa. The methodology involved fine-tuning each model on the provided dataset for binary classification. The study revealed a surprising finding: the multilingual XLM-RoBERTa model achieved the highest performance with an F1 score of 0.7701, surpassing specialized Arabic-language models. This work emphasizes the complexity of AI-generated text detection and highlights the strong generalization capabilities of multilingual models.

Research Background and Motivation

Problem Definition

With the increasing sophistication of Large Language Models (LLMs), the boundary between human-authored and machine-generated text has become blurred. This reality presents significant societal risks, ranging from accelerating misinformation dissemination to undermining academic integrity. Consequently, developing reliable AI-generated text detectors has become an urgent research priority.

Research Significance

1. Social Impact: Misuse of AI-generated text may lead to misinformation propagation and academic misconduct
2. Technical Challenges: Modern LLMs generate highly fluent text, limiting the effectiveness of traditional detection methods
3. Language Specificity: Arabic, as a relatively low-resource language, still lacks mature tools in the AI text detection domain

Limitations of Existing Approaches

1. Inadequacy of Traditional Methods: Early statistical stylometry-based approaches (such as n-gram frequency, readability scores, syntactic structure) perform poorly on detecting fluent text from modern LLMs
2. Scarcity of Language Resources: Arabic AI text detection tools lag behind those for other languages
3. Unclear Model Selection: Lack of systematic comparison of different Transformer architectures on Arabic AI text detection tasks

Core Contributions

1. Comparative Model Study: Provides direct comparison between monolingual and multilingual models on Arabic text detection tasks
2. Counterintuitive Findings: Demonstrates that multilingual models can achieve superior performance compared to specialized language models
3. Preprocessing Impact Analysis: Analyzes how preprocessing choices such as text normalization can unexpectedly harm model performance
4. Practical Validation: Achieves 5th place in the AraGenEval shared task, validating the effectiveness of the approach

Methodology Details

Task Definition

• Input: An Arabic text string
• Output: Binary label ('human' or 'machine')
• Task Type: Binary text classification problem

Model Architecture

The researchers implemented systems based on three different pre-trained models:

System 1: AraELECTRA

• Model: aubmindlab/araelectra-base-discriminator
• Characteristics: Specialized Arabic ELECTRA model
• Preprocessing: Applies aggressive Arabic text normalization
  • Normalizes various Arabic characters (e.g., alef variants to standard alef)
  • Converts ta marbuta to ha
  • Removes all Arabic diacritics and non-alphanumeric characters

System 2: CAMeLBERT

• Model: CAMeL-Lab/bert-base-arabic-camelbert-mix
• Characteristics: Widely-used Arabic BERT model
• Preprocessing: No specific text normalization applied; relies entirely on the model's pre-trained tokenizer

System 3: XLM-RoBERTa

• Model: xlm-roberta-base
• Characteristics: Large-scale multilingual model
• Preprocessing: Similar to CAMeLBERT setup; no language-specific normalization performed

Technical Innovations

1. Systematic Comparison: First systematic comparison of monolingual vs. multilingual models on Arabic AI text detection tasks
2. Differentiated Preprocessing Strategies: Explores the impact of different preprocessing strategies on model performance
3. Data-Driven Analysis: Guides model selection and optimization based on dataset characteristics

Experimental Setup

Dataset

• Dataset: AraGenEval dataset
• Scale: 4,734 training samples after cleaning
• Class Distribution: Nearly balanced
  • Machine-generated: 2,399 samples (50.68%)
  • Human-authored: 2,335 samples (49.32%)

Data Characteristics Analysis

1. Significant Text Length Differences:
   • Average length of human-authored text: 4,059.13 characters
   • Average length of machine-generated text: 1,934.53 characters
2. Vocabulary and N-gram Differences:
   • Human text: Frequently contains current-event-related vocabulary such as "Gaza," "the war," "Israel"
   • Machine text: Uses more generic formal vocabulary, such as "can be," "in a way"

Data Splitting Strategy

• AraELECTRA & CAMeLBERT: Use all 4,734 training samples for training and development phase evaluation
• XLM-RoBERTa: Split training data in 80/20 ratio
  • Training set: 3,787 samples
  • Validation set: 947 samples
  • Employs stratified sampling to maintain label distribution

Evaluation Metrics

• Primary Metric: Macro-averaged F1 score
• Auxiliary Metrics: Accuracy, Precision, Recall, Specificity, Balanced Accuracy

Implementation Details

Experimental Results

Main Results

Key Findings

1. Multilingual Model Advantage: XLM-RoBERTa achieves the best performance across all metrics, significantly outperforming specialized Arabic models
2. Preprocessing Strategy Impact: AraELECTRA's aggressive text normalization strategy may be counterproductive
3. Performance Ranking: XLM-RoBERTa > CAMeLBERT > AraELECTRA

Results Analysis

Reasons for XLM-RoBERTa's Success

• Diverse Pre-training Corpus: Extensive pre-training on 100 languages may provide stronger feature extraction capabilities for generalization
• Style Sensitivity: Better captures stylistic differences between human text (news-focused) and machine text (formal and analytical)

Reasons for AraELECTRA's Poor Performance

• Over-normalization: Aggressive text normalization and diacritic removal may eliminate critical fine-grained signals
• Information Loss: Removes important distinguishing features such as lexical style choices and specific named entities

Error Pattern Analysis

• Precision vs. Recall: All models show lower precision than recall, indicating a tendency to misclassify human text as machine-generated
• Possible Causes: Domain mismatch or formulaic human-authored text may resemble AI generation patterns

Related Work

Historical Development

1. Early Methods: Statistical stylometry-based authorship attribution and machine text detection
   • Features: n-gram frequency, readability scores, syntactic structure
   • Limitations: Limited effectiveness on modern LLMs
2. Neural Network Methods: Current mainstream research
   • Fine-tuning pre-trained Transformers (such as BERT)
   • Detecting statistical artifacts in LLM generation processes
   • Embedding "watermarks" in text generation processes

Paper Positioning

• Follows the fine-tuning paradigm
• Inspired by comprehensive comparative studies (such as Al-Shboul et al., 2024)
• Focuses on AI text detection in resource-scarce Arabic language domain

Conclusions and Discussion

Main Conclusions

1. Unexpected Advantage of Multilingual Models: XLM-RoBERTa surpasses specialized Arabic models on Arabic AI text detection tasks
2. Double-Edged Effect of Preprocessing: Excessive text normalization may harm model performance
3. Importance of Data Characteristics: Text length and vocabulary choice are key features for distinguishing human from machine text

Limitations

1. Poor AraELECTRA Performance: Primarily due to inappropriate preprocessing strategy choices
2. Insufficient Error Analysis: Lacks detailed qualitative error analysis
3. Limited Single-Dataset Validation: Validation only on the AraGenEval dataset

Future Directions

1. Preprocessing Optimization: Explore less aggressive text normalization methods
2. Model Ensemble: Experiment with model ensemble techniques
3. In-Depth Error Analysis: Better understand failure patterns in the task
4. Cross-Domain Generalization: Validate the approach on multiple Arabic datasets

In-Depth Evaluation

Strengths

1. Systematic Comparison: Provides comprehensive comparison of different types of Transformer models
2. Counterintuitive Findings: The finding that multilingual models outperform specialized language models is significant
3. Practical Value: Achieves good performance in actual competition, validating method effectiveness
4. Sufficient Data Analysis: Provides in-depth analysis of dataset characteristics, informing model selection
5. Reasonable Experimental Design: Appropriate hyperparameter settings and evaluation metric selection

Weaknesses

1. Inconsistent Preprocessing Strategies: Three models use different preprocessing strategies, affecting comparison fairness
2. Inconsistent Data Splitting: Different models use different data splitting strategies
3. Missing Error Analysis: Lacks in-depth analysis of model failure cases
4. Insufficient Ablation Studies: Does not fully verify component contributions
5. Limited Generalization Validation: Validation only on a single dataset

Impact

1. Academic Contribution: Provides important benchmarks for Arabic AI text detection
2. Practical Guidance: Offers reference for model selection in similar tasks
3. Methodological Value: Systematic comparison methodology applicable to other languages and tasks
4. Reproducibility: Provides detailed experimental settings for easy reproduction

Applicable Scenarios

1. Arabic Content Moderation: AI text detection for social media and news platforms
2. Academic Integrity Checking: Assignment and paper originality verification in educational institutions
3. Multilingual Environments: Scenarios requiring AI text detection across multiple languages
4. Resource-Constrained Environments: Provides methodological reference for AI text detection in other low-resource languages

References

This paper cites multiple important related works, including:

• Transformer architecture foundational paper (Vaswani et al., 2017)
• BERT model (Devlin et al., 2019)
• ELECTRA model (Clark et al., 2020)
• XLM-RoBERTa model (Conneau et al., 2020)
• Specialized Arabic models: AraELECTRA (Antoun et al., 2021) and CAMeLBERT (Inoue et al., 2021)
• Arabic text classification survey (Al-Shboul et al., 2024)

Overall Assessment: This is a solid empirical research paper that reveals through systematic comparison the unexpected advantage of multilingual models on Arabic AI text detection tasks. Despite some methodological limitations, its findings hold important value for the field and provide beneficial direction for future research.