Toward a Safer Web: Multilingual Multi-Agent LLMs for Mitigating Adversarial Misinformation Attacks

Basic Information

• Paper ID: 2510.08605
• Title: Toward a Safer Web: Multilingual Multi-Agent LLMs for Mitigating Adversarial Misinformation Attacks
• Authors: Nouar Aldahoul, Yasir Zaki (New York University Abu Dhabi)
• Categories: cs.CL (Computational Linguistics), cs.AI, cs.CR, cs.LG
• Publication Date: October 7, 2025 (arXiv preprint)
• Paper Link: https://arxiv.org/abs/2510.08605

Abstract

The rapid dissemination of misinformation on digital platforms threatens public discourse, emotional stability, and decision-making. While prior work has explored various adversarial attacks in misinformation detection, the specific transformations investigated in this paper have not been systematically studied. Specifically, this paper investigates language switching across English, French, Spanish, Arabic, Hindi, and Chinese, along with subsequent translation. It also examines query length expansion prior to summarization and structural reformatting into multiple-choice questions. The paper proposes a multilingual, multi-agent large language model framework incorporating retrieval-augmented generation techniques, deployable as a web plugin on online platforms. This work emphasizes the importance of AI-driven misinformation detection in protecting online factual integrity while demonstrating the feasibility of plugin-based deployment in real-world web applications.

Research Background and Motivation

Problem Definition

The core problem addressed in this research is the lack of effective misinformation detection capabilities in large language models (LLMs) when facing adversarial attacks, which can inadvertently amplify the spread of false information.

Problem Significance

1. Social Impact: Rapid dissemination of misinformation severely threatens public discourse, emotional stability, and decision-making
2. Technical Challenges: Existing LLMs perform near random guessing in misinformation detection
3. Security Requirements: Need for robust detection systems against diverse attacks

Limitations of Existing Methods

1. Embedded Knowledge Constraints: LLMs rely solely on knowledge embedded during training, lacking real-time fact-checking capabilities
2. Language Bias: Significantly degraded performance on non-English languages
3. Adversarial Attack Vulnerability: Lack of resistance to format conversion, translation, and summarization attacks
4. Lack of Systematic Study: Existing work has not systematically evaluated multilingual, multi-structural adversarial attacks

Research Motivation

The authors propose the need to develop a multilingual misinformation detection system capable of resisting multiple adversarial attacks and deploy it as a practical web plugin.

Core Contributions

1. Proposed Multi-Agent RAG Framework: Multi-agent architecture combining Llama 3.1-8B and retrieval-augmented generation techniques
2. Constructed Novel Adversarial Attack Dataset: Dataset containing three attack forms: multiple-choice questions (MCQ), translation, and summarization
3. Implemented Multilingual Detection Capability: Support for six languages: English, French, Spanish, Arabic, Hindi, and Chinese
4. Verified Practical Deployment Feasibility: Designed as a deployable web plugin
5. Provided Comprehensive Experimental Evaluation: Achieved over 95% accuracy in misinformation detection

Methodology Details

Task Definition

Input: Text content from the web (news articles, user comments, social media posts, etc.), potentially containing adversarial transformations Output: Binary classification result (True/False) determining whether the input text contains misinformation Constraints: System must operate in a black-box setting, making judgments based solely on binary feedback

Model Architecture

RAG-Llama Core Components

1. Embedding Models: Comparison of three multilingual embedding models
   • OpenAI's text-embedding-3-large (proprietary)
   • jina-embeddings-v3 (proprietary)
   • multilingual-e5-large (open-source)
2. Retrieval Mechanism: Cosine similarity-based retrieval system
   • Store false headline embeddings in CSV files
   • Retrieve false headlines most relevant to the query
   • Use Llama for contextual analysis to make final judgment

Multi-Agent Architecture

The system comprises four collaborative agents:

1. Web Crawler Agent
   • Extract structured content from dynamic websites
   • Segment text into manageable chunks
   • Pass to manager agent for processing
2. Manager Agent
   • Interact with web crawler to receive text
   • Route to topic and misinformation detection agents
   • Send notifications to users
3. Misinformation Detection Agent
   • Leverage RAG-Llama for detection
   • Retrieve from database containing 5,000 verified false headlines
   • Use open-source Llama model for final judgment
4. Topic Agent (Optional)
   • Classify queries into 10 predefined categories
   • Accelerate RAG search process
   • Use GPT-4o-mini for topic classification
5. Adjudicator Agent
   • Ensure all text chunks are processed
   • Verify consistency across system components
   • Serve as additional validation layer to enhance robustness

Technical Innovations

1. Multi-Modal Adversarial Attack Handling: First systematic treatment of MCQ, translation, and summarization attacks
2. Multilingual Retrieval Capability: Leverage multilingual embedding models for cross-lingual detection
3. Negative Sample Matching Strategy: Use only false information database for negative matching detection
4. Modular Plugin Design: Directly deployable as web browser plugin

Experimental Setup

Dataset

Data Sources

• False Headlines: 20,950 false headlines collected from Snopes and Politifact
• Factual Headlines: 4,000 true headlines collected
• Experimental Data: Selected 5,000 false headlines and 2,000 factual headlines

Three Attack Datasets

1. MCQ Dataset: Convert headlines to "why"-prefixed multiple-choice questions
2. Translation Dataset: Translate expanded text into six languages
3. Summarization Dataset: Generate 500-word long texts for summarization tasks

Evaluation Metrics

• Factual Accuracy: Percentage of correctly classified factual information
• False Accuracy: Percentage of correctly classified false information
• Attack Success Rate (ASR): Ratio of adversarial inputs causing system failure (lower is better)

Comparison Methods

• Baseline Model: Original Llama 3.1-8B-Instruct
• RAG-Llama Variants with Different Embedding Models
• System Variants With/Without Topic Classification

Implementation Details

• Model: Llama 3.1-8B-Instruct
• Hardware: GPU A100 80GB
• Hyperparameters: temperature=0.1, top-p=1
• Embedding Storage: CSV file format

Experimental Results

Main Results

Baseline Model Vulnerability

• Direct Question ASR: 46.74%
• MCQ Attack ASR: 97.72%
• Translation Attack ASR: 100%
• Summarization Attack ASR: 100%

RAG-Llama Performance

Embedding Model Comparison

Topic Classification Performance

• Speed Improvement: Median 2x or more, average 3x or more
• Accuracy: Ranges from 78.27%-91.18%
• Lower MCQ Task Accuracy: Due to classification difficulty caused by multiple-topic answers in multiple-choice questions

Experimental Findings

1. RAG Significantly Outperforms Baseline: Substantial improvements across all attack types
2. Multilingual Capability: Maintained over 95% false detection accuracy across six languages
3. Embedding Model Impact: multilingual-e5-large performed best in balancing performance and accessibility
4. Topic Classification Acceleration: Effectively improved retrieval speed, but accuracy decreased on complex queries

Related Work

Fine-Tuning Methods

• BERT-based approaches (FakeBERT, etc.)
• T5 instruction fine-tuning
• Llama-2 PEFT/LoRA fine-tuning
• Reinforcement learning methods

RAG Methods

• Mixtral-8x7B combined with RAG
• Real-time web data integration
• Adaptive Topic RAG (AT-RAG)

Multi-Agent Systems

• LLM-Consensus for visual misinformation detection
• TruEDebate (TED) structured debate system
• Complete misinformation lifecycle processing framework

Adversarial Attacks

• Gradient-based token-level substitution
• Reinforcement learning-driven claim perturbation
• Black-box attack strategies

Conclusions and Discussion

Main Conclusions

1. Significant LLM Vulnerabilities: Vanilla LLMs are highly susceptible to spreading misinformation under adversarial attacks
2. RAG Effectively Enhances Robustness: RAG-Llama significantly outperforms baseline across various attacks
3. Multilingual Detection Feasibility: System effectively handles misinformation in six major languages
4. Practical Deployment Potential: Multi-agent architecture suitable for deployment as web plugin

Limitations

1. Topic Classification Accuracy: Topic misclassification affects retrieval precision
2. Database Dependency: System performance heavily depends on quality and completeness of misinformation database
3. Dynamic Update Requirements: Requires continuous database updates to address emerging misinformation
4. Security Vulnerabilities: RAG systems may face database poisoning and embedding attacks

Future Directions

1. Improve Topic Classification: Enhance classification accuracy for complex queries
2. Explore Other LLMs: Evaluate performance of different language models in RAG
3. Enhance Security: Develop protective mechanisms against embedding attacks and database poisoning
4. Expand Attack Types: Investigate more varieties of adversarial transformations

In-Depth Evaluation

Strengths

1. Problem Importance: Addresses critical security issues in LLM-based misinformation detection
2. Methodological Innovation: First systematic study of multilingual, multi-structural adversarial attacks
3. Experimental Comprehensiveness: Comprehensive evaluation covering six languages and three attack types
4. Practical Value: Provides deployable plugin solution
5. Technical Advancement: Incorporates latest RAG and multi-agent technologies

Weaknesses

1. Dataset Scale Limitation: Uses only 7,000 headlines, relatively small scale
2. Limited Attack Types: Considers only three specific attack forms
3. Single Evaluation Metric: Primarily focuses on accuracy, lacking efficiency and cost metrics
4. Insufficient Theoretical Analysis: Lacks theoretical explanation for method effectiveness
5. Unverified Long-Term Stability: Has not evaluated performance degradation during extended use

Impact

1. Academic Contribution: Provides new research direction for multilingual misinformation detection
2. Practical Value: Directly applicable to social media and news platforms
3. Reproducibility: Uses open-source models, facilitating reproduction and improvement
4. Industry Impact: Provides technical foundation for content moderation and fact-checking

Applicable Scenarios

1. Social Media Platforms: Real-time detection of false information posted by users
2. News Aggregation Websites: Verify authenticity of news articles
3. Educational Platforms: Help users identify misinformation
4. Enterprise Content Moderation: Automated review of large-scale content
5. Government Regulation: Assist relevant departments in monitoring online misinformation

References

This paper cites 50 relevant references covering important works in multiple domains including LLMs, RAG, multi-agent systems, and adversarial attacks, providing a solid theoretical foundation for the research.

Overall Assessment: This is an important contribution to the misinformation detection field, proposing an innovative multi-agent RAG framework and achieving excellent experimental results under multilingual, multi-attack-type settings. Despite some limitations, its practical value and technical innovation make it a significant advance in the field.