Modelling the Spread of New Information on Social Networks

Basic Information

• Paper ID: 2505.15370
• Title: Modelling the Spread of New Information on Social Networks
• Authors: Ziming Xu, Shi Zhou, Vasileios Lampos, Ingemar J. Cox
• Classification: cs.SI (Social and Information Networks)
• Publication Date: October 14, 2025 (arXiv v3)
• Paper Link: https://arxiv.org/abs/2505.15370v3

Abstract

This paper investigates the problem of predicting information spread on social networks, specifically predicting whether users will retweet information about previously unseen topics. Existing research typically assigns information and users randomly to training and test sets, ensuring both sets come from the same distribution. However, the new information spread problem is fundamentally an out-of-distribution generalization classification task. Experimental results demonstrate that existing algorithms, which primarily rely on message content features, perform well when training and test distributions are identical, but show significant performance degradation when the test set is out-of-distribution (i.e., test topics do not exist in training data). The study finds that supplementing or replacing message features with user profile and historical behavior features substantially improves out-of-distribution prediction performance, with F1 scores improving from 0.117 to 0.705. Results indicate that retweet behavior for unseen topics can be largely predicted through user profiles and historical behavior, and is essentially content-independent.

Research Background and Motivation

1. Core Problem

The core problem addressed in this paper is new information spread prediction, namely predicting whether users will retweet information about previously unseen topics. This is a typical out-of-distribution generalization problem, as test topics are completely absent from training data.

2. Problem Significance

• Interdisciplinary Importance: Information spread prediction is significant across multiple disciplines including computer science, social sciences, political science, and marketing
• Practical Application Value: Possesses important applications in marketing campaigns, political propaganda, misinformation, and rumor propagation scenarios
• Theoretical Significance: Contributes to understanding the underlying mechanisms of information diffusion on social media

3. Limitations of Existing Methods

• Over-reliance on Message Content: Existing algorithms primarily use features extracted from message text content
• Lack of Out-of-Distribution Evaluation: Existing research typically employs random dataset partitioning, ensuring training and test data come from identical distributions
• Underestimation of User-Related Data: Important information such as user profiles, follow lists, and historical behavior are underutilized

4. Research Motivation

New topics frequently emerge on social media platforms (e.g., breaking news), necessitating out-of-distribution prediction capabilities beyond traditional same-distribution classification, which is more challenging and valuable in practical applications.

Core Contributions

1. Proposed a New Evaluation Paradigm: First explicitly distinguishes between same-distribution and out-of-distribution prediction, providing a more comprehensive evaluation framework for retweet prediction research
2. Constructed a Comprehensive Feature System: Identified and constructed 303 features, including 78 message-related features and 225 user-related features
3. Revealed the Importance of User Features: Experiments demonstrate that user-related features are crucial for out-of-distribution prediction, with F1 scores improving from 0.117 to 0.705
4. Provided Important Theoretical Insights: Discovered that retweet behavior is largely content-independent and primarily determined by user characteristics ("It is who we are, not what we see")

Methodology Details

Task Definition

Retweet prediction is defined as predicting whether a recipient will retweet a message received from a sender:

$f : \{M, U_S, U_R\} \rightarrow y \in \{0, 1\}$

Where:

• $M$: Message
• $U_S$: Sender
• $U_R$: Recipient
• $y=1$: Recipient will retweet the message; $y=0$: Will not retweet

Data Architecture

1. Message Data Data(M)

Contains text content from 111,401 X (Twitter) messages, extracting 78 message-related features:

• Topic Features (39): Topic identification using Twitter-roBERTa and LDA models
• Linguistic Features (10): Grammatical correctness, polarity, subjectivity, etc.
• Readability Features (11): Flesch reading difficulty, SMOG index, etc.
• Sentiment Features (5): Positive, negative, and neutral sentiment scores
• Emotion Features (8): Probabilities of anger, joy, fear, and other emotions
• Hate Speech Features (4): Aggressiveness and hate measures
• Tag Features (1): Presence of specific hashtags

2. User Data Data(U)

Contains three categories of user-related data:

User Profile Data Data(U-P):

• User profiles and follow lists
• Extract 30 features: follower count, influence measures, network relationships, etc.

User Historical Behavior Data Data(U-HA):

• Metadata from the most recent 50 historical messages
• Extract 38 features: retweet rate, interaction patterns, inter-user interactions, etc.

User Historical Message Data Data(U-HM):

• Text content from the most recent 50 historical messages
• Extract 157 features: aggregated features of historical messages, topic similarity, etc.

Model Architecture

1. Decision Tree (DT) Model

Uses XGBoost decision trees, discovering the critical role of user features through feature importance analysis. Main hyperparameters:

• Maximum depth: 8
• Learning rate: 0.3
• Number of estimators: 100

2. Neural Network (NN) Model

Extended from the SUA-ACNN model with added MLP components for processing user data:

• NN-M: Uses only message data
• NN-U: Uses only user data
• NN-ALL: Uses all data types

3. BERT Model

Uses BERT-base to process message text, generating semantic embeddings for prediction.

Technical Innovations

1. Out-of-Distribution Evaluation Design: For each hashtag, trains on data from 13 other hashtags and tests on that specific hashtag
2. Negative Sample Generation Strategy: Selects the most similar negative sample for each positive sample, ensuring evaluation relevance
3. Multi-level Feature System: Systematically extracts features from multiple dimensions including messages, user profiles, and historical behavior

Experimental Setup

Dataset

• Data Source: X platform (formerly Twitter) Academic API
• Time Range: July 27 to August 14, 2022
• Data Scale:
  • 111,401 messages
  • 44,014 retweet events (positive samples)
  • 79,707 unique users
  • 3.8 million historical messages
• Topic Coverage: 14 popular hashtags

Dataset Construction

Creates three datasets with different positive-negative sample ratios:

• 1:1 Dataset: Each positive sample paired with one most similar negative sample
• 1:5 Dataset: Each positive sample paired with 5 most similar negative samples
• 1:10 Dataset: Each positive sample paired with 5 similar + 5 random negative samples

Evaluation Metrics

Primarily uses F1 score: $F1 = \frac{TP}{TP + \frac{1}{2}(FP + FN)}$

For results across multiple hashtags, calculates overall mean and standard deviation.

Experimental Design

Conducts three types of experiments:

1. Experiment I: Same-distribution prediction with mixed hashtags
2. Experiment II: Same-distribution prediction with single hashtag
3. Experiment III: Out-of-distribution prediction

Experimental Results

Main Results

Same-Distribution Prediction (Experiment I)

F1 scores on 1:5 dataset:

Out-of-Distribution Prediction (Experiment III)

Overall F1 scores (μ̄±σ̄):

Key Findings

1. Critical Role of User Features:
   • Models using only message features show dramatic performance degradation in out-of-distribution prediction
   • Models using only user features perform comparably to models using all features in out-of-distribution prediction
2. Feature Importance Analysis:
   • Among the top 20 most important features, 17 are user-related
   • The most important feature is "whether the recipient follows the sender" (U-P_R_FollowS)
3. Significant Performance Improvement:
   • Out-of-distribution prediction F1 score improves from 0.117 to 0.705 (502% improvement)
   • Demonstrates the importance of user features for new topic prediction

Ablation Studies

Through comparative experiments with different feature combinations, findings include:

• U-P and U-HA Features: Contribute most to out-of-distribution prediction
• U-HM Features: Show similar performance to message features with limited out-of-distribution capability
• Message Features: Nearly ineffective in out-of-distribution settings

Related Work

Information Diffusion Research

Existing research falls into several categories:

1. Message Popularity Prediction: Predicting message propagation scale
2. Diffusion Tree Prediction: Predicting propagation paths and timing
3. Retweet Prediction: Predicting specific user retweet behavior

Limitations of Existing Methods

1. Feature Dependency: Over-reliance on message text features
2. Evaluation Limitations: Lack of out-of-distribution evaluation
3. Insufficient Data Utilization: Underutilization of user profile and behavioral data

Advantages of This Work

1. First systematic out-of-distribution evaluation
2. Comprehensive consideration of user-related features
3. Provides more realistic evaluation scenarios

Conclusions and Discussion

Main Conclusions

1. Content Independence: Retweet behavior is largely content-independent and primarily determined by user characteristics
2. Generalization Capability of User Features: User profiles and historical behavior possess cross-topic generalization capability
3. Importance of Evaluation Paradigm: Out-of-distribution evaluation is more meaningful for practical applications

Limitations

1. Platform Limitations: Research based solely on X platform data
2. Time Window: Only considers retweet behavior within 24 hours
3. Feature Engineering: Some feature extraction depends on specific tools and models
4. Cultural Context: Does not consider behavioral differences across cultural backgrounds

Future Directions

1. Cross-Platform Research: Extend to other social media platforms
2. Dynamic Modeling: Consider temporal evolution of user behavior
3. Causal Inference: Deepen understanding of causal relationships between user features and retweet behavior
4. Real-Time Applications: Develop real-time prediction systems

In-Depth Evaluation

Strengths

1. Innovative Problem Formulation:
   • First explicitly proposes out-of-distribution retweet prediction
   • More aligned with practical application scenarios
2. Rigorous Experimental Design:
   • Multiple model comparisons for validation
   • Detailed ablation studies
   • Statistical significance analysis
3. Comprehensive Feature Engineering:
   • Systematic construction of 303 features
   • Multi-dimensional feature importance analysis
4. Profound Theoretical Contributions:
   • Important insight of "It is who we are, not what we see"
   • Provides new perspective for understanding social media behavior

Weaknesses

1. Data Representativeness:
   • Uses only 14 hashtags, potentially insufficient
   • Short time span, lacking long-term observation
2. Feature Interpretability:
   • Psychological mechanisms of some user features unclear
   • Lacks in-depth analysis of feature interactions
3. Practical Considerations:
   • Obtaining complete user history data may be difficult in practical applications
   • Insufficient consideration of privacy protection
4. Model Complexity:
   • 303 features may contain redundancy
   • Lacks feature selection and dimensionality reduction analysis

Impact

1. Academic Contribution:
   • Provides new evaluation paradigm for information spread research
   • Challenges assumptions of existing methods
2. Practical Value:
   • Provides guidance for social media platform recommendation algorithms
   • Offers new insights for marketing and public opinion monitoring
3. Reproducibility:
   • Detailed experimental setup and parameter descriptions
   • Open feature engineering methodology

Applicable Scenarios

1. Social Media Platforms: Content recommendation and user behavior prediction
2. Digital Marketing: Target user identification and content strategy
3. Public Opinion Monitoring: Trending topic propagation prediction
4. Academic Research: Social network analysis and behavior modeling

References

The paper cites 48 related references, covering:

• Information diffusion theory research
• Machine learning method applications
• Social media behavior analysis
• Natural language processing techniques

Key references include classical retweet prediction work, neural network models (such as BERT and SUA-ACNN), and foundational research in social network analysis.

Overall Assessment: This is a high-quality research paper with significant contributions in problem formulation, methodological innovation, and experimental validation. Particularly, the proposal of out-of-distribution prediction and the discovery of user feature importance open new directions for social media information spread research. Despite some limitations, its theoretical value and practical significance are prominent, and it is expected to have important impact on related fields.