A Graph Laplacian Eigenvector-based Pre-training Method for Graph Neural Networks

Basic Information

• Paper ID: 2509.02803
• Title: A Graph Laplacian Eigenvector-based Pre-training Method for Graph Neural Networks
• Authors: Howard Dai, Nyambura Njenga, Hiren Madhu, Siddharth Viswanath, Ryan Pellico, Ian Adelstein, Smita Krishnaswamy
• Classification: cs.LG (Machine Learning)
• Publication Date: October 11, 2025 (arXiv preprint)
• Paper Link: https://arxiv.org/abs/2509.02803v2

Abstract

This paper proposes a pre-training method for Graph Neural Networks (GNNs) based on graph Laplacian eigenvectors. Addressing the insufficiency of structured pre-training methods in graph foundation models (GFMs), the authors develop the Laplacian Eigenvector Learning Module (LELM), which performs pre-training by predicting the low-frequency eigenvectors of the graph Laplacian. The method introduces novel architectural designs that overcome the over-smoothing problem, enabling GNN models to learn long-range dependencies. Experiments demonstrate that models pre-trained using this framework outperform baseline models on molecular property prediction tasks.

Research Background and Motivation

Problem Definition

1. Insufficient Structured Pre-training Methods: Existing GNN pre-training approaches primarily rely on feature reconstruction and contrastive learning, while pre-training methods based on graph structural properties remain relatively unexplored.
2. Over-smoothing Problem: Traditional message-passing GNNs face challenges in capturing global and regional structures, and tend to exhibit over-smoothing phenomena as network depth increases.
3. Difficulty in Learning Long-range Dependencies: Existing GNN architectures have limited expressiveness in learning long-range interdependencies within graphs.

Research Significance

• Development of graph foundation models requires effective self-supervised pre-training tasks
• Structure-aware downstream applications require pre-training methods capable of capturing underlying graph structures
• Applications such as molecular property prediction depend on understanding global graph structures

Limitations of Existing Methods

• Contrastive Methods: Primarily use Jensen-Shannon estimators or InfoNCE objectives, lacking direct modeling of structural information
• Prediction Methods: Mostly focus on graph reconstruction tasks, with few methods based on graph property prediction
• Structural Representation Capacity: Existing methods struggle to effectively capture global graph structure information

Core Contributions

1. Proposes LELM Framework: The first method using graph Laplacian eigenvectors as pre-training targets
2. Innovative Architectural Design: Introduces graph-level MLP heads enabling GNNs to capture large-scale structures without requiring excessively deep networks
3. Node Feature Enhancement: Proposes enhanced node features based on graph diffusion operators, overcoming GNN expressiveness limitations
4. Experimental Validation: Demonstrates method effectiveness on molecular datasets, serving as both a standalone pre-training method and a plug-in for existing pipelines

Methodology Details

Task Definition

Given a graph $G = (V,E)$, the objective is to pre-train a GNN model to predict the $k$ lowest-frequency eigenvectors $\psi_1, \psi_2, \ldots, \psi_k$ of the graph Laplacian matrix $L = D - A$, where $L\psi_i = \lambda_i\psi_i$.

Model Architecture

The LELM framework comprises three core components:

1. Node Feature Enhancement

Wavelet Positional Encoding: Encodes relative position information between nodes

• Randomly select two nodes $i, j$ and construct Dirac signals $\delta_i, \delta_j$
• Apply wavelet operator $\Psi_k = P^{2^{j-1}} - P^{2^j}$, where $P = D^{-1}A$ is the diffusion operator
• Wavelet positional encoding for node $m$: $w_m = [w_{m,1} \ldots w_{m,J}]$

Diffusion Dirac Encoding: Encodes local connectivity structure

• For each node $m$, compute $d_{m,k} = \Psi_k(m, \cdot) P(m, \cdot)^T$
• Diffusion Dirac encoding: $d_m = [d_{m,1} \ldots d_{m,J}]$

2. Graph-level MLP

• Base GNN: Processes the enhanced feature graph to generate node representations
• Graph-level Aggregation: Concatenates all node representations into a graph-level vector $Z = [z_1, \ldots, z_n] \in \mathbb{R}^{nd}$
• MLP Prediction Head: $\tilde{U} = \text{MLP}(Z)$ outputs predicted eigenvectors

3. Eigenvector Prediction

Orthogonality constraints are imposed via QR decomposition: $\hat{U} = \text{QR}(\tilde{U})$

Loss Function:

• Energy Loss: $L_{\text{energy}} = \frac{1}{k}\sum_{i=1}^k \hat{u}_i^T L \hat{u}_i$
• Eigenvector Loss: $L_{\text{eigvec}} = \frac{1}{k}\sum_{i=1}^k \|L\hat{u}_i - \lambda_i\hat{u}_i\|$
• Total Loss: $L = \alpha \cdot L_{\text{energy}} + \beta \cdot L_{\text{eigvec}}$

Technical Innovations

1. Graph-level MLP Design: Avoids the problem of node-level MLPs failing to learn long-range interactions
2. Eigenvector Objectives: Low-frequency Laplacian eigenvectors naturally encode global, regional, and local graph structures
3. Diffusion Operator Enhancement: Provides structural context information, enhancing GNN expressiveness
4. Dual Loss Mechanism: Energy loss ensures subspace correctness, eigenvector loss ensures proper ordering

Experimental Setup

Datasets

• ZINC-12k: 12,000 molecular graphs
• ZINC-250k: 250,000 molecular graphs
• QM9: 134,000 molecular graphs with multiple quantum chemical properties

Evaluation Metrics

• MAE (Mean Absolute Error): Primary evaluation metric
• ROC-AUC: Used for binary classification tasks

Comparison Methods

• Baseline Models: Untrained GIN and GPS models
• Alternative Pre-training Targets: Node degree, local clustering coefficient, cycle counting, Laplacian eigenvalues
• Existing Pre-training Methods: ContextPred, Masking, etc.

Implementation Details

• Pre-training Epochs: 100-200 rounds
• Fine-tuning Epochs: 150-500 rounds
• Number of Eigenvectors: $k = 6$
• Loss Weights: $\alpha = 2, \beta = 1$ (main experiments)
• Optimizer: Adam
• Learning Rate: 0.001

Experimental Results

Main Results

Performance Comparison on ZINC and QM9 Datasets:

LELM significantly improves performance on most tasks, with particularly notable improvements on the ZINC dataset.

Ablation Studies

Graph-level MLP vs Node-level MLP:

Graph-level MLP significantly outperforms node-level MLP across both architectures.

Comparison of Alternative Structural Pre-training Targets:

LELM clearly outperforms other structured pre-training targets.

Enhancement of Existing Pre-training Methods

Adding LELM as a plug-in to existing pre-training pipelines on molecular prediction tasks:

• Masking + LELM: Shows improvements across all 5 datasets
• ContextPred + LELM: Shows improvements on most tasks

Experimental Findings

1. Importance of Graph-level Architecture: Graph-level MLP effectively learns long-range dependencies
2. Superiority of Eigenvectors: Laplacian eigenvectors are more suitable for pre-training than other structural targets
3. Generalizability: LELM can be combined with existing pre-training methods
4. Scalability: The method applies to different GNN architectures (GIN, GPS)

Related Work

Classification of Graph Pre-training Methods

1. Contrastive Methods:
   • Graph-node contrast (Deep Graph Infomax, etc.)
   • Subgraph-node contrast (InfoGraph, etc.)
   • Subgraph-subgraph contrast (GraphCL, etc.)
2. Prediction Methods:
   • Graph reconstruction (node/edge masking, autoencoders)
   • Property prediction (k-hop connectivity, meta-paths)

Applications of Laplacian Eigenvectors

• Positional Encoding: Standard positional encoding in graph Transformers
• Spectral Graph Neural Networks: Learning filters in the signal domain
• Spectral Clustering: Generating low-dimensional embeddings for clustering
• Graph Partitioning: Fiedler vector generating optimal graph cuts

Positioning of This Work

LELM is the first property prediction method using graph Laplacian eigenvectors as pre-training targets, filling a gap in structured pre-training methods.

Conclusions and Discussion

Main Conclusions

1. Effectiveness Validation: LELM significantly improves GNN performance on molecular property prediction tasks
2. Architectural Innovation: Graph-level MLP effectively addresses the over-smoothing problem
3. Universal Framework: Can serve as both a standalone method and an enhancement component for existing pipelines
4. Theoretical Guarantees: Loss function possesses necessary sign and basis invariance properties

Limitations

1. Unexplored Transfer Learning Capability: Currently validated only on same or related domain datasets
2. Computational Complexity: Requires Laplacian eigendecomposition, which may be challenging for large graphs
3. Cross-domain Generalization: Effects on synthetic or cross-domain datasets remain unknown
4. Statistical Significance: Error bars not reported due to computational cost constraints

Future Directions

1. Cross-domain Pre-training: Explore pre-training effects on synthetic or cross-domain datasets
2. Large-scale Applications: Investigate scalability on larger-scale graphs
3. Theoretical Analysis: Provide deeper analysis of why Laplacian eigenvectors are good pre-training targets
4. Architecture Optimization: Further optimize graph-level MLP design

In-depth Evaluation

Strengths

1. Strong Novelty: First application of Laplacian eigenvectors to GNN pre-training with innovative approach
2. Solid Theoretical Foundation: Laplacian eigenvectors have deep theoretical foundations in graph theory
3. Clever Architectural Design: Graph-level MLP effectively addresses long-range dependency learning
4. Comprehensive Experiments: Includes multiple comparative, ablation, and enhancement experiments
5. Good Generalizability: Compatible with different GNN architectures and existing pre-training methods

Weaknesses

1. Limited Application Domains: Primarily validated on molecular data; effects on other graph types unknown
2. Computational Overhead: Eigendecomposition computational cost may limit large-scale applications
3. Hyperparameter Sensitivity: Lack of systematic analysis for hyperparameter selection (e.g., loss weights)
4. Insufficient Theoretical Explanation: Lacks deep theoretical analysis of why the method is effective

Impact

1. Academic Value: Provides new research directions for graph pre-training
2. Practical Value: Potential value in practical applications such as molecular property prediction
3. Reproducibility: Complete code and experimental settings provided
4. Inspirational Value: May inspire more pre-training methods based on graph spectral properties

Applicable Scenarios

1. Molecular Property Prediction: Validated effective application scenario
2. Social Network Analysis: Tasks requiring understanding of global structure
3. Knowledge Graphs: Graph reasoning tasks where structural information is important
4. Biological Networks: Biological applications such as protein interaction networks

References

The paper cites multiple important related works, including:

• Hu et al. (2019): "Strategies for pre-training graph neural networks" - Classical work on graph pre-training
• Shaham et al. (2018): "SpectralNet" - Neural network approach to spectral clustering
• Dwivedi et al. (2021): "Graph neural networks with learnable structural and positional representations" - Structural positional representation learning
• Rampášek et al. (2022): "Recipe for a general, powerful, scalable graph transformer" - GPS architecture

Overall Assessment: This is a high-quality research paper proposing an innovative graph neural network pre-training method. While there is room for improvement in certain aspects, its core ideas are novel, experimental validation is comprehensive, and it makes important contributions to the graph pre-training field. The method's generalizability and extensibility demonstrate good application prospects.