Large Language Models Inference Engines based on Spiking Neural Networks

Basic Information

• Paper ID: 2510.00133
• Title: Large Language Models Inference Engines based on Spiking Neural Networks
• Authors: Adarsha Balaji (Argonne National Laboratory), Sandeep Madireddy (Argonne National Laboratory), Prasanna Balaprakash (Oak Ridge National Laboratory)
• Classification: cs.LG (Machine Learning)
• Publication Date: October 14, 2025 (arXiv preprint)
• Paper Link: https://arxiv.org/abs/2510.00133v3

Abstract

Foundation models based on the Transformer architecture represent the current state-of-the-art in general language modeling and scientific domains such as materials science and climate science. However, training and deploying these models presents significant computational challenges due to their quadratic time and space complexity with respect to input sequence length. This paper explores the use of Spiking Neural Networks (SNNs) to design Transformer models. Existing surrogate learning methods for training large-scale SNNs are inefficient and time-consuming, while techniques for converting existing Transformer models to equivalent SNNs lack scalability. To address this, the authors propose the NeuTransformer method, which designs SNN-based Transformer inference engines through supervised fine-tuning combined with existing conversion methods. The approach includes: (1) replacing the self-attention mechanism with spike-based self-attention (SSA), (2) converting feedforward blocks to equivalent SNNs, and (3) fine-tuning SSA blocks using SNN surrogate learning algorithms. Experiments demonstrate that the converted GPT-2 Small model incurs a 5-12% loss in cosine similarity, achieves a 9.7% reduction in perplexity, and SSA blocks achieve 64.71%-85.28% energy consumption reduction compared to ASA blocks.

Research Background and Motivation

Core Problems

1. Computational Complexity Issue: The self-attention mechanism in Transformer models exhibits O(n²) time and space complexity, where n is the sequence length. This causes computational and memory requirements to increase dramatically when processing long sequences.
2. Energy Consumption Problem: Traditional Transformer models require expensive GPUs or custom accelerators for training and inference, resulting in massive energy consumption.
3. SNN Training Difficulties: Existing SNN training methods have two major limitations:
   • Direct training of large-scale SNNs using backpropagation learning rules is inefficient
   • ANN-SNN conversion methods require numerous spike time steps to achieve optimal performance, leading to increased inference latency

Research Motivation

The authors aim to leverage the bio-inspired characteristics and event-driven computing advantages of spiking neural networks to design Transformer models that can execute efficiently on neuromorphic hardware, thereby achieving data-efficient, low-energy, and resource-efficient large language model inference.

Core Contributions

1. Proposes NeuTransformer Method: A method for designing SNN-based Transformers from pre-trained Transformer models, combining supervised fine-tuning to enhance model performance.
2. Designs Sparse Spike-Based Self-Attention Mechanism: Replaces energy and latency-intensive matrix multiplication and softmax operations in traditional self-attention with sparse spike-based computations.
3. Implements Large-Scale SNN-based LLM: Successfully converts GPT-2 and its variants to SNN versions. To the authors' knowledge, GPT-2 Large is the largest parameter SNN-based Transformer model to date.
4. Comprehensive Performance Evaluation: Evaluates model performance across multiple dimensions including application accuracy, cosine similarity, perplexity, and bits per byte, while analyzing energy consumption and throughput performance.

Methodology Details

Task Definition

Convert pre-trained Transformer models to equivalent spiking neural network versions while maintaining acceptable performance degradation and achieving significant energy consumption reduction and hardware efficiency improvements.

Model Architecture

1. Spiking Neuron Model

Uses Integrate-and-Fire (IF) neurons with fixed thresholds and adjustable membrane potential decay characteristics:

S(t) = {
  1,   if Vmem ≥ 1
  -1,  if Vmem ≤ -1  
  0,   otherwise
}

Where Vmem is the membrane potential and S(t) is the spike activation output by the neuron.

2. Spike-Based Self-Attention (SSA) Mechanism

Traditional self-attention mechanism:

ASA(Q,K,V) = softmax(Q·K^T)V

Spike-based self-attention mechanism:

AttentionScore(AS) = LIF((Q⊗K^T)_Columnwise)
SSA(Q,K,V) = (AS ⊗ V)

Key innovations:

• Replaces N-bit matrix multiplication with AND operations and accumulators
• Replaces dot product operations with column-wise Hadamard products
• Replaces softmax function with LIF neuron activation

3. Spike-Based Feedforward Layer Conversion

Based on ANN-SNN conversion principles, converts ReLU-activated feedforward layers to IF neurons:

• ReLU function: ReLU(y) = max(0, y)
• IF neuron: τm ∂Vmem/∂t = -Vmem(t) + R*I(t)

Weight Normalization:

s^l_norm = max(a^l)
W̃^l ← W^l / s^l_norm

NeuTransformer Three-Step Conversion Pipeline

1. Replace Self-Attention Blocks: Replace ASA with SSA while retaining trained weights
2. Convert Feedforward Blocks: Convert ReLU/GeLU feedforward layers to SNN equivalent versions
3. Fine-tune SSA Blocks: Fine-tune SSA block weights using surrogate gradient learning algorithms

Fine-tuning Objective Function

Minimizes mean squared error between ASA and SSA attention scores:

Σ(i=1 to d_model) (ASA_as - SSA_as)²

Experimental Setup

Datasets

• Shakespeare Dataset: Contains 40,000 lines of Shakespeare dramatic text
• OpenWebText Dataset: Open-source reproduction of OpenAI's WebText dataset

Model Scales

• GPT-2 Small: 117M parameters
• GPT-2 Medium: 345M parameters
• GPT-2 Large: 763M parameters

Evaluation Metrics

• Character Accuracy: Character-by-character comparison of generated characters between ANN and SNN
• Cosine Similarity: Cosine of the angle between two non-zero vectors in multidimensional space
• Perplexity: Metric for measuring language model quality
• Bits per Byte (BpB): Average number of bits required to predict the next token

Hardware Platforms

• Computational Resources: Swing HPC cluster at Argonne National Laboratory's LCRC
• Configuration: 6 nodes, each with 2×AMD EPYC 7742 processors and 8×NVIDIA A100 GPUs
• Evaluation Platforms: NVIDIA A100 GPU and Graphcore IPU platforms

Experimental Results

Main Performance Results

Energy Consumption Analysis Results

Estimated energy consumption reduction of SSA blocks compared to ASA blocks:

• GPT-2 Small: 85.28%
• GPT-2 Medium: 85.22%
• GPT-2 Large: 64.71%

Throughput Evaluation

On the Graphcore platform, the SNN version demonstrates superior throughput performance compared to the baseline ANN in most configurations, with particular advantages in handling irregular and sparse data access patterns.

Key Findings

1. Scale Effect: Performance degradation of the SNN version gradually increases with model size
2. Energy Efficiency Advantage: All model scales achieve significant energy consumption reduction
3. Hardware Adaptability: SNNs demonstrate excellent performance on MIMD processing architectures, particularly suited for sparse spike workloads

Related Work

SNN-based Transformer Research

• Spikformer (Li et al., 2024): First implementation of SNN-based Transformer on vision tasks
• Spikingformer (Zhou et al., 2023): Improved version using pre-activation skip connections
• SGLFormer (Zhang et al., 2024): Combines local and global Transformer blocks

ANN-SNN Conversion Methods

• Early work primarily focused on small-scale networks for vision tasks
• Existing methods have limited applicability to large-scale language models

Advantages of This Work

Compared to existing methods, NeuTransformer avoids the computational overhead of training SNNs from scratch, achieving large-scale SNN-based LLM construction through a conversion plus fine-tuning strategy.

Conclusions and Discussion

Main Conclusions

1. Feasibility Verification: Successfully converts GPT-2 series models to SNN versions, demonstrating the feasibility of large-scale SNN-based LLMs
2. Performance Trade-off: Achieves significant energy consumption reduction with acceptable performance degradation
3. Scale Limitations: Identifies that performance degradation exceeds acceptable thresholds when model parameters exceed 300M

Limitations

1. Scale Bottleneck: Performance degradation issues in large-scale models remain unresolved
2. Conversion Accuracy: Imperfect conversion of feedforward blocks affects overall performance
3. Fine-tuning Constraints: Limitations of surrogate gradient learning on deep SNNs

Future Directions

1. Improve training and conversion methods for large-scale SNNs
2. Optimize surrogate gradient learning algorithms
3. Explore more efficient spike encoding and decoding strategies
4. Validate performance on actual neuromorphic hardware

In-Depth Evaluation

Strengths

1. Strong Innovation: First implementation of large-scale SNN-based language models with novel technical approach
2. High Practical Value: Significant energy consumption reduction has important implications for practical applications
3. Comprehensive Evaluation: Evaluates model performance across multiple dimensions with rigorous experimental design
4. Clear Presentation: Detailed technical descriptions and clear methodology exposition

Weaknesses

1. Obvious Scale Limitations: Severe performance degradation in large models, limiting method applicability
2. Insufficient Theoretical Analysis: Lacks in-depth theoretical analysis of performance degradation causes
3. Limited Hardware Verification: Primarily based on estimations, lacking verification on actual neuromorphic hardware
4. Insufficient Comparative Experiments: Limited direct comparisons with other SNN-based methods

Impact

1. Academic Contribution: Opens new directions for SNN applications in large-scale language models
2. Practical Prospects: Provides new technical pathways for low-power AI inference
3. Reproducibility: Detailed method descriptions enable good reproducibility

Applicable Scenarios

1. Edge Computing: Resource-constrained mobile devices and IoT scenarios
2. Low-Power Inference: Applications sensitive to energy consumption
3. Neuromorphic Computing: Specialized neuromorphic hardware platforms

References

The paper cites important works in the field, including:

• Original Transformer paper (Vaswani et al., 2017)
• Classical SNN conversion methods (Rueckauer et al., 2016; Diehl & Cook, 2015)
• Recent SNN-based Transformer research (Li et al., 2024; Zhou et al., 2023)
• Surrogate gradient learning methods (Eshraghian et al., 2023)

Summary: This paper proposes an innovative method for converting large-scale language models to spiking neural network versions, achieving significant energy consumption reduction while maintaining acceptable performance. Despite scale limitations, it provides valuable technical contributions to the fields of low-power AI inference and neuromorphic computing.