Local Timescale Gates for Timescale-Robust Continual Spiking Neural Networks

Basic Information

• Paper ID: 2510.12843
• Title: Local Timescale Gates for Timescale-Robust Continual Spiking Neural Networks
• Authors: Ansh Tiwari, Ayush Chauhan
• Classification: cs.LG (Machine Learning)
• Publication Date: October 13, 2025 (Preprint)
• Paper Link: https://arxiv.org/abs/2510.12843

Abstract

Spiking Neural Networks (SNNs) offer energy efficiency advantages on neuromorphic hardware but perform poorly on tasks requiring rapid adaptation and long-term memory, particularly in continual learning scenarios. This paper proposes Local Timescale Gates (LT-Gate), a neuron model combining dual timescale constant dynamics with adaptive gating mechanisms. Each spiking neuron parallelly tracks fast and slow timescale information, with their influence locally regulated through learned gates. This design enables individual neurons to maintain slow contextual information while responding to fast signals, addressing the stability-plasticity dilemma. The research also introduces variance-tracking regularization inspired by biological homeostasis to stabilize firing activity. Experiments demonstrate that LT-Gate achieves approximately 51% final accuracy on temporal classification benchmarks, a significant improvement over the recent Hebbian continual learning baseline (HLOP) at 46%.

Research Background and Motivation

Core Problems

1. Catastrophic Forgetting in Continual Learning: SNNs, like conventional neural networks, forget previously learned knowledge when learning new tasks
2. Stability-Plasticity Dilemma: Requires balancing the stability of retained knowledge with the plasticity for learning new knowledge
3. Timescale Adaptability: Existing methods struggle to handle tasks requiring different timescale responses

Significance

• Neuromorphic computing has gained considerable attention due to its event-driven energy efficiency advantages
• Continual learning is a key technology for achieving lifelong learning AI systems
• Biological neural systems address this challenge through multiple adaptive processes and different intrinsic timescales

Limitations of Existing Methods

1. Weight Regularization and Experience Replay: Require additional storage and computational overhead
2. HLOP Method: Requires iterative subspace computation with high computational complexity
3. DSD-SNN: Requires complex network expansion strategies
4. Lack of Biological Plausibility: Most methods do not comply with local computation constraints of neuromorphic hardware

Core Contributions

1. Proposes LT-Gate Neuron Model: Combines dual timescale dynamics with local gating mechanisms, enabling flexible temporal credit assignment
2. Designs Variance Regularization Mechanism: Based on biological homeostasis theory, maintains neuronal activation within healthy ranges and prevents instability during task switching
3. Achieves Significant Performance Improvement: Substantially improves continual learning performance on temporally heterogeneous tasks without requiring external memory or architecture expansion
4. Neuromorphic Hardware Compatible: Efficiently maps to Intel Loihi 2 chip, supporting local filtering and gating operations for on-chip learning

Methodology Details

Task Definition

The research focuses on temporal domain incremental learning tasks: the network must perform continual learning across input domains with different timescales, such as transitioning from high-frequency (1000Hz) input domains to low-frequency (50Hz) input domains, while maintaining classification ability on previous domains.

Model Architecture

Dual Timescale Neuron Model

The LT-Gate neuron contains two parallel Leaky Integrate-and-Fire (LIF) units:

Fast Membrane Potential Dynamics:

U^f_i(t+1) = ρ_f · U^f_i(t) + I_i(t)

Slow Membrane Potential Dynamics:

U^s_i(t+1) = ρ_s · U^s_i(t) + I_i(t)

where 0 < ρ_f < ρ_s < 1 are decay factors, and ρ = e^(-dt/τ)

Local Timescale Gating

The two membrane potentials are combined through learnable gating variables γ_i ∈ 0,1:

U_i(t) = γ_i · U^s_i(t) + (1-γ_i) · U^f_i(t)

• γ_i ≈ 1: Neuron behaves as a long timescale unit, retaining past inputs
• γ_i ≈ 0: Prioritizes fast responses

Variance-Tracking Homeostatic Regularization

Introduces a regularization term encouraging each neuron to maintain target firing rate and variance:

L_var = λ_var Σ_i [(μ_i - μ*)² + (σ_i - σ*)²]

where (μ*, σ*) are target values and λ_var is the weighting coefficient.

Technical Innovations

1. Distributed Gating Mechanism: Unlike LSTM's explicit gating units, LT-Gate's gating mechanism is localized to each spiking neuron state
2. Dual Homeostatic Process: Fast dynamics respond to input changes, while slow dynamics and gating provide negative feedback to maintain overall firing rate statistics
3. Hardware-Aware Design: All computations (decay, accumulation, thresholding, gating) are local to each neuron or synapse, requiring no global coordination
4. Adaptive Timescale Constant Allocation: Through learning γ, the network can distribute an effective timescale spectrum across neurons

Experimental Setup

Datasets

• Frequency-Variant MNIST Dataset: Encodes standard MNIST images as spiking sequences at different frame rates
  • Fast domain: 1000Hz input presentation rate
  • Slow domain: 50Hz input presentation rate

Evaluation Metrics

• Final Accuracy: Classification accuracy on combined test set
• Forgetting Rate: Accuracy drop on old tasks after learning new tasks
• Convergence Speed: Training epochs required to reach 90% final accuracy

Comparison Methods

• HLOP: Hebbian Learning Orthogonal Projection method
• DSD-SNN: Dynamic Structural Development Spiking Neural Network
• Standard SNN: Single timescale LIF baseline

Implementation Details

• Network Architecture: 3-layer convolutional SNN (32→64→128 units)
• Optimizer: Adam, learning rate 0.001
• Time Constants: τ_f = 5ms, τ_s = 50ms
• Target Firing Rate: 2%, variance regularization weight λ_var = 0.01

Experimental Results

Main Results

Key Findings

1. Significant Performance Improvement: LT-Gate achieves 51.0% final accuracy, 5.8 percentage points higher than HLOP
2. Minimal Forgetting: Only 3.2% forgetting rate, retaining approximately 95% of original performance
3. Fast Convergence: Converges in 25 epochs, 40-60% faster than baseline methods

Ablation Studies

• LT-Gate without Gating: Accuracy drops to 46.5%, forgetting rate increases to 8.9%
• Without Variance Regularization: Accuracy 49.8%, occasional training instability

Cross-Timescale Generalization

After training only on 1000Hz, LT-Gate achieves 48% accuracy on untrained slow domain (10Hz), compared to 20% for standard SNN, demonstrating strong cross-domain generalization capability.

Related Work

SNN Continual Learning Methods

1. Weight Regularization: Spiking versions of Elastic Weight Consolidation
2. Experience Replay: Memory replay mechanisms protecting old knowledge
3. Brain-Inspired Methods: HLOP uses Hebbian/anti-Hebbian updates, DSD-SNN dynamically grows/prunes neurons

Multi-Timescale Neural Computation

1. LSNN Model: Heterogeneous neuron mixtures with different adaptation time constants
2. Biological Neuron Models: Threshold adaptation and slowly-changing current adaptive neurons

Neuromorphic Hardware

Intel Loihi chip supports multiple synaptic traces with different decay rates, providing natural hardware support for LT-Gate.

Conclusions and Discussion

Main Conclusions

1. Multi-timescale Processing is an Effective Solution to the Stability-Plasticity Dilemma
2. Local Gating Mechanisms Outperform Centralized Gating and Better Suit Neuromorphic Hardware
3. Bio-Inspired Homeostatic Mechanisms are Critical for Network Stability
4. Hardware-Aware Design Enables Efficient On-Chip Learning

Limitations

1. Dependence on Surrogate Gradient Backpropagation: Limits biological plausibility
2. Limited Evaluation Scope: Only tests two sequential domains; multi-task scenarios require further verification
3. Fixed Time Constants: Does not implement per-neuron adaptive timescale learning
4. Complex Task Scalability: May require lightweight replay or regularization supplements

Future Directions

1. Local Learning Rules: Develop STDP-based gating adaptation mechanisms
2. Multi-Timescale Extension: Utilize three or more timescales
3. Hardware Co-Design: Develop dedicated temporal gating circuits
4. Large-Scale Applications: Extend to deep vision and speech SNNs

In-Depth Evaluation

Strengths

1. Strong Novelty: First implementation of distributed multi-timescale gating in SNNs
2. Comprehensive Experiments: Thorough ablation studies and hardware compatibility verification
3. Convincing Results: Significant improvements over SOTA methods across multiple metrics
4. High Practical Value: Directly maps to commercial neuromorphic chips

Weaknesses

1. Insufficient Theoretical Analysis: Lacks theoretical analysis of gating learning dynamics
2. Dataset Limitations: Primarily validated on synthetic temporal data, lacking real-world benchmarks
3. Comparison Fairness: Some baseline methods may not be fully optimized

Impact

1. Domain Contribution: Provides new design paradigm for SNN continual learning
2. Hardware Advancement: Promotes multi-timescale functionality development in neuromorphic hardware
3. Reproducibility: Provides complete code and detailed implementation details

Applicable Scenarios

1. Edge AI Devices: Low-power intelligent systems requiring continuous adaptation
2. Robotics: Real-time learning and adaptation in dynamic environments
3. Sensor Networks: Signal processing under varying sampling rates
4. Neuromorphic Computing: Applications leveraging specialized hardware characteristics

References

1 Bellec et al. "Long short-term memory and learning-to-learn in networks of spiking neurons." NeurIPS 2018. 2 Cannon & Miller. "Stable control of firing rate mean and variance by dual homeostatic mechanisms." J Math Neurosci 2017. 3 Davies et al. "Loihi: A neuromorphic manycore processor with on-chip learning." IEEE Micro 2018. 7 Xiao et al. "Hebbian learning based orthogonal projection for continual learning of snns." ICLR 2024.