Lifting Manifolds to Mitigate Pseudo-Alignment in LLM4TS

Basic Information

• Paper ID: 2510.12847
• Title: Lifting Manifolds to Mitigate Pseudo-Alignment in LLM4TS
• Authors: Liangwei Nathan Zheng, Wenhao Liang, Wei Emma Zhang, Miao Xu, Olaf Maennel, Weitong Chen
• Classification: cs.LG (Machine Learning)
• Publication Date: October 14, 2024 (arXiv preprint)
• Paper Link: https://arxiv.org/abs/2510.12847

Abstract

Pseudo-alignment is a prevalent challenge in many Large Language Models for Time Series (LLM4TS), frequently resulting in performance inferior to linear models or randomly initialized backbone networks. However, community discussion regarding the underlying causes of pseudo-alignment remains limited. This paper conducts an in-depth investigation into the fundamental causes of pseudo-alignment in LLM4TS and establishes a connection between pseudo-alignment and the cone effect in LLMs. The research demonstrates that pseudo-alignment originates from the interaction between the cone effect in pretrained LLM components and the inherent low-dimensional manifold of time series data. Furthermore, this paper introduces TimeSUP, a novel technique designed to mitigate this problem and improve the predictive performance of existing LLM4TS methods.

Research Background and Motivation

Problem Definition

1. Core Issue: The prevalent pseudo-alignment phenomenon in LLM4TS models, leading to suboptimal performance, even underperforming simple linear models
2. Phenomenon Description: Time series and language representations appear aligned at the first-order statistics level (e.g., mean), yet the complete distributions remain different, indicating failure of true semantic alignment and distortion of modality-specific features

Research Significance

• Practical Application Value: Time series analysis has important applications in medical diagnosis, weather forecasting, traffic flow, and energy load prediction
• Theoretical Significance: Understanding LLM adaptation mechanisms in non-linguistic domains provides theoretical foundations for cross-modal learning
• Technical Challenge: Existing LLM4TS methods lack systematic investigation into the mechanistic origins of pseudo-alignment

Limitations of Existing Methods

1. Lack of in-depth analysis of the fundamental causes of pseudo-alignment
2. Absence of effective architectural modifications or training strategies to activate LLM's rich knowledge for time series prediction
3. Existing methods often underperform lightweight baseline models

Core Contributions

1. First-time revelation of pseudo-alignment from the data manifold dimensionality perspective, providing new insights for LLM4TS models and demonstrating the impact of low dimensionality on time series through comprehensive experiments
2. Proposal of the TimeSUP method, a simple yet effective large language model time series reprogramming approach that effectively addresses pseudo-alignment by lifting the intrinsic dimensionality of time series data
3. Achievement of consistent performance improvements, where TimeSUP consistently outperforms state-of-the-art LLM4TS baselines across various long-term prediction datasets and is easily adaptable to other LLM4TS methods

Methodology Details

Task Definition

This paper focuses on long-term time series forecasting tasks, with inputs being historical time series data and outputs being predicted values for future time steps. The core challenge is how to effectively leverage the linguistic knowledge of pretrained LLMs to enhance time series prediction performance.

Theoretical Foundation

Time Series Manifold Analysis

Through PCA analysis, the following is discovered:

• Time series tokens (patch size=16, stride=8) require only 21 principal components for good representation
• GPT-2 language tokens retain 712 out of 768 components
• Time series modality lies on a lower-dimensional manifold compared to the language modality

Pseudo-Alignment Theoretical Analysis

Theorem 1: When manifold dimensionality m→0 and n→0, cosine similarity converges only to the similarity between the means of time series and language distributions, causing pseudo-alignment.

Mathematical expression:

E[cos(x_ts, x_l)] = (μ_ts μ_l) / (√(||μ_ts|| + mσ_ts) √(||μ_l|| + nσ_l))

When m≪n and mσ_ts is negligible, due to the cone effect, cosine similarity increases significantly, and the equation converges to high similarity between μ_ts and the entire language distribution.

TimeSUP Architecture

1. Patched Time Series Embedding

• Input sequence length L, patch size P, stride S
• Number of patches generated: N = ⌈(P-L)/S⌉ + 1
• Linear mapping to shared language embedding space R^d

2. Top-K Text Prototype Selection

• Generate 1000 text prototypes through linear combinations of vocabulary
• Use asymmetric cross-attention to find Top-K prototypes best describing time patches
• Attention weight calculation: A_k = TopK(Softmax(QK^T/√d))

3. Temporal Manifold Enhancer

Design two lightweight MLPs:

• M_c ∈ R^((K+1)×N)×n: Operating across token dimensions
• M_f ∈ R^(d×d): Operating across feature channels

Fusion process:

T* = M_f(M_c^T T_t)^T

where T_t is the concatenated representation of temporal-textual pairs.

Effect Verification

Through PCA probing experiments, enhanced representations lift the intrinsic manifold dimensionality of time series from 21 to 224 (compared to 712 dimensions for GPT-2 language tokens), significantly increasing data manifold dimensionality.

Experimental Setup

Datasets

Eight widely-adopted long-term forecasting benchmark datasets are used:

• ETT Series: ETTh1, ETTh2, ETTm1, ETTm2 (Electricity Transformer Temperature data)
• Illness: Disease data (7 dimensions, weekly frequency)
• Weather: Weather data (21 dimensions, 10-minute frequency)
• Traffic: Traffic data (862 dimensions, hourly frequency)
• ECL: Electricity Consuming Load data (862 dimensions, hourly frequency)

Evaluation Metrics

• MSE: Mean Squared Error
• MAE: Mean Absolute Error

Comparison Methods

LLM4TS Methods: FSCA, CALF, S2IP, TimeLLM, UniTime, OFA Lightweight Baselines: TimeMixer, TimesNet, iTransformer

Implementation Details

• Hardware: 4×RTX 4090 24GB and 4×A100 40GB
• Optimizer: Adam
• Loss Function: Mean Squared Error
• Visualization analysis based on official OFA implementation

Experimental Results

Main Results

TimeSUP achieves best performance in 60 out of 80 test configurations, significantly outperforming all baseline methods:

Representative Results:

• ETTh1 Average: MSE 0.412 vs best baseline 0.426 (3.3% improvement)
• ETTh2 Average: MSE 0.353 vs best baseline 0.355 (0.6% improvement)
• Illness Average: MSE 1.885 vs best baseline 2.056 (8.3% improvement)
• Weather Average: MSE 0.231 vs best baseline 0.233 (0.9% improvement)

Layer-wise Analysis Experiments

Through layer-by-layer visualization analysis of 6-layer GPT-2:

• Baseline Model: Cosine similarity skyrockets to nearly 1 in the first layer and remains above 0.9 in subsequent layers
• TimeSUP: Starting from layer 2, time series embeddings begin to fan out and map onto the language manifold, with cosine similarity gradually increasing but eventually stabilizing at approximately 0.6643

Adaptability Experiments

TimeSUP seamlessly integrates into multiple existing LLM4TS methods:

• S2IP+TimeSUP: MSE reduction of 3% on ETTh1, MAE reduction of 2%
• OFA+TimeSUP: MSE reduction of 4.8%, MAE reduction of 1.3%
• Average Improvement: MSE reduction of 11% on Illness dataset, 2% on ETTh1

Ablation Studies

Through controlling the pretraining/fine-tuning states of LayerNorm (LN) and Multi-Head Attention (MHA):

• LN-PT & MHA-PT: Produces the most severe pseudo-alignment
• Randomly Initialized Components: Significantly reduces prediction performance
• LN-PF & MHA-RF: Largest performance degradation
• LN-RT & MHA-PF: Smallest performance degradation, indicating most linguistic knowledge is preserved in MHA layers

Related Work

Lightweight Time Series Models

• RNN-based: Learn temporal features through recurrence, but suffer from long-term dependency issues
• CNN-based: Learn convolutional kernels to extract temporal and local features
• Transformer-based: PatchTST, iTransformer, AutoFormer, etc., utilize global receptive fields
• MLP-based: DLinear, TimesNet, TimeMixer, etc., simplify model parameters

LLM4TS Methods

• OFA: Reprogram GPT-2 for time series multi-task adaptation through LayerNorm fine-tuning
• TimeLLM: Use prompts and cross-attention to find text tokens from vocabulary best describing temporal features
• CALF: Leverage LoRA fine-tuning and text-temporal consistency loss
• S2IP: Decompose time series and align language tokens to STL components

Conclusions and Discussion

Main Conclusions

1. Root Cause of Pseudo-alignment: Demonstrates that pseudo-alignment is a combined effect of the cone effect and the low-dimensional manifold of time series
2. Effective Solution: TimeSUP effectively mitigates pseudo-alignment by lifting the manifold dimensionality of time series
3. Broad Applicability: The method can be integrated as a "plug-and-play" module into various LLM4TS architectures

Limitations

1. Computational Overhead: Although TimeSUP is relatively lightweight, the added dimensionality lifting still incurs certain computational costs
2. Hyperparameter Sensitivity: Hyperparameters such as Top-K selection and compressed token count require tuning for different datasets
3. Theoretical Analysis: While mathematical proofs are provided, theoretical coverage for complex real-world scenarios remains limited

Future Directions

1. Adaptive Dimensionality Lifting: Develop methods that automatically determine optimal manifold dimensionality
2. Multi-modal Extension: Extend this idea to other modality alignment problems
3. Efficiency Optimization: Investigate more efficient manifold enhancement techniques

In-Depth Evaluation

Strengths

1. Outstanding Theoretical Contribution: First-time in-depth analysis of pseudo-alignment from manifold dimensionality perspective, providing clear mathematical theoretical support
2. Simple Yet Effective Method: TimeSUP has simple design but significant effectiveness, easy to understand and implement
3. Comprehensive Experiments: Full comparison with 10 baseline methods across 8 datasets, results are convincing
4. In-depth Visualization Analysis: Clear demonstration of method mechanisms through UMAP and layer-wise analysis
5. Broad Applicability: Demonstrates integration into multiple existing architectures

Weaknesses

1. Insufficient Computational Efficiency Analysis: Lacks detailed analysis of added computational costs and training time
2. Hyperparameter Sensitivity: Different datasets require different hyperparameter settings, lacking unified selection strategies
3. Limited Long-term Effect Verification: Primarily focuses on long-term forecasting; effectiveness on short-term forecasting and other time series tasks requires further verification
4. Theoretical Assumptions: Some mathematical derivations are based on idealized assumptions; applicability in practical scenarios may be limited

Impact

1. Academic Value: Provides important theoretical insights for the LLM4TS field, potentially inspiring subsequent related research
2. Practical Value: As a plug-and-play module, possesses strong practical application potential
3. Reproducibility: Paper provides detailed implementation details and parameter settings, facilitating reproduction

Applicable Scenarios

1. Long-term Time Series Forecasting: Particularly suitable for complex time series prediction tasks requiring LLM knowledge
2. Multi-modal Learning: The idea can be extended to other cross-modal learning problems with dimensionality mismatches
3. Pretrained Model Adaptation: Provides new perspectives for adapting pretrained language models to other domains

References

This paper cites 35 relevant references, covering important works in time series forecasting, large language models, multi-modal learning, and other domains, providing solid theoretical foundations for the research.

Overall Assessment: This is a high-quality paper with sufficient theoretical analysis and experimental validation. The paper identifies and addresses an important problem in the LLM4TS field, proposing a simple yet effective method with strong practical value and academic significance.