litespecformer 0.0.2

A lightweight Transformer-based model for spectrum prediction and time series forecasting

LiteSpecFormer

✨Overview

LiteSpecFormer is the first lightweight wireless foundation model for zero-shot confidence spectrum prediction. Built on a channel-independent Transformer with a sliding autoregressive paradigm and a novel linear correlation loss to mitigate error accumulation, it achieves state-of-the-art performance across arbitrary frequency bands and sequence lengths without downstream fine-tuning.

We present Large-Spectrum-Prediction-Dataset (LSPD), the first large-scale dataset specifically designed for pre-training spectrum prediction foundation models. It comprises 18 billion timestamps and integrates two learnable data generation mechanisms to generate high-quality, diverse spectrum samples.

🧭 Quickstart

Installation

First create a Python virtual environment with 3.11+, then install the required dependencies through PyPI:

pip install litespecformer

Then, you can refer to our demo file to create our pipeline and then use the prediction method for zero-shot prediction:

from s2generator.utils import generate_nonstationary_sine
from litespecformer import LiteSpecFormerPipeline


# Load our model using our pipeline.
pipeline = LiteSpecFormerPipeline.from_pretrained("FlowVortex/LiteSpecFormer")

# Set the context length and prediction length
context_length = 256
# Generate a non-stationary sine wave time series with the specified lengths
time_series = np.vstack(
    [
        generate_nonstationary_sine(
            seq_length=context_length + prediction_length, freq=2 + i
        )
        for i in range(5)
    ]
)

# Prediction the inputs through our pipeline
outputs = pipeline.predict(
    time_series[:, :context_length], prediction_length=prediction_length
)

Data Preparation

Download Large-Spectrum-Prediction-Dataset (LSPD) for pre-training or downstream evaluation:

hf download FlowVortex/Large-Spectrum-Prediction-Dataset \
  --local-dir ./data \
  --type dataset

To fetch only the test split (useful for quick benchmarking):

hf download FlowVortex/Large-Spectrum-Prediction-Dataset \
  --local-dir ./data \
  --type dataset \
  --include "test/*"

The simulation and augmentation pipelines used to build LSPD are available in S2Generator: see the simulator and augmentation modules.

Model Pre-Train and Fine-Tune

We provide scripts for pre-training, fine-tuning, and zero-shot out-of-distribution evaluation on downstream tasks:

# Pre-train LiteSpecFormer
bash scripts/pre-training.sh

# Fine-tune LiteSpecFormer
bash scripts/fine_tuning.sh

Zero-Shot Evaluation

After downloading the test split, run the evaluation scripts for zero-shot out-of-distribution prediction. At inference time, LiteSpecFormer performs channel-independent forecasting:

# Example: Madrid dataset
bash scripts/zero_shot_prediction/Madrid.sh

Other benchmark datasets are available under scripts/zero_shot_prediction/ (e.g., Alcorcon1, IADAM, Nudelsalat, Oreland, PiSDR1).

📊 Results

Benchmark Results

After large-scale pre-training, LiteSpecFormer achieves stronger out-of-distribution generalization and scalability than existing supervised baselines:

Forecasting Visualization

Predictions on the Madrid dataset. Compared with supervised in-distribution models and other out-of-distribution zero-shot baselines, LiteSpecFormer produces more accurate forecasts, especially in fine-grained spectral detail:

🎖️ Acknowledgement

We appreciate the following GitHub repos a lot for their valuable code and efforts.

• chronos-forecasting (https://github.com/amazon-science/chronos-forecasting);
• PySDKit (https://github.com/wwhenxuan/PySDKit);
• S2Generator (https://github.com/wwhenxuan/S2Generator);
• Gift-Eval (https://huggingface.co/spaces/Salesforce/GIFT-Eval);

🤗 Contact

If you have any questions or are interested in our view on the complex dynamics of time series, feel free to contact:

• Whenxuan Wang ([e-mail])
• Dan Wang ([e-mail])