MADI: From Consistency to Complementarity — Aligned and Disentangled Multi-modal Learning for Time Series Understanding and Reasoning

This repository contains the official code of MADI, a multi-modal large language model (MLLM) for time series understanding and reasoning (TSUR). MADI hybridizes numerical time series with their visualized line plots and integrates the two modalities with three components:

• Patch-level Alignment (PA) — physically grounded, fine-grained contrastive alignment between numerical time series, their visualized line plots, and patch-wise textual captions;
• Discrete Disentangled Interaction (DDI) — hierarchical vector quantization that separates modality-common semantics into compact discrete latents, followed by cross-attention fusion of the purified modality-unique components;
• Critical-token Highlighting (CTH) — prepends question-relevant summary tokens to the multimodal sequence for robust reasoning.

MADI is built on Qwen2.5-VL-7B-Instruct and trained/evaluated on the public ChatTS datasets (synthetic understanding, real-world understanding, and reasoning).

This codebase contains the training, evaluation, and model code for the main model reported in the paper (exp name M2TQ-vq-pall4-tunevis-twos2-try1-align-from23, final checkpoint checkpoint-950). Baseline implementations are released by their original authors; see Baselines.

Repository structure

├── model/               # Custom MADI model code (overlay onto Qwen2.5-VL-7B-Instruct)
│   ├── config.json                  # Model config incl. PA / DDI / prepend settings
│   ├── configuration_qwen2_5_vl.py  # Custom HF config class
│   ├── modeling_qwen2_5_vl.py       # TS encoder + PA + DDI + prepended highlight tokens
│   ├── modeling_timeseries_encoder.py
│   ├── processing_qwen2_5_vl_ts.py  # Processor: normalization, statistics-preserved prompt, plotting
│   ├── vector_quantize.py           # VQ library used by DDI
│   └── ...                          # Tokenizer extras (<ts>, <|ts_pad|>, <|query_prompt|>, ...)
├── src/                 # LLaMA-Factory-based training framework (fork with TS support)
├── ds_config/           # DeepSpeed ZeRO-3 configs for the two training stages
├── chatts/              # Data-construction pipeline and DeepSpeed inference utilities
├── evaluation/          # Metrics (accuracy / F1 / relative accuracy / answer correctness)
├── config/              # Data-generation configs
├── scripts/
│   ├── prepare_base_model.sh   # Download Qwen2.5-VL + overlay model/
│   ├── train_madi.sh           # Two-stage training of MADI
│   ├── eval_generate.sh        # Answer generation on the eval datasets (DeepSpeed)
│   └── eval_metrics.sh         # Metric computation + reasoning aggregation
├── data/                # Place training data here (not included, see below)
└── exp/                 # Evaluation outputs (created at runtime, not included)

Installation

conda create -n madi python=3.11 -y && conda activate madi
pip install -r requirements.txt
# install the training framework
pip install -e .

Required hardware: 4 × NVIDIA A800 (80G) for training as reported in the paper; fewer GPUs work with a reduced batch size (with CPU offload in the DeepSpeed config). Evaluation generation runs on 1–4 GPUs (the paper uses 4).

Data preparation

Training data (ChatTS release, Apache-2.0 / MIT):

• align_256 and ift splits from ChatTSRepo/ChatTS-Training-Dataset. See data/README.md for the exact file layout.

Evaluation data (ChatTS release, CC BY 4.0):

• dataset_a.json and dataset_b.json from Zenodo. See evaluation/dataset/README.md.

No data files are distributed with this repository.

1. Prepare the base model

bash scripts/prepare_base_model.sh

This downloads Qwen/Qwen2.5-VL-7B-Instruct into Models/Qwen25-VL-7B-Instruct-for-MADI and overlays the custom model code from model/ (set LOCAL_SRC=/path/to/local/Qwen2.5-VL-7B-Instruct to use a local copy instead of downloading).

2. Train MADI

bash scripts/train_madi.sh

This reproduces the paper's two-stage training:

	Stage 1 (warmup)	Stage 2 (full fine-tuning)
Trainable	Vision tower + new modules (LLM frozen)	All parameters
Steps	230	1200 (save every 50)
Dataset mix	align_256 : ift = 0.9 : 0.1	same
Per-device batch × grad-accum	1 × 128	2 × 64
LR schedule	1e-5, cosine, warmup 0.02	same
DeepSpeed	ZeRO-3, fp16	ZeRO-3, fp16

The final checkpoints are saved under Models/M2TQ-vq-pall4-tunevis-twos2-try1-align-from23/ (the paper reports checkpoint-950, selected by evaluation every 50 steps).

3. Evaluate MADI

# 3.1 Generate answers (temperature 0.01, matching the paper)
bash scripts/eval_generate.sh

# 3.2 Compute metrics + aggregate reasoning results
bash scripts/eval_metrics.sh

Answers are written to exp/M2TQ-vq-pall4-tunevis-twos2-try1-align-from23/checkpoint-950_0.01/<dataset>/ and the metrics to result_gpt.json (overall) and detailed_result_gpt.json (per sample) therein:

• Categorical tasks — accuracy and F1;
• Numerical tasks — relative accuracy max(0, 1 − |v_pred − v_label| / |v_label|);
• Open-ended reasoning tasks — Answer Correctness (RAGAS v0.1.9) with a GPT-4o-mini judge. Set your API credentials in evaluation/ragas/config/config.toml before evaluating open-ended questions.

scripts/eval_metrics.sh finally runs evaluation/merge_reason.py, which merges the reasoning results across dataset_a + dataset_b + dataset_mcq100 into exp/.../dataset_reason/result_gpt_merged.json (reasoning numbers of the paper).

Useful env vars: CHECKPOINTS (space-separated steps), DATASETS, NUM_GPUS, TEMP, MODEL_DIR.

Hyperparameters

The full model hyperparameters are stored in model/config.json (ts, ts_img, and align sections), including: embedding dimension D = 3584, numerical patch size p_n = 8, codebook size 1024 with EMA decay 0.8 and the rotation trick, the hierarchical codebook settings (3 levels, ratio 2), the contrastive alignment temperature/loss weights (α = 5 modality-common, β = 1 orthogonality, λ1 = 0.02 PA, λ2 = 0.2 DDI), and decoding temperature 0.01. See the paper's Appendix B.4 for the full implementation details.

Baselines

For comparison we fine-tuned the following models under the same backbone (Qwen2.5-7B-Instruct / Qwen2.5-VL-7B-Instruct) and training data:

• ChatTS (MIT)
• ChatTime
• ITFormer (MIT)
• InstructTime
• GEM (Apache-2.0)

General-purpose LLMs/MLLMs (GPT-4o, GPT-5.2, Qwen3, Qwen3-VL, DeepSeek-V3.2, Gemini 3 Pro) were queried through official APIs and are not part of this repository.

License

This repository is released under the Apache-2.0 license. It builds on LLaMA-Factory (Apache-2.0), ChatTS (MIT), and Qwen2.5-VL (Apache-2.0). The training/evaluation datasets are released by ChatTS under their own licenses (see the links above) and are not redistributed here.

Citation

@misc{ni2026madi,
      title={From Consistency to Complementarity: Aligned and Disentangled Multi-modal Learning for Time Series Understanding and Reasoning}, 
      author={Hang Ni and Weijia Zhang and Fei Wang and Zezhi Shao and Hao Liu},
      year={2026},
      eprint={2601.21436},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2601.21436}, 
}

Acknowledgements

We thank the authors of ChatTS, LLaMA-Factory, Qwen, and vector-quantize-pytorch for releasing their code, models, and datasets.