When Does Text Improve Time Series Forecasting?

Peter Bubenik · Amazon Research Publication · Nov 3, 2026 · Source

Image for When does multimodality lead to better time series forecasting?

After studying this material, you should be able to:

Explain what multimodal time series forecasting is and why it matters
Distinguish between the two main multimodal forecasting paradigms
Identify the specific conditions under which text integration improves forecasting
Evaluate whether a given scenario would benefit from multimodal approaches
Critically assess claims about multimodal forecasting performance

Step-by-Step Study Material

Step 1: Foundation — What is Time Series Forecasting?

Core Concept

Time series forecasting means predicting future values based on historical patterns.

Example:
Past data:  [10, 12, 11, 13, 14, ?]
Goal:        Predict the next value(s)

Real-World Examples by Domain

Domain	What We Forecast
Health	Patient vital signs
Environment	Temperature, pollution
Economics	Stock prices, GDP

Why It's Challenging

Data can be noisy
Patterns can change over time
Numbers alone may miss context

💡 Key Question this article asks: Can adding text information make these predictions better?

Step 2: What is Multimodality?

Simple Definition

Multimodality = using more than one type of data simultaneously

Traditional forecasting:
[Numbers only] → Model → Prediction

Multimodal forecasting:
[Numbers + Text] → Model → Prediction

What Kind of Text?

News headlines about the economy
Medical notes about a patient
Weather reports describing conditions
Analyst commentary on markets

The Core Assumption

Text might carry complementary information that numbers alone don't capture

Example:

Numbers say: Sales dropped 30% this week
Text adds:   "Major competitor launched a rival product"

The text EXPLAINS and PREDICTS what numbers alone cannot

Step 3: The Two Multimodal Paradigms

This is a critical distinction you must understand.

Paradigm 1: Aligning-Based Methods

How it works:

Time Series Data  →  [Encoder] → Numerical Representation
                                          ↓
                                    [Alignment Layer]
                                          ↓
Text Data         →  [Text Encoder] → Text Representation
                                          ↓
                                  Combined Representation
                                          ↓
                                      Prediction

Key idea: Train the model to map time series and text into a shared space

Analogy: Like teaching two people who speak different languages to communicate through a shared translator

Paradigm 2: Prompting-Based Methods

How it works:

Time Series Data → [Convert to text/tokens]
                              +
Text Context    → [Combine into prompt]
                              ↓
              Large Language Model (LLM)
                              ↓
                          Prediction

Key idea: Use an LLM directly by feeding it both the numbers and text as a prompt

Analogy: Like asking a very knowledgeable expert a question by giving them all available information

Side-by-Side Comparison

Feature	Aligning-Based	Prompting-Based
Approach	Learn shared representation	Directly use LLM
Training needed	Yes	Minimal/None
Flexibility	Lower	Higher
Dependency	Custom architecture	Relies on LLM quality

Step 4: The Central Finding — Benefits Are Conditional

⚠️ Critical Insight

The article's most important finding: Text integration does NOT universally improve forecasting

This challenges a common assumption. Let's break down exactly when it helps.

Step 5: Modeling-Side Conditions (3 Factors)

Condition 1: High-Capacity Text Models

What this means: The text encoder/LLM must be large and powerful enough to extract meaningful signals

Weak text model:
"Economy slowing down" → [Small model] → Poor representation → No gain

Strong text model:
"Economy slowing down" → [Large model] → Rich representation → Gain ✓

Why it matters: A small text model may fail to capture nuance, making the text signal useless or even harmful

📌 Rule: Bigger, better text models = more likely to benefit from multimodality

Condition 2: Comparatively Weaker Time Series Models

What this means: If your time series model is already very strong, text adds little marginal value

Scenario A (Text helps):
Weak TS model alone:     60% accuracy
+ Text integration:      72% accuracy  ✓ Big gain

Scenario B (Text doesn't help):
Strong TS model alone:   89% accuracy
+ Text integration:      90% accuracy  ✗ Marginal gain

Intuition:

If the numerical model already captures most patterns, text has less room to contribute

Analogy: Adding a flashlight in a fully lit room vs. a dark room — it only truly helps when there's darkness (weakness) to fill

Condition 3: Appropriate Aligning Strategies

What this means: How you combine text and time series representations matters enormously

Poor alignment:
[TS representation] + [Text representation] → Simple concatenation → Noise added

Good alignment:
[TS representation] + [Text representation] → Learned fusion → Signal amplified

Key takeaway: Not all integration methods are equal — the architecture of fusion determines whether text helps or hurts

Step 6: Data-Side Conditions (2 Factors)

Condition 4: Sufficient Training Data

What this means: Multimodal models are more complex — they need more data to learn properly

Data availability spectrum:

Small dataset → Model overfits → Text adds noise → Performance drops ↓
Large dataset → Model generalizes → Text adds signal → Performance rises ↑

Visual representation:

Performance
    ↑
    |                              ●●●  (Multimodal)
    |                         ●●●
    |              ●●●●●●●●●●
    |    ●●●●●●●●●
    |●●●
    |________________________→ Training Data Size
    
    (Multimodal only wins after sufficient data threshold)

📌 Rule: More training data = more likely multimodality helps

Condition 5: Text Offers Complementary Predictive Signal

This is the most conceptually important condition

What "complementary" means: The text must tell you something the numbers don't already tell you

NOT complementary (redundant):
Numbers: Temperature rising steadily for 5 days
Text:    "It has been getting warmer this week"
→ Text adds nothing new → No gain

IS complementary:
Numbers: Sales figures dropping
Text:    "New government regulation announced affecting this industry"
→ Text explains cause numbers can't show → Gain ✓

How to think about it:

Information in Text
        ↓
Already captured    |    NEW information
by time series?     |    not in numbers
        ↓                      ↓
    No gain              Potential gain

Step 7: Synthesizing All 5 Conditions

The Complete Decision Framework

Should I use multimodal forecasting?

START
  │
  ▼
Is my text model high-capacity? ──No──→ Unlikely to help
  │ Yes
  ▼
Is my TS model relatively weak? ──No──→ Marginal gains only
  │ Yes
  ▼
Do I have a good aligning strategy? ──No──→ Risk of hurting performance
  │ Yes
  ▼
Do I have sufficient training data? ──No──→ Likely to overfit
  │ Yes
  ▼
Does text offer NEW signal beyond numbers? ──No──→ Redundant, no gain
  │ Yes
  ▼
✅ Multimodality likely to help!

Summary Table of All 5 Conditions

#	Side	Condition	If Met	If Not Met
1	Model	High-capacity text model	Text signal is rich	Text signal is weak/noisy
2	Model	Weaker TS model	Room for improvement	Ceiling effect
3	Model	Appropriate alignment	Signals fuse well	Signals interfere
4	Data	Sufficient training data	Model learns fusion	Model overfits
5	Data	Complementary text signal	New info added	Redundant info

Step 8: Why This Matters — Critical Thinking

The Danger of Assuming "More Data = Better"

A common intuition is:

"Adding more information (text) should always help"

This article proves that's WRONG. Here's why:

More information can:
✓ Add signal (good)
✗ Add noise (bad)
✗ Overwhelm the model (bad)
✗ Create redundancy (neutral/bad)

The Benchmark Methodology

The researchers tested across:

16 forecasting tasks
7 domains (health, environment, economics, etc.)

This diversity is important because it allows generalizable conclusions rather than domain-specific ones

Step 9: Key Vocabulary Review

Term	Definition
Multimodal	Using multiple data types (e.g., numbers + text)
Foundation Model	Large pre-trained model adaptable to many tasks
Aligning-based	Method that maps TS and text to shared representation space
Prompting-based	Method that feeds data directly to an LLM as a prompt
Complementary signal	Information in text NOT already captured by numbers
High-capacity model	Large model with strong representational power
Benchmark	Standardized set of tasks for systematic evaluation

Step 10: Self-Assessment Questions

Test your understanding:

Q1: A researcher has a small dataset of 500 samples and wants to add news text to improve stock forecasting. Based on the article, what would you advise?

Answer: Caution — insufficient training data (Condition 4 not met) means the multimodal model may overfit and perform worse

Q2: What's the difference between aligning-based and prompting-based methods?

Answer: Aligning-based methods train a model to create a shared representation space for both modalities. Prompting-based methods directly feed both time series and text to an LLM as input prompts.

Q3: If a time series model already achieves 95% accuracy, should you add text?

Answer: Likely not — Condition 2 says text helps most when the TS model is comparatively weak. A strong TS model leaves little room for improvement.

Q4: Give an example of text that IS complementary vs. NOT complementary to sales data.

Answer:

Complementary: "Factory strike announced, production halted" — explains future drop numbers can't predict

Not complementary: "Sales have been declining recently" — just restates what the numbers already show

Final Summary

┌─────────────────────────────────────────────────────┐
│         WHEN DOES MULTIMODALITY HELP?               │
├─────────────────────────────────────────────────────┤
│                                                     │
│  MODEL SIDE:                                        │
│  ✓ Strong text model (high capacity)               │
│  ✓ Weak time series model (room to improve)        │
│  ✓ Good alignment strategy                         │
│                                                     │
│  DATA SIDE:                                         │
│  ✓ Enough training data                            │
│  ✓ Text adds NEW information beyond numbers        │
│                                                     │
│  KEY INSIGHT:                                       │
│  Benefits are CONDITIONAL, not universal           │
│  More data ≠ always better                         │
│                                                     │
└─────────────────────────────────────────────────────┘

More to study

Evaluating Safety Across Multimodal AI Models

How LLMs Learn Auditable Rules for Trustworthy Decisions

Distilling AI Reasoning for Smarter Product Upgrades