Modular LLMs at scale: How the Danish Foundation Models project is using FlexOlmo to pool national expertise without pooling sensitive data | Ai2

Concept 1: The Problem with Monolithic Language Models

What is a monolithic model?

A traditional large language model (LLM) is a single, unified system. Every piece of text you feed it gets processed by the entire model, every time.

Why is this a problem?

Real-world example: A Danish hospital has valuable medical text data, but cannot legally share it due to GDPR. Under a monolithic approach, that data simply can't contribute to a shared model.

---

Concept 2: Mixture-of-Experts (MoE) — The Core Modular Idea

Instead of one big system, what if you had many specialists?

A Mixture-of-Experts (MoE) model breaks the model into multiple expert modules, each trained on different types of content.

Input Token → [Router] → selects relevant expert(s) → Output
                              ↓
                    [Legal Expert]
                    [Code Expert]
                    [Medical Expert]
                    [Math Expert]

Key insight: Only the selected experts run at any given time — not all of them.

Analogy: Instead of one doctor who knows everything, you have specialists. When you have a legal question, you consult the legal expert. When you have a coding problem, you consult the coding expert. You don't consult all of them every time.

Why this matters:

• Experts can be trained separately, on separate data
• Data never needs to leave its owner
• The combined model is more capable than any single expert alone

---

Concept 3: FlexOlmo — Modular Training Without Data Sharing

FlexOlmo is Ai2's framework that operationalizes the MoE idea for independent, distributed training.

How it works:

Team A trains Expert A on their private data  ──┐
Team B trains Expert B on their private data  ──┼──→ Merge into shared model
Team C trains Expert C on their private data  ──┘

• Each team trains their own expert in isolation
• No raw data is ever shared
• The experts are then merged into one combined model

The breakthrough: You can pool expertise without pooling sensitive data.

The limitation FlexOlmo had:

Each expert is the full size of a standalone model. Add more experts → model grows very large → requires expensive hardware to run.

---

Concept 4: Low-Rank Adapters — Making Experts Smaller

The core innovation in FlexMoRE

Instead of keeping every expert at full size, FlexMoRE replaces most experts with low-rank adapters (LoRA).

What is a low-rank adapter?

Think of it as a compressed summary of what a full expert learned, using far fewer parameters (the numbers that define the model's behavior).

Full Expert:     [████████████████████]  ← many parameters
Low-Rank Adapter: [████]                 ← far fewer parameters, similar behavior

The "rank" controls how much compression happens:

• Higher rank = larger adapter, more expressive
• Lower rank = smaller adapter, more compressed

The key finding: Not all tasks need the same rank.

Analogy: A detailed engineering blueprint needs high resolution to be useful. A simple map of a city can be compressed more without losing what matters.

---

Concept 5: FlexMoRE — Scaling Modular Models to Consumer Hardware

FlexMoRE = FlexOlmo's architecture + low-rank adapters for most experts

What changes:

FlexOlmo:
[Full Expert] [Full Expert] [Full Expert] [Full Expert] [Full Expert]
     ↑              ↑              ↑              ↑              ↑
  Full size      Full size      Full size      Full size      Full size

FlexMoRE:
[Full Expert] [Small Adapter] [Small Adapter] [Small Adapter] [Full Expert]
     ↑                ↑               ↑               ↑              ↑
  Full size        Compressed     Compressed      Compressed     Full size

The result:

• Uses less than one-third the parameters of a full FlexOlmo model
• Outperforms the full-size baseline in testing
• Runs on consumer hardware (laptops, standard workstations)

Why this matters for the Danish project:

Hospitals, universities, and small companies can now download only the experts they need (e.g., Danish language + medical domain) and run the model on their own machines — no expensive cloud infrastructure required.

---

Concept 6: Federated / Distributed Training — The Bigger Picture

What FlexMoRE enables at a systems level

This approach is an example of federated learning — a paradigm where:

Institution A ──trains local expert──→ contributes module
Institution B ──trains local expert──→ contributes module   →  Shared model
Institution C ──trains local expert──→ contributes module

No raw data ever moves. Only the trained model weights (the expert modules) are shared.

Why this is significant:

---

Concept 7: Extending Modularity Beyond Pretraining (EMO & BAR)

The article introduces two more projects that push modularity into other stages of model development.

Stage 1: Pretraining (what FlexOlmo/FlexMoRE address)

Training the model on raw text data to learn language and knowledge.

Problem FlexOlmo had: Expert specializations had to be defined in advance. EMO's solution: Let experts discover their own specializations during training.

FlexOlmo: You decide → "This expert handles legal text"
EMO:      Model decides → experts self-organize around topics naturally

---

Stage 2: Post-training (what BAR addresses)

After pretraining, models are further trained to:

• Follow instructions
• Reason through problems
• Use tools
• Refuse unsafe requests

Problem: In a standard pipeline, all these behaviors get tangled together — improving one often breaks another.

BAR's solution: Give each capability its own isolated post-training pipeline.

Standard:  [Base Model] → one big post-training → all behaviors mixed together

BAR:       [Base Model] → [Instruction pipeline]
                        → [Reasoning pipeline]
                        → [Safety pipeline]
                        → [Tool-use pipeline]
           Each runs in isolation, then combined

---

Summary: The Full Conceptual Arc

PROBLEM: Monolithic models require centralized data and expensive hardware

SOLUTION LAYER 1 — MoE Architecture:
  Route tokens to specialized experts instead of one big model

SOLUTION LAYER 2 — FlexOlmo:
  Train experts independently, merge without sharing data

SOLUTION LAYER 3 — FlexMoRE:
  Replace most experts with low-rank adapters → smaller, faster, accessible

SOLUTION LAYER 4 — EMO:
  Remove the need to pre-define expert specializations

SOLUTION LAYER 5 — BAR:
  Extend modularity into post-training stages

OUTCOME: Powerful AI built collaboratively, openly, without centralizing
         sensitive data or requiring expensive infrastructure

---

The unifying principle across all these concepts:

Powerful models don't have to be monolithic or centralized. Modularity allows expertise to be pooled without pooling the sensitive data underneath — making AI development more open, more private, and more accessible.