M* Maximizes Multimodal Model Serving Efficiency with Higher Throughput and Flexibility

Concept 1: The Problem — Modern Models Are No Longer Simple Loops

What traditional LLM inference looks like

Systems like vLLM and SGLang were built on one core assumption:

Input prompt → [Prefill] → [Decode token 1] → [Decode token 2] → ... → Stop

This is called autoregressive (AR) generation — one token at a time, in a single loop.

---

Why this breaks for modern models

Today's multimodal models are composite — they are built from structurally different pieces wired together:

These models introduce three things a single AR loop cannot handle:

1. Non-AR loops — e.g., diffusion runs 49 denoising steps, not one token at a time
2. Internal parallelism — e.g., Classifier-Free Guidance (CFG) runs 3 forward passes simultaneously
3. Input-dependent paths — e.g., BAGEL takes a completely different route through its components depending on whether you ask it to generate or understand an image

---

The core insight

The structure of modern model inference is best described as a dataflow graph, not a single loop. A request is a walk through that graph.

---

Concept 2: The Walk Graph — M*'s Core Abstraction

What is a Walk Graph?

In M*, a model is declared as:

• A graph of nodes (model components like encoders, LLMs, decoders)
• Connected by tensor edges (data flowing between components)
• Plus a set of named Walks — labeled subgraphs for each phase of behavior

Model = Graph of Components + Set of Named Walks
Request = A sequence of Walks chosen by a state machine

---

Concrete example: BAGEL

BAGEL has 4 core components:

vit_encoder → (understands images)
vae_encoder → (encodes images for editing)
LLM         → (backbone for text and image generation)
vae_decoder → (decodes latents to pixels)

Different requests take different walks through these components:

Generate image  (text → image):
  prefill_text → image_gen

Understand image (image → text):
  prefill_text → prefill_vit → decode

Edit image (image → image):
  prefill_text → prefill_vae → prefill_vit → image_gen

Key benefit: An image-understanding request never touches the diffusion loop or VAE decoder. An image-generation request never runs the ViT understanding path. You only execute what you need.

---

The state machine

The author writes a small function that picks the next Walk based on current state:

def next_walk(self, state):
    if state.prefill_steps:           # still consuming inputs
        return state.prefill_steps.pop(0)   # prefill_text / prefill_vae / prefill_vit
    if state.target == "image":
        return "image_gen"            # go to diffusion loop
    return "decode"                   # otherwise, autoregressive text

The author only writes the graph and the walks. Everything physical — placement, scheduling, batching, tensor transport — is the runtime's job.

---

Concept 3: Graph Nodes and Edges — Building Blocks

A Node

A node represents one model component. It declares:

• Its input names (what tensors it expects)
• Its output edges (where each output tensor goes next)

vae_decoder = GraphNode(
    name="vae_decoder",
    input_names=["latents"],
    outputs=[
        GraphEdge(next_node=EMIT_TO_CLIENT, name="image_output",
                  output_modality="image"),
    ],
)

The graph only describes wiring. The actual computation lives in a NodeSubmodule:

class VAEDecoderSubmodule(NodeSubmodule):  # is a torch.nn.Module
    def prepare_inputs(self, graph_walk, fwd_info, inputs):
        return NodeInputs(tensor_inputs={"latents": inputs["latents"][0]})

    def forward(self, graph_walk, engine_inputs, latents):
        image = self.vae_model.decode(unpatchify(latents))
        return {"image_output": [image]}

Separation of concerns:

GraphNode  → describes structure (wiring)
NodeSubmodule → describes compute (math)
Runtime    → handles batching, KV cache, CUDA graphs, transport

---

Concept 4: Loops — First-Class Repeated Execution

The problem loops solve

Diffusion models don't generate an image in one step — they run the same denoising operation 49 times, each time refining the latents. World models run rollouts for many steps. Even autoregressive text decode is a loop.

Traditional systems handle loops inside a single stage — they can't express loops that span multiple components generically.

---

M*'s Loop primitive

image_gen = Sequential([
    Loop(
        section=GraphNode(
            name="LLM",
            input_names=["latents", "time_index"],
            outputs=[
                GraphEdge(next_node="LLM", name="latents"),      # feeds back to itself
                GraphEdge(next_node="LLM", name="time_index"),
            ],
        ),
        max_iters=49,                                             # 49 denoising steps
        outputs=[GraphEdge(next_node="vae_decoder", name="latents")],
    ),
    vae_decoder,
])

The loop:

1. Runs the LLM node 49 times, feeding each step's output back as the next step's input
2. After 49 iterations, routes the final latents to vae_decoder

---

Why this matters

Because Loop is a generic primitive, M* applies the same optimizations to every loop:

World model rollout gets up to 12.5× speedup over Meta's native implementation because M* keeps the KV cache alive across loop steps instead of recomputing it from scratch each step.

---

Concept 5: Parallelism — Running Branches Simultaneously

What is Classifier-Free Guidance (CFG)?

CFG is a technique used in diffusion image generation. Each denoising step requires 3 forward passes through the LLM:

Pass 1: unconditional (no guidance)
Pass 2: text-conditioned
Pass 3: image-conditioned
→ Combine all three → refined latents

Running these in parallel minimizes latency. But flat pipeline systems like vLLM-Omni can't express this natively — they need bespoke per-model plugin code.

---

M*'s Parallel primitive

image_gen_cfg = Sequential([
    Loop(
        section=Sequential([
            Parallel([
                GraphNode(name="LLM",          ...outputs to combine_cfg...),
                GraphNode(name="LLM_cfg_text", ...outputs to combine_cfg...),
                GraphNode(name="LLM_cfg_img",  ...outputs to combine_cfg...),
            ]),
            GraphNode(
                name="combine_cfg",
                input_names=["v_main", "v_cfg_text", "v_cfg_img", "latents", "time_index"],
                outputs=[...routes updated latents back to all three LLM branches...],
            ),
        ]),
        max_iters=49,
        outputs=[GraphEdge(next_node="vae_decoder", name="latents")],
    ),
    vae_decoder,
])

The three LLM "views" are the same model weights accessed under three names, each reading/writing its own guidance cache. The runtime fans them out, runs them in parallel, and merges results — no per-model glue code needed.

---

Concept 6: Placement — Mapping Nodes to GPUs

The idea

Placement is declared in a YAML file that maps logical node names to physical GPU ranks. The model code never changes.

---

Single GPU

model: "bagel"
node_groups:
  - { node_names: [vit_encoder, vae_encoder, vae_decoder, LLM], ranks: [0] }

Everything runs on GPU 0.

---

Multi-GPU with CFG parallelism

model: "bagel"
node_groups:
  - { node_names: [vit_encoder, vae_encoder, vae_decoder], ranks: [0] }
  - { node_names: [LLM, combine_cfg],                      ranks: [0] }
  - { node_names: [LLM_cfg_text], ranks: [1], graph_walks: [image_gen_cfg] }
  - { node_names: [LLM_cfg_img],  ranks: [2], graph_walks: [image_gen_cfg] }

The graph_walks key means: only use this placement during the image_gen_cfg walk. During text understanding, LLM_cfg_text and LLM_cfg_img don't exist — no wasted GPU memory.

---

What placement enables

---

Concept 7: Streaming — Overlapping Components in Time

The problem

Qwen3-Omni generates speech through a pipeline of 3 components:

Thinker (LLM) → produces hidden states + text
    ↓ streams hidden states
Talker (AR model) → produces audio codec tokens
    ↓ streams codec frames
Code2Wav (codec) → produces audio waveform

To start playing audio before the full response is computed, these components must overlap in time — the Talker should start as soon as the Thinker produces its first hidden state.

---

M*'s streaming edges

# Inside Thinker's walk: stream hidden states to Talker
StreamingGraphEdge(next_node="Talker", name="thinker_states",
                   target_partition="Talker")

# Inside Talker's walk: stream codec frames to Code2Wav
StreamingGraphEdge(next_node="Code2Wav", name="codec_tokens",
                   target_partition="Code2Wav")

---

Chunk policies

Each streaming edge declares how the consumer reassembles the stream:

PartitionTopology(
    connections=[
        Connection(
            from_partition="Thinker", to_partition="Talker",
            edge_name="thinker_states",
            chunk_policy_factory=lambda: FixedChunkPolicy(chunk_size=1,
                                                          continue_after_done=True)
            # Feed Talker one hidden state at a time
        ),
        Connection(
            from_partition="Talker", to_partition="Code2Wav",
            edge_name="codec_tokens",
            chunk_policy_factory=lambda: LeftContextChunkPolicy(chunk=25,
                                                                 left_context=25)
            # Feed Code2Wav 25 frames + 25 frames of left context
            # (needed to warm up causal convolutions)
        ),
    ],
)

Result: Three components on three GPUs, running simultaneously, emitting audio incrementally — with no bespoke per-model streaming code.

---

Concept 8: The Runtime — What Happens Under the Hood

Architecture overview

Client Request
      ↓
  Conductor  ← tracks each request's current Walk, dispatches work
      ↓
  Workers (one per GPU)  ← execute nodes, route tensors to each other
      ↓
  Engines  ← the actual compute

---

Key runtime components

1. Engines

Both support continuous batching and CUDA-graph replay.

2. Pluggable data plane

Components exchange tensors over the best available transport:

Same GPU    → shared memory (zero copy)
Different GPUs, same machine → RDMA
Across machines → TCP (via Mooncake)

3. Overlapped scheduling

GPU:  [Step N running] [Step N+1 running] [Step N+2 running]
CPU:       [Prepare N+1]     [Prepare N+2]     [Prepare N+3]

While the GPU executes step N, the CPU prepares the next batch and attention plan on a separate stream. Loop stop checks are deferred by one iteration so the GPU rarely stalls waiting for the CPU.

4. Tensor parallelism

Set tp_size: 2 in the placement file → the runtime automatically handles parallel linear layers, vocab-parallel embeddings, sharded MoE, sharded KV cache, and NCCL collectives.

---

Concept 9: Performance Results — Does It Work?

Summary table

---

Where each win comes from

BAGEL image editing (2.6×):

• vLLM-Omni must choose between a 2-stage config (slow cross-stage transfer) or a 1-stage config (loses AR engine optimizations)
• M* uses Walks — each request type gets the right engine automatically, no tradeoff needed

Qwen3-Omni TTS (2.7–4×):

• M*'s streaming edges overlap Thinker, Talker, and Code2Wav in time
• Continuous batching applies across all components uniformly

V-JEPA 2 rollout (12.5×):

• M*'s Loop primitive keeps the KV cache persistent across steps
• Meta's native implementation recomputes KV cache from scratch each rollout step

---

Concept 10: Comparison to Prior Systems

How M* generalizes prior approaches

Key insight: vLLM-Omni and SGLang-Omni are restricted subsets of the Walk Graph. M* strictly generalizes them.

---

Why flat DAGs fall short

vLLM-Omni / SGLang-Omni model:
  Stage 1 → Stage 2 → Stage 3
  (each stage is a black box; loops stay inside a stage)

M* model:
  Any subgraph can loop, run in parallel, or stream to any other subgraph

CFG parallelism in vLLM-Omni requires a bespoke CFGParallelMixin plugin using torch.distributed.all_gather. In M*, it's just a Parallel block — the same primitive used for any other parallelism.

---

Concept 11: What's Next — Three Future Directions

1. SLO-aware placement and autoscaling

Automatically search for the best node-to-GPU placement that meets a Service Level Objective (throughput, latency, or cost). Scale only the components on hot walks; offload cold components to host memory.

2. Agentic serving layer

An AI agent is itself a graph of model calls — the same shape M* runs inside one model. The goal: place the inter-model agent graph and the intra-model component graph under one runtime, so calls across many agents share scheduling, batching, and cached state.

3. Walk Graph as a compiler IR

Treat the Walk Graph as an Intermediate Representation for a compiler that can:

• Eliminate components a request never touches
• Fuse operations across nodes
• Map each component to the hardware it runs best on (GPU, CPU, specialized accelerator)

---

Summary: The M* Mental Model

┌─────────────────────────────────────────────────────┐
│                    M* System                        │
│                                                     │
│  Model Author provides:                             │
│    • Graph of nodes (components)                    │
│    • Named Walks (phases of behavior)               │
│    • State machine (which walk comes next)          │
│    • Placement YAML (which GPU each node uses)      │
│                                                     │
│  Runtime provides:                                  │
│    • Scheduling & batching                          │
│    • KV cache management                            │
│    • CUDA graphs & FlashInfer kernels               │
│    • Tensor transport (shared mem / RDMA / TCP)     │
│    • Loop execution with persistent state           │
│    • Parallel branch execution                      │
│    • Streaming with chunk policies                  │
└─────────────────────────────────────────────────────┘

The core philosophy: Decouple what the model does (the graph + walks) from how it runs (the runtime). This decoupling is what enables both flexibility for new model architectures and performance through reusable, generic optimizations.