Capturing token IDs during agentic interactions for better reinforcement learning

Concept 1: What Tokens Are and Why They Matter

The Basic Idea

Computers don't read text the way humans do. Language models convert text into tokens — numbered units represented as integer IDs.

Breaking It Down

• A sentence like "Solve this Python task" might become something like [12045, 419, 7823, 1902]
• This conversion is done by a tokenizer — a deterministic function (same input always gives same output)
• Each model is paired with one specific tokenizer — you cannot swap them

The Critical Sensitivity Problem

Even tiny changes produce different token IDs:

Key takeaway: What looks identical to a human can be completely different to a model at the token level.

---

Concept 2: Reinforcement Learning (RL) for Language Models

The Basic Idea

RL is a training technique where a model learns by attempting tasks, getting scored, and adjusting its behavior toward what worked.

The Process Step by Step

1. The model attempts a task (e.g., write code, navigate a website)
2. The attempt is scored (did it succeed?)
3. The score is used to nudge the model's parameters toward successful choices

The Mathematical Requirement

RL math (specifically policy-gradient RL) has a strict requirement:

The trainer must optimize the model against the exact context the model actually experienced — not a reconstruction, not a summary, not an approximation.

Why This Is Hard

To compute a parameter update, the trainer needs:

• ✅ Exact token IDs the model saw
• ✅ Log probabilities (the model's confidence for each token choice, expressed as logarithms)
• ✅ A loss mask (which tokens were generated by the model vs. provided by the user/tools)

---

Concept 3: Rollouts and the Behavior Policy

Definitions

• A rollout = one complete recorded attempt at a task (prompt + tool calls + tool results + model responses + final outcome)
• The behavior policy = the version of the model that produced the rollout

Why the Exact Record Matters

Imagine the model saw token sequence [A, B, C] and made decision D.

If the trainer later reconstructs [A, B', C] (slightly different due to reformatting):

• The trainer thinks it's optimizing decision D given context [A, B', C]
• But the model never saw [A, B', C]
• The training signal degrades — sometimes invisibly, because the model still appears to be learning

Key takeaway: Training against a slightly wrong past is worse than it looks, because the error is silent.

---

Concept 4: Agent Harnesses and Why They Make the Problem Worse

What a Harness Is

A harness is software that wraps a language model to enable multi-step tasks:

• Lets the model call tools (e.g., run code, search the web)
• Observes results and feeds them back
• Manages the conversation flow across many turns

The Problem: Harnesses Rewrite History

During a single rollout, a harness might:

• Compact older messages to save memory
• Retry a malformed tool call
• Branch into subagents and merge their results
• Summarize conversation history

Each of these rewrites is a chance for token drift — the next request's token sequence diverges from what the model actually generated last turn.

Two Named Types of Drift

The Core Problem

A harness transcript is a faithful record of the conversation. It is not a faithful record of the tokens — and it is the tokens the trainer needs.

---

Concept 5: Capturing Tokens at the Right Moment (Turnstile's Core Design)

The Insight

Instead of trying to reconstruct token-level state from text after the rollout ends, capture it at the moment of generation — the only point where it is unambiguously correct.

How Turnstile Works as a Proxy

Agent Harness → [Turnstile Proxy] → Inference Backend (SGLang/vLLM)
                      ↓
               Records exact tokens,
               log probs, loss masks

Step-by-Step Flow

1. The harness points its API client at Turnstile's address instead of the real backend
2. Turnstile forwards every request to the actual inference backend
3. As the model generates tokens, Turnstile records:
   • Exact token IDs sampled
   • Per-token log probabilities
   • Loss mask (model-generated vs. input tokens)
4. When the rollout finishes, the harness retrieves a TrainingSequence object

The Key Benefit

The harness runs completely unchanged — it doesn't need to know anything about RL training.

---

Concept 6: Prefix-Aware Multiturn Trajectories

The Naive (Wrong) Approach

Treat every request as an independent training example.

Problems:

• Wasteful: Each new request includes the whole conversation so far → same tokens duplicated repeatedly
• Wrong: Loses the relationship between turns

Turnstile's Approach: A Single Growing Token Path

When Turn 2 is just Turn 1 + new tokens at the end:

Turn 1: [T1, T2, T3, T4, T5]
Turn 2: [T1, T2, T3, T4, T5, T6, T7, T8]
                              ↑ new tokens

Turnstile verifies (at the token ID level, not by comparing text strings) that the prefix is identical, then merges them into one continuous trainable sequence.

When Merging Is Not Safe: "Exploding the Trajectory"

If the harness rewrote earlier messages or tokens drifted:

Turn 1: [T1, T2, T3, T4, T5]
Turn 2: [T1, T2, T3', T4, T5, T6, T7]  ← T3 changed to T3'

Turnstile starts a new sequence rather than pretend the prefix is the same.

Design philosophy: The goal is not maximum compactness. The goal is never lying to the trainer about what happened.

---

Concept 7: Mixture-of-Experts (MoE) Routing as a Hidden Variable

What MoE Architecture Is

Some models don't activate all their parameters for every token. Instead:

• A small router network selects a subset of "experts" (parameter groups) for each token
• Only those experts process that token
• The routing decision depends on the activations from previous tokens

Why This Creates an RL Problem

Even if two requests have identical token IDs, the tokens might be routed to different experts on different runs — because tiny differences in how previous tokens were processed change the routing decisions.

Turnstile's Solution

When MoE capture is enabled:

1. Turnstile records the routing trace alongside the tokens
2. When extending a trajectory, it verifies the routing for the shared prefix matches what was recorded
3. If routing differs (e.g., due to a cache miss forcing recomputation) → split the trajectory

Key insight: Token IDs alone are not sufficient to fully characterize what the model experienced in MoE models — the routing path is also part of the computation.

---

Concept 8: Multimodal Rollouts (Images as Input)

The Problem

When a model takes images as input, the model doesn't see raw image bytes. It sees the output of an image processor:

• Resizes and crops the image
• Converts it to a tensor of pixel values
• Inserts placeholder tokens into the text

The same image bytes can produce different tensors if:

• The processor version changes
• The resize policy differs
• The patch geometry changes

Turnstile's Solution

Turnstile treats image processing as part of the rollout:

1. Decodes the image
2. Hashes and stores original bytes (for audit)
3. Runs the model's configured processor
4. Records both raw and processed visual data in the trajectory

class TrainingSequence:
    tokens: list[int]           # token IDs
    logprobs: list[float]       # per-token log probabilities
    segment_info: list[...]     # loss mask info
    weight_versions: list[...]  # which model weights were active
    routed_experts: str | None  # MoE routing trace
    images: list[TrainingImage]              # raw images
    processed_images: list[TrainingProcessedImage]  # processed tensors

---

Summary: The Complete Picture

The unifying principle: The model sampled the tokens. Record them at that moment. Never reconstruct, never approximate, never lie to the trainer about what happened.