Step-by-Step Teaching

Step 1: The Core Problem — Why a Patch Is Not a Fix

Before learning the solution, you need to understand the gap it closes.

When a vulnerability is discovered, the typical response is:

1. Someone writes a patch or opens a pull request (PR)
2. CI (automated tests) passes
3. The PR waits for review

This is where most people stop thinking. Here is why that is dangerous:

Key mental model:

Patch created → [TIME PASSES] → Repo changes → Patch is now stale
                                              → CI results are invalid
                                              → Fix may already exist elsewhere

A patch is a claim about code at a specific moment. Code moves. The claim expires.

---

Step 2: What River Actually Is

River is Shopify's AI agent that lives in Slack and works inside their monorepo (called World).

Why this matters architecturally:

River does not need to be taught how Shopify's codebase works from scratch. It shares the same:

• Development environments
• Engineering conventions
• Written-down skills (documented processes)

This means River starts with organizational context built in. The team only had to add remediation logic on top of an already-contextual foundation.

Practical takeaway: When building agentic workflows, the agent's effectiveness is proportional to how much organizational context it inherits automatically. Starting from a shared substrate beats teaching context from scratch every time.

---

Step 3: The Two Entry Points Into the Pipeline

The remediation loop handles two types of vulnerabilities, each with a different starting point:

Dependency Vulnerability          Application Vulnerability
        │                                   │
        ▼                                   ▼
Existing upgrade PR             Agent-discovered finding
(may link to vulnerability)     + recommended fix
        │                                   │
        └──────────────┬────────────────────┘
                       ▼
              Same remediation loop

Important detail: Even when a PR already exists and links to a vulnerability, River still verifies that link. It does not trust the link as fact. This is the first instance of a pattern you will see repeatedly: treat every record as a claim to verify, not a fact to accept.

---

Step 4: The Remediation Loop — Four Phases

This is the core of the system. Each phase has a specific purpose and a specific reason it cannot be skipped.

Phase 1: Validate Live State

What happens: River compares its internal ledger (memory of past work) against three live sources:

• The actual GitHub repository
• Open/closed PR status
• The vulnerability tracker

Why this is necessary:

The ledger is useful memory, not operational truth. Examples of ledger drift:

• A PR listed as open that already merged
• Work listed as in-progress on a PR that closed without merging

River's rule: Every ledger entry is a claim to be checked, not a fact.

Real result from the article: On one run, reconciling the ledger against GitHub removed a large share of the apparent open backlog. Most had merged; a few were closed by the system due to newer upgrades being available.

Then: River checks whether the vulnerability still exists before touching any code:

For dependency vulnerabilities:
  → Does the codebase still resolve to the vulnerable version?
  → Has a sibling PR already fulfilled the upgrade?

For application vulnerabilities:
  → Is the vulnerable code still present at repository HEAD?
  → Is someone else's fix already in flight?

If the vulnerability is already gone, River records the evidence and stops. It does not create a redundant patch.

Why this matters: A technically correct patch built on a stale premise wastes engineering time and risks changing production behavior unnecessarily.

---

Phase 2: Repair and Iterate on Current HEAD

What happens: Only after confirming the vulnerability is still active does River touch code.

For dependency upgrades:

1. Replay the upgrade on the current repository HEAD (not the stale branch)
2. Regenerate the lockfile using the repository's own tooling
3. Repair mechanical failures caused by the upgrade

Two critical problems this phase solves:

Problem A — Lockfile drift:

Original PR branch:  lockfile at version X
Repository HEAD:     lockfile at version X + 2 unrelated upgrades

If you simply rebase the branch:
  → You revert those 2 unrelated upgrades (silent regression)

River's solution:
  → Graft the new upgrade INTO the current lockfile
  → Preserve all unrelated changes

In the article's example, three PRs shared a lockfile. All three landed without reverting adjacent changes because River grafted rather than rebased naively.

Problem B — CI result invalidation:

Every force-push (rebase) creates a new commit. Any CI results from before that push are now invalid — they tested a different version of the code.

River's rule: Discard CI notifications that arrived after a force-push but before new CI completes. Wait for results tied to the actual current HEAD.

When River does NOT rebase:

River detected that a human had taken over one branch with their own commits, and a second engineer had an open review on it. Rebasing would have:

• Overwritten someone's work
• Pre-empted an ongoing design discussion

River returned that PR to its author untouched. This is a deliberate stopping condition, not a failure.

---

Phase 3: Hand Off the Decision, Not the Investigation

The key distinction: River hands off decisions, not investigations.

When River reaches something it cannot resolve — a question about product behavior, runtime type safety, acceptable risk — it stops and hands off with:

1. The current HEAD state
2. What it tried
3. What the evidence shows
4. The smallest single question the owner needs to answer

What the owner does NOT have to do:

• Shuttle state between Git, CI, Slack, and the tracker
• Re-investigate what River already investigated

The example that illustrates this best:

River was asked to harden a long-standing feature. It confirmed the behavior, prepared a fix, updated tests. A developer asked: what was this feature originally designed to do?

River investigated and found:

• The behavior was deliberate, introduced years earlier for a specific reason
• It had its own security review at the time
• Platform-level protections added since then had reduced residual risk
• The proposed fix would have broken the flow it was trying to protect

River's recommendation: drop its own patch.

This was the correct outcome. The evidence changed, so the recommendation changed. This is not a failure — it is the system working correctly.

Why unexplained stops destroy trust:

If River had simply stopped without explanation, engineers would not know whether to restart the work, escalate it, or close it. Every stop must:

• Preserve evidence in a shared location
• Name the boundary (why it stopped)
• Route to the right decision-maker
• Ask one clear question

---

Phase 4: Verify Closure

The critical insight: Ownership transferring to a steward does not mean the vulnerability is closed. It means a human is now responsible for closing it.

River keeps watching after handoff.

What "done" actually means:

✓ Repository HEAD no longer uses the vulnerable dependency/code
✓ Vulnerability record in the tracker reflects the fix
✓ Internal ledger reflects the fix
✓ All three agree

The example that makes this concrete:

Two green PRs. Both passed CI.

Both "worked." Only one removed the vulnerability. Green CI on a PR is not the same as a fixed repository.

---

Step 5: The Architectural Principle — Prompts vs. Code

This is the most transferable engineering lesson in the article.

The problem: River's workflows are driven by prompts that specify behaviors like "enumerate completely" and "humans must approve merges." But prompts are not guarantees.

Proof from the article: On one run, River rechecked only the oldest in-flight claim instead of all of them — and said so in its own report. The prompt said to enumerate completely. River did not. The prompt did not enforce the behavior.

The rule:

Prompts are for:          Code is for:
─────────────────         ─────────────────────────────
Judgment calls            Pagination (don't miss items)
Contextual reasoning      Deduplication (don't double-act)
Stopping decisions        Current-HEAD identity verification
Risk assessment           Accounting (ledger reconciliation)

In security specifically: Any rule that determines whether a vulnerability is closed must be enforced in code, not expressed in a prompt.

---

Step 6: The Three Principles You Can Apply Without Shopify's Tools

These are the portable lessons, stated directly:

Principle 1: Revalidate Before You Edit

Security work is a claim about moving code. Before an agent touches anything:

• Compare against repository HEAD
• Check live PR state
• Check the vulnerability tracker

Do not let an agent work from cached or assumed state.

Principle 2: Make Evidence Follow the Code

• Bind tests and CI results to the specific commit SHA they tested
• After merge, inspect the default branch
• Reconcile the tracker

What not to measure: PR count, green branch count. These measure motion, not remediation.

What to measure: Did the vulnerable code leave the default branch? Do all records agree?

Principle 3: Make Handoff a Deliberate Workflow Step

• Let the agent complete work it can fully verify
• When it reaches a judgment boundary, it must preserve evidence, identify the right owner, and ask one clear question
• Measure whether each finding reaches an evidence-supported outcome: fixed, rejected, or escalated

Do not measure: How many patches the agent created.

---

Step 7: The Defensive Asymmetry — Why This Matters Beyond Shopify

The article names something important about the security context specifically:

Attackers can tolerate repeated failure. Defenders have to preserve production-intended behavior with every fix.

This asymmetry means:

This is why the pipeline is built around evidence-backed stopping points rather than maximum patch throughput. Generating more patches faster is not the goal. Generating correct, verified, non-regressive fixes is the goal.

---

Summary: The Complete Mental Model

VULNERABILITY FOUND
        │
        ▼
[Phase 1] VALIDATE LIVE STATE
  • Reconcile ledger against GitHub, PR system, tracker
  • Confirm vulnerability still exists at HEAD
  • Check if fix already exists elsewhere
  • If already fixed → record evidence, stop
        │
        ▼
[Phase 2] REPAIR ON CURRENT HEAD
  • Replay fix on current HEAD (not stale branch)
  • Graft into current lockfile (don't revert adjacent changes)
  • Discard CI results invalidated by force-push
  • If human has taken over → return to author, stop
        │
        ▼
[Phase 3] HAND OFF DECISION (not investigation)
  • Carry: current HEAD, what was tried, evidence, one clear question
  • Human supplies: product context, risk acceptance, approval, merge
  • If evidence changes → update recommendation (even to "drop the patch")
        │
        ▼
[Phase 4] VERIFY CLOSURE
  • Confirm vulnerable code/dependency gone from HEAD
  • Confirm tracker updated
  • Confirm ledger updated
  • All three must agree before marking "done"

The patch is the beginning of remediation, not the end.