After studying this material, you should be able to:

1. Define the cold-start problem in natural language search
2. Explain how CASTLE generates synthetic training data using contrastive listing pairs
3. Compare CASTLE against prior approaches (InPars, Promptagator) using KL divergence
4. Understand why relevance labels are correct "by construction" in CASTLE
5. Evaluate the trade-offs and extensions of the CASTLE framework

---

Step-by-Step Teaching

Step 1: The Problem — Why Cold-Start Is Hard

What is Natural Language Search?

Instead of clicking filters, users type free-form queries like:

• "romantic cabin near skiing"
• "pet-friendly beach house that sleeps eight"

The Core Challenge

To train a search model, you need two things:

This is the cold-start problem: You need data to train the model, but you need the model running to collect data.

---

Step 2: How Others Tried to Solve It (And Why It Falls Short)

Prior Approach 1: InPars

• Uses an LLM to generate queries based on a single listing document
• Then uses another LLM call to assign relevance labels
• Problem: Two separate steps = two sources of error compounding each other

Prior Approach 2: Promptagator

• Improves query diversity using few-shot examples
• Problem: Generated queries still don't match how real users actually search

How Do We Measure "How Far Off" They Are?

Using Kullback–Leibler (KL) Divergence:

KL divergence measures how much one probability distribution differs from a reference distribution. Lower = closer to real user behavior.

Key insight: Prior methods treat query generation and relevance labeling as separate problems. CASTLE solves both simultaneously.

---

Step 3: CASTLE's Core Mechanism — Contrastive Listing Pairs

The Clever Observation from Booking Data

When a guest views multiple listings but books only one, that behavior implicitly encodes preference:

Guest views: Listing A, Listing B, Listing C
Guest books: Listing A

→ Listing A = POSITIVE (preferred)
→ Listing B or C = NEGATIVE (viewed but rejected)

This creates a (positive, negative) contrastive pair — grounded in real human behavior, not LLM assumptions.

Why This Solves the Relevance Label Problem

The LLM is instructed:

"Generate a query satisfied by the POSITIVE listing but NOT the negative listing."

Because the distinction is defined before the LLM acts, the relevance label is correct by construction — no post-hoc human judgment needed.

Positive Listing: Cabin with fireplace, ski resort nearby
Negative Listing: Beach house, no fireplace

Generated Query: "cozy cabin near ski resort with fireplace"

Relevance Label: Positive listing ✓ (automatically correct)

No circularity: What counts as positive/negative comes from user booking data, not from the LLM itself.

---

Step 4: How CASTLE Generates Diverse, Realistic Queries

Three prompt variants work together:

Variant 1: Template-Based Prompts

• Ensures structural diversity across common query patterns
• Prevents the model from always generating the same sentence structure

Variant 2: Few-Shot Seed Prompts

• Uses ~500 real queries collected from user research
• Grounds generated language in authentic guest phrasing

Variant 3: Attribute-Grounded Prompts

• Focuses on specific differences between the two listings
• Example: "this one has a fireplace, that one doesn't" → query mentions fireplace

Anti-Collapse Mechanism

Without intervention, LLMs tend toward "query collapse" — converging on the same short phrase repeatedly.

CASTLE explicitly instructs the model to vary length and phrasing.

Result:

CASTLE nearly perfectly matches real user query length variation. Baselines are far off.

---

Step 5: Handling Edge Cases — Flexible Input Design

Problem: New Listings Have No Booking History

No booking history → no contrastive pairs → CASTLE can't apply its core mechanism.

Solution: Virtual Judge (VJ)

• An LLM evaluator that assigns relevance labels to additional query–listing pairs
• Used specifically for long-tail, newly listed properties
• Maintains quality of the core training set while extending coverage

CASTLE-R: For When Real Traffic Exists

• A variant called CASTLE-R incorporates real user queries as seeds
• Designed for production settings where some traffic has already accumulated

---

Step 6: Validation — Does It Actually Work?

Test 1: Distributional Alignment (KL Divergence)

Test 2: Label Quality (Human Evaluation)

• 200 sampled triplets reviewed by 3 independent experts
• Agreement with CASTLE's relevance assignments: 91–93%
• Achieved with zero human annotation in the training loop

---

Step 7: Beyond Cold-Start — Why CASTLE Keeps Providing Value

Even after real traffic accumulates, CASTLE remains useful because:

Long-Tail Coverage

Real traffic naturally underrepresents rare searches like:

• Unusual property types
• Rare amenity combinations

CASTLE's generation is conditioned on listing characteristics, so it can deliberately oversample these rare but important intents.

Think of it as filling gaps that organic traffic will never fill adequately.

---

Summary: The Key Insight

Authentic Booking Behavior  +  LLM Generation
        ↓                              ↓
  Contrastive pairs              Query diversity &
  (real preference signal)       linguistic variety
  + Relevance labels                    ↓
  by construction                Realistic phrasing
        ↓                              ↓
        └──────────────┬───────────────┘
                       ↓
            Closes the gap between
          synthetic and real queries

---

Quick Self-Check Questions

1. What are the two things you need to train a natural language search model, and why are both hard to get at cold-start?
2. Why does CASTLE's relevance label approach avoid circularity?
3. What does a lower KL divergence score mean, and how does CASTLE compare to baselines?
4. What is "query collapse" and how does CASTLE prevent it?
5. Why does CASTLE remain useful even after real traffic accumulates?