Imagine a delivery robot navigating an outdoor environment. Its operator has trained it to prefer certain terrains (e.g., paved sidewalks over muddy grass) for safety and efficiency.
Operator Preference Example:
✅ Concrete path → Safe, preferred
⚠️ Gravel → Acceptable
❌ Mud/wet grass → Avoid
The robot encounters:
🔑 Key Problem: The robot's visual system sees something unfamiliar. How does it know whether the operator would approve or disapprove of traversing it?
| Approach | Method | Problem |
|---|---|---|
| Manual Retraining | Collect new data, relabel | Labor-intensive, not scalable |
| Handcoded Rewards | Engineer reward functions | May not align with actual operator preferences |
| Pure Visual Systems | Camera-based classification | Fails under lighting changes or novel appearances |
🔑 Core Insight Gap: Visual appearance is fragile. The same terrain can look completely different under different lighting. Existing methods don't handle this gracefully.
This is the central intellectual contribution of the paper.
"Even if a terrain looks visually novel, how it feels to traverse it remains consistent."
Consider walking on gravel:
Same Gravel Terrain:
Noon sunlight → Looks bright/washed out ─┐
Overcast → Looks grey/dull ─┤→ But FEELS the same!
Dusk → Looks dark/shadowy ─┘
| Modality | What It Captures | Example Signals |
|---|---|---|
| Inertial | Motion/vibration patterns | IMU accelerometer, gyroscope |
| Proprioceptive | Robot's internal state | Joint torques, wheel speeds |
| Tactile | Physical contact properties | Terrain texture, resistance |
PATERN = Preference extrApolation for Terrain awarE Robot Navigation
┌─────────────────────────────────────────────────────┐
│ PATERN Pipeline │
│ │
│ Robot Sensors │
│ [IMU + Proprioception + Tactile] │
│ │ │
│ ▼ │
│ ┌─────────────────┐ │
│ │ Encoder Network │ ← Learns shared │
│ │ (Learned │ representation │
│ │ Representation)│ │
│ └────────┬────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────┐ ┌──────────────────────┐ │
│ │ Representation │ │ Reference Database │ │
│ │ Space (Embedding│◄───│ (Known terrains + │ │
│ │ Vector) │ │ operator labels) │ │
│ └────────┬────────┘ └──────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────┐ │
│ │ K-Nearest │ ← "What does this │
│ │ Neighbor Search │ FEEL like?" │
│ └────────┬────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────┐ │
│ │ Preference │ → Used for path planning │
│ │ Estimation │ │
│ └─────────────────┘ │
└─────────────────────────────────────────────────────┘
Representation Space (conceptual):
[Mud cluster] ●●●
●●● [Gravel cluster]
[Grass cluster] ●●
●●● [Concrete cluster]
Database Entry Example:
Terrain: "Gravel path (experienced before)"
Embedding: [0.23, -0.41, 0.87, ...]
Operator Preference Score: 0.6 (acceptable)
🔑 Why KNN? It's non-parametric — no retraining needed for new terrains. Just search!
Novel Terrain Encountered:
New embedding: [0.21, -0.38, 0.91, ...]
Nearest neighbors found:
→ Gravel (similarity: 0.95) → preference: 0.6
→ Packed dirt (similarity: 0.89) → preference: 0.5
Estimated preference for novel terrain: ~0.55
Once preferences are estimated, they feed into the navigation/path planning module:
Preference Score → Cost Map → Path Planner → Robot Motion
High Preference → Low Cost → Robot prefers this path
Low Preference → High Cost → Robot avoids this path
This ensures the robot navigates in a preference-aligned manner — doing what the operator would want, even on terrains never explicitly labeled.
Visual System: Tactile/Inertial System:
Noon → [bright img] Noon → [vibration pattern A]
Dusk → [dark img] Dusk → [vibration pattern A] ← SAME!
↑ Different ↑ Consistent
The authors validated PATERN through physical robot experiments outdoors:
| Test Condition | What Was Evaluated |
|---|---|
| Novel terrains | Can robot correctly estimate preferences for unseen terrain types? |
| Lighting variations | Does performance hold at dawn/dusk/shadows? |
| Comparison to baselines | Does PATERN outperform manual labeling / handcoded reward approaches? |
PATERN robustly generalizes to diverse terrains and varied lighting while maintaining preference alignment — outperforming all baseline approaches.
┌──────────────────────────────────────────────────────────┐
│ PATERN in One Slide │
│ │
│ PROBLEM: Robot sees novel/differently-lit terrain │
│ → Doesn't know operator preference │
│ │
│ INSIGHT: Physical feel of terrain is lighting-invariant │
│ and consistent across appearances │
│ │
│ SOLUTION: Learn embedding from inertial/proprioceptive/ │
│ tactile data → KNN search → Extrapolate │
│ operator preferences │
│ │
│ RESULT: Preference-aligned navigation that generalizes │
│ to novel terrains and lighting conditions │
└──────────────────────────────────────────────────────────┘