After studying this material, you should be able to:
Imagine watching a 3-hour football match recording. You want a computer to automatically find:
This is Precise Event Spotting.
| Term | Meaning |
|---|---|
| Untrimmed video | A long, unedited video (e.g., full match recording) |
| Event | A specific action of interest (goal, card, etc.) |
| Spotting | Locating the exact timestamp of an event |
| Precise | Not just "somewhere in this 10-second clip" โ the exact moment |
Full Match Video (90+ minutes)
|-----------------------------------------------|
โ โ โ
Goal at Foul at Card at
23:14 67:02 78:45
โ Must find these exact moments among thousands of frames
โ Check your understanding: PES is harder than simply classifying a short clip because you must also locate the event in time within a very long video.
Events in sports don't happen in isolation. Context matters:
Frame 1000: Player receives ball
Frame 1050: Player dribbles
Frame 1100: Player shoots
Frame 1110: โ GOAL (the event to detect)
To correctly identify frame 1110 as a goal, the model needs to "remember" what happened in frames 1000โ1109.
A long-range dependency means the model needs information from far-away frames to understand the current frame.
Short-range: [Frame 1108] โ [Frame 1109] โ [Frame 1110] โ easy
Long-range: [Frame 500] โ ... โ [Frame 1110] โ hard
Most existing methods use convolutional neural networks (CNNs), which look at small local windows of frames. They are like reading a book one sentence at a time without remembering earlier chapters.
CNN Window: |--5 frames--| slides along video
โ Only sees nearby context
โ Misses long-range context
This module captures global context โ it can connect information from distant parts of the video.
Think of it like this:
Without long-range module:
[Frame 1] [Frame 2] ... [Frame 1000] [Frame 1001] โ isolated
With long-range module:
[Frame 1] โโโโโโโโโโโโโโโโโโโโโโโโโโโโ [Frame 1001]
All frames can "talk" to each other
๐ Key Insight: Technologies like Transformers (used in ChatGPT) are commonly used for this because they use attention mechanisms โ they let every frame attend to every other frame.
In a sports video, not all events happen equally often:
Event Distribution in a Football Match:
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
Background (no event): โโโโโโโโโโโโโโโโโโโโ 95%
Ball out of play: โโ 3%
Goal: โ 0.5%
Red card: โ 0.1%
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
If you train a model on this data, it learns:
"If I just predict 'background' every time, I'm right 95% of the time!"
This is called a lazy/biased model โ it ignores rare but important events.
Naive Model Prediction:
Frame 1: Background โ (correct, but trivial)
Frame 2: Background โ (correct, but trivial)
...
Frame 1110: Background โ (WRONG โ this was a goal!)
To understand SoftIC loss, we need to first understand what a loss function does.
A loss function measures how wrong the model is and guides it to improve.
Model Output โ Loss Function โ "You were this wrong" โ Model Updates
Contrastive learning teaches the model:
Feature Space (simplified to 2D):
Before training: After contrastive training:
G B G G G
B R B โ B B
G B G R
G=Goal, B=Background, R=Red Card
(scattered) (clustered by class)
Standard contrastive loss treats all negatives (wrong classes) equally. SoftIC is smarter:
Standard Contrastive:
"Goal" vs "Background" โ push apart (same force)
"Goal" vs "Red Card" โ push apart (same force)
SoftIC (Soft):
"Goal" vs "Background" โ push apart STRONGLY (very different)
"Goal" vs "Red Card" โ push apart GENTLY (both are rare events, somewhat similar)
๐ Key Insight: "Soft" means the degree of separation is proportional to how different the classes actually are. This is more nuanced and effective.
Without SoftIC:
Rare class (Goal) features โ scattered, confused with background
With SoftIC:
Rare class (Goal) features โ compact cluster, clearly separated
even with few training examples
Before classifying events, the model must convert raw video frames into meaningful numerical representations called features.
Raw Video Frames โ Feature Extractor โ Feature Vector โ Classifier โ Event Label
[pixels] [numbers]
Spatial: [Frame 1110] โ Where is the ball? Where is the goal?
Temporal: [Frame 1108โ1109โ1110] โ How fast is the ball moving?
Spatio-Temporal: Both together โ Ball moving fast toward goal = likely a shot!
Adaptive Spatio-Temporal Refinement Module (ASTRM) is a component that enhances the features extracted from video by adding rich spatio-temporal information.
Basic CNN Features: [spatial info only]
โ
ASTRM
โ
Enhanced Features: [spatial + temporal + adaptive refinement]
"Adaptive" means it adjusts its behavior based on the input โ it doesn't apply the same transformation to every video the same way.
Slow play scenario: ASTRM focuses more on spatial details
Fast action scenario: ASTRM focuses more on temporal motion
Existing methods often use features from large networks pre-trained on general tasks (like ImageNet โ classifying cats, dogs, etc.).
Pre-trained on: General images
Used for: Sports event spotting
Problem: Mismatch! General features โ Sports-specific features
The proposed method trains end-to-end โ meaning the feature extractor is trained specifically for PES:
End-to-End Training:
Raw Video โ [Feature Extractor + ASTRM + Temporal Module] โ Event Prediction
โ_____________________________________________โ
All trained together for PES
Result: Features optimized specifically for spotting sports events
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ COMPLETE PES PIPELINE โ
โ โ
โ Long Sports Video โ
โ โ โ
โ โผ โ
โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ
โ โ Convolutional Spatial- โ โ Extracts basic โ
โ โ Temporal Feature Extractor โ spatial+temporal โ
โ โโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโ features โ
โ โ โ
โ โผ โ
โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ
โ โ ASTRM โ โ Refines features โ
โ โ (Adaptive Spatio-Temporal โ adaptively โ
โ โ Refinement Module) โ โ
โ โโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโ โ
โ โ โ
โ โผ โ
โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ
โ โ Long-Range Temporal Module โ โ Captures global โ
โ โโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโ context โ
โ โ โ
โ โผ โ
โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ
โ โ Classifier + SoftIC Loss โ โ Handles class โ
โ โโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโ imbalance โ
โ โ โ
โ โผ โ
โ Event: "GOAL" at timestamp 23:14 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
Let's compare systematically:
| Challenge | Existing Methods | Proposed Method |
|---|---|---|
| Feature quality | Pre-trained general features | End-to-end trained, task-specific |
| Spatial-temporal info | Basic | Enhanced via ASTRM |
| Long-range context | Limited (local windows) | Full (long-range temporal module) |
| Class imbalance | Ignored or basic handling | SoftIC loss |
| Training | Partial (only classifier) | Full end-to-end |
PES = Find WHAT event + EXACTLY WHEN in long sports videos
Problem 1: Long-Range Dependency
โ Solution: Long-Range Temporal Module (global attention)
Problem 2: Class Imbalance
โ Solution: SoftIC Loss (soft contrastive learning)
Enhancement: ASTRM
โ Adaptively refines spatio-temporal features
Training: End-to-End
โ Everything optimized together for PES specifically
Test yourself before moving on:
What is the difference between "event detection" and "precise event spotting"?
Hint: Think about the word "precise" and what it implies about timing
Why would a model trained on imbalanced data tend to ignore rare events?
Hint: Think about what "being right most of the time" means when data is skewed
What does "end-to-end training" mean and why is it beneficial here?
Hint: Think about what gets optimized and for what purpose
In your own words, explain what contrastive learning teaches a model to do.
Hint: Think about "pulling" and "pushing" in feature space
Why is "adaptive" an important property of ASTRM?
Hint: Think about how different sports scenarios require different focus
If you want to deepen your understanding, explore these related topics: