After studying this material, you should be able to:
Traditional Neural Networks:
┌─────────────────────────┐
│ Load ENTIRE image │ ← Requires HUGE memory
│ Process all at once │
└─────────────────────────┘
On-Chip Reality:
┌─────────────────────────┐
│ Limited Memory (SRAM) │ ← Cannot store full image
│ Must process smartly │
└─────────────────────────┘
Why does this matter?
A line buffer is a memory structure that stores only a few rows (lines) of an image at a time.
Full Image (too large to store):
Row 1: [pixel][pixel][pixel]...[pixel]
Row 2: [pixel][pixel][pixel]...[pixel]
Row 3: [pixel][pixel][pixel]...[pixel]
...
Row N: [pixel][pixel][pixel]...[pixel]
Line Buffer (stores only what's needed):
┌────────────────────────────┐
│ Row k-2: [p][p][p]...[p] │
│ Row k-1: [p][p][p]...[p] │ ← Active window
│ Row k: [p][p][p]...[p] │
└────────────────────────────┘
Slides down ↓
Key Insight: Process image line by line, discarding what's no longer needed.
Image Grid:
←————— Intra-line (horizontal) —————→
↑ [p][p][p][p][p][p][p][p][p][p] Row k-2
| [p][p][p][p][p][p][p][p][p][p] Row k-1
Inter [p][p][p][p][p][p][p][p][p][p] Row k
line [p][p][p][p][p][p][p][p][p][p] Row k+1
↓ [p][p][p][p][p][p][p][p][p][p] Row k+2
| Correlation Type | Direction | Meaning |
|---|---|---|
| Intra-line | Horizontal (within a row) | Pixels in the same line depend on each other |
| Inter-line | Vertical (across rows) | Pixels in different rows depend on each other |
Example: Denoising a pixel X
[a][b][c]
[d][X][e] ← X needs context from ALL neighbors
[f][g][h]
Intra-line: b, d, X, e, g (same or adjacent horizontal)
Inter-line: a, b, c, f, g, h (rows above and below)
A strip convolution operates on a narrow horizontal band of pixels.
Standard 3×3 Convolution: Strip Convolution (1×5):
[■][■][■] [ ][ ][ ][ ][ ]
[■][X][■] ← 2D kernel [■][■][X][■][■] ← Horizontal strip
[■][■][■] [ ][ ][ ][ ][ ]
The key idea: start narrow, expand gradually
Stage 1 — Small strip:
[■][X][■] Receptive field: 3 pixels
Stage 2 — Wider strip:
[■][■][X][■][■] Receptive field: 5 pixels
Stage 3 — Even wider:
[■][■][■][X][■][■][■] Receptive field: 7 pixels
Why progressive?
Pipeline:
Raw Line → [Stage 1] → [Stage 2] → [Stage 3] → Enhanced Features
↑ ↑ ↑
narrow medium wide
strip strip strip
Store ALL previous lines:
Line 1 features: [f1][f1][f1]...[f1] ← Still in memory
Line 2 features: [f2][f2][f2]...[f2] ← Still in memory
Line 3 features: [f3][f3][f3]...[f3] ← Still in memory
...
Line k features: [fk][fk][fk]...[fk] ← Current
MEMORY EXPLODES!
Core Idea: Compress and summarize older lines into progressively compact representations.
Hierarchical Levels:
Level 0 (Most Recent — Full Detail):
[Line k-1 features: full resolution]
Level 1 (Older — Compressed):
[Lines k-3 to k-2: compressed ↓2×]
Level 2 (Even Older — More Compressed):
[Lines k-7 to k-4: compressed ↓4×]
Level 3 (Oldest — Highly Compressed):
[Lines k-15 to k-8: compressed ↓8×]
Visual Analogy:
Think of it like human memory:
Yesterday → Remember in DETAIL (Level 0)
Last week → Remember key events (Level 1)
Last month → Remember rough summary (Level 2)
Last year → Remember only highlights (Level 3)
As new line arrives:
┌──────────────────────────────┐
New Line k ────────►│ Level 0 Buffer │
│ [Full features of line k-1] │
└──────────┬───────────────────┘
│ Compress (↓2)
┌──────────▼───────────────────┐
│ Level 1 Buffer │
│ [Compressed older lines] │
└──────────┬───────────────────┘
│ Compress (↓2)
┌──────────▼───────────────────┐
│ Level 2 Buffer │
│ [More compressed lines] │
└──────────┬───────────────────┘
│ Compress (↓2)
┌──────────▼───────────────────┐
│ Level 3 Buffer │
│ [Highly compressed history] │
└──────────────────────────────┘
Features from all levels are combined when processing the current line:
Current Line Processing:
┌─────────────────────────────────┐
Current features ──►│ │
Level 0 features ──►│ Feature Fusion & Processing │──► Output
Level 1 features ──►│ │
Level 2 features ──►│ │
Level 3 features ──►│ │
└─────────────────────────────────┘
Method | Memory Usage | Performance
--------------------|--------------|------------
Full Image Buffer | 100% (base) | Best
Naive Line Buffer | High | Good
Proposed Hierarchical| ~20% (1/5) | Near-best
Traditional approach stores N lines at full resolution:
Memory = N × W × C (lines × width × channels)
Hierarchical approach:
Level 0: 1 line × W × C = W×C
Level 1: 1 line × W/2 × C = W×C/2
Level 2: 1 line × W/4 × C = W×C/4
Level 3: 1 line × W/8 × C = W×C/8
Total ≈ W×C × (1 + 0.5 + 0.25 + 0.125)
= W×C × 1.875 ← Much less than N×W×C!
| Task | What it does | Why memory matters |
|---|---|---|
| RAW Denoising | Remove sensor noise from raw images | Real-time camera pipeline |
| Gaussian Denoising | Remove artificial Gaussian noise | General image restoration |
| Super-Resolution | Increase image resolution | Display enhancement |
RAW Denoising Comparison:
Previous Best Method:
├── Memory: 100% (full)
└── PSNR: X dB
Proposed Method:
├── Memory: 20% (1/5 of previous!)
└── PSNR: X + 1 dB (BETTER performance!)
Performance ↑ AND Memory ↓ ← This is the breakthrough!
PSNR (Peak Signal-to-Noise Ratio): Higher = better image quality. A 1 dB gain is considered significant in image processing.
┌─────────────────────────────────────────────────────────┐
│ IMAGE SENSOR │
│ (outputs line by line) │
└─────────────────────────┬───────────────────────────────┘
│ One line at a time
▼
┌─────────────────────────────────────────────────────────┐
│ INTRA-LINE PROCESSING │
│ Progressive Strip Convolutions (Stage 1→2→3) │
│ Captures horizontal pixel relationships │
└─────────────────────────┬───────────────────────────────┘
│ Enhanced line features
▼
┌─────────────────────────────────────────────────────────┐
│ INTER-LINE PROCESSING │
│ Hierarchical Line Buffer │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Level 0 │ │ Level 1 │ │ Level 2 │ │ Level 3 │ │
│ │ (recent) │ │(compress)│ │(compress)│ │(compress)│ │
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
│ Captures vertical pixel relationships │
└─────────────────────────┬───────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────┐
│ PROCESSED OUTPUT │
│ (Denoised / Super-resolved image) │
└─────────────────────────────────────────────────────────┘
| Concept | Key Point |
|---|---|
| Line Buffer | Process image row-by-row to save memory |
| Intra-line correlation | Horizontal dependencies within a row |
| Inter-line correlation | Vertical dependencies across rows |
| Strip Convolution | Horizontal-only kernel for efficient intra-line processing |
| Progressive Enhancement | Gradually widen receptive field across stages |
| Hierarchical Buffer | Compress older line features to save memory |
| Trade-off achieved | 5× less memory + 1 dB better quality |
💡 Core Insight: The entire method is about being smart with what you remember — keeping recent information in full detail and older information in compressed summaries, just like efficient human memory — while still capturing all the spatial relationships needed for high-quality image processing.