What they are: Diffusion models are a type of AI that learns to generate images or videos by learning how to reverse a "noising" process.
Simple analogy:
Imagine taking a clear photo and slowly adding static/noise until it's unrecognizable. A diffusion model learns to reverse that process β starting from pure noise and gradually recovering a clean image.
Why it matters here: Many papers (BAgger, BulletTime, Choreographing, Dual Ascent) are built on top of diffusion models.
What it means: Generating content one piece at a time, where each new piece depends on what came before.
Simple analogy:
Like writing a story word by word β each word is influenced by all previous words.
In video: Instead of words, you generate frames one after another.
Problem this creates: Called Exposure Bias or Drift β small errors in early frames compound over time, making later frames look wrong.
π BAgger directly addresses this drift problem in autoregressive video diffusion.
What it is: When a model is trained on perfect data but at test time must use its own (imperfect) outputs as inputs, errors accumulate.
Analogy:
A GPS trained on perfect maps but given slightly wrong location data will give increasingly wrong directions the longer you drive.
BAgger's solution β Backwards Aggregation: Instead of only looking forward, the model also looks backwards at previously generated frames to self-correct drift.
What they are: Diffusion models extended to generate entire video sequences, not just single images.
Extra challenges vs. image generation:
π Relevant to: BAgger, BulletTime, Choreographing, Stand-In, Generated Reality
What it means: Rather than generating random videos, you want specific control over what happens β like controlling:
π BulletTime focuses on controlling both time and camera pose independently.
What it means: "Camera pose" = the position and orientation of the camera in 3D space.
Camera trajectory = the path the camera takes over time.
Why it's hard: Video generators naturally mix together what's happening in the scene and how the camera moves β separating these is non-trivial.
BulletTime's approach: "Decoupled control" β control time (what's happening) and camera pose (viewpoint) as separate, independent inputs.
"Bullet Time" reference: Named after the famous Matrix movie effect β freezing time while the camera rotates around a scene.
What "4D" means:
So 4D generation means creating content that is spatially 3D AND changes over time.
π Relevant to: BulletTime, Choreographing a World of Dynamic Objects
What it is: A technique to use a pretrained diffusion model as a "teacher" to guide the creation of 3D or 4D content.
How it works (simplified):
Analogy:
Like having an art critic (diffusion model) give feedback on your sculpture (3D object) from many angles until it looks photorealistic.
π Used in Choreographing a World of Dynamic Objects
What it means: Ensuring that a specific person's face/appearance is consistently maintained throughout a generated video.
Why it's hard: Video generators tend to drift in appearance β a person might look slightly different in each frame.
π Stand-In proposes a "lightweight and plug-and-play" solution β meaning it can be added to existing video generators without retraining them from scratch.
What they are: AI models that combine:
Analogy:
Like a robot that can read "pick up the red cup" (language), see the cup (vision), and then physically grab it (action).
π Relevant to: Ego-Pi, Scaling Verification
What it is: Teaching a robot by showing it examples of a task being performed, rather than programming rules explicitly.
Types:
Challenge: The robot must generalize β do the task in new situations it hasn't seen before.
π HoMMI learns whole-body mobile manipulation from human demonstrations
What it means: A robot that can move around (mobile) AND manipulate objects with its arms (manipulation) β doing both simultaneously.
Why it's hard:
π HoMMI addresses this with whole-body control
What it means: Instead of making a model better by training it more, you make it better by doing more computation at inference time (when it's actually being used).
Analogy:
Instead of studying more before an exam (training), you spend more time thinking through each question during the exam (test time).
In robotics: Spending more compute to verify whether an action is correct before executing it.
π Scaling Verification paper argues this is more effective than just training a better policy
What it is: A training technique where the model learns by comparing similar and dissimilar examples.
Core idea:
Analogy:
Teaching someone what a "chair" is by showing them many chairs (similar) and many non-chairs (dissimilar) side by side.
π Used in Scaling Verification for VLA alignment
What it is: A technique to represent a 3D scene as a collection of millions of tiny 3D "blobs" (Gaussians), each with:
Why it's popular: Much faster to render than traditional neural rendering methods, while still looking photorealistic.
Analogy:
Imagine describing a cloud not as a solid object but as millions of tiny transparent colored puffs overlapping each other.
π Used in GaussFusion for 3D reconstruction
What it means: Given photos of a scene from some angles, generate what the scene would look like from a completely new angle you've never photographed.
Analogy:
You have photos of a building from the front and sides. Novel view synthesis lets you generate what it looks like from the back or from above.
π Relevant to: GaussFusion
What it is: Using neural networks to render (generate images of) 3D scenes, rather than traditional computer graphics pipelines.
Advantage: Can capture complex lighting, reflections, and textures that are hard to model manually.
What they are: AI models that can process multiple types of input β typically combining:
Examples: GPT-4V, Gemini, LLaVA
π Downscaling Intelligence studies small multimodal models β exploring what capabilities are lost when you make them smaller
Perception: The ability to correctly understand what's in an image
Reasoning: The ability to draw conclusions from what's perceived
Why the distinction matters: A model might fail at a task because it didn't see correctly (perception failure) OR because it didn't think correctly (reasoning failure) β these require different fixes.
π Downscaling Intelligence explores which of these is the bottleneck in small models
What it means: The ability to understand and reason about positions, distances, and relationships between objects in space.
Examples:
Why it's hard for AI: Language models are trained on text, which is inherently non-spatial β space must be learned indirectly.
π Theory of Space tests whether foundation models can build spatial understanding through active exploration
What it means: Rather than passively receiving information, an agent actively chooses where to look or move to gather more useful information.
Analogy:
Instead of waiting for someone to show you a room, you walk around it yourself, looking in corners and opening doors.
In AI: A model that can decide "I need to move to the left to see what's behind that wall" rather than just describing what's already visible.
π Central to Theory of Space
What they are: Large pretrained models (like GPT but for images) that have learned general visual features and can be fine-tuned for specific medical tasks.
In brain MRI: Models trained on thousands of brain scans that learn general brain structure representations.
π GeoSAE studies these models for brain MRI analysis
What they are: Neural networks that learn to compress information into a representation where most values are zero (sparse) β only a few "features" are active at once.
Why useful for interpretability: Each active feature can potentially correspond to a meaningful concept (like "this region looks like a tumor").
Analogy:
Instead of describing a face with 1000 numbers, use only 5 numbers that each mean something specific: "nose size," "eye spacing," etc.
π GeoSAE uses SAEs to interpret what brain MRI foundation models have learned
What it means: Understanding how a neural network works internally β not just what outputs it produces, but which internal components are responsible for which behaviors.
Why it matters: In medical AI, you need to trust the model β understanding its reasoning helps catch errors and biases.
π Central to GeoSAE
What they are: Problems where you observe an output and want to recover the input that caused it.
Examples:
| Observed Output | Want to Recover |
|---|---|
| Blurry image | Sharp original |
| Noisy MRI scan | Clean scan |
| Partial image | Complete image |
Why they're hard: Many different inputs could produce the same output β the problem is underdetermined.
π Dual Ascent Diffusion uses diffusion models as "priors" to solve inverse problems
What they are: Using a pretrained diffusion model's knowledge of "what natural images look like" to constrain the solution to an inverse problem.
Analogy:
If you're trying to reconstruct a blurry photo of a face, a diffusion prior says "the answer should look like a realistic face" β ruling out solutions that are technically correct mathematically but look unnatural.
What it means: Finding the best solution to a problem while satisfying certain rules (constraints).
Example:
Dual Ascent is a specific mathematical technique for solving such constrained optimization problems.
What it means: Having an AI system that can understand the 3D structure of a scene β where objects are, how they relate spatially, what can be moved where.
Goes beyond: Just recognizing objects (2D) β understanding their 3D positions and relationships
π VULCAN uses this for 3D object arrangement
What they are: Multiple AI agents working together, each potentially with different roles or specializations.
Analogy:
Like a team of workers β one measures, one plans, one executes β rather than one person doing everything.
In VULCAN: Different agents handle different aspects of arranging 3D objects in a scene.
What it means: Giving an AI model access to external tools it can call β like calculators, search engines, or 3D rendering software.
Why useful: Some tasks are better done by specialized tools than by the neural network itself.
π VULCAN uses tool-augmented multi-agents for 3D arrangement
What it means: Converting images into discrete tokens (like words in a sentence) so they can be processed by language-model-style architectures.
Why needed: Transformers (the architecture behind GPT, etc.) work on sequences of tokens β images must be converted to this format.
Challenge: Images have continuous pixel values β converting to discrete tokens loses some information.
π Spherical Leech Quantization proposes a better way to do this
What it means: Reducing the precision of values β converting from many possible values to fewer possible values.
Example:
Trade-off: Less memory/computation β some loss of information
"Spherical" quantization: Arranges the quantization points on a sphere in high-dimensional space for more uniform coverage.
What they are: AI systems that build an internal model of how the world works β predicting what will happen next given current observations and actions.
Key capability: Can simulate the consequences of actions without actually performing them.
Analogy:
Like mental simulation β you can imagine "if I push this glass, it will fall off the table" without actually pushing it.
π Physical Object Understanding paper focuses on world models that understand physical properties of objects
What it means: Understanding not just what objects are, but their physical properties:
Why it matters for robotics: A robot needs to know that a glass is fragile and a ball will roll before it can interact with them safely.
π Physical Object Understanding paper received a Highlight nomination at CVPR
Here's how all concepts connect:
DIFFUSION MODELS (Concept 1)
βββ Autoregressive Generation β Drift Problem β BAgger
βββ Video Diffusion β Controllable Video β BulletTime
βββ Score Distillation β 4D Generation β Choreographing
βββ Diffusion Priors β Inverse Problems β Dual Ascent
βββ Visual Tokenization β Quantization β Spherical Leech
VISION-LANGUAGE MODELS
βββ Multimodal Models β Small Models β Downscaling Intelligence
βββ Spatial Reasoning β Active Exploration β Theory of Space
βββ VLA Models β Robot Learning β Ego-Pi, Scaling Verification
3D UNDERSTANDING
βββ Gaussian Splatting β Novel View Synthesis β GaussFusion
βββ 3D Scene Understanding β Multi-Agent β VULCAN
βββ World Models β Physical Understanding β Physical Object Understanding
ROBOTICS
βββ Imitation Learning β Mobile Manipulation β HoMMI
βββ Test-Time Scaling β Verification β Scaling Verification
MEDICAL AI
βββ Foundation Models β SAEs β Interpretability β GeoSAE
Would you like me to dive deeper into any specific concept, or explore the connections between specific papers in more detail?