Step-by-Step Study Guide

STEP 1: Understanding the Foundation — What Is Local AI?

The Core Concept

Local AI means running artificial intelligence models directly on your own hardware rather than sending data to remote cloud servers.

Cloud AI:          Your Device → Internet → Remote Server → Response
Local AI:          Your Device → Local GPU → Response

Why It Matters

Key Takeaway

Local AI enables always-on agents that process sensitive data, credentials, and private files without relying on external services.

---

STEP 2: Understanding AI Model Specifications

Before exploring specific models, you need to understand how to read model specifications.

Critical Terms Explained

Parameters

• Think of parameters as the "knowledge capacity" of a model
• More parameters = generally more capable but requires more memory
• Example scale:

4B parameters   → Runs on edge devices (phones, small computers)
30B parameters  → Runs on consumer GPUs (RTX 5090)
100B+ parameters → Requires multiple GPUs or specialized hardware

Mixture of Experts (MoE)

• A model architecture where only a fraction of parameters activate per task
• Benefit: Large knowledge base with lower computational cost

Example: DeepSeek-V4-Flash
Total parameters:  284 billion
Active per token:  13 billion  ← Only this portion works at once
Result: Frontier intelligence at fraction of the compute cost

Context Window

• How much text the model can "remember" and process at once
• Measured in tokens (roughly ¾ of a word per token)

1 million token context = approximately 750,000 words
                        = roughly 10 full novels simultaneously

Quantization (GGUF, NVFP4, BF16)

• Process of compressing model weights to use less memory
• Trade-off: Slightly reduced precision for dramatically lower hardware requirements

Original Model:  100% accuracy, 100% memory usage
NVFP4 Version:   ~99% accuracy,  ~50% memory usage  ← Practical choice
GGUF Version:    ~97% accuracy,  ~40% memory usage  ← Maximum accessibility

---

STEP 3: Mapping Models to Use Cases

Category 1 — Coding and Agentic Tasks

Meta Muse Glimmer (30B)

• Purpose: Always-on local coding and agentic AI
• Key capability: 200+ tokens per second on RTX 5090
• Runs on: Single consumer GPU

Poolside Laguna S 2.1 (118B)

• Purpose: Extended coding sessions (hours-long tasks)
• Key capability: Works through complex, multi-step problems
• Runs on: Single NVIDIA DGX Spark with NVFP4 checkpoint

NVIDIA Nemotron 3.5 Lightning (30B MoE)

• Purpose: Fast, specialized agentic tasks
• Key capability: 4x faster token generation than comparable models
• Runs on: RTX PCs, DGX Spark, Jetson devices

Category 2 — Video and Creative Generation

LTX-2.5

• Purpose: High-quality video generation
• Key capability: Multishot support, maintains continuity across cuts
• Performance: 20% faster, 40% memory savings on RTX 6000 PRO

MiniMax-H3 (33B)

• Purpose: Video + synchronized stereo audio generation
• Input types: Text, images, video, audio, or combinations

Wan-Animate-2 (14B)

• Purpose: Motion transfer from video to static images
• Performance: Up to 26x faster on RTX 5090

Category 3 — Robotics and Vision

NVIDIA Cosmos 3 Edge (4B)

• Purpose: Robotics, autonomous vehicles, vision AI
• Key advantage: Runs on edge devices (Jetson, DGX Spark)
• Size advantage: Quarter the size of Cosmos 3 Nano

Category 4 — Multimodal Reasoning

Thinking Machines Inkling-Small (276B)

• Purpose: Frontier-class reasoning across text, images, and audio
• Architecture: Activates only 12B parameters per token (MoE principle)
• Runs on: Single DGX Station or two DGX Spark systems

---

STEP 4: Understanding the Hardware Ecosystem

Hardware Hierarchy

INCREASING POWER AND COST
↑
│  Data Centers / Cloud
│  ─────────────────────────────────────────
│  DGX Station (GB300)     → 276B+ models
│  ─────────────────────────────────────────
│  DGX Spark               → 30B-118B models
│  ─────────────────────────────────────────
│  RTX 5090 / RTX PRO      → 30B models
│  ─────────────────────────────────────────
│  RTX Consumer GPUs       → Up to 30B models
│  ─────────────────────────────────────────
↓  Jetson / Edge Devices   → 4B models

Scaling with NVIDIA Sync

When a single system isn't enough:

Single DGX Spark → Limited to certain model sizes
         +
NVIDIA Sync Cluster Assistant
         ↓
Multiple DGX Sparks → Connected via ConnectX-7 ports
                    → Automatic network configuration
                    → Larger models become accessible
                    → Shared memory and compute

---

STEP 5: Understanding Agentic AI Workflows

What Makes an AI "Agentic"?

An agent doesn't just answer questions — it takes actions over time to complete complex goals.

Agent Capability Requirements

Simple Chatbot:          Single question → Single answer
                         
Agentic AI:              Goal → Plan → Step 1 → Tool Use → 
                         Step 2 → Error Recovery → Step 3 → 
                         ... → Final Result (hours later)

Key Agent Features in These Models

1. Tool Use Models can call external functions, APIs, and applications

2. Long Context Maintains memory across many steps (120K+ tokens in Muse Glimmer)

3. Error Recovery Detects unexpected results and adjusts approach

4. Session Resumption Tracks progress and continues interrupted workflows

Practical Agent Examples

---

STEP 6: Cost Optimization with Model Routing

The Problem

Running frontier models for every task is expensive and slow.

The Solution — NeMo Switchyard

Agent Workflow Step → NeMo Switchyard → Evaluates: Accuracy needed?
                                                    Speed required?
                                                    Cost budget?
                                        
                                      → Routes to BEST-FIT model

Simple task  → Small, fast, cheap model
Complex task → Large, capable, expensive model

Real-World Impact

Without routing:  All tasks → Opus 4.8 → 100% cost baseline
With routing:     Tasks distributed → ~33% of original cost
                  While maintaining frontier-level completion

---

STEP 7: Fine-Tuning for Personalization

What Is Fine-Tuning?

Taking an existing model and training it further on your specific examples to improve performance on your particular tasks.

Fine-Tuning Use Cases

Base Model (Nemotron 3.5 Lightning)
         ↓
Your Training Examples
         ↓
Fine-Tuned Model that can:

✓ Write in YOUR preferred style
✓ Understand YOUR domain terminology  
✓ Follow YOUR coding conventions
✓ Handle YOUR specific workflows

Tools for Local Fine-Tuning

• NVIDIA NeMo Automodel — Fine-tune Muse Glimmer locally
• Unsloth Desktop — First desktop app combining training AND inference
• Unsloth Studio — Optimized quantized models for efficient deployment

---

STEP 8: Deployment Frameworks

Choosing the Right Inference Framework

---

Concept Summary Map

LOCAL AI ECOSYSTEM
│
├── MODELS (by capability)
│   ├── Edge (4B): Cosmos 3 Edge
│   ├── Consumer (30-33B): Muse Glimmer, Nemotron 3.5, MiniMax-H3
│   ├── Workstation (100-120B): Laguna S 2.1
│   └── Multi-system (276-284B): Inkling-Small, DeepSeek V4 Flash
│
├── HARDWARE (by scale)
│   ├── Jetson → Edge deployment
│   ├── RTX GPUs → Consumer/prosumer
│   ├── DGX Spark → Developer workstation
│   └── DGX Station → High-end local
│
├── OPTIMIZATION TECHNIQUES
│   ├── Quantization (NVFP4, GGUF) → Less memory needed
│   ├── MoE Architecture → Fewer active parameters
│   └── Clustering (NVIDIA Sync) → Scale across systems
│
└── WORKFLOWS
    ├── Inference → Run models locally
    ├── Fine-tuning → Customize for your needs
    ├── Agentic tasks → Multi-step autonomous work
    └── Cost routing → NeMo Switchyard optimization

---

Quick Knowledge Check

Test your understanding with these questions:

1. Why would a developer choose GGUF format over BF16? (Answer: Lower memory requirements at slight accuracy trade-off)

2. How does MoE architecture make large models practical locally? (Answer: Only a fraction of parameters activate per token, reducing compute needs)

3. What problem does NeMo Switchyard solve? (Answer: Routes tasks to appropriately-sized models, reducing cost while maintaining quality)

4. What makes Muse Glimmer suitable for "always-on" agents? (Answer: Dense architecture with hybrid attention keeps memory manageable across long tasks; runs on single consumer GPU)

5. How does NVIDIA Sync enable running larger models? (Answer: Clusters multiple DGX Spark systems together, combining their memory and compute)