New tools enhance AI collaboration through annotation and verification for improved workflows.

Concept 1: What is a Perception Agent?

The core idea: A perception agent is an AI that can see and interpret visual interfaces — like screens, web pages, or documents — the way a human does.

What makes it different from regular AI:

Practical example: A perception agent can notice that:

• A button is 2 pixels off-center
• A border radius doesn't match the design spec
• A navigation flow breaks on the third click due to a silent state failure

Key insight: Seeing is not enough on its own. The agent also needs ways to act on what it sees and communicate with humans about it.

---

Concept 2: What is a "Harness"?

Simple definition: A harness is a structured set of tools and interaction patterns built around the perception agent to make it more useful and collaborative.

Think of it like this:

• The perception agent = the engine
• The harness = the controls, dashboard, and communication system around that engine

The harness being built here has two first primitives:

1. Annotation
2. Verification

---

Concept 3: What is a "Primitive"?

Simple definition: A primitive is a basic, reusable building block — a fundamental tool that can be combined with others to build more complex workflows.

Why it matters: Rather than building one large complex tool, the team is releasing small, focused capabilities that anyone can build on top of.

Think of primitives like LEGO bricks — simple individually, powerful when combined.

---

Concept 4: The Annotation Primitive

The problem it solves:

When you and an agent are looking at the same screen, today you have to type a paragraph to explain what you see. This is slow and loses context.

Example of the old way:

"The button in the top right corner of the navigation bar, the one that says 'Submit', seems to be slightly misaligned and the border doesn't match the design..."

Example of the new way:

✏️ Draw a circle around the button directly on the screen.

---

How annotation works technically:

On web pages:

• You click an element
• The tool automatically captures:
  • DOM selector — the code identity of the element
  • Bounding box — its exact position and size on screen
  • Computed styles — its visual properties (color, font, spacing, etc.)

On visual surfaces (documents, diagrams):

---

The output:

Your annotation is saved as a structured artifact — a precise, machine-readable piece of feedback the agent can act on directly.

Key insight: The feedback itself becomes the context. Nothing is lost in translation.

---

Collaborative by design:

Annotation is not just a 1-to-1 loop between you and the agent.

Anyone on a team can annotate:

• A designer circles a layout issue
• A QA engineer captures a broken flow
• A developer marks what needs refactoring

That structured feedback can be sent to an agent or another human — replacing tickets and messages that lose context.

---

Concept 5: The Verification Primitive

The problem it solves — "The Vibe Coding Paradox":

With large language models, generation is nearly instant. You can get a full application in 60 seconds.

But validation is still entirely manual and slow. You then spend 3 hours checking if it actually works.

This is the paradox:

Fast generation + Slow validation = No net time saved

---

The key concept: Shared Perception

Definition: Shared perception is the condition where both a human and an agent observe the same visual output and can each reason about its contents.

Why this matters: When the agent can see what it built — just like a human reviewer would — generation and validation can become one continuous loop instead of two separate phases.

Old model:          Build → [stop] → Manually test → [stop] → Fix
New model:          Build → Auto-verify → Fix → Auto-verify → Done

---

How verification works technically:

Step 1 — Trigger: After code generation, the agent invokes the Nova Act Visual Verifier skill

Step 2 — Render: The skill spins up the rendered application (the actual visual output)

Step 3 — Two layers of checks:

Important detail: Deterministic checks run first and don't require AI — they are fast, reliable, and rule-based. AI-driven behavioral checks follow for deeper functional testing.

---

Concept 6: How Annotation and Verification Work Together

These two primitives address two sides of the same problem:

Together they create a tighter feedback loop:

Human annotates issue
        ↓
Agent receives structured feedback
        ↓
Agent makes changes
        ↓
Agent verifies its own output
        ↓
Human reviews verified result
        ↓
(repeat as needed)

---

Concept 7: Why Open Source?

The team is releasing these tools publicly because:

"These interaction patterns get better with more people using them, breaking them, and building on top of them."

The open-source strategy:

• More users = more edge cases discovered
• More builders = more primitives contributed
• Community feedback shapes which primitives come next

---

Summary: The Full Picture

PERCEPTION AGENT
(sees visual interfaces like a human)
            |
            | needs structured ways to
            | communicate and validate
            ↓
PERCEPTION AGENT HARNESS
            |
    ________|________
    |               |
ANNOTATION      VERIFICATION
(Human → Agent) (Agent → Self/Human)
Point, draw,    Check rendered output
describe what   against intent,
needs changing  automatically

The big idea in one sentence:

Perception agents become truly useful not just by seeing, but by having structured primitives that let humans and agents communicate precisely and validate continuously.