How Uber Uses AI Agents to Automate Figma Specs

Peter Bubenik ยท Uber Data / ML ยท ยท Source
Image for How Uber Built an Agentic System to Automate Design Specs in Minutes

Step-by-Step Study Material

Step 1: Understanding the Problem First

Before learning the solution, you need to feel the weight of the problem.

What is a Design Spec?

A design spec is structured documentation that tells engineers exactly how to build a UI component. Think of it as a blueprint.

For a single component like a button, a complete spec includes:

SectionWhat It Covers
AnatomyEvery visual element labeled and described
APIEvery configurable property, its values, defaults
PropertiesVariant axes and toggle options
Color AnnotationDesign tokens mapped to each element and state
StructureHeights, padding, spacing across sizes
Screen ReaderAccessibility for VoiceOver, TalkBack, ARIA

๐Ÿ’ก Key Insight: That is 6 sections for one component, covering only the implementation side. The screen reader section alone references 3 separate accessibility APIs, each with hundreds of properties.

Why Does This Become a Crisis at Scale?

Let's do the math Uber faced:

Hundreds of components
ร— 7 implementation stacks (UIKit, SwiftUI, Android XML, 
  Android Compose, Web React, Go, SDUI)
ร— Multiple density variants per stack
ร— Strict accessibility requirements
= Thousands of spec pages that must stay accurate

The core problems with manual spec writing:

  • โŒ Slow โ€” weeks per component
  • โŒ Inconsistent โ€” quality varies by author
  • โŒ Drifts out of date โ€” the moment a component changes, the spec is already wrong
  • โŒ Doesn't scale โ€” documentation becomes a dedicated team that still can't keep up

Step 2: Understanding the Solution Architecture

Uber built uSpec โ€” an agentic system with three connected layers.

The Three Layers Visualized

โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚           YOU (Designer)                โ”‚
โ”‚   Provide: Figma link + context         โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
                   โ”‚
                   โ–ผ
โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚         AI AGENT (in Cursor)            โ”‚
โ”‚   Runs agent "skills" with domain       โ”‚
โ”‚   knowledge encoded per spec section    โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
                   โ”‚ local connection only
                   โ–ผ
โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚      Figma Console MCP (open-source)    โ”‚
โ”‚   Bridge between agent and Figma        โ”‚
โ”‚   Read/write access to your file        โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
                   โ”‚
                   โ–ผ
โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚         FIGMA DESKTOP (local)           โ”‚
โ”‚   Finished spec rendered directly       โ”‚
โ”‚   into your file                        โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜

๐Ÿ”’ Critical Detail: Everything runs locally. No design data leaves your machine. This is what makes it viable for an enterprise like Uber.


Step 3: Understanding Agent Skills

This is where the intelligence lives.

What is an Agent Skill?

An agent skill is a specialized instruction set that encodes domain expertise for one specific section of a spec.

Think of it like hiring a specialist instead of a generalist:

SkillWhat Domain Knowledge It Carries
AnatomyHow to identify and label component elements
APIHow to document properties, values, and defaults
PropertiesHow to represent variant axes and toggles
Color AnnotationHow to map design tokens to states
StructureHow to document dimensions and spacing
Screen ReaderVoiceOver modifiers, TalkBack semantics, ARIA roles

Why Skills Matter โ€” The Screen Reader Example

Without a skill, an AI agent would guess at accessibility property names.

With the screen reader skill:

  • The agent loads platform documentation first
  • It selects property names from documented APIs, not inference
  • It covers all 3 platforms (iOS, Android, Web) in a single pass
  • Result: under 2 minutes for a complete screen reader spec

๐Ÿ’ก Key Insight: The skill is what separates "AI that guesses" from "AI that knows." Domain knowledge is encoded, not improvised.


Step 4: Understanding the MCP Bridge

What is the Figma Console MCP?

MCP stands for Model Context Protocol โ€” it is the technical bridge that gives the AI agent direct access to your Figma file.

What Can the MCP Do That Screenshots Cannot?

This distinction is important. Many AI tools analyze Figma by taking screenshots. The MCP goes deeper:

CapabilityScreenshot AnalysisFigma Console MCP
Read token namesโŒ Guesses from visualsโœ… Reads exact token values
Traverse component treeโŒ Sees flat imageโœ… Crawls parent-child layers
Detect variable modesโŒ Cannot seeโœ… Reads directly
Identify sub-componentsโŒ Limitedโœ… Full structure access
Write back to FigmaโŒ Cannotโœ… Renders spec directly

What the MCP Extracts Per Skill

Different skills need different data, and the MCP is configured accordingly:

Color Annotation Skill  โ†’  Needs: token values, variable modes
Screen Reader Skill     โ†’  Needs: screenshots + component structure
API Skill               โ†’  Needs: component properties, variant axes
Structure Skill         โ†’  Needs: dimensions, spacing values

Step 5: Understanding the Full Workflow

Now that you understand the parts, here is the complete workflow reduced to its simplest form.

The 2-Step Process

Step 1 โ€” You provide input:

  • Open Cursor
  • Reference an agent skill
  • Paste your Figma component link
  • Add any context the agent cannot infer (specific states, platform behavior, edge cases)

Step 2 โ€” The agent does the rest:

Agent connects to Figma via MCP
        โ†“
Extracts component structure, tokens, variables, styles
        โ†“
Maps parent-child layer relationships
        โ†“
Loads skill-specific domain knowledge
        โ†“
Imports matching documentation template from your library
        โ†“
Detaches template, populates it:
  - Fills text fields
  - Clones sections
  - Builds tables
  - Places markers
        โ†“
Finished spec appears in your Figma file

๐Ÿ’ก Key Insight: There is no intermediate output. No export, no copy-paste, no formatting step. The agent reads the component and renders the spec in one pass.


Step 6: Understanding Why This Works at Enterprise Scale

Uber identified 6 properties that make this viable beyond a prototype:

The 6 Properties Explained

1. Security

  • Local WebSocket connection only
  • No cloud API calls
  • No proprietary data leaves the machine
  • Why it matters: Enterprise adoption requires this. Without it, the system cannot exist at Uber.

2. Consistency

  • Structured schemas and templates enforce uniform output
  • Quality does not depend on which designer runs it
  • Why it matters: Manual specs vary by author. Automated specs do not.

3. Speed

  • Full screen reader spec across 3 platforms: under 2 minutes
  • Hundreds of components: days instead of months
  • Why it matters: Speed enables keeping pace with component changes.

4. Accuracy

  • Agent reads real token names and real variant values from Figma
  • No transcription, no transcription errors
  • Why it matters: A wrong accessibility property breaks the experience for assistive technology users.

5. Multi-platform

  • One prompt generates iOS, Android, and Web specs simultaneously
  • Why it matters: Uber has 7 stacks. Generating each separately would eliminate the speed advantage.

6. Maintainability

  • Changelogs update directly in Figma via MCP with a single prompt
  • Why it matters: Specs that cannot be maintained will drift. Drift is the original problem.

Step 7: The Broader Significance

Design is Moving Closer to Code

The article points to a larger trend worth understanding:

Traditional separation:
  Designer โ†’ writes spec โ†’ Engineer reads spec โ†’ builds component

Emerging reality:
  Designer โ†” Code (designers submitting PRs, fixing issues in code)
  Spec โ†” Component (automation shrinking the gap between intent and implementation)

Why Open Source Matters Here

The Figma Console MCP is open source. This is a deliberate choice with a practical reason:

  • The documentation bottleneck is not unique to Uber
  • Every design systems team at scale faces the same problem
  • When infrastructure is shared, teams build on each other's work
  • uSpec (the application layer) could not exist without the MCP (the infrastructure layer)

๐Ÿ’ก Key Insight: The article distinguishes between infrastructure layer (Figma Console MCP โ€” shared, open source) and application layer (uSpec โ€” built on top). Understanding this separation helps you think about how to build on existing tools rather than rebuilding from scratch.


Summary: The Complete Mental Model

PROBLEM
  Manual spec writing is slow, inconsistent, and drifts out of date
  At enterprise scale this becomes unsustainable
        โ†“
SOLUTION COMPONENTS
  Agent Skills     = Domain knowledge encoded per spec section
  Figma MCP        = Local bridge giving agent read/write Figma access
  Cursor Agent     = Orchestrates skills and MCP calls
        โ†“
WORKFLOW
  You: Figma link + context
  Agent: reads โ†’ interprets โ†’ renders spec directly in Figma
        โ†“
RESULT
  Consistent, accurate, multi-platform specs
  Minutes instead of weeks
  Fully local, enterprise-safe
  Maintainable as components evolve

Check Your Understanding

Test yourself with these questions:

  1. Why is screenshot-based Figma analysis insufficient for generating accurate specs?
  2. What is the role of an agent skill, and why does it matter for the screen reader section specifically?
  3. What problem does the local-only architecture solve, and for what type of organization does this matter most?
  4. Why does consistency improve when specs are generated by an agent rather than written manually?
  5. What is the difference between the infrastructure layer and the application layer in this system?

More to study