Build Real-Time Document AI Apps and Agents

Peter Bubenik ยท Databricks AI ยท ยท Source
Image for Power real-time applications and agents with Document Intelligence

Step-by-Step Teaching Guide

Step 1: Understanding the Problem (The "Why")

The Core Challenge

Enterprises receive millions of documents daily โ€” invoices, contracts, reports, etc.

These documents are:

  • ๐Ÿ“„ Messy and unstructured (PDFs, scanned files, images)
  • ๐Ÿ’ผ Packed with critical business data (financial info, customer details, legal terms)
  • โณ Requiring manual reading before the data can be used

The Bottleneck Visualized

Document Arrives โ†’ Someone Reads It โ†’ Extracts Data โ†’ Business Decision
                        โ†‘
                   THIS IS SLOW
                   (and gets worse as volume grows)

Key Insight: The manual extraction step is the bottleneck. As document volume scales, costs and delays explode.


Step 2: The Solution โ€” Document Intelligence REST APIs

What Are They?

Databricks released real-time REST APIs that automatically:

  • Parse documents
  • Classify documents
  • Extract structured data from documents

Why REST APIs Specifically?

REST APIs allow developers to:

  • Call document processing from any application
  • Integrate into existing workflows
  • Process documents in real-time (not batch)
  • Power both human-facing apps and AI agents

Step 3: The Three Core APIs (+ 1 Beta)

Think of these as an assembly line for document understanding:

๐Ÿ”ง API #1: ai_parse_document

AttributeDetail
What it doesConverts raw files into structured content
InputRaw document (PDF, image, etc.)
OutputStructured, machine-readable content
AnalogyLike OCR but smarter โ€” it understands document structure

๐Ÿท๏ธ API #2: ai_classify

AttributeDetail
What it doesIdentifies the type of document
ExampleIs this an NDA, MSA, or Statement of Work?
OutputDocument category/label
AnalogyLike sorting mail into labeled folders automatically

๐Ÿ” API #3: ai_extract

AttributeDetail
What it doesPulls out specific fields from the document
ExampleParties involved, effective date, renewal terms
OutputStructured key-value data
Special FeatureCitations โ€” links each extracted value back to its source location
AnalogyLike a smart highlighter that finds exactly what you need

๐Ÿ”€ API #4: Decisions API (Beta)

AttributeDetail
What it doesRoutes documents to the right person/process
StatusCurrently in Beta
Use caseDocument routing and workflow decisions

Step 4: How the APIs Work Together โ€” A Real Example

Use Case: Contract Intake for a Law Firm

STEP 1: Lawyer uploads a contract
            โ†“
STEP 2: ai_parse_document
        โ†’ Converts PDF into structured content
            โ†“
STEP 3: ai_classify
        โ†’ "This is an NDA" (not an MSA or SOW)
        โ†’ App knows which review form to display
            โ†“
STEP 4: ai_extract
        โ†’ Pulls: parties, effective date, renewal terms, notice period
        โ†’ Provides citations so lawyer can verify each value
            โ†“
STEP 5: Review form opens with fields PRE-FILLED
        โ†’ Lawyer reviews and approves (human-in-the-loop)

Result: What used to take 30+ minutes of manual reading now happens in real-time


Step 5: Two Major Use Cases

Use Case Type 1: ๐Ÿ‘ค Human-in-the-Loop Applications

  • Built with Databricks Apps
  • Human reviews AI-extracted data
  • Example: Lawyer reviewing pre-filled contract form
  • Best for: High-stakes decisions requiring human verification

Use Case Type 2: ๐Ÿค– Autonomous AI Agents

  • Built with Agent Bricks
  • Agent calls APIs to understand documents before making decisions
  • No human required in the loop
  • Best for: High-volume, routine document processing
Human-in-the-Loop:  Document โ†’ APIs โ†’ Pre-filled Form โ†’ Human Reviews โ†’ Decision
Autonomous Agent:   Document โ†’ APIs โ†’ Agent Analyzes โ†’ Agent Decides โ†’ Action

Step 6: Key Concepts Summary

ConceptSimple Definition
Document IntelligenceUsing AI to automatically understand document content
REST APIA way for applications to request services over the internet
Real-time processingDocuments processed instantly as they arrive
Structured dataData organized in a predictable format (tables, key-value pairs)
CitationsReferences linking extracted data back to its source in the document
Human-in-the-loopWorkflow where a human reviews/approves AI output

Step 7: Quick Knowledge Check โœ…

Test yourself with these questions:

  1. What problem do Document Intelligence APIs solve?
  2. Which API would you use to determine if a document is an invoice vs. a purchase order?
  3. Which API would you use to pull out the total amount from an invoice?
  4. What is the difference between a human-in-the-loop app and an autonomous agent workflow?
  5. Why are citations an important feature of ai_extract?

Summary Diagram

                    DOCUMENT ARRIVES
                          โ†“
              โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
              โ”‚  ai_parse_document  โ”‚ โ†’ Structured Content
              โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
                          โ†“
              โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
              โ”‚    ai_classify      โ”‚ โ†’ Document Type
              โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
                          โ†“
              โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
              โ”‚    ai_extract       โ”‚ โ†’ Key Fields + Citations
              โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
                          โ†“
              โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
              โ†“                       โ†“
    Human Reviews App          AI Agent Acts
    (Databricks Apps)          (Agent Bricks)

Bottom Line: Document Intelligence APIs eliminate the manual bottleneck between document arrival and business action โ€” enabling faster, scalable, and more accurate document workflows.

More to study