Building an AI-Powered IT Knowledge Base Pipeline

Peter Bubenik · AWS ML · Aug 19, 2026 · Source

Image for KnowledgeForge: mining gold from the ITSM ticket graveyard

🎯 Learning Outcomes

After studying this material, you should be able to:

Explain the core problem KnowledgeForge solves (trapped ITSM knowledge + decaying knowledge bases)
Describe the two-subsystem architecture (Generation + Curation) and how they form a closed loop
Identify the AWS services used and justify why each was chosen
Understand key engineering patterns: RAG grounding, vector-based duplicate detection, distributed orchestration, quality scoring, and multi-tenant isolation
Apply these architectural patterns to design similar document-processing pipelines

📚 Step-by-Step Teaching Guide

STEP 1: Understand the Problem Before the Solution

Before touching any technology, you must clearly understand what problem exists.

The Two-Sided Problem

SIDE 1: The Ticket Graveyard          SIDE 2: The Messy Knowledge Base
─────────────────────────────         ────────────────────────────────
Thousands of resolved tickets    VS   Knowledge base exists but is:
each month, each containing:          • Full of duplicate articles
  • A symptom                         • Stale/outdated content
  • A root cause                      • Inconsistent quality
  • A working fix                     • Hard to search effectively

Result: Knowledge is TRAPPED          Result: Knowledge is UNUSABLE

Why This Matters

Think of it like a library where new books are locked in the basement (tickets), while the public shelves (knowledge base) are filled with outdated, duplicate copies of the same book.

Key insight: The resolutions already exist. The problem is access and organization, not missing information.

STEP 2: Understand the High-Level Architecture

KnowledgeForge has two subsystems that feed each other in a closed loop.

┌─────────────────────────────────────────────────────────┐
│                    CLOSED LOOP                          │
│                                                         │
│  Incident Tickets                                       │
│       │                                                 │
│       ▼                                                 │
│  ┌──────────────┐    vectors    ┌──────────────────┐   │
│  │  GENERATION  │◄──────────────│    CURATION      │   │
│  │              │               │                  │   │
│  │ Tickets →    │──new articles─►│ Classify         │   │
│  │ Articles     │               │ Deduplicate      │   │
│  └──────────────┘               │ Score Quality    │   │
│                                 │ Improve Content  │   │
│                                 └────────┬─────────┘   │
│                                          │              │
│                                          ▼              │
│                                  Knowledge Manager      │
│                                  Reviews & Approves     │
└─────────────────────────────────────────────────────────┘

The Five Stages an Article Travels Through

Stage	What Happens
1. Ingestion	Ticket clusters arrive as JSON in S3
2. Generation	AI writes draft articles from ticket clusters
3. Classification	Articles are sorted by type
4. Curation	Deduplication → Quality Scoring → Improvement
5. Review	Knowledge manager approves before publishing

STEP 3: Learn the Generation Subsystem

How Generation Works — Step by Step

Step 3.1: Input arrives

Resolved Tickets (grouped by theme)
         │
         ▼
    JSON file in S3
    (per customer, per theme)
    Contains:
    • Keywords
    • Article scope  
    • Sample ticket descriptions
    • Work notes from engineers

Step 3.2: The queue and compute layer

New S3 file
    │
    ▼
Amazon SQS Queue  ←── event notification
    │
    ▼
Amazon ECS + AWS Fargate container
(polls queue, processes up to 5 themes at once)

Why ECS + Fargate here?

Requirement	Why Fargate Fits
Work takes several minutes per theme	Long-running containers (not short Lambda functions)
Workload arrives in bursts	Scales task count up/down based on queue depth
Team wants to focus on logic, not servers	Serverless containers — no EC2 management

Step 3.3: RAG Grounding (Critical Concept)

Before writing anything, the system asks: "What already exists on this topic?"

Theme arrives
    │
    ▼
Query S3 Vectors index
"Find 5 most similar existing articles"
    │
    ▼
Pass those articles as CONTEXT to the AI model
    │
    ▼
AI generates new article GROUNDED in existing content

Why this matters: Without grounding, the AI might invent procedures or use inconsistent terminology. With grounding, new articles match the style and accuracy of proven content.

Step 3.4: What gets generated

For each ticket cluster/theme, the model produces two documents:

A Knowledge Base Article (the fix, step-by-step)
A Root Cause Analysis document (why it happened)

The system uses response streaming — it assembles the document as tokens arrive rather than waiting for the complete response. This reduces perceived latency on long documents.

STEP 4: Learn the Curation Subsystem

Curation handles all articles — both newly generated ones and existing knowledge base articles.

The Four Curation Steps

Article enters curation
        │
        ▼
┌───────────────────┐
│  1. CLASSIFY      │  What type of article is this?
└────────┬──────────┘
         │
         ▼
┌───────────────────┐
│  2. DEDUPLICATE   │  Does this already exist?
└────────┬──────────┘
         │
         ▼
┌───────────────────┐
│  3. SCORE QUALITY │  How good is this article?
└────────┬──────────┘
         │
         ▼
┌───────────────────┐
│  4. IMPROVE       │  If score is low, rewrite it
└────────┬──────────┘
         │
         ▼
    Published to ServiceNow
    for knowledge manager review

STEP 5: Deep Dive — Duplicate Detection with S3 Vectors

This is one of the most elegant patterns in the system.

Why Keyword Matching Fails

Article A: "Restart the network adapter to fix connectivity"
Article B: "To resolve connection issues, reboot the NIC"
Article C: "Network problems? Cycling the adapter usually works"

Keyword match: These look DIFFERENT
Meaning match: These are the SAME article

How Vector-Based Deduplication Works

Step 5.1: Every article gets converted to a vector

Article text
    │
    ▼
Amazon Titan Text Embeddings V2
    │
    ▼
1,024-dimension vector
(a list of 1,024 numbers representing the article's MEANING)
    │
    ▼
Stored in Amazon S3 Vectors index (per customer)

Step 5.2: New article is compared against existing vectors

New article → embed → query S3 Vectors index
                              │
                              ▼
                    Find top-5 nearest vectors
                              │
                              ▼
                    Cosine distance < 0.05?
                    (similarity > 95%?)
                         │         │
                        YES        NO
                         │         │
                    DUPLICATE    UNIQUE
                         │
                         ▼
                  Keep NEWER article,
                  retire OLDER one

Understanding Cosine Distance Threshold

Distance 0.0 = Identical articles
Distance 0.05 = 95%+ similar = DUPLICATE (flagged)
Distance 0.10 = Related but different = NOT duplicate
Distance 1.0 = Completely unrelated

Tuning insight: Too loose (0.10) → false duplicates flagged. Too tight (0.01) → reworded copies missed. The team sampled flagged pairs and adjusted until behavior was correct.

Why S3 Vectors Instead of a Dedicated Vector Database?

Factor	S3 Vectors	Standalone Vector DB
Infrastructure	None (uses existing S3)	New service to manage
Cost model	Per query + per GB	Per running node (always-on cost)
Integration	Already in S3 ecosystem	Requires new connections
Scale	Grows with your content	May need resizing

Bonus: The same index serves double duty — it's used for RAG grounding in Generation AND for duplicate detection in Curation.

STEP 6: Understand the Orchestration with Step Functions

Why Orchestration Is Needed

Curation processes tens of thousands of articles across many customers. You need:

Work distributed across multiple workers
Automatic retry on failures
State tracking across long-running processes
Protection against one bad article breaking everything

The Two-Phase Distributed Map

EventBridge (daily schedule)
        │
        ▼
Lambda: Find customers with new/changed articles
        │
        ▼
Group into batches → SQS FIFO Queue
        │
        ▼
Dispatcher: Pull one batch → Start Step Functions execution
        │
        ▼
┌───────────────────────────────────────┐
│         STEP FUNCTIONS STATE MACHINE  │
│                                       │
│  PHASE 1: Distributed Map             │
│  ┌─────────────────────────────────┐  │
│  │ For each article (concurrent):  │  │
│  │  • Classify type                │  │
│  │  • Deduplicate (sequential)     │  │
│  └─────────────────────────────────┘  │
│                                       │
│  PHASE 2: Distributed Map             │
│  ┌─────────────────────────────────┐  │
│  │ For each article (concurrent):  │  │
│  │  • Score quality                │  │
│  │  • Improve if below threshold   │  │
│  └─────────────────────────────────┘  │
└───────────────────────────────────────┘

Key Configuration Decisions Explained

Decision 1: STANDARD vs EXPRESS mode

Mode	Max Duration	Execution History	Use When
EXPRESS	5 minutes	Limited	Short, high-volume tasks
STANDARD	1 year	Full history	Long AI calls + debugging

KnowledgeForge uses STANDARD because Bedrock calls can exceed 5 minutes.

Decision 2: S3 pointer instead of inline state

❌ WRONG approach:
State machine carries full article HTML in its state
→ Hits 256 KB limit immediately with large articles

✅ CORRECT approach:
State machine carries only: { "s3_location": "bucket/article123.json" }
→ Workers read directly from S3
→ State payload stays tiny regardless of article size

Decision 3: FIFO Queue for ordering

Customer A articles: [A1, A2, A3, A4, A5] → processed in order
Customer B articles: [B1, B2, B3, B4, B5] → processed in order
                                              (parallel to Customer A)

Using customer ID as the message group key ensures:

Each customer's batches process in order (prevents dedup race conditions)
Different customers still run in parallel (no bottleneck)

STEP 7: Quality Scoring and Content Improvement

The Quality Score

Each article is scored across 10 weighted dimensions (highest to lowest weight):

Dimension                    Weight
─────────────────────────────────────
Technical accuracy           ████████ (highest)
Completeness                 ███████
Clarity                      ██████
Structure/formatting         █████
Step-by-step instructions    ████
Troubleshooting coverage     ███
Prerequisites listed         ██
Examples included            ██
Links/references             █
Metadata quality             █ (lowest)
─────────────────────────────────────
Weighted total = Quality Score (0-100)

The Improvement Decision

Article scored
      │
      ▼
Score ≥ threshold?
   │          │
  YES         NO
   │          │
Keep as-is   Send for AI improvement
              │
              ▼
         Score AGAIN after improvement
              │
              ▼
         New score > Original score?
              │          │
             YES         NO
              │          │
         Keep new    Keep original
         version     (don't make it worse!)

Critical safety check: The before-and-after scoring ensures the pipeline never silently degrades content quality.

The Media Placeholder Pattern

This solves a subtle but important problem: AI models tend to corrupt image URLs and hyperlinks when rewriting prose.

BEFORE improvement:
"See <img src='https://cdn.example.com/diagram.png'> for reference.
Visit <a href='https://docs.example.com/fix'>this guide</a>."

STEP 1 - Replace with tokens:
"See [IMG_1] for reference. Visit [LINK_1]."
(Store: IMG_1 = original img tag, LINK_1 = original link)

STEP 2 - AI improves prose:
"Refer to [IMG_1] for a visual overview. 
Consult [LINK_1] for detailed instructions."

STEP 3 - Restore originals:
"Refer to <img src='https://cdn.example.com/diagram.png'> for a visual overview.
Consult <a href='https://docs.example.com/fix'>this guide</a> for detailed instructions."

The AI never sees raw URLs, so it cannot break them.

STEP 8: Resilience Patterns Under Load

Running tens of thousands of AI model calls concurrently requires careful engineering.

Three Controls for Predictable Generation

Control 1: Rate Limiting
─────────────────────────
Limit concurrent Bedrock calls per batch
→ Prevents hitting API quotas
→ Keeps costs predictable

Control 2: Guardrail Fallback
──────────────────────────────
If guardrail API throttles:
  → Process article WITHOUT guardrail
  → Log a WARNING
  → Continue (don't fail the whole batch)
Why: Keeps pipeline healthy during load spikes

Control 3: Token Budget Retry
──────────────────────────────
If response is cut off (hit token limit):
  → Detect "stopped at token limit" signal
  → Retry with DOUBLE the token budget
  → Up to model's maximum ceiling
Why: Long articles need more tokens; short articles pay nothing extra

Two Fault-Tolerance Mechanisms

Mechanism 1: Tolerated Failure Percentage
──────────────────────────────────────────
A few bad articles in a batch?
→ Continue processing the rest
→ Don't abort the entire batch

Mechanism 2: Dead-Letter Queue (DLQ)
──────────────────────────────────────
Repeatedly failing articles?
→ Move to DLQ for investigation
→ Don't block other articles
→ Don't stall the entire run

STEP 9: Multi-Tenant Isolation

The system serves multiple customers on shared infrastructure. Isolation is non-negotiable.

Per-Customer Resources

Customer A                    Customer B
──────────────────────────    ──────────────────────────
S3 Vectors index (A only)     S3 Vectors index (B only)
Custom prompts for A          Custom prompts for B  
Encryption key (A only)       Encryption key (B only)
SQS FIFO message group A      SQS FIFO message group B

Why This Design Is Smart

❌ Shared pool approach:
One index for all customers
→ Customer A's articles could appear in Customer B's results
→ Data leakage risk
→ One customer's bad data affects others

✅ Per-customer approach:
Duplicate check runs against THIS customer's index only
Content improvement uses THIS customer's prompts only
Data encrypted under THIS customer's key only
→ Complete isolation by design

Onboarding new customers: No code changes needed. The system discovers customers from a data catalog at runtime. Adding a customer = provisioning their resources only.

STEP 10: Synthesize — The Four Key Lessons

These are the transferable patterns you can apply to your own pipelines:

Lesson	Pattern	Benefit
1. Reuse your vector index	Same index for RAG grounding AND duplicate detection	Solves two problems with one infrastructure component
2. Pass pointers, not payloads	S3 location through Step Functions, not article content	Eliminates state-size failures at scale
3. Score before AND after improvement	Quality check brackets the AI call	Safe to run automatically without degrading content
4. Degrade gracefully under throttling	Guardrail throttle = warning, not failure	Pipeline stays healthy through load spikes

🧠 Knowledge Check

Test your understanding with these questions:

Why does KnowledgeForge use ECS+Fargate for generation but Step Functions for curation?
Explain why cosine distance of 0.05 was chosen for duplicate detection. What happens if you set it to 0.20?
Trace the path of a single article from ticket cluster to published knowledge base article through all five stages.
Why is the media placeholder pattern necessary during content improvement?
What would happen if the system used a shared S3 Vectors index for all customers instead of per-customer indexes?

📋 Summary Diagram

ITSM Tickets (resolved, clustered by theme)
              │
              ▼
    ┌─────────────────────────────────────────────────────┐
    │              GENERATION (ECS + Fargate)             │
    │  RAG grounding → Claude Sonnet → KB Article + RCA  │
    └──────────────────────┬──────────────────────────────┘
                           │ new articles
                           ▼
    ┌─────────────────────────────────────────────────────┐
    │           CURATION (Step Functions)                 │
    │                                                     │
    │  Classify → Deduplicate    → Score → Improve        │
    │             (S3 Vectors)     (10     (Claude +      │
    │             cosine dist)     dims)   placeholders)  │
    └──────────────────────┬──────────────────────────────┘
                           │ vectors stored back
                           ▼
              S3 Vectors Index (per customer)
                           │
                           ▼ (also feeds back to Generation)
              Knowledge Manager Review (ServiceNow)
                           │
                           ▼
              ✅ Published Knowledge Base Article

More to study

Asynchronous patterns for calling Amazon Bedrock AgentCore agents in serverless pipelines

Control Web Search with Domain and Date Filters

Build a Multi-Agent Support System on AWS