After studying this material, you should be able to:

1. Explain what agentic engineering is and how organizations adopt it at scale
2. Describe the infrastructure components needed to support LLM access across large engineering organizations
3. Identify the challenges and governance considerations when deploying AI coding tools broadly
4. Analyze how AI coding impacts code quality, PR behavior, and team workflows
5. Apply knowledge-sharing and governance frameworks to manage AI tool adoption

---

Step-by-Step Study Material

Step 1: What Is Agentic Engineering?

Definition First

Agentic Engineering = using AI agents (LLM-powered systems) that can autonomously take sequences of actions, use tools, and complete multi-step tasks — going beyond simple chat or autocomplete

The Progression of AI Coding Tools

Think of it as a ladder:

Level 1: Autocomplete (GitHub Copilot early days)
         ↓
Level 2: Chat UI (ask questions, get answers)
         ↓
Level 3: API-based LLM access (programmatic use)
         ↓
Level 4: CLI tools with agent mode
         ↓
Level 5: Full agentic loops (agent plans, acts, evaluates, repeats)

Zalando climbed this ladder over 2.5 years, and their experience teaches us what each step requires.

---

Step 2: The Infrastructure Foundation — LLM Proxy

Why You Need a Proxy

When 250+ engineering teams all need LLM access, you cannot have each team manage their own API keys, model choices, and billing. A central proxy solves this.

What Zalando Built

Engineer's Tool (Claude Code, CLI, Chat UI)
        ↓
   LiteLLM Proxy  ←── Single control point
        ↓
┌───────────────────────────────┐
│  OpenAI  │  AWS Bedrock  │  Google Vertex  │
└───────────────────────────────┘

Key Proxy Features — and WHY Each Matters

Practical Insight: Scale Numbers

• 2,000 Monthly Active Users (MAU)
• Only 6 small pods (2k CPU, 4GB RAM each)
• This is efficient because the proxy is stateless and lightweight

---

Step 3: Beyond the API — Tools Built on Top

Three Complementary Tools

┌─────────────────────────────────────────────┐
│              LLM Proxy (foundation)          │
├──────────────┬──────────────┬───────────────┤
│   Chat UI    │     CLI      │  IDE Plugins  │
│ (fork of OSS)│(custom-built)│(GitHub Copilot│
│              │              │  + others)    │
└──────────────┴──────────────┴───────────────┘

The CLI Tool — A Case Study in Organic Growth

The CLI started as a hackathon project (August 2024) for simple terminal model access. It grew organically into something much more powerful:

Original purpose: Run LLM commands in terminal scripts

What it became:

• Multi-turn interactive chat with file context management
• Agent mode with MCP (Model Context Protocol) support
• Automatic Bearer token injection (no hardcoded secrets)
• HTTP-to-stdio MCP proxy (makes internal MCP servers accessible everywhere)
• Configuration installer for other coding agents

Key Security Concept: Token Injection

WITHOUT token injection:
  User → hardcodes API secret in config file → SECURITY RISK

WITH token injection (Zalando's approach):
  User → CLI handles auth automatically → No secrets in config files

This matters especially as non-engineers start using LLM tools — they have less security intuition.

---

Step 4: MCP — Model Context Protocol

What Is MCP?

MCP is a standard that lets AI agents connect to external tools and data sources. Think of it as a plugin system for AI agents.

Agent
  ↓
MCP Server (e.g., internal API catalogue, documentation)
  ↓
Real data / actions

Zalando's MCP Approach

• Teams build and host their own MCP servers
• Servers are automatically protected by OAuth (ingress filter)
• The CLI tool handles auth token injection so users never deal with credentials
• A centralized collection of MCP servers is discoverable through their developer portal

Why This Architecture Works

Problem: Each MCP server needs authentication
         Each user needs to configure credentials
         Non-engineers can't handle this complexity

Solution: 
  MCP Server → Protected by default OAuth filter (automatic)
  User's CLI → Injects auth token automatically
  Result: User just uses the tool, security is handled

---

Step 5: Measuring Impact — What Changes When AI Coding Scales

PR (Pull Request) Size Changes

Zalando observed measurable changes in how code is submitted:

Before widespread AI coding:
  Most PRs: small, focused changes

After (especially post-Sonnet 4, Q2 2025):
  Growth in larger PR buckets: [500-1k lines], [1k-2k lines]

Why this happens: AI agents generate more code faster, leading engineers to submit larger batches of changes.

Team responses varied:

• Some teams: set hard PR size limits
• Other teams: adopted tools that semantically group changes (GitHub PR grouping, Linear Reviews)

Code Complexity Impact

Zalando tracked Cyclomatic Complexity Number (CCN) — a measure of how complex code logic is.

Cyclomatic Complexity = number of independent paths through code. Higher = harder to test and maintain.

What they found across 4 codebases:

Codebase Type          | AI Adoption Pattern | Complexity Pattern
-----------------------|---------------------|-------------------
New (Go, AI from day1) | Full from start     | Builds up fast, then plateaus
Existing (Go, OSS)     | Added at commit 3000+| Inflection point visible
Existing (Java)        | Gradual adoption    | Gradual increase
Reference (Java, no AI)| None                | Baseline/stable

Key insight: AI amplifies existing patterns — both good practices AND bad ones.

Commit Message Anomalies

AI-generated commit messages cluster around 5,000 characters — much longer than human-written ones. In one extreme case, a commit message included a full unit test execution log.

Lesson: Add pre-commit hooks to enforce commit message length limits.

---

Step 6: Risk-Based PR Approval — A Practical Innovation

The Problem

More PRs + larger PRs = reviewers become bottlenecks. Lead time (time from PR creation to merge) increases.

The Solution: Automated Risk Classification

PR Created
    ↓
Risk Assessment Bot evaluates:
  - What files changed?
  - Does it break backwards compatibility?
  - Is it documentation only?
  - Does it touch configuration files?
    ↓
┌─────────────────────────────────────────┐
│ LOW RISK (33% of PRs) → Auto-approved   │
│ MEDIUM RISK → Needs 1 human reviewer    │
│ HIGH RISK → Full review required        │
└─────────────────────────────────────────┘

Results

• 20-40% reduction in PR lead time for auto-approved PRs
• Engineers building prototypes no longer need to interrupt colleagues for rubber-stamp approvals

Behavioral Side Effect (Important!)

Engineers started structuring their PRs differently to maximize low-risk approvals

Example:

Before: One big PR mixing backwards-compatible changes + breaking changes
After:  PR #1 (low risk, backwards-compatible) → ships fast
        PR #2 (medium risk, breaking change) → separate review

This is governance through incentives, not mandates.

---

Step 7: Governance at Scale

The Core Tension

250+ teams innovating independently
        vs.
Need for consistency, security, and shared learning

Zalando's Answer: Transparency Over Mandates

They explicitly chose NOT to mandate a single tool or approach. Instead:

1. Tech Radar — tracks which AI practices are proven vs. early-stage
2. Developer Portal (Sunrise) — central entry point for AI guidance
3. Auto-detection — scans deployed Docker images for AI model usage, auto-registers in portal
4. Legal assessment entry points — per use-case compliance checks

Vendor Independence Strategy

Risk: Becoming locked into one LLM provider
Mitigation:
  - Proxy supports multiple providers (OpenAI, AWS Bedrock, Google Vertex)
  - No mandated single tool
  - Reference configurations for multiple coding agents
  - Open tools preferred over closed ecosystems

Observed human challenge: Even when switching costs are low, engineers become psychologically attached to their preferred coding agent and model style. This is a real adoption barrier.

---

Step 8: Knowledge Sharing Formats

Format Selection by Audience Maturity

Early Adopters (need cutting-edge exchange)
    → LLM Guild weekly sessions (1hr, 20-min slots, recorded)
    → Hackathons with guided topics

Intermediate Users (need structured learning)
    → GenAI Labs (on-site, 20 people, 1-4 hours, pair exercises)

Broad Organization (need scalable training)
    → Monthly trainings (converted from successful Labs)
    → Trainer pool recruited from Lab attendees

Current Monthly Trainings

1. Using MCP servers — onboards everyone to MCP concept + internal servers
2. Building agents with pydantic-ai — teaches tool calling and agent loops fundamentals

Critical Training Insight

"Using coding agents usually inhibits learning"

When training sessions aim to build new skills, explicitly tell attendees when to code manually. The temptation to use AI as a shortcut prevents skill development.

---

Step 9: Agent Skills — Organizational Knowledge Codified

What Are Agent Skills?

Agent skills = reusable instructions/prompts that guide AI agents to perform specific organizational tasks correctly.

Generic AI Agent + Agent Skill = Organizationally-aware AI Agent

How Zalando Organizes Skills

Centralized Skill Collection
├── By Discipline
│   ├── Data engineering skills
│   ├── Frontend skills
│   ├── SRE skills
│   └── Backend engineering skills
├── By Language
│   ├── Java skills
│   └── Go skills
└── Migration Skills (most popular)
    ├── Multi-arch build adoption
    ├── Platform tool migrations
    └── Infrastructure practice updates

Why Migration Skills Are Most Popular

Migration skills solve a real pain point: when a platform team wants 250 teams to adopt a new tool, they can encode the migration steps as an agent skill. Teams run the skill against their codebase instead of reading documentation and figuring it out manually.

---

Step 10: What's Next — The Agent Platform

The Emerging Architecture

Current State:
  Individual engineers use coding agents locally

Future State:
  Teams deploy agents as services on shared infrastructure

Key Components Being Built

The Hard Problem: Authentication in Agentic Systems

Human User → Agent → MCP Server → Internal API

Each arrow requires authentication.
Who is acting? The human? The agent? Both?
What permissions does the agent have on behalf of the user?

Identity Broker solves this by:
- Capturing delegation chains
- Brokering between different OAuth2 systems
- Implementing on-behalf-of flows

---

Summary: Key Mental Models

1. The Infrastructure Pyramid

        Agent Platform (future)
       /                      \
  Agent Skills          Identity/Auth
       \                      /
        LLM Proxy (foundation)
              |
    Multiple LLM Providers

2. Governance Philosophy

Mandate → Resistance → Slow adoption
Transparency + Incentives → Natural alignment → Faster adoption

3. AI Amplification Effect

Good engineering practices + AI = Faster good outcomes
Bad engineering practices + AI = Faster bad outcomes

Therefore: Engineering fundamentals matter MORE, not less, with AI

4. The Learning Paradox

Using AI to learn AI tools → Inhibits skill development
Manual practice first → Builds genuine understanding → Better AI use later

---

Self-Check Questions

1. Why does a large organization need an LLM proxy rather than direct API access per team?
2. What is MCP and why does authentication complexity increase with agentic systems?
3. How does risk-based PR approval change engineer behavior beyond just speeding up reviews?
4. Why did Zalando choose transparency over mandating a single AI tool?
5. What does cyclomatic complexity tell us about the impact of AI coding on codebases?
6. Why are migration skills the most popular type of agent skill in an organization?
7. What is the "learning paradox" in AI training sessions and how should trainers address it?