After studying this material, students should be able to:

1. Explain what Jev is and how it differs from traditional classifiers and LLMs
2. Understand Jev's three question types and when to use each
3. Set up the Python AI SDK and authenticate with AI Gateway
4. Write working Python code using the evaluate() API
5. Evaluate appropriate vs. inappropriate use cases for Jev

---

Step-by-Step Teaching

Step 1: The Problem Jev Solves (Foundation)

Before understanding Jev, you need to understand what existed before it.

Traditional Machine Learning Classifiers

A classifier is a model that sorts input into predefined categories.

Email → [Classifier] → "Spam" or "Not Spam"

The painful workflow:

Collect data → Label data → Train model → Deploy → Repeat for every new task

Example problem: You want to classify customer support tickets into categories.

Step 1: Collect 10,000 tickets
Step 2: Manually label each one ("billing", "technical", "shipping")
Step 3: Train a model specifically for this task
Step 4: Deploy it
Step 5: Need a new category? Start over.

The core limitation: Every new classification task = new data collection + new training cycle.

---

What Jev Changes

Jev is built on a Large Language Model (LLM) that was already trained on a broad slice of human knowledge. The key insight:

LLMs have weights → Classifiers have weights → 
Can we make an LLM behave like a classifier?

Answer: Yes. That is Jev.

Traditional Classifier:    Input → [Task-specific trained model] → Category
Jev:                       Input → [Pre-trained LLM repurposed] → Category + Confidence

What this means practically:

Key mental model: Jev is a universal classifier. You define the question and choices. Jev answers using knowledge it already has.

---

Step 2: Understanding Jev's Three Question Types

Jev has exactly three ways to ask it something. Learn these well.

Question Type 1: ChoiceQuestion

Purpose: Pick exactly one answer from a set you define.

Real-world analogy: Multiple choice exam question.

ChoiceQuestion(
    instructions="Which is bigger, the Sun or the Earth?",
    criteria={"Sun": None, "Earth": None}
)

When to use it:

• Categorizing input into named buckets
• Any "is this A, B, or C?" decision
• Classification tasks (spam/not spam, Python/English, etc.)

---

Question Type 2: ScoreQuestion

Purpose: Rate something on a numeric scale you define.

Real-world analogy: Rating a movie 1-10.

ScoreQuestion → "Rate this code quality from 1 to 5"

When to use it:

• Quality assessment
• Ranking or prioritization
• Any "how much?" or "how well?" question

---

Question Type 3: NoulQuestion

Purpose: Estimate the probability that a statement is true.

Real-world analogy: A weather forecast saying "70% chance of rain."

NoulQuestion → "Is this text written in a formal tone?"
→ Returns: 0.85 (85% probability it is)

When to use it:

• Uncertainty estimation
• Soft yes/no decisions where confidence matters
• Risk assessment

---

Quick Reference Summary

ChoiceQuestion  → "Which one is it?"     → Returns: a label
ScoreQuestion   → "How much/well?"       → Returns: a number
NoulQuestion    → "How likely is this?"  → Returns: a probability

---

Step 3: Setting Up Your Environment

Now let's get hands-on. Follow these steps exactly.

3.1 Install the SDK

Jev uses uv as the package manager. Run this in your terminal:

uv add ai

This installs Vercel's AI SDK for Python, which includes the evaluate() API.

3.2 Get Your API Key

1. Go to Vercel AI Gateway
2. Create an AI Gateway key
3. Set it as an environment variable:

# On Mac/Linux
export AI_GATEWAY_API_KEY="your_key_here"

# On Windows (Command Prompt)
set AI_GATEWAY_API_KEY=your_key_here

# Or create a .env file
AI_GATEWAY_API_KEY=your_key_here

3.3 Verify Your Setup

Your project structure should look like this:

my_jev_project/
├── .env                  ← your API key lives here
├── main.py               ← your code goes here
└── pyproject.toml        ← created automatically by uv

---

Step 4: Understanding the evaluate() Function

This is the only function you need to know. Let's dissect it.

Function Signature

result = await jev.evaluate(
    model,      # which Jev model to use
    state,      # the data/context you're asking about
    questions   # a dictionary of questions to ask
)

Breaking Down Each Parameter

Parameter 1: model

ai.get_model("typesafe-ai/jev")
# This selects the Jev model through Vercel's AI Gateway

Parameter 2: state

# Can be a simple string:
state = "Hello, how are you?"

# Or complex JSON:
state = {
    "text": "Hello, how are you?",
    "context": "customer support chat",
    "timestamp": "2024-01-01"
}

Think of state as: "Here is the thing I want you to evaluate."

Parameter 3: questions

questions = {
    "my_question_name": jev.ChoiceQuestion(
        instructions="What language is this?",
        criteria={"Python": None, "English": None}
    )
}
# The key ("my_question_name") is how you retrieve the answer later

Reading the Result

result = await jev.evaluate(...)

# Access the answer by the question name you defined
answer = result.value["my_question_name"]

print(answer)
# Output includes:
# - The chosen label
# - Confidence score
# - Probabilities for each option
# - Latency

---

Step 5: Your First Complete Program

Let's build the simplest possible working example, then expand it.

Version 1: The Minimal Example

import asyncio
import ai
from ai.ops import experimental as jev

async def main():
    # Step 1: Ask Jev a question it cannot get wrong
    result = await jev.evaluate(
        ai.get_model("typesafe-ai/jev"),  # the model
        "You're Neil deGrasse Tyson",      # the state (context)
        {
            "bigger": jev.ChoiceQuestion(  # question name: "bigger"
                instructions="Which is bigger, the Sun or the Earth?",
                criteria={
                    "Sun": None,    # option 1
                    "Earth": None   # option 2
                },
            )
        },
    )
    
    # Step 2: Print the answer
    print(result.value["bigger"])

if __name__ == "__main__":
    asyncio.run(main())

What happens when you run this:

→ Jev receives: context="You're Neil deGrasse Tyson", question="Sun or Earth, which is bigger?"
→ Jev returns: "Sun" with high confidence
→ You print: the result object for "bigger"

Why async/await? Jev calls are network requests. Using async means your program doesn't freeze while waiting for the response. asyncio.run(main()) starts the async event loop.

---

Version 2: Interactive Classifier (Python vs. English)

This is the practical example from the article. Let's build it step by step.

import asyncio
import ai
from ai.ops import experimental as jev

async def classify_line(line: str) -> None:
    """Classify a single line as Python or English."""
    
    result = await jev.evaluate(
        ai.get_model("typesafe-ai/jev"),
        line,  # the text to classify is the state
        {
            "language": jev.ChoiceQuestion(
                instructions="Does this look like Python code or English text?",
                criteria={
                    "Python": None,
                    "English": None
                },
            )
        },
    )
    
    answer = result.value["language"]
    
    # Print what we know about the answer
    print(f"Input:      '{line}'")
    print(f"Choice:     {answer.value}")           # "Python" or "English"
    print(f"Confidence: {answer.confidence:.2%}")  # e.g. "94.30%"
    print(f"Latency:    {result.latency_ms}ms")
    print("-" * 40)

async def main():
    print("Jev Python/English Classifier")
    print("Type text and press Enter. Ctrl+C to exit.\n")
    
    try:
        while True:
            line = input("> ")
            if line.strip():  # skip empty lines
                await classify_line(line)
    except KeyboardInterrupt:
        print("\nGoodbye!")

if __name__ == "__main__":
    asyncio.run(main())

Sample interaction:

> for i in range(10):
Input:      'for i in range(10):'
Choice:     Python
Confidence: 98.10%
Latency:    210ms
----------------------------------------
> what time is it?
Input:      'what time is it?'
Choice:     English
Confidence: 99.50%
Latency:    198ms
----------------------------------------
> if i
Input:      'if i'
Choice:     Python
Confidence: 71.20%
Latency:    205ms
----------------------------------------

---

Step 6: Knowing When to Use Jev (Critical Thinking)

This is where many developers go wrong. Jev is powerful but narrow.

The Right Mental Model

Jev is a DECISION MAKER, not a CONTENT CREATOR

Good Use Cases ✅

# 1. Content moderation
ChoiceQuestion(
    instructions="Does this comment violate community guidelines?",
    criteria={"Violates": None, "Safe": None}
)

# 2. Sentiment analysis
ChoiceQuestion(
    instructions="What is the sentiment of this review?",
    criteria={"Positive": None, "Negative": None, "Neutral": None}
)

# 3. Code quality scoring
ScoreQuestion(
    instructions="Rate the readability of this code",
    scale={"min": 1, "max": 10}
)

# 4. Intent detection
ChoiceQuestion(
    instructions="What does the user want to do?",
    criteria={"Buy": None, "Browse": None, "Get Support": None, "Return": None}
)

Bad Use Cases ❌

# DON'T use Jev to generate text
# DON'T use Jev for open-ended creative tasks
# DON'T use Jev for tasks requiring explanation or reasoning output
# DON'T use Jev for high-stakes decisions without testing accuracy first

The Decision Framework

Ask yourself:
│
├── Can the answer be expressed as a choice, score, or probability?
│   ├── YES → Jev might work
│   └── NO  → Use an LLM instead
│
└── Does accuracy matter critically? (medical, financial, legal)
    ├── YES → Test extensively before deploying
    └── NO  → Experiment freely

---

Step 7: Understanding Jev's Limitations

The article is honest about this. You should be too.

Limitation 1: Ambiguous Inputs Confuse It

"if i"    → Jev says: Python  ✅ (correct)
"if i i"  → Jev says: English ❌ (wrong, it's partial Python)
"if i is" → Jev says: Python  ✅ (correct again)

Lesson: Jev's confidence is not always calibrated to actual accuracy. Always test with your real data.

Limitation 2: It Cannot Generate Content

# This is what the article's "dumb example" proved:
# Making Jev write Python code letter-by-letter = mostly broken output
# Jev is a classifier, not a writer

Limitation 3: Complex Tasks Need LLM Preprocessing

From the article's AST example:

User Prompt
    ↓
[GPT/LLM] ← expands prompt into detailed instructions
    ↓
[Jev] ← makes structured choices based on those instructions
    ↓
Output

Lesson: For complex tasks, use an LLM to plan and Jev to decide.

---

Step 8: Putting It All Together — Mental Model Review

┌─────────────────────────────────────────────────────┐
│                    JEV OVERVIEW                     │
├─────────────────────────────────────────────────────┤
│  What it is:  Universal classifier built on an LLM  │
│  What it does: Answers structured questions fast    │
│  What it returns: JSON with choices + confidence    │
│  What it costs: Cheap (no text generation)          │
├─────────────────────────────────────────────────────┤
│  THE API                                            │
│                                                     │
│  evaluate(model, state, questions)                  │
│      │         │        │                           │
│      │         │        └── ChoiceQuestion          │
│      │         │            ScoreQuestion           │
│      │         │            NoulQuestion            │
│      │         │                                    │
│      │         └── "the thing to evaluate"          │
│      │                                              │
│      └── ai.get_model("typesafe-ai/jev")            │
├─────────────────────────────────────────────────────┤
│  SETUP                                              │
│  1. uv add ai                                       │
│  2. Create AI Gateway key                           │
│  3. export AI_GATEWAY_API_KEY=...                   │
│  4. import ai / from ai.ops import experimental     │
└─────────────────────────────────────────────────────┘

---

Knowledge Check

Test yourself before moving on:

Q1: What are the three question types in Jev's API and when would you use each?

Q2: What is the difference between Jev and a traditional classifier?

Q3: Why is async/await used when calling evaluate()?

Q4: Why did the "Jev writes Python code" example fail?

Q5: Write the code to ask Jev to classify a product review as "Positive", "Negative", or "Neutral".

---

<details> <summary>Answers</summary>

A1: ChoiceQuestion (pick one label), ScoreQuestion (rate on a scale), NoulQuestion (estimate probability of a statement being true)

A2: Traditional classifiers need task-specific training data. Jev uses a pre-trained LLM so it works across domains without retraining.

A3: evaluate() makes a network request. async/await prevents blocking the program while waiting for the response.

A4: Jev is a classifier, not a text generator. Forcing it to generate content character-by-character goes against its design. It lacks the sequential generation capability of an LLM.

A5:

result = await jev.evaluate(
    ai.get_model("typesafe-ai/jev"),
    review_text,
    {
        "sentiment": jev.ChoiceQuestion(
            instructions="What is the sentiment of this product review?",
            criteria={"Positive": None, "Negative": None, "Neutral": None}
        )
    }
)
print(result.value["sentiment"].value)

</details>