How LLMs Learn New Languages: Syntax, Algorithms, and Bugs

Learning Outcomes

By the end of this material, you should be able to:

1. Define the three components of code generation: syntax, algorithm selection, and implementation fidelity
2. Explain what post-training (fine-tuning) actually contributes to an LLM's coding ability
3. Distinguish between in-distribution and out-of-distribution generalization in code models
4. Identify why implementation fidelity is the primary bottleneck in cross-language code generation
5. Evaluate the experimental logic of using a novel language to isolate fine-tuning contributions

---

Step 1: Setting Up the Core Problem

The Fundamental Question

When you teach an LLM a new programming language through fine-tuning, what exactly does it learn?

There are three candidates:

Why This Is Hard to Study Normally

If you fine-tune a model on Python, you cannot separate:

• What it learned from fine-tuning
• What it already knew from pretraining on billions of Python files

Analogy: Imagine testing whether a cooking class improved someone's skills, but they already cooked professionally before the class. You cannot isolate the class's contribution.

---

Step 2: The Experimental Design — PyLang as a Controlled Probe

The Key Innovation

The researchers created PyLang — a completely novel programming language that:

• The model has never seen during pretraining
• Has its own syntax and structure
• Can express the same algorithms as Python

The Experimental Logic

Python output  =  Pretraining knowledge  +  Fine-tuning knowledge
PyLang output  =                             Fine-tuning knowledge ONLY
                                             ↑
                              (model never saw PyLang before)

Therefore:
Fine-tuning contribution = PyLang output
Pretraining contribution = Python output − PyLang output

This is a controlled subtraction experiment — elegant in its simplicity.

Setup Details

• Models tested: Qwen 3 (4B and 8B parameter versions)
• Training data: 2,250 examples in PyLang
• Comparison: Same models generating Python vs. PyLang solutions

---

Step 3: Finding #1 — In-Distribution Results Are Surprisingly Strong

What "In-Distribution" Means

In-distribution problems are similar to the training examples — same problem types, similar complexity.

The Finding

2,250 PyLang examples are sufficient to completely match Python performance on in-distribution problems.

What This Tells Us

For familiar problem types, fine-tuning successfully teaches:

• ✅ New syntax rules
• ✅ How to structure code in the new language
• ✅ Applying known algorithms in the new language

The gap closes entirely — meaning fine-tuning is highly efficient when problems resemble training data.

Analogy: If you learn to drive in an automatic car, switching to a manual car on the same familiar roads is manageable with modest practice.

---

Step 4: Finding #2 — Out-of-Distribution Reveals a Persistent Gap

What "Out-of-Distribution" Means

Problems that are different from training examples — novel problem types, higher complexity, unseen combinations.

The Finding

A persistent ~19 percentage point gap remains between Python and PyLang performance on out-of-distribution problems.

Seven Interventions That All Failed to Close the Gap

The researchers tried every reasonable technique:

None of them worked.

What This Tells Us

The gap is not caused by:

• Insufficient training data
• Wrong training strategy
• Lack of planning ability
• Syntax confusion

The gap must be caused by something more fundamental — which leads to Finding #3.

---

Step 5: Finding #3 — Models Choose Different Algorithms in Different Languages

The LLM Judge Analysis

An independent LLM evaluated whether models used the same algorithm when solving the same problem in Python vs. PyLang.

The Shocking Result

For 64–75% of problems, models produced fundamentally different algorithms in PyLang vs. Python. Cross-language algorithmic agreement was as low as 25–33%.

Visualizing This

Same Problem → Python  →  Algorithm A (e.g., dynamic programming)
Same Problem → PyLang  →  Algorithm B (e.g., brute force recursion)

This happens the majority of the time.

Why Does This Happen?

The model's algorithm selection is entangled with language familiarity:

• In Python (familiar): Model confidently selects optimal algorithms
• In PyLang (unfamiliar): Model defaults to simpler, more "writable" algorithms it can actually implement

Analogy: A chef who knows a complex French recipe might simplify it when cooking in an unfamiliar kitchen with unknown tools — not because they forgot the recipe, but because they're uncertain about execution.

---

Step 6: The Interpretability Evidence

What Was Measured

The researchers examined the model's internal representations — the mathematical vectors that encode how the model "thinks" about a problem.

Two Key Findings

Finding A: PyLang requires more restructuring

• Fine-tuning on PyLang causes substantially more internal reorganization than fine-tuning on Python
• Python fine-tuning is "easier" because it builds on existing structure
• PyLang fine-tuning must build new representational pathways

Finding B: Both models converge to the same problem understanding

• Despite different outputs, both Python-tuned and PyLang-tuned models develop nearly identical internal representations of the problem itself
• The model understands the problem the same way — it just executes differently

What This Proves

Problem Understanding  →  SAME in both languages  ✓
Algorithm Selection    →  DIFFERENT 64-75% of time  ✗
Code Execution         →  DIFFERENT quality         ✗

The bottleneck is not comprehension — it is execution.

---

Step 7: The Central Conclusion — Implementation Fidelity

Defining Implementation Fidelity

Implementation fidelity = the ability to translate a correct, known algorithm into bug-free, working code in a specific language

The Hierarchy of What Fine-Tuning Teaches

Level 1: SYNTAX          → Fine-tuning teaches this well ✓
Level 2: ALGORITHMS      → Partially transferred, but language-dependent ⚠️
Level 3: IMPLEMENTATION  → This is the bottleneck ✗
         FIDELITY

Why Implementation Fidelity Is Hard

Implementation fidelity requires:

• Knowing exact language idioms (not just general syntax)
• Avoiding subtle bugs specific to the language
• Handling edge cases correctly in unfamiliar constructs
• Trusting your algorithm enough to implement it fully

With only 2,250 examples of a new language, the model learns the shape of the language but not the deep fluency needed for reliable execution.

---

Step 8: Synthesizing the Full Picture

Complete Mental Model

┌─────────────────────────────────────────────────────────┐
│                    CODE GENERATION                       │
├─────────────────┬───────────────────┬───────────────────┤
│    SYNTAX       │   ALGORITHM       │  IMPLEMENTATION   │
│                 │   SELECTION       │  FIDELITY         │
├─────────────────┼───────────────────┼───────────────────┤
│ Fine-tuning     │ Partially         │ Primary           │
│ handles this    │ language-tied     │ bottleneck        │
│ well            │                   │                   │
├─────────────────┼───────────────────┼───────────────────┤
│ Closes gap      │ Causes 64-75%     │ Causes persistent │
│ in-distribution │ algorithm         │ ~19pp OOD gap     │
│                 │ divergence        │                   │
└─────────────────┴───────────────────┴───────────────────┘

Key Takeaways

1. Fine-tuning is efficient for familiar problems — 2,250 examples can match extensive pretraining on in-distribution tasks

2. Algorithm selection is not language-independent — models choose different algorithms based on what they can implement, not just what is optimal

3. Understanding ≠ Execution — models can understand a problem identically across languages but execute solutions very differently

4. Implementation fidelity is the frontier — future research should focus on teaching models how to execute known algorithms reliably, not just which algorithm to choose

---

Quick Self-Check Questions

1. Why was PyLang necessary for this experiment? What problem does it solve?
2. What does it mean that seven interventions all failed? What does this rule out?
3. If a model understands a problem identically in Python and PyLang, why does it still perform worse in PyLang?
4. What is the difference between algorithm selection and implementation fidelity?
5. Why might algorithm selection be entangled with language familiarity?

---

Summary in One Sentence

Post-training efficiently teaches syntax and enables in-distribution performance, but the persistent out-of-distribution gap reveals that the true bottleneck is implementation fidelity — the deep, language-specific fluency needed to translate correct algorithms into bug-free code — which 2,250 examples and seven training interventions all fail to provide.