Results

The optimizer ran for about 10 hours, cost roughly $2, and explored over 200 prompt variants. Through genetic mutation and Pareto selection, it whittled those down to 9 “survivors”—prompts that excelled at different subsets of the problem. The best performer jumped from 72% to 81% accuracy, a lift I hadn’t achieved in months of manual tuning.

The intermediate prompts evolving caught my eye. I could see the optimizer discovering nuances I’d never thought to include: explicit definitions for each category, step-by-step rules for edge cases, domain-specific guidance about soft pulls versus hard pulls. The quality of the reasoning it produced while iterating was genuinely impressive.

Hopefully I have convinced you that this method is powerful, lets see how i did it and you can follow similar steps for yours as well.

How GEPA Works

GEPA (Genetic-Pareto Algorithm) differs from traditional optimization in three key ways:

1. Reflective Mutation: The LLM reads failure feedback and proposes targeted improvements. It’s not random guessing—it’s reasoning about what went wrong.

2. Pareto Selection: Instead of keeping only the single best prompt, GEPA maintains a “frontier” of diverse specialists. One prompt might excel at detecting objection handling; another at predicting outcomes. This prevents catastrophic forgetting.

3. Text-as-Feedback: Traditional RL uses scalar rewards. GEPA exploits rich textual feedback (“You incorrectly marked this as rapport-building because…”) to guide mutations precisely.

Prerequisites

To Use GEPA, we need 3 components.

class CallAnalysis(dspy.Signature):
    """
    Read the provided call transcript and analyze it comprehensively.
    Determine both: (1) which categories the agent displayed, and 
    (2) whether the call will lead to conversion or customer retention.
    """
    message: str = dspy.InputField()
    categories: List[Literal["introduction_rapport_building", "need_assessment_qualification", 
                             "value_proposition_feature_mapping", "objection_handling", 
                             "benefit_reinforcement", "risk_reduction_trust_building", 
                             "call_to_action_closing"]] = dspy.OutputField()
    final_result: Literal['good', 'bad'] = dspy.OutputField()

program = dspy.ChainOfThought(CallAnalysis)

def call_qual_metric(gold, pred):
    return 1.0 if gold == pred else 0.0

def category_qual_metric(gold, pred):
    """Compute score for categories using set operations."""
    pred_set = set(pred)
    gold_true = {k for k, v in gold.items() if v}
    gold_false = {k for k, v in gold.items() if not v}
    
    correct = len(gold_true & pred_set) + len(gold_false - pred_set)
    return correct / len(gold)

def comb_metric(gold, pred, trace=None, pred_name=None, pred_trace=None):
    """Overall metric combining both scores."""
    call_qual = call_qual_metric(gold.final_result, pred.final_result)
    category_qual = category_qual_metric(gold.categories, pred.categories)
    return (call_qual + category_qual) / 2

def comb_metric(example, pred):
    gold_cat = json.loads(example['answer'])
    gold_final = example['final_result']

    # Category score
    correct = sum(1 for k, v in gold_cat.items() 
                  if (v and k in pred.categories) or 
                     (not v and k not in pred.categories))
    cat_score = correct / len(gold_cat)

    # Final result score
    final_score = 1.0 if gold_final == pred.final_result else 0.0

    return (cat_score + final_score) / 2

def comb_metric_with_feedback(example, pred, pred_name=None):
    # ... same scoring logic as above ...

    # Generate textual feedback
    if gold_final != pred.final_result:
        fb = f"Incorrect: predicted {pred.final_result}, actual {gold_final}"
    else:
        fb = f"Correct: {gold_final}"

    if incorrectly_included:
        fb += f"\nFalse positives: {incorrectly_included}"
    if incorrectly_excluded:
        fb += f"\nMissed categories: {incorrectly_excluded}"

    return dspy.Prediction(score=score, feedback=fb)

def call_qual_feedback(gold, pred):
    """ Generate feedback for final result module. """
    if gold == pred:
        fb = f"You correctly classified the sales call as `{gold}`. This sales call is indeed `{gold}`."
    else:
        fb = f"You incorrectly classified the sales call as `{pred}`. The correct sales call is `{gold}`. Think about how you could have reasoned to get the correct sales call label."
    return fb

def category_qual_feedback(gold, pred):
    """Generate feedback using set operations."""
    pred_set = set(pred)
    gold_true = {k for k, v in gold.items() if v}
    gold_false = {k for k, v in gold.items() if not v}
    
    correctly_included = gold_true & pred_set
    incorrectly_included = gold_false & pred_set
    incorrectly_excluded = gold_true - pred_set
    correctly_excluded = gold_false - pred_set
    
    score = (len(correctly_included) + len(correctly_excluded)) / len(gold)
    
    if score == 1.0:
        return f"Perfect. Correctly identified: `{correctly_included}`.", score
    
    fb = f"Correctly identified: `{correctly_included}`.\n"
    if incorrectly_included:
        fb += f"False positives: `{incorrectly_included}`.\n"
    if incorrectly_excluded:
        fb += f"Missed: `{incorrectly_excluded}`.\n"
    return fb

def comb_metric_with_feedback(gold, pred, trace=None, pred_name=None, pred_trace=None):
    """
    Computes a score and provides feedback for the call analysis prediction.
    Returns total score if pred_name is None, otherwise returns dspy.Prediction with score and feedback.
    """
    # Compute feedback and scores
    cal_fb = call_qual_feedback(gold.final_result, pred.final_result)
    cat_fb = category_qual_feedback(gold.categories, pred.categories)
    fb = cal_fb + '\n' + cat_fb
    score = comb_metric(gold, pred)
    return dspy.Prediction(score=score, feedback=fb)

Running GEPA

With prerequisites in place, optimization is straightforward:

from dspy import GEPA

optimizer = GEPA(
    metric=comb_metric_with_feedback,
    auto="light",
)

optimized_program = optimizer.compile(program, trainset=tset, valset=vset)

Post GEPA run

GEPA ran for 10 hrs on my PC. I saw the 2-line prompt evolve into ~1,500 words of instruction discovering nuances like “a bare greeting isn’t rapport-building; look for warmth and time acknowledgment”.

Read the provided call transcript and analyze it comprehensively.
Determine both: (1) which categories the agent displayed, and (2) whether the call will lead to conversion or customer retention.

New Instructions for Analyzing Banking/Card Transaction Call Transcripts

Overview
You are an analysis assistant whose job is to evaluate sales/transact‑ion-focused call transcripts in the banking/credit-card domain. For each transcript, produce a compact, structured analysis with two main objectives:
  (a) identify the agent behavior categories demonstrated (from the seven pillars below), and
  (b) judge whether the call outcome is good or bad.

Inputs you will receive
- A complete transcript of a single call between an agent and a customer. Transcripts may include labels such as "agent:" and "customer:" and may cover topics like card offers, fees, rewards, security, and next steps.

What you must produce (three sections exactly)
1) reasoning
   - Provide a concise, bullets-style justification for every pillar category you detected in the transcript.
   - Include short quotes or paraphrases from the transcript to illustrate why the category applies. Do not introduce facts or assumptions beyond what is in the transcript.
   - If you detect a strength/weakness signal about the outcome, include a brief, one- to two-sentence note here describing how strong the signal is and what would push it toward conversion or toward retention.
   - This section may contain a small, optional note about outcome strength, but must not introduce information outside the transcript.

2) categories
   - Output a Python-like list of the detected pillar categories in the exact order they first appeared in the transcript.
   - Example format: ['introduction_rapport_building', 'need_assessment_qualification', ...]

3) final_result
   - A single word indicating the call outcome:
     - good — the call demonstrates strong agent performance and is likely to lead to conversion or retention.
     - bad — the call shows weak agent performance or missed opportunities.
   - Do not add any qualifiers in this field; use exactly one of the two keywords above.

Optional but encouraged: assess the strength of the outcome
- If you include it (recommendation), place this assessment only in reasoning as the optional strength_of_outcome note. Keep it concise (one or two sentences). It should address:
  - How strong is the good/bad signal?
  - What would most likely push the outcome toward good or toward bad?

Pillar definitions (seven bank/card-specific categories)
- introduction_rapport_building
  - Includes opening greetings, courtesy, acknowledgment of time, and attempts to establish rapport.
  - Examples: greetings, confirming time, polite introductions, small talk about fit or time constraints.
- need_assessment_qualification
  - Involves asking about customer needs, usage, spend patterns, eligibility checks, and whether the product fits (e.g., business vs personal, employee cards, annual fees, soft vs hard pulls).
- value_proposition_feature_mapping
  - Linking card features to tangible, customer-relevant benefits (rewards, protections, credits) and showing how those features align with stated needs.
- objection_handling
  - Addressing concerns about price, complexity, trust, enrollment, or process obstacles. Includes acknowledging concerns and offering clarifications or mitigations.
- benefit_reinforcement
  - Reiterating concrete benefits and value after objections or hesitations, often tying back to the customer's stated needs.
- risk_reduction_trust_building
  - Providing security assurances, privacy protections, non-hard-pull options, guarantees, terms clarity, or brand trust signals.
- call_to_action_closing
  - Concrete next steps or commitments: soft checks, secure links, email/mail options, scheduling follow-ups, or instructions to apply/get more information.

How to apply the rules
- For every transcript, read from start to finish. Mark each pillar as soon as its criteria are clearly demonstrated.
- If a single utterance clearly satisfies more than one pillar, count it under all applicable pillars.
- If a pillar is not clearly demonstrated anywhere in the transcript, do not include it in the categories list.
- Record the detected pillars in the exact order of their first appearance in the transcript.
- The final_result should reflect the overall trajectory of the call as described above.

Output constraints and format
- Do not introduce any facts not present in the transcript.
- Do not insert subjective opinions beyond what is grounded in the transcript.
- Use the exact section headings and formatting:
  reasoning
  categories
  final_result
- Do not include extraneous content beyond the three sections above.

Domain-specific considerations
- You may encounter references to soft pulls vs hard pulls, online applications, secure links, email follow-ups, or scheduled follow-ups. Treat these as legitimate "call_to_action_closing" or "risk_reduction_trust_building" elements as appropriate.
- When quoting or paraphrasing, keep quotes brief and focused on the reason for the pillar.
- If PII appears in the transcript (e.g., partial SSN or addresses), quote minimally and do not reveal full sensitive data in your justification. You may paraphrase or reference the presence of sensitive data without reproducing it.

Example behavior (not to reproduce here)
- A transcript with strong agent behaviors and clear next steps is more likely good; a transcript with missed opportunities, customer hesitation, or weak closing is more likely bad.

End result
- Return exactly three sections for every transcript analyzed, with the content governed by the rules above. This format enables consistent, comparable, and transparent analysis across transcripts.

Result: 72% → 81% accuracy. More importantly, the process was repeatable.

For a deeper technical dive: GEPA Deepdive

The Solution: Targeted Feedback in the Metric

My hunch was simple: if I could teach the optimizer why these specific cases failed, it could learn the distinctions. I added a feedback column to my dataset and filled it only for mistagged cases—writing out precisely why my label was correct. For example:

“The agent said ‘Hi, this is Mark from JP’ without warmth, time acknowledgment, or rapport-building. A bare introduction doesn’t qualify as introduction_rapport_building.”

Then I passed this feedback directly to GEPA through the metric function:

def comb_metric_with_feedback(gold, pred, trace=None, pred_name=None, pred_trace=None):
    """Metric that returns score + feedback, including targeted hints for known failure cases."""
    # Compute scores
    call_qual = call_qual_metric(gold.final_result, pred.final_result)
    category_qual = category_qual_metric(gold.categories, pred.categories)
    score = (call_qual + category_qual) / 2
    
    # Generate base feedback
    cal_fb = call_qual_feedback(gold.final_result, pred.final_result)
    cat_fb = category_qual_feedback(gold.categories, pred.categories)
    fb = cal_fb + '\n' + cat_fb

    # Append targeted feedback from the dataset (if present)
    fb += gold.feedback

    return dspy.Prediction(score=score, feedback=fb)

Now when GEPA’s reflective mutations analyze failures, they see specific guidance like “look for warmth signals, not just greetings” instead of generic “wrong answer” feedback.

The Result

81% → 90% accuracy in just 3 hours and ~$0.50.

The key insight: GEPA’s genetic mutations work best when they have precise feedback to reason about. Generic “wrong answer” feedback produces generic improvements. Targeted feedback like “you’re conflating greetings with rapport” produces targeted fixes.

When NOT to Use GEPA

GEPA only works when you can clearly define “correct.” If you can’t reliably label examples yourself or if the target is fuzzy or subjective, the optimizer will chase noise. I tried it on a sentiment task where human annotators disagreed 30% of the time. The results were inconsistent. Rule: if you can’t do it clearly as a human, don’t expect GEPA to figure it out.

My Recommended Workflow

1. Start with a clear metric and a personally-validated dataset. 20-50 examples you’ve labeled yourself, understanding the edge cases.

2. Run GEPA → Review failures → Add targeted feedback. Study misclassified examples. Write specific explanations of why your label is correct.

3. Run GEPA again → Stop when acceptable → Deploy. One or two iterations usually suffice. Don’t chase perfection.

4. Post-deployment: monitor, collect failures, iterate. Production surfaces edge cases. Add them with feedback and re-run as needed.

The complete Jupyter notebook: github link

How They Compare

RL would need hundreds of rollouts to statistically isolate that x→y is the fix. The LLM gets it from one error message.

From Fixes to Generalizable Rules

Modern LLMs don’t just extract point fixes—they can derive generalizable lessons:

• Beyond “use y instead of x”: “always verify symbol names against the library’s namespace before generating code”
• Beyond “response was too long”: “for Slack contexts, limit responses to 150 words and use bullet points”
• Beyond “leaked company name”: “scan for proper nouns and replace with generic placeholders”

“LLMs can reflect on their own entire trajectories and extract lessons or generalizable rules that can be incorporated into the prompt.” — Lakshya A Agrawal

These rules get folded directly into the prompt as instructions—accumulating improvements across examples rather than treating each failure in isolation.

LLMs that can genuinely reflect and generalize make this approach viable now.

Bootstrap Few-Shot (DSPy, 2023)

The original insight: you don’t need hand-crafted demonstrations. Given a task and metric, run the pipeline on your training examples, score the outputs, and keep the high-scoring (input, output) pairs as few-shot demonstrations for future runs. The system bootstraps its own examples from successful executions.

Example: Your QA system correctly answers “What’s the capital of France?” → “Paris”. That (question, answer) pair becomes a demonstration shown to the model on future queries.

Limitation: Demonstrations are static snapshots. Once selected, they don’t adapt when new failure modes emerge. And there’s no instruction optimization — the system prompt stays identical whether you’re handling edge cases or common inputs.

OPRO (Google DeepMind, 2023)

OPRO (Optimization by PROmpting, NeurIPS 2023) introduced the idea of using an LLM as the optimizer itself. The key mechanism: show the LLM a history of prompts and their scores, then ask it to propose a better one. Higher-scoring prompts appear more frequently in this history, nudging the LLM toward successful patterns.

Example: The optimizer sees:

• "Solve the math problem step by step" → score 0.65
• "Show your work and verify the answer" → score 0.72
• "Break the problem into cases and check each" → score 0.78

It proposes: "Systematically enumerate cases and verify each solution" → score 0.81

Limitation: Score-only signal. The optimizer sees that prompt_v3 scored 0.72 but not why. Did it make algebraic errors? Miss edge cases? The number alone doesn’t say.

MiPRO (2024)

MiPRO recognized that instructions and demonstrations interact—the same instruction performs differently with different example sets. The best instruction with bad demos might score worse than a mediocre instruction with perfect demos.

The search space problem: Say you have 10 candidate instructions and 5 possible demo sets. That’s 50 combinations. Now add instruction variants (“Be concise” vs “Be brief” vs “Answer in one sentence”)—suddenly you have hundreds of candidates. Each full evaluation means running your pipeline on your entire dev set. At $0.10 per run with 100 dev examples, evaluating all 500 combinations costs $5,000. Not feasible.

MiPRO’s solution: a cheap surrogate model. Instead of running the full pipeline, MiPRO trains a small predictor (think: logistic regression) on the evaluations you have run. The predictor learns patterns like “instructions mentioning ‘step-by-step’ tend to score higher” or “demos with longer reasoning traces correlate with better performance.”

The workflow:

1. Bootstrap: Run a small random sample of combinations (say, 30 out of 500)
2. Train surrogate: Fit the predictor on those 30 (instruction, demos) → score pairs
3. Predict cheaply: Score all 500 combinations using the surrogate (milliseconds, not dollars)
4. Evaluate selectively: Only run full evaluation on the top-predicted candidates
5. Repeat: Add new results to training data, retrain surrogate, sample again

Example: After 30 random evaluations, the surrogate learns:

• Instructions with “step-by-step” → +0.08 average
• Demo set B (which has chain-of-thought examples) → +0.05 average
• Combining both → predicted 0.79

MiPRO focuses budget on high-predicted combinations rather than exhaustive search.

Limitation: The surrogate learns correlations, not causation. It knows “step-by-step instructions score higher” but not why—maybe they help on math problems but hurt on simple lookups. And MiPRO optimizes for aggregate score: a prompt that’s 0.75 on everything beats one that’s 0.95 on hard cases but 0.60 overall—even though that hard-case specialist might contain crucial insights.

SIMBA (DSPy, 2024)

SIMBA (Stochastic Introspective Mini-Batch Ascent) introduced self-reflection into prompt optimization. Rather than treating prompts as black boxes with hidden payout rates, SIMBA has the LLM analyze its own performance and propose improvements.

How it works:

1. Sample mini-batches: Instead of evaluating on the full dataset, SIMBA samples small batches of examples
2. Identify hard cases: Track which examples show high output variability or consistent failures
3. Generate reflective rules: Ask the LLM to analyze why those cases failed and propose improvement rules
4. Update prompts: Incorporate the rules as new instructions, or add successful examples as demonstrations
5. Repeat: Iterate with new mini-batches, accumulating insights

Example: After several mini-batches, SIMBA notices examples 7 and 12 consistently fail. It prompts the LLM: “These examples failed. What pattern do you see?” The LLM reflects: “Both involve multi-step calculations where I lost track of units.” SIMBA adds: “Always track units through each calculation step.”

The key innovation: SIMBA bridges the gap between score-only optimization (OPRO, MiPRO) and full trajectory reflection (GEPA). It uses introspection on failure patterns rather than just comparing aggregate scores.

Limitation: SIMBA’s reflection is still relatively shallow—it identifies patterns across examples but doesn’t deeply analyze individual execution traces. GEPA extends this by feeding the optimizer rich textual feedback (compiler errors, rubrics, reasoning traces) from each trajectory.

GEPA (2025): The Reflection Shift

GEPA breaks from this trajectory in two ways:

1. From scalar scores to textual feedback — Instead of just knowing that a prompt scored 0.6, GEPA sees why: the compiler error, the rubric failures, the reasoning trace where hallucination occurred. The optimizer reads the feedback directly.

Example: OPRO sees score = 0.6. GEPA sees: > “Failed on example 7: response was 340 words (limit: 200). Failed on example 12: missing required keyword ‘disclaimer’. Passed examples 1-6, 8-11.”

The LLM reflects: “I should add an instruction about word limits and required keywords.”

2. From greedy to Pareto selection — Instead of always promoting the highest-scoring candidate, GEPA maintains a Pareto frontier: candidates that each excel at something no other candidate beats.

Example: Three candidates evaluated on 10 examples:

• prompt_A: 8/10 overall, but fails hard cases #7 and #9
• prompt_B: 6/10 overall, but nails hard cases #7 and #9
  
• prompt_C: 7/10 overall, no unique strengths

Greedy selection keeps only prompt_A. Pareto selection keeps both prompt_A and prompt_B—because B’s insights about hard cases might combine with A’s general strength. prompt_C gets dropped (dominated by A on everything).

The difference: OPRO knows the score dropped from 0.8 to 0.6. GEPA reads the compiler error that explains why—and proposes the fix directly.

Quick Start: Using GEPA in 30 Seconds

📖 Full tutorial: GEPA for AIME (Math) — optimizing GPT-4.1 Mini from 46.6% → 56.6% on AIME 2025.

Before diving deep, here’s what using GEPA looks like in practice:

Step 1: Configure your language model

import dspy

lm = dspy.LM("openai/gpt-4.1-mini", temperature=1, max_tokens=32000)
dspy.configure(lm=lm)

Step 2: Define your program

program = dspy.ChainOfThought("problem -> answer")

Step 3: Define a metric that returns feedback (not just a score)

This is the key difference from other optimizers—your metric explains why something failed:

def metric_with_feedback(example, prediction, trace=None, **kwargs):
    correct_answer = example.answer
    pred_answer = prediction.answer
    
    score = int(correct_answer == pred_answer)
    
    if score == 1:
        feedback = f"Correct! The answer is '{correct_answer}'."
    else:
        feedback = f"Incorrect. Expected '{correct_answer}', got '{pred_answer}'."
        # Add any additional context that could help improvement:
        if hasattr(example, 'solution'):
            feedback += f" Solution: {example.solution}"
    
    return dspy.Prediction(score=score, feedback=feedback)

Step 4: Optimize with GEPA

from dspy import GEPA

optimizer = GEPA(
    metric=metric_with_feedback,
    auto="light",           # Budget preset: "light", "medium", or "heavy"
    num_threads=32,         # Parallel evaluation threads
)

optimized_program = optimizer.compile(
    program,
    trainset=train_set,
    valset=val_set,
)

GEPA handles the evolutionary loop, Pareto selection, and reflective mutation internally. If you just want to use it, the code above is sufficient—see the DSPy GEPA API Reference for full parameter details.

But what’s actually happening? Your metric returns why something failed (not just a score), an LLM reads that feedback and proposes improved instructions, Pareto selection preserves diverse specialists rather than just the highest-scoring prompt, and merge operations combine insights from divergent lineages.

The rest of this section builds the core mechanism from scratch.

GEPA: REFLECTIVE PROMPT EVOLUTION (Flow diagram from paper)

The diagram above shows GEPA’s loop. Each prompt is evaluated on every training task, producing a per-instance score matrix. Pareto filtering preserves prompts that excel at something no other prompt beats. New candidates come from reflective mutation (analyzing textual feedback) or merge (combining two specialists’ insights). Only candidates passing a minibatch screen get full evaluation.

The next section implements reflective mutation: the “Reflect and Propose New Prompt” step in the diagram.

Hands-On: Building Reflective Mutation from Scratch

To see how this works by implementing GEPA’s core mechanism on a real task.

The Problem: AIME Math Competition

We’ll optimize prompts for solving AIME (American Invitational Mathematics Examination) problems — challenging competition math that tests algebra, number theory, geometry, and combinatorics. These problems are hard: even frontier LLMs struggle without careful prompting.

from datasets import load_dataset
dset = load_dataset("AI-MO/aimo-validation-aime")['train']
# 90 problems with solutions and integer answers

problem	solution	answer
Quadratic polynomials and have l…	Let Since the -terms of…	116
Three spheres with radii , , and …	This solution refers to the Diagram section…	756

Why AIME for testing prompt optimization?

1. Clear ground truth — Every answer is an integer (0-999), so evaluation is unambiguous
2. Rich failure modes — Wrong answers come from algebraic errors, missed cases, misread constraints
3. Domain knowledge helps — Prompts that encode strategies (“subtract equations pairwise”, “enumerate all cases”) measurably improve performance
4. Small dataset — Only 90 problems, so sample efficiency matters

The Setup

# Split: 10 train, 10 validation (simulating scarce labeled data)
tdf = df.sample(45).iloc[:10]  # Training mini-batches drawn from here
vdf = df.drop(tdf.index).iloc[:10]  # Held-out validation

# Base model: Gemini 2.5 Flash via LiteLLM
# Metric: Exact match (predicted integer == ground truth)
def metric(ground_truth, prediction):
    return int(ground_truth) == prediction['answer']

Seed Prompt

We start with a minimal instruction:

seed_prompt = """You are given a problem and you have to give the answer 
along with reasoning. Do not return anything apart from json. 
It should be parsable by json.loads()"""

Baseline validation accuracy: 10% (1/10 correct)

Can reflective mutation improve this? Let’s find out.

Step 1: The Feedback Function

First, we need a function that tells the reflection LLM what went wrong. We’ll start with minimal feedback—just the correct answer:

def feedback(ground_truth, prediction):
    if int(ground_truth) != prediction['answer']:
        return f'You got it wrong! The solution is {ground_truth}'
    return 'You got it right!'

This is deliberately simple. Later we’ll discuss how richer feedback (like expert solutions) can improve results.

Step 2: The Reflection Prompt

Following GEPA’s structure, we build a prompt that shows the LLM its failures:

REFLECTION_TEMPLATE = """I provided an assistant with the following instructions:
<curr_instructions>
{current_prompt}

The following are examples with assistant's responses and feedback:
<inputs_outputs_feedback>
{examples}

Your task: write a new instruction for the assistant.

- Read inputs carefully and identify the input format and task description
- Read all responses and feedback. Identify niche/domain-specific factual information
- If the assistant used a generalizable strategy, include that in the instruction

Provide the new instructions.
"""

def mk_reflection_prompt(df, curr_prompt):
    """Build reflection prompt from minibatch results."""
    examples = []
    for i, row in df.reset_index().iterrows():
        example = f"""# Example {i+1}
## problem
{row['problem']}
## prediction
{row['pred']}
## feedback
{feedback(row.answer, row.pred)}
"""
        examples.append(example)
    
    return REFLECTION_TEMPLATE.format(
        current_prompt=curr_prompt,
        examples="\n".join(examples)
    )

Example filled-in reflection prompt:

I provided an assistant with the following instructions:
<curr_instructions>
You are given a problem and you have to give the answer along with reasoning. 
Do not return anything apart from json. It should be parsable by json.loads()

The following are examples with assistant's responses and feedback:
<inputs_outputs_feedback>
# Example 1
## problem
Quadratic polynomials P(x) and Q(x) have leading coefficient 1. The sum of the roots of P(x) is 7...
## prediction
{"answer": 42, "reasoning": "I solved the system and got x=7, y=6"}
## feedback
You got it wrong! The solution is 116

# Example 2
## problem
Three spheres with radii 11, 13, and 19 are mutually externally tangent...
## prediction
{"answer": 756, "reasoning": "Using the tangent sphere formula..."}
## feedback
You got it right!

# Example 3
## problem
Find the remainder when 2^2024 is divided by 1000...
## prediction
{"answer": 16, "reasoning": "I computed powers of 2 mod 1000..."}
## feedback
You got it wrong! The solution is 896

Your task: write a new instruction for the assistant.

- Read inputs carefully and identify the input format and task description
- Read all responses and feedback. Identify niche/domain-specific factual information
- If the assistant used a generalizable strategy, include that in the instruction

Provide the new instructions.

Step 3: The Complete Optimization Loop

def mk_reflection_prompt(mb, curr_prompt):
    """Build reflection prompt from minibatch results."""
    examples = []
    for i, row in df.reset_index().iterrows():
        example = f"""# Example {i+1}
## problem
{row['problem']}
## prediction
{row['pred']}
## feedback
{feedback(row.answer, row.pred)}
"""
        examples.append(example)
    
    return REFLECTION_TEMPLATE.format(
        current_prompt=curr_prompt,
        examples="\n".join(examples)
    )

def reflect(mb, curr_prompt):
    """Ask LLM to reflect on failures and propose improved instruction."""
    refl_prompt = mk_reflection_prompt(mb, curr_prompt)
    return _call(refl_prompt, format=ReflectionModel)['new_instruction']

def optimize_prompt(seed_prompt, traindf, valdf, n_iters=3, mb_size=3):
    """Greedy reflective prompt optimization."""
    prompts, train_scores, val_scores = [seed_prompt], [], []
    mb = traindf.sample(mb_size)  # Fixed minibatch for this run
    
    print(f'Baseline validation: {eval_val(seed_prompt, valdf):.2%}')
    
    for i in range(n_iters):
        # Evaluate current prompt on minibatch
        mb_eval, mb_score = eval_mb(prompts[-1], mb)
        print(f"📊 Minibatch: {mb_score:.2%}")
        
        # Reflect and propose new instruction
        new_instr = reflect(mb_eval, prompts[-1])
        new_prompt = new_instr  # The new instruction becomes the new prompt
        
        # Evaluate on validation set
        val_score = eval_val(new_prompt, valdf)
        print(f"📊 Validation: {val_score:.2%}")
        
        prompts.append(new_prompt)
        val_scores.append(val_score)
    
    return dict(prompts=prompts, val_scores=val_scores)

What Actually Happened

Running this on AIME problems with Gemini 2.5 Flash:

The good: Iteration 2 extracted genuinely useful domain knowledge. Despite 0% minibatch accuracy, the reflection LLM identified patterns from the problems themselves and added this to the prompt:

“When dealing with systems of equations like , , , consider subtracting equations pairwise to find relationships between variables, such as , which implies or . Systematically explore all such cases.”

This is directly from the actual output—the reflection LLM read the failed attempt on a systems-of-equations problem and generalized a useful heuristic.

The bad: Iteration 3 achieved 67% on its minibatch but dropped to 10% validation. Why? The minibatch happened to contain number theory problems, so the reflection added specialized number-theory guidance:

“When the problem involves number theory and remainders (e.g., , ), pay close attention to inconsistencies or contradictions that might arise from given conditions, especially when distinct remainders are required, as these can lead to an answer of 0.”

This over-specialized advice (“can lead to an answer of 0”) actively hurt performance on non-number-theory problems, dropping validation from 30% back to 10% (1/10)—though notably, it did solve problem #9, which earlier prompts couldn’t.

The Greedy Selection Problem

This demonstrates exactly why GEPA uses Pareto selection instead of always taking the “best” prompt:

1. Iteration 2’s prompt was a specialist—it learned something valuable about systems of equations
2. Iteration 3 tried to improve on iteration 2, but the minibatch had different problems
3. The reflection overwrote the systems-of-equations insight while adding number-theory tips that were too specific
4. Result: catastrophic forgetting

With greedy selection, we would have discarded iteration 2’s valuable insight. Pareto selection would keep it—because it was best on at least one validation instance.

The Missing Ingredient: Rich Feedback

Our minimal feedback ("You got it wrong! The solution is 349") only tells the model that it failed, not why or how to fix it.

The AIME dataset includes expert solutions. A richer feedback function could use them:

def feedback_rich(row):
    if int(row.answer) != row.pred['answer']:
        sol = row.solution[:500] + "..." if len(row.solution) > 500 else row.solution
        return f"""Wrong! Expected {row.answer}, got {row.pred['answer']}.
        
Model's reasoning: {row.pred['short_reasoning']}

Expert solution approach:
{sol}"""
    return "Correct!"

Example output for a wrong answer:

Wrong! Expected 116, got 42.

Model's reasoning: I set up the system of equations and solved for x=7, y=6, giving 7*6=42.

Expert solution approach:
Let R(x)=P(x)+Q(x). Since the x²-terms of P and Q have leading coefficient 1, 
R(x) is quadratic with leading coefficient 2. Given the roots condition, we can 
write R(x) = 2(x-r₁)(x-r₂). Expanding and comparing coefficients...

With rich feedback, the reflection LLM can extract specific strategies from the expert solution rather than having to guess what went wrong. This is what makes GEPA sample-efficient: the feedback contains the fix.

Compare this to RL, which would only see reward = 0 and have to statistically infer what went wrong across thousands of trajectories.

What We Learned

1. Reflective mutation works — Even with minimal feedback, the LLM extracted useful domain knowledge
2. Greedy selection fails — Iteration 3’s collapse shows why we need to preserve specialist insights
3. Feedback quality matters — Rich feedback (expert solutions, compiler errors, rubric explanations) gives the reflection LLM more to work with
4. Sample efficiency is real — We saw meaningful optimization with just 3 iterations on 3 examples each

This is the core limitation of greedy optimization: catastrophic forgetting. The solution? Pareto selection—which we’ll build from scratch next.

Hands-On: Building the Pareto Frontier

In the reflective mutation section, we saw greedy selection fail—iteration 3’s over-specialized prompt dropped validation from 30% to 10%, losing iteration 2’s valuable systems-of-equations insights. The fundamental problem: always improving your “best” prompt discards specialist knowledge.

GEPA’s solution: Pareto selection. Instead of keeping one best prompt, maintain a frontier of prompts where each excels at something no other prompt beats.

import numpy as np

def dominates(candidate_scores, other_scores):
    """Does candidate dominate other? (>= everywhere, > somewhere)"""
    candidate = np.array(candidate_scores)
    other = np.array(other_scores)
    return (candidate >= other).all() and (candidate > other).any()

def is_dominated_by_any(new_scores, frontier_scores):
    """Is new_scores dominated by ANY prompt in the frontier?"""
    new = np.array(new_scores)
    for existing in frontier_scores:
        if dominates(np.array(existing), new):
            return True
    return False

def get_dominated_indices(new_scores, frontier_scores):
    """Which frontier prompts does new_scores dominate?"""
    new = np.array(new_scores)
    return [i for i, existing in enumerate(frontier_scores) 
            if dominates(new, np.array(existing))]

P0 = [1,0,0,0,0,0,0,0,0,0]  # Solves: #0
P1 = [1,0,0,0,0,0,0,0,0,0]  # Solves: #0
P2 = [1,1,1,0,0,0,0,0,0,0]  # Solves: #0, #1, #2
P3 = [0,0,0,0,0,0,0,0,0,1]  # Solves: #9

# Does P2 dominate P0?
dominates(P2, P0)  # True: P2 >= P0 everywhere, P2 > P0 on #1, #2

# Does P2 dominate P1?
dominates(P2, P1)  # True: P2 >= P1 everywhere, P2 > P1 on #1, #2

# Does P2 dominate P3?
dominates(P2, P3)  # False! P2 loses on #9 (0 < 1)

# Does P3 dominate P2?
dominates(P3, P2)  # False! P3 loses on #0, #1, #2

class ParetoFrontier:
    def __init__(self):
        self.prompts = []
        self.scores = []  # scores[i][j] = prompt i's score on instance j
    
    def add(self, prompt, instance_scores):
        """Try to add a prompt. Returns True if it joins the frontier."""
        # Reject if dominated by existing frontier member
        if is_dominated_by_any(instance_scores, self.scores):
            return False
        
        # Remove any frontier members this prompt dominates
        dominated = get_dominated_indices(instance_scores, self.scores)
        for i in sorted(dominated, reverse=True):  # Remove from end first
            del self.prompts[i]
            del self.scores[i]
        
        # Add to frontier
        self.prompts.append(prompt)
        self.scores.append(instance_scores)
        return True
    
    def sample(self):
        """Sample a prompt, weighted by unique wins."""
        weights = []
        scores_arr = np.array(self.scores)
        for i in range(len(self.prompts)):
            # How many instances is this prompt *uniquely* best on?
            others_best = np.delete(scores_arr, i, axis=0).max(axis=0) if len(self.prompts) > 1 else np.zeros_like(scores_arr[i])
            unique_wins = (scores_arr[i] > others_best).sum()
            weights.append(unique_wins + 1)  # +1 smoothing
        
        return np.random.choice(self.prompts, p=np.array(weights)/sum(weights))
    
    def best_aggregate(self):
        """Return the prompt with highest aggregate score."""
        aggregates = [sum(s) for s in self.scores]
        return self.prompts[np.argmax(aggregates)]

def optimize_with_pareto(seed_prompt, traindf, valdf, n_iters=5, mb_size=3):
    """Reflective mutation with Pareto frontier selection."""
    frontier = ParetoFrontier()
    
    # Initialize with seed
    seed_scores = evaluate_per_instance(seed_prompt, valdf)
    frontier.add(seed_prompt, seed_scores)
    print(f'Baseline: {sum(seed_scores)/len(seed_scores):.1%}')
    
    for i in range(n_iters):
        print(f"\n{'='*40}\nIteration {i+1}\n{'='*40}")
        
        # Sample parent from frontier (weighted by unique wins)
        parent = frontier.sample()
        
        # Run on minibatch, reflect, propose mutation
        mb = traindf.sample(mb_size)
        mb_results = evaluate_with_traces(parent, mb)
        new_prompt = reflect_and_mutate(parent, mb_results)
        
        # Evaluate on full validation set
        new_scores = evaluate_per_instance(new_prompt, valdf)
        new_agg = sum(new_scores) / len(new_scores)
        print(f"New prompt: {new_agg:.1%} aggregate")
        
        # Try to add to frontier
        if frontier.add(new_prompt, new_scores):
            print(f"✓ Added to frontier (size: {len(frontier.prompts)})")
        else:
            print(f"✗ Dominated, rejected")
    
    return frontier.best_aggregate()

Why Pareto Works: Quality-Diversity and Map Elites

We’ve now built both core mechanisms from scratch—reflective mutation and Pareto selection. Before diving into the full algorithm, let’s step back and understand why this approach works so well.

The Pareto frontier isn’t a novel invention—it draws from quality-diversity (QD) algorithms, a family of techniques from evolutionary computation that work well when diversity itself is valuable.

Traditional optimization asks: “What’s the single best solution?”

Quality-diversity asks: “What’s the best solution of each type?”

Complex problems often have multiple valid approaches. A chess engine that always plays aggressively might beat one that always plays defensively—but the best engine knows both styles and picks situationally. Maintaining diverse specialists, then combining their insights, outperforms converging prematurely on one “best” approach.

The Lineage Tree and System-Aware Merge

Pareto selection preserves specialists—but it creates a new problem: insights get siloed in separate branches.

Consider what happens after 10 iterations of GEPA on AIME problems:

The P2→P4 lineage accumulated algebra insights. The P3→P5 lineage accumulated number theory insights. Both survive on the Pareto frontier because each solves problems the other can’t.

But what about a problem requiring both?

The Complete Algorithm

Here’s how the pieces combine into the full optimization loop.

import random
import numpy as np
from typing import Callable

def gepa(
    base_prompt: str,
    trainset: list,
    valset: list,
    evaluate_fn: Callable,          # (prompt, example) -> score (0 or 1)
    run_with_feedback_fn: Callable, # (prompt, examples) -> (traces: list[str], feedback: list[str])
    reflect_fn: Callable,           # (parent_prompt, traces, feedback) -> new_prompt: str
    merge_fn: Callable,             # (prompt1, prompt2) -> merged_prompt: str
    max_iterations: int = 20,
    minibatch_size: int = 3,
):
    """
    GEPA: Genetic-Pareto prompt optimization.
    
    Returns the best aggregate prompt; access full frontier via returned dict.
    """
    
    # --- Helper functions ---
    
    def evaluate_all(prompt, dataset):
        """Return per-instance scores as list."""
        return [evaluate_fn(prompt, ex) for ex in dataset]
    
    def evaluate_minibatch(prompt, minibatch):
        """Return aggregate score on minibatch."""
        return sum(evaluate_fn(prompt, ex) for ex in minibatch) / len(minibatch)
    
    def dominates(scores_a, scores_b):
        """Does A dominate B? (>= everywhere, > somewhere)"""
        a, b = np.array(scores_a), np.array(scores_b)
        return (a >= b).all() and (a > b).any()
    
    def is_dominated_by_frontier(new_scores, frontier, scores):
        """Is new_scores dominated by ANY frontier member?"""
        return any(dominates(scores[c], new_scores) for c in frontier)
    
    def sample_from_frontier(frontier, scores):
        """Sample weighted by unique wins."""
        n_instances = len(next(iter(scores.values())))
        weights = []
        for candidate in frontier:
            if len(frontier) == 1:
                unique_wins = n_instances
            else:
                others_max = np.array([scores[o] for o in frontier if o != candidate]).max(axis=0)
                unique_wins = (np.array(scores[candidate]) > others_max).sum()
            weights.append(unique_wins + 1)  # +1 smoothing
        return random.choices(frontier, weights=weights)[0]
    
    def get_root(prompt, lineage):
        """Trace lineage back to root."""
        while lineage.get(prompt) is not None:
            prompt = lineage[prompt]
        return prompt
    
    def should_merge(frontier, lineage, iteration):
        """Decide whether to merge or mutate."""
        if len(frontier) < 2 or iteration < 5:
            return False
        n_lineages = len(set(get_root(c, lineage) for c in frontier))
        return random.random() < (n_lineages - 1) / len(frontier)
    
    def select_divergent(frontier, parent, lineage):
        """Select a candidate from a different lineage."""
        parent_root = get_root(parent, lineage)
        others = [c for c in frontier if get_root(c, lineage) != parent_root]
        return random.choice(others) if others else random.choice(frontier)
    
    def best_aggregate(frontier, scores):
        """Return prompt with highest aggregate score."""
        return max(frontier, key=lambda c: sum(scores[c]))
    
    # --- Main loop ---
    
    candidates = [base_prompt]
    scores = {base_prompt: evaluate_all(base_prompt, valset)}
    pareto_frontier = [base_prompt]
    lineage = {base_prompt: None}
    
    for iteration in range(1, max_iterations + 1):
        
        # 1. SAMPLE: Select parent from Pareto frontier
        parent = sample_from_frontier(pareto_frontier, scores)
        
        # 2. PROPOSE: Either mutate or merge
        minibatch = random.sample(trainset, min(minibatch_size, len(trainset)))
        
        if should_merge(pareto_frontier, lineage, iteration):
            other_parent = select_divergent(pareto_frontier, parent, lineage)
            new_prompt = merge_fn(parent, other_parent)
        else:
            traces, feedback = run_with_feedback_fn(parent, minibatch)
            new_prompt = reflect_fn(parent, traces, feedback)
        
        # 3. EVALUATE: Mini-batch gate
        parent_mb_score = evaluate_minibatch(parent, minibatch)
        new_mb_score = evaluate_minibatch(new_prompt, minibatch)
        if new_mb_score <= parent_mb_score:
            continue  # Reject: didn't improve on mini-batch
        
        # Full evaluation
        new_scores = evaluate_all(new_prompt, valset)
        
        # 4. UPDATE: Pareto frontier maintenance
        if is_dominated_by_frontier(new_scores, pareto_frontier, scores):
            continue  # Reject: dominated by existing candidate
        
        # Remove dominated candidates
        pareto_frontier = [c for c in pareto_frontier 
                          if not dominates(new_scores, scores[c])]
        
        # Add new candidate
        candidates.append(new_prompt)
        scores[new_prompt] = new_scores
        pareto_frontier.append(new_prompt)
        lineage[new_prompt] = parent
    
    return {
        'best': best_aggregate(pareto_frontier, scores),
        'frontier': pareto_frontier,
        'scores': scores,
        'lineage': lineage,
    }

What GEPA Learns: Domain-Specific Knowledge Encoding

We’ve built the full algorithm. What does all this machinery actually produce? The output is prompts that encode domain expertise.

GEPA can encode domain-specific knowledge directly into prompts—turning what experts know implicitly into explicit instructions that persist across examples.

Two Paradigms of Operation

Train-then-generalize (what we’ve built so far):

• Optimize prompts on a training set
• Select the best-aggregate prompt from the Pareto frontier
• Deploy that prompt on new, unseen tasks
• Goal: learn generalizable lessons that transfer

Test-time search (inference-time optimization):

• You have a batch of hard tasks you need to solve now
• Optimize directly on the tasks themselves
• GEPA searches for solutions, storing the best prompt per task
• Goal: maximize performance on these specific instances

The mechanics are identical—reflective mutation, Pareto selection, merge. What changes is the intent: instead of learning transferable knowledge, you’re using GEPA as a search algorithm over the solution space. This is an instance of inference-time compute scaling—investing more computation at inference to solve harder problems.

The key mechanical change: In normal GEPA, you have separate trainset (to learn from via reflection) and valset (to evaluate generalization). The Pareto frontier tracks per-instance performance on valset, preserving prompts that generalize well.

For test-time search, pass the same problems as both trainset and valset:

# Test-time search: optimize directly on the tasks you want to solve
optimized = GEPA(metric=metric_with_feedback, auto="medium").compile(
    program,
    trainset=hard_problems,  # The actual tasks you need solved
    valset=hard_problems,    # Same held-out validation
)

This tells GEPA: “I don’t care about generalization—optimize directly on these specific problems.” The Pareto frontier now tracks “best prompt for each problem” rather than “prompts that transfer to unseen data.” See the GEPA API documentation for full parameter details.

“Given a batch of tasks that we want to solve and given some budget… GEPA can propose and update its own strategy to solve that particular task iteratively till that task is solved.” — Lakshya A Agrawal

Why GEPA Beats High-Temperature Sampling

Traditional inference-time strategies sample at high temperature to generate many candidates, then pick the best. But these samples tend to be similar—variations on the same approach. GEPA induces genuine diversity through Pareto tracking (maintaining candidates that excel at something different) and reflective mutation (proposing structurally different strategies based on what went wrong, not random perturbations).

When feedback says “memory bandwidth bottleneck,” the next candidate might switch from a naive loop to shared-memory tiling—a qualitative change that temperature variation rarely discovers. On the MATH benchmark, this approach achieves 93% accuracy compared to 67% with basic DSPy ChainOfThought.

Self-Bootstrapping at inference

During training, you iterate a fixed number of times and deploy the result. At inference time, you can keep iterating on a single hard problem until it’s solved—and GEPA’s reflective loop creates a self-bootstrapping dynamic:

1. Round 1: Generate rollout → compiler error (“undefined variable x”) → reflect → propose fix
2. Round 2: Code compiles → runtime error (division by zero) → reflect → propose fix
   
3. Round 3: Runtime works → wrong output (“expected 42, got 41”) → reflect → propose fix
4. Round 4: Correct output ✓

Each iteration surfaces the next failure mode—you can’t discover the runtime error until the compile error is fixed. Traditional sampling generates 100 candidates that all hit the same compiler error. GEPA’s iterative reflection progresses through the failure cascade.

“It identifies new challenges that the system is going to encounter as well as at every step it is going to propose a solution to that challenge. So it’s kind of like self-bootstrapping data to train itself.” — Lakshya A Agrawal

This is why test-time GEPA can solve problems that stumped training: it has the budget to chase failure modes deeper than any fixed training run.

Note: When optimizing a batch of tasks, Pareto selection ensures that fixing one problem doesn’t discard prompts that solved others—see “Cross-Task Transfer Within a Batch” below.

Cross-Task Transfer Within a Batch

When solving related tasks (e.g., a batch of CUDA kernels), insights compound across the batch:

“All of these tasks are highly related. So if I discover an insight that works well on task one, there is a high possibility that it will also work well on task two. So what happens is I use the rollout from task one to update my prompt and then I use that prompt to generate the solution for task two.” — Lakshya A Agrawal

The frontier maintains multiple specialized prompts simultaneously:

“One prompt will be highly specialized to the convolution one and this can work well for three-four task instances. Another might specialize to the summation one and this could cater to another three-four instances.” — Lakshya A Agrawal

When a new problem arrives, GEPA tries candidates from across the frontier—the convolution specialist might crack it, or the reduction specialist, or insights from both might merge.

Background Optimization Loops

The self-bootstrapping pattern suggests a natural application: background GEPA loops for personalization in tools like Cursor and other AI-assisted environments.

“For your cursor agent use case, I can imagine that there can be a background GEPA loop that runs continuously and every time you give it feedback, it iterates and generates a new generalizing prompt… cursor can learn a user-specific prompt that works well specifically for you.” — Lakshya A Agrawal

Every correction you provide—rejecting a verbose explanation, fixing a code style—becomes a training signal that accumulates into a personalized prompt. The infrastructure isn’t widespread yet, but the pattern points toward continuous adaptation rather than one-shot optimization.

What GEPA Stores

For each task in the batch, GEPA tracks both artifacts:

“GEPA tracks the best prompt per test case and you can store both the prompt as well as the output generated by that prompt.” — Lakshya A Agrawal

• Best outputs — the actual solutions, ready to use
• Best prompts — specialized strategies representing different subdomains of your problem space

You can deploy these domain-specific prompts for future similar tasks, or simply extract the outputs and move on.

When the conditions are right—rich feedback, high-value tasks worth the compute, domains where the LLM has strong priors—test-time GEPA can beat sampling-based approaches. The next section demonstrates this on code generation: GEPA-optimized CUDA kernels that exceed human-written PyTorch baselines, in a domain where even frontier models like OpenAI-o1 and DeepSeek-R1 match the baseline on less than 20% of tasks.

Case Study: Code Optimization for Novel Hardware

The GEPA paper demonstrates test-time search on code optimization for hardware with limited pre-training data—a domain well-suited to GEPA’s reflective approach.

Use GEPA for Train-Then-Generalize When:

• You have rich textual feedback (compiler errors, profiler output, LLM-as-judge rubrics, expert solutions)
• Your evaluation budget is limited (50-500 examples, not 5,000)
• You’re optimizing compound AI systems where prompts orchestrate multi-step pipelines
• You need interpretable results—prompts you can read, edit, and reason about

Use GEPA for Test-Time Search When:

• You have a batch of high-value tasks worth the compute investment
• Each task produces execution feedback (tests, profilers, validators)
• Tasks are related enough for cross-task transfer to help

Stick with Traditional Approaches When:

• You have abundant labeled data and compute budget for fine-tuning
• Feedback is purely scalar with no explanatory signal
• The task is already solved by few-shot prompting
• You need sub-second latency

Get Started

Ready to try GEPA on your own pipelines?

• GEPA for AIME Tutorial — Complete walkthrough from setup to optimized results
• GEPA API Reference — Full parameter documentation
• Paper — Algorithm details and experimental methodology

The core insight is simple: your LLM pipelines already produce rich textual feedback. GEPA just reads it—and learns.

Hashing vs. Python’s built-in hash()

Python’s hash() uses SipHash-1-3 — a fast, non-cryptographic hash designed for dictionary lookups. It’s: - Randomized per session (to prevent HashDoS attacks) - NOT collision-resistant or suitable for security

Bitcoin needs SHA-256 — deterministic, collision-resistant, and one-way.

SHA-256 isn’t used to hide data — it’s used as a fingerprint. Given a hash, you can never work backwards to the original input. But given an input, you’ll always get the same hash. With possible outputs, accidental collisions are essentially impossible.

import hashlib

# SHA-256 takes bytes and returns a 64-char hex string.
# Same input -> same output, every time.
hashlib.sha256(b'123').hexdigest()

'a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3'

# A helper: .digest() returns raw bytes (not hex), which is what secp256k1 expects.
def hash_fn(msg): return hashlib.sha256(msg).digest()

What is ECC?

Elliptic Curve Cryptography builds on the math of elliptic curves — curves defined by:

Bitcoin uses the SECP256k1 curve where :

The hard problem: given a starting point (the Generator), your private key is a big random number , and your public key is (point on the curve). Computing is easy, but reversing it (finding from the public key) is computationally impossible with a 256-bit key.

Why not RSA? (And why ECC wins)

The most common public-key system before ECC was RSA (Rivest–Shamir–Adleman, 1977), still widely used in TLS/HTTPS, email encryption, and digital certificates. RSA’s security relies on the integer factorization problem: given two large primes and , it’s easy to compute their product , but given alone, factoring it back into and is computationally infeasible for large keys.

• Public key: — used to encrypt or verify
• Private key: — used to decrypt or sign

RSA works, but it has a problem. Factoring is vulnerable to sub-exponential algorithms like the General Number Field Sieve — attacks smarter than brute force. To compensate, RSA key sizes must keep growing: 2048-bit is the bare minimum today, 3072+ recommended.

Other systems (DSA, Diffie-Hellman, ElGamal) have the same weakness — they all need large keys. This is what makes ECC special: the elliptic curve discrete logarithm problem has no known sub-exponential attack. It’s fundamentally harder per bit:

A 256-bit ECC key provides security equivalent to a ~3072-bit RSA key — that’s over 10x smaller for the same security. For Bitcoin, this matters enormously:

• Smaller signatures & keys → less data per transaction → lower fees
• Faster verification → nodes can validate blocks more quickly
• Less storage & bandwidth → critical for a global decentralized network with thousands of nodes

In short: RSA works, but ECC gives you the same (or better) security in a fraction of the size. For a blockchain where every byte counts, that’s a big deal.

Further Reading

• ECC: Cloudflare’s Primer on ECC · Wikipedia: Elliptic-curve cryptography
• RSA: Brilliant: RSA Encryption · Wikipedia: RSA cryptosystem · CrypTool: RSA step-by-step

Below, we’ll first understand why Bitcoin uses these curves, and then visualize them — first over the real numbers (smooth), then over a finite field (discrete points).

What Do Elliptic Curves Actually Achieve?

Before we look at the curves visually, it helps to understand why Bitcoin uses them at all.

The core problem: Alice wants to prove she owns some bitcoin, without revealing her secret (private key). She needs a math operation that’s:

• Easy to do forward: take a number, multiply it by a point on the curve → get a new point
• Impossible to reverse: given that new point, figure out the original number

Elliptic curves give us exactly this. Your private key is just a big random number . Your public key is that number multiplied by the generator point on the curve: . Anyone can verify your signature using the public key, but no one can work backwards to find .

Why curves specifically? Because point multiplication on an elliptic curve is the hardest one-way function we know of per bit of key size — harder than RSA’s factoring problem. Think of it like mixing paint: easy to blend two colors together, but nearly impossible to look at the result and figure out exactly which two colors were used.

With the why established, let’s talk about what to look for in the diagrams.

Lets look at the Elliptical curves for intuition The diagrams below aren’t security proofs — they’re intuition for two things that matter later:

1. Symmetry: for every point there’s a matching . This is why Bitcoin can compress a public key to just the -coordinate plus a single parity bit — saving ~50% space on every transaction.

2. Discrete points: Bitcoin doesn’t use the smooth curve — it uses a finite field (mod a prime ). All the point addition and multiplication still works, but on a scattered cloud of discrete points. The smooth curve is just for intuition.

With that in mind, here’s what they look like:

import numpy as np
import matplotlib.pyplot as plt

# --- Smooth curve (over real numbers, for intuition) ---
x = np.linspace(-2, 3, 1000)
y_sq = x**3 + 7
mask = y_sq >= 0
x = x[mask]
y_pos = np.sqrt(x**3 + 7)
y_neg = -y_pos

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 5))

ax1.plot(x, y_pos, 'b')
ax1.plot(x, y_neg, 'b')
ax1.axhline(0, color='k', linewidth=0.5)
ax1.axvline(0, color='k', linewidth=0.5)
ax1.set_title('y² = x³ + 7 (over reals)')
ax1.grid(True)

# --- Finite field (mod 97, for visualization) ---
p = 97  # small prime for visualization (Bitcoin uses a 256-bit prime)
points = []
for xf in range(p):
    for yf in range(p):
        if (yf*yf - xf**3 - 7) % p == 0:
            points.append((xf, yf))

xs, ys = zip(*points)
ax2.scatter(xs, ys, s=5, color='blue')
ax2.set_title(f'y² = x³ + 7 (mod {p}) — {len(points)} points')
ax2.grid(True)

plt.tight_layout()
plt.show()

print("Notice the symmetry: for every (x, y), the point (x, -y) also exists.")
print("This is why we only need to store x + a parity bit to compress a public key.")

Notice the symmetry: for every (x, y), the point (x, -y) also exists.
This is why we only need to store x + a parity bit to compress a public key.

Public Key Compression

Since , given there are only two possible values (one even, one odd). So a public key can be compressed from 65 bytes (full ) down to 33 bytes: just the full coordinate + a prefix byte (02 = even , 03 = odd ).

Address Generation Pipeline

private key (32 bytes)
    ↓  ECC multiplication
compressed public key (33 bytes)
    ↓  SHA256 → RIPEMD160
address hash (20 bytes)
    ↓  Base58Check encoding
Bitcoin address (e.g. 1A1zP1...)

How it works

1. Signing (sender): Hash the message, then combine the hash + private key using ECDSA to produce a signature.
2. Verifying (miner/anyone): Feed the message hash + signature + public key into the verification algorithm. Returns True if valid.

Properties

• Proof of Ownership: Only the private key holder could have produced the signature.
• Tamper-Proof: Changing even one character in the message invalidates the signature.

Note on the math

We don’t just hash(private_key + message). That would require the verifier to know the private key. Instead, ECDSA creates a mathematical puzzle that can be verified with just the public key, but only solved with the private key.

# Step 1: Create and hash a message
msg = b'Send 1 satoshi to Bob'
msg_hash = hash_fn(msg)

# Step 2: Generate a key pair
priv_key, pub_key = get_account()

# Step 3: Sign the hash with the private key
# The signature is 64 bytes, mathematically tied to both the message and private key.
signature = priv_key.ecdsa_sign(msg_hash)

# Step 4: Verify using the public key (anyone can do this)
is_valid = pub_key.ecdsa_verify(msg_hash, signature)
print(f"Signature valid: {is_valid}")

# Step 5: Tamper with the message — signature should fail
tampered_msg = b'Send 10 satoshi to Bob'  # 1 satoshi -> 10 satoshis
tampered_hash = hash_fn(tampered_msg)
is_valid_tampered = pub_key.ecdsa_verify(tampered_hash, signature)
print(f"Tampered signature valid: {is_valid_tampered}")

Signature valid: True
Tampered signature valid: False

UTXO = Unspent Transaction Output

Bitcoin doesn’t track “accounts” with balances. Instead, it tracks individual UTXOs — like cash bills. If you have a $20 bill and buy a $5 coffee: - Your $20 UTXO is consumed as an input - Two new UTXOs are created as outputs: $5 to the shop, $15 change back to you

Your “balance” = sum of all UTXOs your private key can unlock.

Spending multiple UTXOs

A transaction can have multiple inputs — just like pulling three $5 bills from your wallet to pay for a $12 item. Each input needs its own signature.

from fastcore.all import *

class UTXO:
    """An Unspent Transaction Output.
    - amount:   value in satoshis
    - pub_key:  the secp256k1.PublicKey that can spend this UTXO
    - spent:    whether this UTXO has been consumed by a transaction
    """
    def __init__(self, amount, pub_key, spent=False):
        store_attr()

class Transaction:
    """A Bitcoin-style transaction.
    - inputs:   list of UTXOs being consumed (must be unspent)
    - outputs:  list of new UTXOs being created
    - signatures: list of ECDSA signatures, one per input
    """
    def __init__(self, inputs, outputs):
        self.inputs  = inputs   # UTXOs we are spending
        self.outputs = outputs  # New UTXOs being created

    def validate_input(self):
        """Check that total inputs >= total outputs (you can't spend more than you have)."""
        input_amount  = sum(o.amount for o in self.inputs)
        output_amount = sum(o.amount for o in self.outputs)
        if input_amount >= output_amount: return True
        return False

    def transact(self):
        """Process the transaction: validate math and mark inputs as spent."""
        if not self.validate_input(): return f'Invalid transaction'
        for o in self.inputs: o.spent = True

    def get_hash(self, input_index):
        """Hash the transaction data for a specific input.
        The input_index ensures each input signs a slightly different hash,
        preventing replay attacks (same concept as real Bitcoin)."""
        tx_data = f"In:{[o.amount for o in self.inputs]} Out:{[o.amount for o in self.outputs]}; this is {input_index}"
        return hash_fn(tx_data.encode())

    def sign_input(self, input_index, priv_key):
        """Sign a specific input with the given private key.
        Each input gets its own signature — the wallet does this automatically."""
        if not hasattr(self, 'signatures'):
            self.signatures = [None] * len(self.inputs)
        self.signatures[input_index] = priv_key.ecdsa_sign(self.get_hash(input_index))

    def verify(self):
        """Verify all signatures. Each input's signature must be valid against
        the corresponding UTXO's public key."""
        return all(
            inp.pub_key.ecdsa_verify(self.get_hash(idx), sig)
            for idx, (inp, sig) in enumerate(zip(self.inputs, self.signatures))
        )

Putting it together: Alice sends Bitcoin to Bob

Alice has two UTXOs (50 + 30 = 80 satoshis). She sends 70 to Bob and keeps 10 as change. She signs each input, then we verify the whole transaction is authentic.

# 1. Generate keys for Alice and Bob
alice_priv, alice_pub = get_account()
bob_priv,   bob_pub   = get_account()

# 2. Create Alice's UTXOs (she owns these from previous transactions)
utxo1 = UTXO(amount=50, pub_key=alice_pub)
utxo2 = UTXO(amount=30, pub_key=alice_pub)

# 3. Build the transaction:
#    Inputs:  50 + 30 = 80 satoshis from Alice
#    Outputs: 70 to Bob, 10 change to Alice
tx = Transaction(
    inputs=[utxo1, utxo2],
    outputs=[UTXO(amount=70, pub_key=bob_pub), UTXO(amount=10, pub_key=alice_pub)]
)

# 4. Alice signs each input (wallet does this automatically in practice)
tx.sign_input(0, alice_priv)
tx.sign_input(1, alice_priv)

# 5. Verify — miners do this before including the transaction in a block
print(f"Transaction valid: {tx.verify()}")

# 6. Process the transaction — mark old UTXOs as spent
tx.transact()
print(f"Inputs spent:     {[u.spent for u in tx.inputs]}")

Transaction valid: True
Inputs spent:     [True, True]

What is a Block?

A block is a container for transactions, plus metadata that links it to the chain: - transactions: list of confirmed transactions - prev_block_hash: hash of the previous block (creates the “chain”) - nonce: the number miners brute-force to find a valid hash

How Mining Works

Miners repeatedly hash the block while incrementing the nonce, looking for a hash that starts with a target number of zeros. This is Proof of Work: - Hard to find: millions of guesses required - Easy to verify: anyone can hash once and check - Difficulty = number of leading zeros required

The first miner to find a valid nonce broadcasts the block. The network verifies it, and the block is added to the chain. The miner earns a block reward + transaction fees.

class Block:
    """A single block in the blockchain.
    - transactions:      list of Transaction objects in this block
    - prev_block_hash:   hex string linking to the previous block
    - nonce:             number used once — miners increment this to find a valid hash
    """
    def __init__(self, transactions, prev_block_hash, nonce=0):
        store_attr()

    def create_hash(self):
        """Hash the block contents: previous hash + transaction data + nonce."""
        trans_str = self.prev_block_hash
        for o in self.transactions: trans_str += str(o)
        trans_str += str(self.nonce)
        return hash_fn(trans_str.encode())

    def mine(self, difficulty: int = 2):
        """Brute-force the nonce until create_hash() starts with `difficulty` zeros.
        difficulty=4 means we need a hash starting with '0000'."""
        target = '0' * difficulty
        while not self.create_hash().hex().startswith(target):
            self.nonce += 1
        print(f"Mined block! Nonce: {self.nonce}")
        print(f"Hash:   {self.create_hash().hex()}")

The Genesis Block

The very first block in a blockchain. Its prev_block_hash is all zeros because there’s nothing before it. (Bitcoin’s real genesis block was mined by Satoshi Nakamoto on January 3, 2009.)

# The genesis block contains our Alice->Bob transaction.
# Its prev_block_hash is all zeros (nothing came before it).
genesis_block = Block(
    transactions=[tx],
    prev_block_hash="0" * 64
)

# Mine it! Higher difficulty = more leading zeros = much longer to compute.
# Try difficulty=2 first, then bump to 4 or 5 to see the difference.
genesis_block.mine(difficulty=4)

Mined block! Nonce: 60699
Hash:   0000599ce1c15af69c019c9b8563475beaeef688a6ced5923fa728edb833a551

Fork Resolution

When two miners find a valid block at almost the same time, the chain temporarily forks. The network resolves this by always following the longest chain (most cumulative work). The tie is broken when the next block is mined on one branch, and the other branch becomes an “orphan” that’s abandoned.

class Blockchain:
    """A simple blockchain that stores blocks and enforces hash linking."""
    def __init__(self):
        self.chain = []
        self.prev_block_hash = "0" * 64  # genesis prev hash

    def add_block(self, block):
        """Add a block only if its prev_block_hash matches the current chain tip."""
        if block.prev_block_hash != self.prev_block_hash: return False
        self.chain.append(block)
        self.prev_block_hash = self.chain[-1].create_hash().hex()

# 1. Initialize the blockchain
my_chain = Blockchain()

# 2. Add the genesis block (already mined above)
my_chain.add_block(genesis_block)
print(f"Chain length after genesis: {len(my_chain.chain)}")

# 3. Mine and add a second block
# Note: it links to the genesis block via my_chain.prev_block_hash
block_2 = Block(
    transactions=[],  # empty for simplicity
    prev_block_hash=my_chain.prev_block_hash
)

print("\nMining block 2...")
block_2.mine(difficulty=4)

my_chain.add_block(block_2)
print(f"\nChain length after block 2: {len(my_chain.chain)}")

Chain length after genesis: 1

Mining block 2...
Mined block! Nonce: 25841
Hash:   0000cc9875df0e8b172014556622fcf8a37289eafb7a5afe518c587c955852d2

Chain length after block 2: 2

What we built

Near-term extensions

• Merkle Trees — Instead of hashing transactions individually, Bitcoin hashes them into a binary tree. This lets you prove a transaction is in a block without downloading the whole block (SPV / “light clients”).
• Transaction Fees — The difference between inputs and outputs isn’t wasted — it goes to the miner. We already allow input > output; we’d just need to track and award that difference.
• Coinbase Transactions — Each block contains a special first transaction that creates new coins out of nowhere (the block reward). This is how new Bitcoin enters circulation.
• Difficulty Adjustment — Real Bitcoin adjusts difficulty every 2016 blocks (~2 weeks) to keep block time at ~10 minutes, regardless of total network hashpower.

Medium-term extensions

• Networking / P2P — Nodes that discover peers, gossip transactions, and propagate blocks. Even a simple simulated network would illustrate consensus.
• Wallet with Address Derivation — Hash the public key through SHA-256 → RIPEMD-160 → Base58Check to produce real-looking Bitcoin addresses.
• Script / Locking Conditions — Bitcoin transactions use a small stack-based scripting language. The simplest case is “must provide a signature matching this public key,” but it supports multisig, timelocks, and more.
• Mempool & Block Templates — A pool of unconfirmed transactions that miners select from, prioritizing higher fees.

Advanced

• SegWit / Witness Data — Separating signatures from the transaction data that gets hashed, fixing transaction malleability.
• SPV Verification — A lightweight client that verifies transactions using Merkle proofs without downloading the full blockchain.
• Simple Payment Verification vs. Full Nodes — Tradeoffs between trust, storage, and verification.
• Double-spend detection — Rejected conflicting transactions that try to spend the same UTXO twice.

The Heart of It All: The Value Class

The core idea is simple: wrap every number in a Value object that tracks: - Its value (the actual number) - Its gradient (how much it affects the final output — computed later during backprop) - Its children (which Values produced it, so we can trace the computation graph) - A _backward function (how to compute gradients for its inputs)

We’ll build this up one method at a time using fastcore’s @patch decorator, so each piece can be understood in isolation.

import math
from fastcore.all import patch

class Value:
    "Stores a single scalar value and its gradient"
    def __init__(self, val, _op='', _children=None):
        self.val  = val          # the actual number
        self.grad = 0            # gradient (filled in during backprop)
        self._op  = _op          # the operation that created this node (e.g. '+', '*')
        self._children = set(_children) if _children else set()  # inputs to this node
        self._backward = lambda: None  # will be replaced by each op

    def __repr__(self): 
        return f'Value(val={self.val}, grad={self.grad})'

    def _v(self, x): 
        "Wrap a plain number in a Value if needed"
        return x if isinstance(x, Value) else Value(x)

Addition and the Chain Rule

When we add two Values, the result is a new Value that remembers its parents. The key insight is the chain rule: if out = a + b, then:

So during backprop, the gradient of out flows directly to both a and b unchanged. We use += because a variable might be used in multiple operations, and gradients accumulate.

@patch
def __add__(self:Value, other):
    other = self._v(other)  # wrap plain numbers
    out = Value(self.val + other.val, '+', (self, other))
    def _backward():
        self.grad  += out.grad   # d(out)/d(self) = 1
        other.grad += out.grad   # d(out)/d(other) = 1
    out._backward = _backward
    return out

Multiplication

For out = a * b, the derivatives are:

So each input’s gradient is the other input’s value, scaled by the output’s gradient. This is where the chain rule really kicks in — we multiply the local derivative by the upstream gradient.

@patch
def __mul__(self:Value, other):
    other = self._v(other)
    out = Value(self.val * other.val, '*', (self, other))
    def _backward():
        self.grad  += out.grad * other.val  # d(out)/d(self) = other
        other.grad += out.grad * self.val   # d(out)/d(other) = self
    out._backward = _backward
    return out

Power

For out = a^b, we need the power rule:

Our implementation goes a bit further than Karpathy’s — he restricts the exponent to plain numbers (int/float). We allow Value exponents too, which is more general. We only compute the gradient for the base here since that’s what we need for division and negation.

@patch
def __pow__(self:Value, other):
    other = self._v(other)
    out = Value(self.val ** other.val, 'pow', (self, other))
    def _backward():
        self.grad += out.grad * other.val * self.val**(other.val - 1)
    out._backward = _backward
    return out

Activation Functions

Neural networks need non-linearities. We implement two:

• ReLU: max(0, x) — gradient is 1 if the input was positive, 0 otherwise. Simple and efficient, which is why it powers most modern deep networks.
• Tanh: squashes values into [-1, 1] — gradient is 1 - tanh²(x). Smoother than ReLU, which we’ll see helps with small networks.

We’ll come back to the tanh vs ReLU debate later — it turned out to be one of the most interesting things we learned!

@patch
def relu(self:Value):
    out = Value(max(0, self.val), 'relu', (self,))
    def _backward():
        self.grad += out.grad * (out.val > 0)  # gradient is 1 if positive, 0 otherwise
    out._backward = _backward
    return out

@patch
def tanh(self:Value):
    out = Value(math.tanh(self.val), 'tanh', (self,))
    def _backward():
        self.grad += out.grad * (1 - out.val**2)  # derivative of tanh is 1 - tanh²(x)
    out._backward = _backward
    return out

Derived Operators (Free Backward!)

Here’s a beautiful trick: subtraction, division, and negation don’t need their own backward functions. We define them in terms of __add__, __mul__, and __pow__:

• a - b = a + (-b)
• a / b = a * b⁻¹
• -a = a * -1

The __radd__, __rmul__, etc. methods handle cases where a plain number appears on the left (e.g. 2 * Value(3)). Python calls these “reflected” operators.

Since each of these just composes existing operators, the backward logic is already handled — the chain rule takes care of everything automatically.

@patch
def __radd__(self:Value, other): return self + other       # other + self
@patch
def __rmul__(self:Value, other): return self * other       # other * self
@patch
def __sub__(self:Value, other): return self + (-other)     # self - other
@patch
def __rsub__(self:Value, other): return (-self) + other    # other - self
@patch
def __truediv__(self:Value, other): return self * other**-1  # self / other
@patch
def __rtruediv__(self:Value, other): return self**-1 * other # other / self
@patch
def __neg__(self:Value): return self * -1                  # -self

The Backward Pass: Topological Sort

To compute gradients, we need to process nodes in the right order — from outputs back to inputs. This is called a topological sort: arrange the computation graph so that every node appears after all its children.

Then we:
1. Set the output’s gradient to 1 (seed)
2. Walk the sorted list in reverse, calling each node’s _backward()

Each _backward uses the chain rule to push gradients to its inputs. Because we go in reverse topological order, every node’s gradient is fully computed before we process it.

@patch
def backward(self:Value):
    # Build topological order: children before parents
    topo = []
    seen = set()
    def build(v):
        for c in v._children:
            if c not in seen:
                seen.add(c); build(c)
        topo.append(v)
    build(self)

    # Seed the output gradient, then propagate backwards
    self.grad = 1.0
    for n in topo[::-1]:  # reverse topo = parents before children
        n._backward()

Does It Work? A Simple Sanity Check

Before doing anything fancy, let’s test the basics. We’ll create two Values, do some arithmetic, and call backward() to see if the gradients make sense.

For d = a*b + c where c = a + b, with a=1, b=2: - d = 1*2 + 3 = 5 - ∂d/∂a = b + 1 = 3 (from both the a*b and a+b paths) - ∂d/∂b = a + 1 = 2 (same reasoning)

Let’s see if our engine gets this right:

a = Value(1.0)
b = Value(2.0)
c = a + b
d = a * b + c
d.grad = 1.0
d.backward()
a,b,c,d

((val=1.0, op=, grad=3.0, child=[]) ,
 (val=2.0, op=, grad=2.0, child=[]) ,
 (val=3.0, op=+, grad=1.0, child=[2.0, 1.0])    ,
 (val=5.0, op=+, grad=1.0, child=[2.0, 3.0])    )

Pushing Further: Division and Activation

Let’s try a more complex expression that uses division and ReLU: f = relu((a*b + c) / c).

This exercises the __truediv__ (which decomposes into __mul__ and __pow__), the relu activation, and the accumulation of gradients through multiple paths. If our engine handles this, we can be fairly confident it’s correct.

a = Value(-1.5)
b = Value(2.0)
c = a + b
d = a * b + c
f = d / c
f.relu()
f.backward()
a,b,c,d

((val=-1.5, op=, grad=16.0, child=[])   ,
 (val=2.0, op=, grad=9.0, child=[]) ,
 (val=0.5, op=+, grad=12.0, child=[-1.5, 2.0])  ,
 (val=-2.5, op=+, grad=2.0, child=[0.5, -3.0])  )

Are Our Gradients Actually Correct?

There’s a simple, beautiful way to verify analytical gradients: numerical gradient checking.

Instead of using our backprop engine, we compute the function twice — once normally, once with a tiny perturbation h to one input — and approximate the gradient as:

If our autograd is correct, the analytical and numerical gradients should match. it matches! 🎉

def act_grad(h):
    a = Value(-1.5); b = Value(2.0)
    c = a + b
    d = a * b + c
    f0 = d / c
    f0.relu()
    
    a = Value(-1.5); b = Value(2.0) + h
    c = a + b
    d = a * b + c
    f1 = d / c
    f1.relu()
    return (f1 - f0) / h
act_grad(0.0001)

(val=8.998200359942032, op=*, grad=0, child=[10000.0, 0.0008998200359942032])   

Seeing the Computation Graph

One of the nicest things about building autograd from scratch is that we can see the computation graph. Each Value knows its children and its operation, so we can walk the graph and draw it with Graphviz.

This makes it much easier to understand what’s happening — you can literally see how a * b + c decomposes into individual operations, and how gradients flow back through the same graph.

from graphviz import Digraph

def trace(root):
    nodes, edges = set(), set()
    def build(v):
        if v not in nodes:
            nodes.add(v)
            for c in v._children:
                edges.add((c, v))
                build(c)
    build(root)
    return nodes, edges

def draw(root):
    dot = Digraph(format='svg', graph_attr={'rankdir': 'LR'})
    nodes, edges = trace(root)
    for n in nodes:
        dot.node(str(id(n)), f"val={n.val} grad={n.grad}", shape='record')
    for c, p in edges:
        dot.edge(str(id(c)), str(id(p)), label=c._op)
    return dot

# Rebuild the computation graph from our relu test and draw it
a = Value(-1.5); b = Value(2.0)
c = a + b
d = a * b + c
f = d / c
f.relu()
f.backward()
draw(f)

From Autograd to Neurons: Building a Network

Now for the fun part — we have a working autograd engine, so let’s build a neural network on top of it!

The architecture mirrors a real framework: - Neuron: holds weights and a bias, computes w·x + b - Layer: a collection of neurons, outputs a list - MLP: stacks layers, applies tanh activation between layers (but not on the output)

We use tanh as our activation — we’ll explain why that matters in the next section. The parameters() methods collect all the learnable weights so we can update them during training.

import random

class Neuron:
    def __init__(self, nin):
        self.w = [Value(random.uniform(-1,1)) for _ in range(nin)]
        self.b = Value(0.0)
    def __call__(self, x):
        act = sum((wi*xi for wi,xi in zip(self.w, x)), Value(0.0))
        return act + self.b
    def parameters(self):
        return self.w + [self.b]

class Layer:
    def __init__(self, nin, nout):
        self.neurons = [Neuron(nin) for _ in range(nout)]
    def __call__(self, x):
        return [n(x) for n in self.neurons]
    def parameters(self):
        return [p for n in self.neurons for p in n.parameters()]

class MLP:
    def __init__(self, nin, nouts):
        sz = [nin] + nouts
        self.layers = [Layer(sz[i], sz[i+1]) for i in range(len(nouts))]
    def __call__(self, x):
        for layer in self.layers:
            x = [n.tanh() for n in layer(x)] if layer is not self.layers[-1] else layer(x)
        return x[0] if len(x)==1 else x
    def parameters(self):
        return [p for l in self.layers for p in l.parameters()]

Teaching It XOR

XOR is the classic “not linearly separable” problem — a single perceptron can’t solve it, but a small MLP can. It’s the perfect test for our engine.

We use mean squared error as the loss, plain SGD with a learning rate of 0.1, and train for 100 epochs. Each epoch:

1. Forward pass: run all 4 examples through the network
2. Compute loss: sum((pred - target)²)
3. Zero gradients (critical — otherwise they accumulate!)
4. Backward pass: loss.backward()
5. Update weights: w -= lr * w.grad

Let’s see if our little network can learn it:

# tiny dataset: binary classification
xs = [[0.0, 0.0], [0.0, 1.0], [1.0, 0.0], [1.0, 1.0]]
ys = [0.0, 1.0, 1.0, 0.0]  # XOR

net = MLP(2, [4, 1])

for epoch in range(100):
    # forward
    pred = [net(x) for x in xs]
    loss = sum(((p - y)**2 for p, y in zip(pred, ys)), Value(0.0))
    # zero grads
    for p in net.parameters():
        p.grad = 0.0

    # backward
    loss.backward()

    # update (SGD)
    for p in net.parameters():
        p.val -= 0.1 * p.grad

    if epoch % 10 == 0:
        print(f"epoch {epoch}: loss={loss.val:.4f}")

epoch 0: loss=1.8480
epoch 10: loss=0.8182
epoch 20: loss=0.6427
epoch 30: loss=0.4399
epoch 40: loss=2.2708
epoch 50: loss=0.3058
epoch 60: loss=0.1274
epoch 70: loss=0.1937
epoch 80: loss=0.9536
epoch 90: loss=0.0420

Why Tanh Works Better Here (And Why Deep Networks Moved On)

During our experiments, we noticed that tanh trained our XOR network much more smoothly than ReLU. This led us down an interesting rabbit hole.