Building an LLM Evaluation Framework in Python

The first time I shipped a prompt change to an assistant without an eval set, the demo looked sharper and support tickets got worse. One reply that used to say "I do not know" started inventing a confident answer. Another that used to cite a doc started paraphrasing it into something close but wrong. Nothing in CI failed, because CI only checked that the process started.

This post is a build guide for a small evaluation harness I would put next to any assistant or coding helper before the next prompt edit. It is not tied to a product brand. The shape works for FAQ desks and coding helpers where "looks good in chat" is not a release criterion. If you already split Node and Python as in Node.js + Python architecture for AI apps, this lives on the Python side next to the model code.

What you are actually measuring

An LLM eval framework answers one question: did this change make the system better on tasks we care about, or did we trade one failure for another?

I keep three layers, in order of trust:

1. Deterministic checks: shape, required fields, banned phrases, citation presence, fixed substrings, hidden unit tests.
2. Rubric scores: a judge (human or model) scores dimensions you define (correctness, refusal, citation fidelity).
3. Regression set: a frozen JSONL of cases you re-run on every prompt, model, or tool change.

Skip dashboards until the JSONL loop is boring and reliable.

A tiny case format

Each case is one row you can edit in a text editor.

{"id": "faq-001", "task": "faq", "input": {"question": "Do I need CAC registration before a business bank account?"}, "expect": {"must_include": ["CAC"], "must_not_include": ["I am a lawyer"], "max_chars": 800}}
{"id": "code-012", "task": "coding", "input": {"prompt": "Write a Python function is_even(n) with type hints."}, "expect": {"must_compile": true, "hidden_tests": [{"call": "is_even(2)", "eq": true}, {"call": "is_even(3)", "eq": false}]}}

Rules I stick to: stable id values (add faq-001b if meaning changes), a task tag for per-category pass rates, and expect as data fields rather than prose judges must interpret.

The runner skeleton

Inject the system under test as a function so the same harness can score a raw chain or a FastAPI client.

from __future__ import annotations

import json
import time
from dataclasses import asdict, dataclass
from pathlib import Path
from typing import Any, Callable

SystemFn = Callable[[dict[str, Any]], dict[str, Any]]


@dataclass
class CaseResult:
    id: str
    task: str
    passed: bool
    checks: dict[str, bool]
    scores: dict[str, float]
    latency_ms: int
    output: dict[str, Any]
    error: str | None = None


def load_cases(path: Path) -> list[dict[str, Any]]:
    return [json.loads(line) for line in path.read_text().splitlines() if line.strip()]


def check_deterministic(output: dict[str, Any], expect: dict[str, Any]) -> dict[str, bool]:
    text = output.get("text", "")
    checks: dict[str, bool] = {}
    for needle in expect.get("must_include", []):
        checks[f"include:{needle}"] = needle.lower() in text.lower()
    for needle in expect.get("must_not_include", []):
        checks[f"exclude:{needle}"] = needle.lower() not in text.lower()
    if "max_chars" in expect:
        checks["max_chars"] = len(text) <= int(expect["max_chars"])
    if expect.get("must_compile"):
        try:
            compile(text, "<case>", "exec")
            checks["compiles"] = True
        except SyntaxError:
            checks["compiles"] = False
    for i, t in enumerate(expect.get("hidden_tests", [])):
        ns: dict[str, Any] = {}
        try:
            exec(compile(text, "<case>", "exec"), ns, ns)
            checks[f"hidden:{i}"] = eval(t["call"], ns, ns) == t["eq"]
        except Exception:
            checks[f"hidden:{i}"] = False
    return checks


def run_suite(cases_path: Path, system: SystemFn, report_path: Path) -> float:
    results: list[CaseResult] = []
    for case in load_cases(cases_path):
        t0 = time.perf_counter()
        err = None
        try:
            out = system(case["input"])
        except Exception as e:
            out, err = {"text": ""}, str(e)
        checks = check_deterministic(out, case.get("expect", {}))
        passed = err is None and (all(checks.values()) if checks else True)
        results.append(
            CaseResult(
                id=case["id"],
                task=case.get("task", "unknown"),
                passed=passed,
                checks=checks,
                scores={},
                latency_ms=int((time.perf_counter() - t0) * 1000),
                output=out,
                error=err,
            )
        )
    report_path.write_text("\n".join(json.dumps(asdict(r)) for r in results) + "\n")
    return sum(1 for r in results if r.passed) / max(len(results), 1)

Wire system to the same prompt, temperature, and timeout production uses. Scoring a different path is how you get green reports and red chats.

Rubric scores (when strings are not enough)

Deterministic checks miss polite, well-formed wrong answers. Add a short rubric and either score 20 to 50 seed cases by hand or use a second model at temperature 0.

Never let a judge override a failed deterministic check. Ask the judge for JSON only ({"correctness": 2, "refusal": 2, "citation": 1, "notes": "..."}). Treat parse failure as needs_review, not a silent pass.

Coding replies: hidden tests matter

For coding assistants: ask for a single function, compile the extracted code, run hidden tests the model never saw, and treat invented big-O claims as a separate check. Locally, exec in a fresh dict is enough to start on a trusted machine. Before untrusted output on a shared host, run the suite in a container with no network and tight CPU/memory limits.

JSONL reports and how to read them

Every run should write reports/<label>.jsonl (one line per case) plus a one-line summary: overall pass rate, pass rate by task, p50/p95 latency. Diff failed ids, not average score. Averages hide the three cases that used to refuse medical advice and now invent dosage. Version the prompt and model in metadata; without that, last week's 92% is meaningless next to this week's 94% on a different model.

Failure modes I plan for

1. Eval set contamination: only demo happy paths. Add adversarial and "should refuse" rows early.
2. Flaky judges: temperature above 0, or free-form essays. Fix the schema.
3. Training on the test: editing prompts until 30 known cases pass. Keep a held-out slice for release candidates.
4. Wrong system under test: scoring a notebook while production uses another system prompt.
5. Cost blindness: a 2% accuracy bump that triples token spend can still be a product loss.

How this fits a side-project setup

evals/
  cases/faq.jsonl
  cases/coding.jsonl
  run_suite.py
  reports/

Run it in CI on PRs that touch prompts or model config. Fail if pass rate drops more than a threshold (for example 3 points) or if any severity: blocker case fails. The same runner works when system calls a real retrieval or tool chain, like the small desk in What LangChain Is.

Takeaways

• Treat eval as a product dependency: no prompt or model change without a report.
• Prefer deterministic checks first; add rubrics only where strings cannot decide.
• Store cases and results as JSONL so diffs and grep stay easy.
• Score the same code path users hit, with temperature and timeouts pinned.
• Read failed case ids, not only the headline pass rate.