How to Evaluate AI-Generated Python Code Safely

A coding helper returned is_even with two examples: is_even(2) was true and is_even(3) was false. It also claimed O(1). I ran values that were not in the prompt. None raised TypeError. True returned true, because bool is a subclass of int. A larger input showed a loop. The sample tests could not catch that. The model had already seen them.

Examples in the prompt are part of the prompt, not an evaluation set. This is the scoring path I put on top of the harness in Building an LLM Evaluation Framework in Python: hidden tests, a process boundary, and a complexity check that does not trust the prose.

Sample tests are not hidden tests

If the case file is the text you pasted into the chat, you are grading recall. I keep two files:

• public.jsonl: the short spec and one or two examples that are allowed in the prompt.
• hidden.jsonl: the cases that decide pass or fail. These never go into the prompt, the system message, or a path the model can read.

A hidden case names the function, the arguments, and either an expected value or an expected exception. For is_even, I want True rejected, not treated as 1.

{"fn": "is_even", "args": [2], "expect": true}
{"fn": "is_even", "args": [3], "expect": false}
{"fn": "is_even", "args": [true], "error": "TypeError"}
{"fn": "is_even", "args": [null], "error": "TypeError"}

JSON true must become Python True, and JSON null must become None, before the call. Floats need an explicit tolerance: 0.1 + 0.2 == 0.3 is false. Use math.isclose only when the case sets tol. And do not use bare == when a boolean is involved. True == 1 is true in Python, so a check that should reject True will pass a function that returns 1 unless you compare types.

import math


def matches(got: object, case: dict) -> bool:
    if "error" in case:
        return False
    exp = case["expect"]
    if isinstance(got, bool) or isinstance(exp, bool):
        return type(got) is type(exp) and got == exp
    tol = case.get("tol")
    if tol is not None and isinstance(got, (int, float)) and isinstance(exp, (int, float)):
        return math.isclose(float(got), float(exp), abs_tol=float(tol))
    return got == exp

Run the snippet in another process

exec inside the eval worker is fine for a toy on a machine you trust. It is the wrong default once the snippet is untrusted, or once a bad reply can touch the disk that stores your hidden cases. I write the extracted function to a temp directory and run a child interpreter.

An import denylist is a fast reject, not a sandbox. Walking the AST catches import os. It does not catch __import__("os") or a name built at runtime. I still refuse the obvious replies before spending a process. Anything that passes the AST still runs with a timeout and a CPU cap.

import ast
import json
import subprocess
import tempfile
from pathlib import Path

BANNED = {"os", "subprocess", "socket", "shutil", "ctypes", "importlib", "pathlib"}


def banned_imports(code: str) -> set[str]:
    found: set[str] = set()
    for node in ast.walk(ast.parse(code)):
        if isinstance(node, ast.Import):
            names = [alias.name for alias in node.names]
        elif isinstance(node, ast.ImportFrom) and node.module:
            names = [node.module]
        else:
            continue
        for name in names:
            root = name.split(".", 1)[0]
            if root in BANNED:
                found.add(root)
    return found


CHILD = r"""
import json, resource, sys
resource.setrlimit(resource.RLIMIT_CPU, (1, 1))
resource.setrlimit(resource.RLIMIT_FSIZE, (1_000_000, 1_000_000))
ns: dict = {}
exec(compile(open("solution.py", encoding="utf-8").read(), "solution.py", "exec"), ns, ns)
fn = ns[sys.argv[1]]
out = []
for case in json.loads(open("cases.json", encoding="utf-8").read()):
    try:
        got = fn(*case["args"])
    except Exception as exc:
        out.append({"ok": type(exc).__name__ == case.get("error")})
        continue
    exp = case.get("expect")
    if "error" in case:
        ok = False
    elif isinstance(got, bool) or isinstance(exp, bool):
        ok = type(got) is type(exp) and got == exp
    else:
        ok = got == exp
    out.append({"ok": ok})
print(json.dumps(out))
"""


def run_hidden(code: str, fn_name: str, cases: list[dict], timeout_s: float = 3.0) -> list[dict]:
    try:
        bad = banned_imports(code)
    except SyntaxError:
        return [{"ok": False, "error": "syntax"}]
    if bad:
        return [{"ok": False, "error": "banned:" + ",".join(sorted(bad))}]
    with tempfile.TemporaryDirectory() as tmp:
        root = Path(tmp)
        (root / "solution.py").write_text(code, encoding="utf-8")
        (root / "cases.json").write_text(json.dumps(cases), encoding="utf-8")
        (root / "child.py").write_text(CHILD, encoding="utf-8")
        try:
            proc = subprocess.run(
                ["python3", "-I", "child.py", fn_name],
                cwd=root,
                capture_output=True,
                text=True,
                timeout=timeout_s,
            )
        except subprocess.TimeoutExpired:
            return [{"ok": False, "error": "timeout"}]
    if proc.returncode != 0 or not proc.stdout.strip():
        return [{"ok": False, "error": "runner_failed"}]
    return json.loads(proc.stdout)

python -I skips user site packages. It does not remove the network. The timeout kills that direct child; if the snippet starts more processes, kill the process group. I skip RLIMIT_AS because a tight address-space cap often kills the interpreter during startup. On a shared host, mount the same directory read-only in a container with no network, a memory cap, and a non-root user. PYTHONDONTWRITEBYTECODE=1 avoids a write to __pycache__ on that mount.

docker run --rm --network none --memory 256m --cpus 0.5 --pids-limit 64 \
  --read-only --tmpfs /tmp:rw,noexec,size=16m --user 65534 \
  -e PYTHONDONTWRITEBYTECODE=1 -v "$WORKDIR":/work:ro python:3.12-slim \
  python -I /work/child.py is_even

If the model can read cases.json, you are grading recall again. If the interpreter is OOM-killed, raise the memory cap. Do not drop it to let a leaky solution pass.

Complexity claims are a separate check

"O(1)" in a comment is a string. I do not fail CI on one wall-clock sample. Timing ratios move with GC, CPU steal, and a cold cache.

If the reply claims O(1) and the body contains a for or while, the claim fails. A loop over a fixed constant is a false positive, so a case can set allow_loops when a loop is honest. Broader claims (a sort, an amortized dict lookup) stay as notes until you have a counter you trust.

def o1_claim_is_false(code: str, claim: str, allow_loops: bool) -> bool:
    if claim.replace(" ", "") not in {"O(1)", "O(1)time"}:
        return False
    if allow_loops:
        return False
    tree = ast.parse(code)
    return any(isinstance(node, (ast.For, ast.While)) for node in ast.walk(tree))

Flaky judges

A second model scoring "is this idiomatic" will change its mind if temperature is above 0. Pin temperature to 0, demand JSON with a fixed schema, and treat a parse failure as needs_review. The judge never overrides a failed hidden test.

Other flakes: tests that depend on set or dict order, solutions that share a module cache because the worker ran them in one process (use a fresh process per solution), and a judge that reads the prose instead of the code. Extract one fenced block, or the named function. Two functions when the contract asked for one is a fail.

What I fail the run on

Pass means every hidden case returned ok, the import gate did not fire, the process exited in time, and any O(1) claim survived the AST check. Write one JSONL row per solution (id, pass, failed case indexes, claim result, latency) and diff the failed ids. A mean score hides three solutions that crash on None. The harness system function returns the code string. This scorer is the deterministic check.

Takeaways

• Examples in the prompt are public. Hidden cases decide pass or fail, and they stay out of the model context.
• Run untrusted Python in a short-lived process: timeout, CPU cap, and a container with no network on a shared host.
• An AST import denylist is a fast reject. It is not a security boundary.
• Block on a narrow complexity check (O(1) plus a loop). Do not block on one timing sample.
• A model judge can add a rubric. It cannot overrule a failed test, and it has to return JSON at temperature 0.