Prompt Evaluation#

Prompt engineering is inherently empirical. A prompt that reads well rarely behaves well across the full diversity of real inputs, and a prompt that seems to improve things on a few manual test cases often regresses on others. The core_genai.prompt module gives you a systematic way to measure the difference.

The problem it solves#

When you refine a prompt: adding guidelines, inserting a few-shot example, tightening the format or changing the tone, you need to know whether the change actually improved outputs on average, not just on the three examples you happened to test by hand. Without measurement, prompt iteration is guesswork:

  • You cannot tell whether version B is better than version A, or just different.

  • You cannot catch regressions introduced while fixing a specific failure.

  • You cannot make an objective case to a team that one approach outperforms another.

PromptEvaluator addresses this by automating the full loop: generating a diverse test dataset, running each prompt version against every case, and grading the outputs with a second model call (the judge). The result is a numeric score per case, an average across the dataset, and an HTML report you can share.

Model-as-judge#

The grading step uses a second LLM call, a judge, to evaluate each output against a set of criteria. This is sometimes called the model-as-judge pattern or LLM-as-evaluator.

The judge receives:

  • The original task description and prompt inputs (so it knows what was asked).

  • The model’s output (what was produced).

  • Per-case solution_criteria: specific quality requirements generated alongside each test case.

  • extra_criteria: mandatory requirements you define globally, applied to every case (for example: “all portion sizes must be in grams”).

The judge returns a score from 1 to 10, a reasoning explanation, and lists of strengths and weaknesses. Crucially, the grading prompt instructs the judge to treat any violation of extra_criteria as an automatic failure (score ≤ 3), which makes hard constraints enforceable at evaluation time.

Why use a model as judge rather than a rule-based scorer? Natural language outputs are hard to check with heuristics. A model can assess coherence, tone, completeness, and domain correctness in ways that regex or keyword checks cannot. The trade-off is that model grading introduces noise: the same output graded twice may receive slightly different scores. That is why dataset size matters, averages over 10+ cases are far more stable than a single graded example.

The evaluation pipeline#

┌─────────────────────────────────────────┐
│          generate_dataset()             │
│                                         │
│  1. Ask the model for N diverse ideas   │
│  2. Expand each idea into a TestCase:   │
│       - prompt_inputs (the variables)   │
│       - solution_criteria (the rubric)  │
│  3. Persist to dataset.json             │
└──────────────────┬──────────────────────┘
                   │  same dataset.json
┌──────────────────▼──────────────────────┐
│           run_evaluation()              │
│                                         │
│  For each TestCase (concurrently):      │
│    a. run_prompt_function(inputs) → str │
│    b. grade_output(output, criteria)    │
│         → score, reasoning,             │
│            strengths, weaknesses        │
│  Aggregate → JSON + HTML report         │
└─────────────────────────────────────────┘

The key design decision is that dataset generation is separated from evaluation. You generate the dataset once and reuse it across every prompt version you want to compare. This is essential: if the dataset changes between runs, differences in scores could come from different test cases rather than from the prompt change.

Reuse the dataset across versions explicitly:

# Generate once; subsequent runs skip regeneration if the file exists.
if not Path("dataset.json").exists():
    evaluator.generate_dataset(..., output_file="dataset.json")

# Compare any number of prompt versions on the same cases.
results_v1 = evaluator.run_evaluation(prompt_v1, dataset_file="dataset.json", ...)
results_v2 = evaluator.run_evaluation(prompt_v2, dataset_file="dataset.json", ...)
results_v3 = evaluator.run_evaluation(prompt_v3, dataset_file="dataset.json", ...)

Scores and what they mean#

The grader uses a 1-10 scale with a hard split at the mandatory criteria:

Score

Meaning

1-3

Failed a mandatory requirement (extra_criteria violation).

4-6

Met mandatory requirements; significant gaps in secondary criteria (per-case solution_criteria).

7-8

Met all mandatory and most secondary criteria; minor issues.

9-10

Met all mandatory and secondary criteria.

The pass threshold (default 7) controls the colour of score badges in the HTML report and the pass-rate statistic in the header. It does not affect grading; it is purely a reporting parameter.

A meaningful average typically requires at least 10 test cases. With 3 cases, a single hard scenario shifts the average by more than 0.6 points per score point, which is enough to reverse an apparent ranking between two prompts.

Designing good extra_criteria#

extra_criteria are the mandatory requirements your prompt must satisfy. They are shown to the judge as automatic-failure conditions. Keep them:

  • Structural, not subjective. “Include daily caloric total” is checkable; “be helpful” is not.

  • Exhaustive for your domain. If a missing piece would make the output useless in production (missing portion sizes, missing citations, wrong currency), make it mandatory.

  • Independent of the per-case rubric. solution_criteria varies per test case and covers scenario-specific requirements. extra_criteria covers baseline requirements that apply to every output regardless of the specific scenario.

Example for a meal-plan task:

EXTRA_CRITERIA = """
The output must include:
- Daily caloric total as a number
- Macronutrient breakdown (protein, fat, carbs in grams)
- Every meal with exact foods, portions in grams, and meal timing
- Only foods that comply with the stated dietary restrictions
"""

A naive prompt that omits gram-level portions will consistently score ≤ 3 on these criteria, creating a clear gap against a version that enforces structure through guidelines and few-shot examples. That gap is the signal you are measuring.

Choosing dataset size#

There is a cost-accuracy trade-off: every test case requires at least two model calls (one to generate the prompt output, one for the judge), so a larger dataset costs more.

  • 3-5 cases - useful for a quick sanity check or debugging a specific failure. Averages are noisy; a single outlier can flip a ranking.

  • 10-20 cases - good for comparing two or three prompt versions. Variance is low enough that a genuine +0.5 average gap is reliable.

  • 50+ cases - appropriate for a production-grade regression suite or when the task has high natural variance (open-ended generation, complex constraints).

A practical approach: start with 10 cases, fix the dataset file, and iterate on prompts. Only regenerate the dataset when the task description or prompt_inputs_spec changes.

Interpreting HTML reports#

The HTML report produced by run_evaluation is self-contained (no external dependencies) and designed for sharing. Each row contains:

  • Scenario - the idea the test case was generated from.

  • Prompt inputs - the exact variable values passed to your prompt function.

  • Criteria - the per-case solution_criteria the judge used.

  • Output - the raw text returned by your prompt function, rendered in a monospace block.

  • Score - colour-coded badge (green = pass, yellow = borderline, red = fail) relative to pass_threshold.

  • Reasoning - the judge’s overall assessment, followed by bullet lists of strengths and weaknesses.

The header shows total cases, average score, and pass rate at a glance. Open multiple reports side by side in a browser to visually compare prompt versions.

Limitations#

  • Grader noise. Model-based grading is not deterministic. A single case graded at temperature 0.0 is reproducible, but the grader’s calibration can shift slightly across runs or models. Averages over large datasets are robust; individual scores are not.

  • Grader bias. The judge may favour verbose, well-formatted outputs over accurate but terse ones. Calibrate extra_criteria to push back against any format biases you observe.

  • Dataset quality. Generated solution_criteria occasionally include constraints that are over-specified (e.g. “exactly 3 meals” when your prompt produces 6). Review the dataset after generation and regenerate if criteria are unreasonably narrow.

  • Model as prompt target. If you use the same model family for both generation and grading, the judge may be more forgiving of outputs that match its own generation style. Using a different model for the judge, or a different temperature, can reduce this.