Prompt Evaluation =============================================================================== Prompt engineering is inherently empirical. A prompt that reads well rarely behaves well across the full diversity of real inputs, and a prompt that *seems* to improve things on a few manual test cases often regresses on others. The ``core_genai.prompt`` module gives you a systematic way to measure the difference. The problem it solves ------------------------------------------------------------------------------- When you refine a prompt: adding guidelines, inserting a few-shot example, tightening the format or changing the tone, you need to know whether the change actually improved outputs **on average**, not just on the three examples you happened to test by hand. Without measurement, prompt iteration is guesswork: * You cannot tell whether version B is better than version A, or just different. * You cannot catch regressions introduced while fixing a specific failure. * You cannot make an objective case to a team that one approach outperforms another. ``PromptEvaluator`` addresses this by automating the full loop: generating a diverse test dataset, running each prompt version against every case, and grading the outputs with a second model call (the *judge*). The result is a numeric score per case, an average across the dataset, and an HTML report you can share. Model-as-judge ------------------------------------------------------------------------------- The grading step uses a second LLM call, a *judge*, to evaluate each output against a set of criteria. This is sometimes called the **model-as-judge pattern** or LLM-as-evaluator. The judge receives: * The original task description and prompt inputs (so it knows what was asked). * The model's output (what was produced). * Per-case ``solution_criteria``: specific quality requirements generated alongside each test case. * ``extra_criteria``: mandatory requirements you define globally, applied to every case (for example: "all portion sizes must be in grams"). The judge returns a score from 1 to 10, a reasoning explanation, and lists of strengths and weaknesses. Crucially, the grading prompt instructs the judge to treat any violation of ``extra_criteria`` as an automatic failure (score ≤ 3), which makes hard constraints enforceable at evaluation time. **Why use a model as judge rather than a rule-based scorer?** Natural language outputs are hard to check with heuristics. A model can assess coherence, tone, completeness, and domain correctness in ways that regex or keyword checks cannot. The trade-off is that model grading introduces noise: the same output graded twice may receive slightly different scores. That is why dataset size matters, averages over 10+ cases are far more stable than a single graded example. The evaluation pipeline ------------------------------------------------------------------------------- .. code-block:: text ┌─────────────────────────────────────────┐ │ generate_dataset() │ │ │ │ 1. Ask the model for N diverse ideas │ │ 2. Expand each idea into a TestCase: │ │ - prompt_inputs (the variables) │ │ - solution_criteria (the rubric) │ │ 3. Persist to dataset.json │ └──────────────────┬──────────────────────┘ │ same dataset.json ┌──────────────────▼──────────────────────┐ │ run_evaluation() │ │ │ │ For each TestCase (concurrently): │ │ a. run_prompt_function(inputs) → str │ │ b. grade_output(output, criteria) │ │ → score, reasoning, │ │ strengths, weaknesses │ │ Aggregate → JSON + HTML report │ └─────────────────────────────────────────┘ The key design decision is that **dataset generation is separated from evaluation**. You generate the dataset once and reuse it across every prompt version you want to compare. This is essential: if the dataset changes between runs, differences in scores could come from different test cases rather than from the prompt change. Reuse the dataset across versions explicitly: .. code-block:: python # Generate once; subsequent runs skip regeneration if the file exists. if not Path("dataset.json").exists(): evaluator.generate_dataset(..., output_file="dataset.json") # Compare any number of prompt versions on the same cases. results_v1 = evaluator.run_evaluation(prompt_v1, dataset_file="dataset.json", ...) results_v2 = evaluator.run_evaluation(prompt_v2, dataset_file="dataset.json", ...) results_v3 = evaluator.run_evaluation(prompt_v3, dataset_file="dataset.json", ...) Scores and what they mean ------------------------------------------------------------------------------- The grader uses a 1-10 scale with a hard split at the mandatory criteria: +----------+----------------------------------------------------------------+ | Score | Meaning | +==========+================================================================+ | 1-3 | Failed a mandatory requirement (``extra_criteria`` violation). | +----------+----------------------------------------------------------------+ | 4-6 | Met mandatory requirements; significant gaps in secondary | | | criteria (per-case ``solution_criteria``). | +----------+----------------------------------------------------------------+ | 7-8 | Met all mandatory and most secondary criteria; minor issues. | +----------+----------------------------------------------------------------+ | 9-10 | Met all mandatory and secondary criteria. | +----------+----------------------------------------------------------------+ The pass threshold (default 7) controls the colour of score badges in the HTML report and the pass-rate statistic in the header. It does not affect grading; it is purely a reporting parameter. A meaningful average typically requires at least 10 test cases. With 3 cases, a single hard scenario shifts the average by more than 0.6 points per score point, which is enough to reverse an apparent ranking between two prompts. Designing good extra_criteria ------------------------------------------------------------------------------- ``extra_criteria`` are the mandatory requirements your prompt **must** satisfy. They are shown to the judge as automatic-failure conditions. Keep them: * **Structural, not subjective.** "Include daily caloric total" is checkable; "be helpful" is not. * **Exhaustive for your domain.** If a missing piece would make the output useless in production (missing portion sizes, missing citations, wrong currency), make it mandatory. * **Independent of the per-case rubric.** ``solution_criteria`` varies per test case and covers scenario-specific requirements. ``extra_criteria`` covers baseline requirements that apply to every output regardless of the specific scenario. Example for a meal-plan task: .. code-block:: python EXTRA_CRITERIA = """ The output must include: - Daily caloric total as a number - Macronutrient breakdown (protein, fat, carbs in grams) - Every meal with exact foods, portions in grams, and meal timing - Only foods that comply with the stated dietary restrictions """ A naive prompt that omits gram-level portions will consistently score ≤ 3 on these criteria, creating a clear gap against a version that enforces structure through guidelines and few-shot examples. That gap is the signal you are measuring. Choosing dataset size ------------------------------------------------------------------------------- There is a cost-accuracy trade-off: every test case requires at least two model calls (one to generate the prompt output, one for the judge), so a larger dataset costs more. * **3-5 cases** - useful for a quick sanity check or debugging a specific failure. Averages are noisy; a single outlier can flip a ranking. * **10-20 cases** - good for comparing two or three prompt versions. Variance is low enough that a genuine +0.5 average gap is reliable. * **50+ cases** - appropriate for a production-grade regression suite or when the task has high natural variance (open-ended generation, complex constraints). A practical approach: start with 10 cases, fix the dataset file, and iterate on prompts. Only regenerate the dataset when the task description or ``prompt_inputs_spec`` changes. Interpreting HTML reports ------------------------------------------------------------------------------- The HTML report produced by ``run_evaluation`` is self-contained (no external dependencies) and designed for sharing. Each row contains: * **Scenario** - the idea the test case was generated from. * **Prompt inputs** - the exact variable values passed to your prompt function. * **Criteria** - the per-case ``solution_criteria`` the judge used. * **Output** - the raw text returned by your prompt function, rendered in a monospace block. * **Score** - colour-coded badge (green = pass, yellow = borderline, red = fail) relative to ``pass_threshold``. * **Reasoning** - the judge's overall assessment, followed by bullet lists of strengths and weaknesses. The header shows total cases, average score, and pass rate at a glance. Open multiple reports side by side in a browser to visually compare prompt versions. Limitations ------------------------------------------------------------------------------- * **Grader noise.** Model-based grading is not deterministic. A single case graded at temperature 0.0 is reproducible, but the grader's calibration can shift slightly across runs or models. Averages over large datasets are robust; individual scores are not. * **Grader bias.** The judge may favour verbose, well-formatted outputs over accurate but terse ones. Calibrate ``extra_criteria`` to push back against any format biases you observe. * **Dataset quality.** Generated ``solution_criteria`` occasionally include constraints that are over-specified (e.g. "exactly 3 meals" when your prompt produces 6). Review the dataset after generation and regenerate if criteria are unreasonably narrow. * **Model as prompt target.** If you use the same model family for both generation and grading, the judge may be more forgiving of outputs that match its own generation style. Using a different model for the judge, or a different temperature, can reduce this.