Prompt

Evaluator

class core_genai.prompt.evaluator.PromptEvaluator(agent: IAgent, model: str, max_concurrent_tasks: int = 3, max_tokens: int = 4000, logger: Logger | None = None)

Bases: object

Generates test datasets, runs prompts, and grades outputs with a model-as-judge.

__init__(agent: IAgent, model: str, max_concurrent_tasks: int = 3, max_tokens: int = 4000, logger: Logger | None = None) → None

static render(template_string: str, variables: dict) → str

Replace {key} placeholders in template_string with values from variables.

generate_unique_ideas(task_description: str, prompt_inputs_spec: dict[str, str], num_cases: int) → list[str]

Ask the model for num_cases distinct scenario ideas for the given task.

generate_test_case(task_description: str, idea: str, prompt_inputs_spec: dict[str, str] | None = None) → TestCase

Generate a single structured test case from a scenario idea.

generate_dataset(task_description: str, prompt_inputs_spec: dict[str, str] | None = None, num_cases: int = 1, output_file: str = 'dataset.json') → list[TestCase]

Generate and persist a test dataset for the given task description.

grade_output(test_case: TestCase, output: str, extra_criteria: str | None) → dict

Score a prompt output against the test case criteria using the model.

run_test_case(test_case: TestCase, run_prompt_function: Callable[[dict[str, str]], str], extra_criteria: str | None = None) → EvaluationResult

Run run_prompt_function on a test case and return the graded result.

run_evaluation(run_prompt_function: Callable[[dict[str, str]], str], dataset_file: str, extra_criteria: str | None = None, json_output_file: str = 'output.json', html_output_file: str = 'output.html', pass_threshold: int = 7) → list[EvaluationResult]

Run the full evaluation pipeline on every test case in dataset_file.

core_genai.prompt.evaluator.add_user_message(messages: list, text: str) → None

Append a user turn to the messages list.

core_genai.prompt.evaluator.add_assistant_message(messages: list, text: str) → None

Append an assistant turn to the messages list.

Report

core_genai.prompt.report.generate_prompt_evaluation_report(evaluation_results: list[EvaluationResult], pass_threshold: int = 7) → str

Build and return a self-contained HTML evaluation report.

Contracts

class core_genai.prompt.types.TestCase

A single generated test case with inputs and grading criteria.

scenario: str

task_description: str

prompt_inputs: dict[str, str]

solution_criteria: list[str]

class core_genai.prompt.types.EvaluationResult

The graded output for one test case.

output: str

score: int

reasoning: str

strengths: list[str]

weaknesses: list[str]

test_case: TestCase