Creating Judges

A judge in Evalm8 scores a single aspect of quality. You can define two kinds, LLM judges and deterministic judges.

→ To wire judges into evaluations, see Building Evals.

Judge Types at a Glance

	LLM Judge	Deterministic Judge
Type	`"llm"`	`"code"`
How it works	Sends a prompt to an LLM and parses a structured verdict	Runs a Python snippet in a sandboxed environment
Best for	Subjective reasoning: coherence, relevance, hallucination	Deterministic checks: schema validation, regex, exact match
Origin	`bespoke` (fixed prompt) or `fine_tuned` (learns from annotations)	`bespoke`

Creating a Judge in the UI

Go to Evaluation → Judges and click Create Judge.

Judge builder

The judge name becomes its slug ID (e.g. Correctness JUDGE → correctness-judge), which evals and pipelines reference.

Common Fields

Both judge types share:

Field	Required	Description
`name`	Yes	Unique name. Becomes the judge's slug ID.
`type`	Yes	`"llm"` or `"code"`
`inputs`	Yes	List of input fields the judge expects (at least one).
`mode`	No	Scoring mode and scale. Defaults to pointwise with integer 1–5.

Declaring Inputs

Every judge declares the variables it expects to receive:

"inputs": [
  {
    "name": "response",
    "target_type": "string",
    "description": "The LLM's raw output text"
  },
  {
    "name": "reference",
    "target_type": "string",
    "description": "The expected correct answer"
  }
]

Property	Required	Description
`name`	Yes	Variable name your code or prompt template references.
`target_type`	No	`string`, `integer`, `float`, `boolean`, `dict`, or `list`. Defaults to `string`. Evalm8 coerces each value to this type.
`description`	No	Human-readable description. Shown in the UI and used as context for LLM judges.
`annotation_template`	No	HTML template for rendering this field in the annotation/review interface.

A judge never extracts data from the trace itself. A pipeline (or the default heuristic pipeline) extracts values from the trace and maps them to these declared input names.

LLM Judges

An LLM judge uses a prompt template and a language model to reason about quality.

{
  "type": "llm",
  "name": "Correctness JUDGE",
  "inputs": [
    { "name": "query", "target_type": "string", "description": "The prompt sent to the model." },
    { "name": "response", "target_type": "string", "description": "The model response to evaluate." }
  ],
  "template": "You are an expert evaluator assessing the correctness of a model response. Score 1 to 5, then explain briefly.\n\nQuery: {{query}}\nResponse: {{response}}",
  "model": "google/gemini-2.5-flash",
  "mode": {
    "type": "pointwise",
    "scale": { "type": "integer", "min": 1, "max": 5 }
  }
}

Field	Required	Description
`template`	Yes	The prompt sent to the LLM. Reference inputs with `{{variable_name}}`.
`model`	No	Model provider URI (e.g. `google/gemini-2.5-flash`). Uses platform default if omitted.
`mode.scale`	No	The scoring scale the LLM should use. Defaults to integer 1–5.

Model providers

Before using a model for LLM judges, register it under Integrations → Model Providers.

Model providers

Bespoke vs Fine-Tuned

bespoke: a fixed prompt template. Scores are consistent but do not adapt to your domain without manual prompt editing.
fine_tuned: starts from the same template and learns from your golden dataset annotations through the refinement workflow. With each refinement cycle, its scores move closer to human judgment.

Deterministic Judges

A deterministic judge runs a Python snippet in a sandboxed environment. Use it for checks that don't need LLM reasoning.

{
  "type": "code",
  "name": "json_schema_validator",
  "inputs": [
    { "name": "response", "target_type": "string" },
    { "name": "schema", "target_type": "dict" }
  ],
  "code": "try:\n    data = json.loads(response)\n    jsonschema.validate(data, schema)\n    score = 1.0\n    reason = \"Valid JSON conforming to schema\"\nexcept json.JSONDecodeError:\n    score = 0.0\n    reason = \"Output is not valid JSON\"\nexcept jsonschema.ValidationError as e:\n    score = 0.0\n    reason = f\"Schema violation: {e.message}\""
}

Deterministic judge inputs

Your snippet runs with each declared input already available as a local variable. If you declared an input named response, write response in your code. You don't need to unpack anything.

Your snippet must set:

score (numeric): the judge's verdict. It defaults to 0.0 if not set.
reason (string, optional): an explanation. If you omit it, Evalm8 uses a generic fallback.

For ranking judges, set rank (a list of integers) instead of score.

Available in the Sandbox

Pre-imported modules (no import needed):

Module	Typical use
`math`	`math.log`, `math.sqrt`, `math.ceil`
`re`	Regex matching and substitution
`json`	Parse/dump JSON strings
`jsonschema`	Validate dicts against JSON Schema
`datetime`	Date/time construction and arithmetic
`collections`	`Counter`, `defaultdict`, `OrderedDict`
`itertools`	`chain`, `product`, `combinations`
`functools`	`reduce`, `partial`
`hashlib`	`md5`, `sha256`
`statistics`	`mean`, `median`, `stdev`
`copy`	`copy.copy`, `copy.deepcopy`

Built-in functions: int, float, str, bool, list, dict, set, tuple, len, sum, max, min, round, sorted, range, enumerate, zip, map, filter, any, all, isinstance, abs, print (for debugging), and standard exception types.

Not available: File I/O, networking, system access, or any module not listed above.

Error Handling

If your snippet raises an unhandled exception, Evalm8 reports the error with the offending line number. Use try/except to catch errors and set a score yourself:

try:
    parsed = json.loads(response)
    score = 1.0 if "name" in parsed else 0.0
    reason = "Has 'name' key" if score else "Missing 'name' key"
except json.JSONDecodeError:
    score = 0.0
    reason = "Not valid JSON"

Deterministic Judge Examples

Regex Pattern Gate

pattern = r"\b\d{4}-\d{2}-\d{2}\b"
matches = re.findall(pattern, response)
score = 1.0 if matches else 0.0
reason = f"Found {len(matches)} date(s)" if matches else "No ISO-8601 date found"

Word-Overlap Similarity

ref_tokens = set(reference.lower().split())
res_tokens = set(response.lower().split())
overlap = len(ref_tokens & res_tokens) / len(ref_tokens) if ref_tokens else 1.0
score = round(overlap, 3)
reason = f"Token overlap: {score}"

Exact Match with Normalisation

normalise = lambda s: re.sub(r'\s+', ' ', s.strip().lower())
score = 1.0 if normalise(response) == normalise(expected) else 0.0
reason = "Exact match" if score else "Does not match expected output"

Length Validation

length = len(response.strip())
if length < 10:
    score = 0.0
    reason = "Response too short"
elif length > 2000:
    score = 0.5
    reason = "Excessively long"
else:
    score = 1.0
    reason = f"Acceptable length ({length} chars)"

JSON Schema Validation (with config)

This judge receives schema via judge configuration on the eval binding, so you can reuse it across dimensions:

try:
    data = json.loads(response)
    jsonschema.validate(data, schema)
    score = 1.0
    reason = "Response conforms to the required schema"
except json.JSONDecodeError:
    score = 0.0
    reason = "Response is not valid JSON"
except jsonschema.ValidationError as e:
    score = 0.0
    reason = f"Schema violation: {e.message}"

Next steps

Guide	Covers
Core Concepts	Traces, Rubrics, Dimensions, and the evaluation flow
Authoring Pipelines	How to extract trace data and route it to judges
Building Evals	Wiring judges to rubric dimensions, configuration, the default pipeline
Workflows	Datasets, annotation, and judge refinement

Judge Types at a Glance​

Creating a Judge in the UI​

Common Fields​

Declaring Inputs​

LLM Judges​

Bespoke vs Fine-Tuned​

Deterministic Judges​

Deterministic judge inputs​

Available in the Sandbox​

Error Handling​

Deterministic Judge Examples​

Regex Pattern Gate​

Word-Overlap Similarity​

Exact Match with Normalisation​

Length Validation​

JSON Schema Validation (with config)​

Next steps​