Building Evals

An Eval is the executable form of a Rubric. It binds each rubric dimension to specific judges, with optional custom pipelines and per-judge configuration.

→ For defining judges, see Creating Judges.

→ For pipeline authoring, see Authoring Pipelines.

Creating an Eval in the UI

Go to Evaluation → Evals and click Create Eval. Select a rubric, then configure a judge binding for each dimension.

Eval Structure

{
  "name": "product_quality_eval",
  "rubric_uri": "eart://rubrics/product_quality",
  "bindings": [ ... ]
}

Field	Description
name	Unique name for this eval.
rubric_uri	URI of the rubric this eval implements.
bindings	One JudgeBinding per rubric dimension, in positional order. The first binding grades the first dimension, the second the second, and so on.

Judge Bindings

Each binding is a recipe for scoring one rubric dimension.

{
  "judges": [
    {
      "id": "hallucination",
      "judge_uri": "eart://judges/bespoke/hallucination_check",
      "config": null
    }
  ],
  "pipeline": null
}

Judge References

Each entry in judges is a reference to a stored judge:

Field	Required	Description
id	Yes	Local identifier for this judge within this eval. Used as judge_id in pipeline call steps.
judge_uri	Yes	URI of the judge artifact. Append ?ref=v3 to pin a version. Omit it for latest.
config	No	Per-judge configuration dictionary merged into the pipeline context.

Use a local id

The id is a local alias scoped to this eval. It lets you configure and reference the same judge as distinct entries:

• The same judge can appear multiple times with different configurations. A pipeline might call a generic "page_comparison" judge once for URL matching and once for title matching, each with a different config.
• Pipeline call steps reference judges by id, not by URI.

{
  "judges": [
    {
      "id": "url_match",
      "judge_uri": "eart://judges/bespoke/page_comparison",
      "config": { "comparison_field": "url" }
    },
    {
      "id": "title_match",
      "judge_uri": "eart://judges/bespoke/page_comparison",
      "config": { "comparison_field": "title" }
    }
  ],
  "pipeline": { "steps": [ ... ] }
}

Both point to the same judge artifact, but the pipeline calls them as two separate judges via "judge_id": "url_match" and "judge_id": "title_match".

Judge Configuration

The config dictionary makes judges reusable across bindings. The executor merges config values into the pipeline execution context before it evaluates input_mapping. You can reference them in mapping paths or auto-wire them by name. A judge input named schema connects to a config key named schema.

Common uses:

• JSON schemas for a generic validation judge
• Thresholds: { "min_length": 100, "max_length": 2000 }
• Comparison parameters: { "comparison_field": "url" }
• Reference data: { "expected_output": "Paris" }

Example: Reusable Schema Judge

Judge (created once):

{
  "type": "code",
  "name": "json_schema_validator",
  "inputs": [
    { "name": "response", "target_type": "string" },
    { "name": "schema", "target_type": "dict" }
  ],
  "code": "..."
}

Eval binding (reused with different schemas):

{
  "judges": [{
    "id": "product_schema",
    "judge_uri": "eart://judges/bespoke/json_schema_validator",
    "config": {
      "schema": {
        "type": "object",
        "required": ["product_id", "name", "price"]
      }
    }
  }]
}

The schema config auto-wires to the judge's schema input. The pipeline only needs to provide response.

The Default Pipeline

When a binding has a single judge and no pipeline, Evalm8 synthesises a default two-step pipeline:

Step 1: heuristic extract  →  Step 2: call the judge

The heuristic resolver extracts common inputs from the last LLM span in the trace.

Heuristic inputs

Variable	Source	Description
query	Last user-role message across all LLM spans	The most recent user turn
response	Last assistant-role output, or last tool call	The model's response
instruction	Last system-role message across all LLM spans	The system prompt
tool_calls	Last message.tool_calls from output messages	Tool invocations
tool_definitions	input.value.tools attribute	Tool schemas available to the model
history	All input messages except the last from the target span, or all inputs from prior LLM spans	Conversation history

Shim Mappings

Evalm8 maps these domain-specific judge input names for you:

Judge input name	Maps to
plan	query
tool_responses	query
action	response
observation	response
reflection	response
reference_response	reference
context	instruction
history	instruction
messages	instruction
memory	instruction
reference_tool_calls	tool_calls

An agent-oriented judge expecting action and observation works without custom extraction.

Context priority

If a requested field name already exists in the execution context from a prior step, the heuristic skips re-extraction and uses the existing value. This lets you override specific fields while using heuristic extraction for the rest.

Default pipeline criteria

Use the default when:

• Your judge's inputs match the standard names above (or their shim aliases).
• Your trace follows OpenInference conventions.
• You don't need guards, list iteration, branching, or complex transforms.

Write an explicit pipeline when:

• Your trace uses non-standard attributes or needs regex/JSON parsing.
• You iterate over a list of items (map/reduce).
• You need guards to reject invalid traces.
• You branch on content type (switch).
• You orchestrate multiple judges together.

Multi-Judge Bindings

When a binding has more than one judge, you must provide a pipeline.

Sequential Judges

{
  "judges": [
    { "id": "extract_facts", "judge_uri": "eart://judges/bespoke/fact_extractor" },
    { "id": "verify_facts", "judge_uri": "eart://judges/bespoke/fact_verifier" }
  ],
  "pipeline": {
    "steps": [
      { "id": "inputs", "type": "extract", "resolver": { "type": "heuristic" } },
      { "id": "facts", "type": "call", "judge_id": "extract_facts",
        "input_mapping": { "response": "inputs.response" } },
      { "id": "verification", "type": "call", "judge_id": "verify_facts",
        "input_mapping": { "facts": "facts", "response": "inputs.response" } }
    ]
  }
}

Map Over Items

{
  "judges": [
    { "id": "item_scorer", "judge_uri": "eart://judges/bespoke/relevance_check" }
  ],
  "pipeline": {
    "steps": [
      { "id": "extracted", "type": "extract", "resolver": { "type": "code",
        "code": "import json; output = json.loads(trace.spans[0].attributes.get('llm.output_messages', [{}])[0].get('message.content', '{}')); result = {'items': output.get('selected_items', [])}" } },
      { "id": "scores", "type": "map", "iterator_ref": "extracted.items",
        "item_alias": "current_item", "parallelism": 5,
        "steps": [
          { "id": "score_item", "type": "call", "judge_id": "item_scorer",
            "input_mapping": { "item": "current_item" } }
        ] },
      { "id": "final", "type": "reduce", "input_ref": "scores",
        "aggregation": { "strategy": "average" } }
    ]
  }
}

Complete Example

A two-dimension eval: relevancy (LLM judge with heuristic extraction + guard) and schema validation (deterministic judge with config).

Rubric

{
  "name": "product_quality",
  "description": "Evaluates product matching quality",
  "dimensions": [
    {
      "name": "relevancy",
      "description": "Are selected products relevant?",
      "scale": { "type": "binary", "min": 0, "max": 1 },
      "weight": 2.0
    },
    {
      "name": "schema",
      "description": "Is the output valid JSON?",
      "scale": { "type": "binary", "min": 0, "max": 1 },
      "weight": 1.0
    }
  ],
  "passing_score_threshold": 0.7
}

Eval

{
  "name": "product_quality_eval",
  "rubric_uri": "eart://rubrics/product_quality",
  "bindings": [
    {
      "judges": [{ "id": "relevance_judge", "judge_uri": "eart://judges/bespoke/product_relevance" }],
      "pipeline": {
        "steps": [
          { "id": "guard_llm", "type": "guard",
            "condition": "trace.spans.filter(s, s['openinference.span.kind'] == 'LLM').size() > 0",
            "on_fail": { "result": { "score": 0, "reason": "No LLM span." }, "reason": "Early exit." } },
          { "id": "inputs", "type": "extract", "resolver": { "type": "heuristic" } },
          { "id": "result", "type": "call", "judge_id": "relevance_judge",
            "input_mapping": { "query": "inputs.query", "response": "inputs.response" } }
        ]
      }
    },
    {
      "judges": [{
        "id": "schema_check",
        "judge_uri": "eart://judges/bespoke/json_schema_validator",
        "config": { "schema": { "type": "object", "required": ["selected_product_ids"] } }
      }],
      "pipeline": {
        "steps": [
          { "id": "output", "type": "extract", "resolver": { "type": "cel",
            "mapping": { "response": "trace.spans[0].attributes['llm.output_messages'][0]['message.content']" } } },
          { "id": "check", "type": "call", "judge_id": "schema_check",
            "input_mapping": { "response": "output.response" } }
        ]
      }
    }
  ]
}

Quick Reference

Concept	Key point
Binding order	Positional. The Nth binding grades the Nth rubric dimension.
Judge id	Local alias within the eval. Referenced by call steps.
Same judge, multiple uses	Different id, different config
config	Merged into pipeline context. Auto-wired by matching name.
No pipeline	Valid only with a single judge. Default: heuristic extract → call.
Multi-judge	Pipeline required.
Versioning	?ref=v3 pins a judge version. Omit for latest.

Next steps

Guide	Covers
Core Concepts	Traces, Rubrics, Dimensions
Creating Judges	Defining judges and writing code snippets
Authoring Pipelines	Full step reference and patterns
Workflows	Datasets, annotation, and refinement