Core Concepts

Evalm8 evaluates traces with the core entities below. Each entity has its own detailed guide linked from this page.

The Five Entities

Evalm8 operates on five primary entities that form a pipeline from raw data to a quality verdict:

Entity	What it is	Analogy
Trace	The raw execution record from your LLM application, captured in OpenInference format. Contains one or more Spans, each representing a unit of work.	The answer booklet
Rubric	A grading scheme that defines what to evaluate. It contains Dimensions (metrics), each with its own scale, passing threshold, and weight.	The grading scheme
Judge	A specific evaluator that scores one aspect. Can be LLM-based (prompt + model) or deterministic (Python snippet).	The subject examiner
Pipeline	The execution graph that extracts data from a Trace and passes it to Judges. Supports guards, branching, iteration, and aggregation.	The scanner
Eval	The executable binding that connects a Rubric's dimensions to specific Judges, with optional custom Pipelines and per-judge configuration.	The board of examiners
Connection	A registered link to an external observability platform (e.g. Divyam Router, Langfuse) that Evalm8 pulls traces from.	The data feed
Model Provider	An LLM endpoint registration with a model name, provider, API credentials, and pricing. LLM judges use it for inference.	The examiner's toolkit

An Eval evaluates a Trace. It uses its Rubric to know which dimensions to measure, and its Judge Bindings to know which Judges to run and how to feed them data.

Traces and Spans

Evalm8 uses the OpenInference standard, in which a Trace is a collection of Spans. Each span carries:

• SpanKind: categorises the work as LLM, TOOL, RETRIEVER, AGENT, CHAIN, RERANKER, EMBEDDING, or EVALUATOR.
• Attributes: a flat key-value bag containing prompts, completions, token counts, and metadata.
• Events: structured records for data too large for flat attributes, such as full message content.

The attributes you will access most often are llm.input_messages, llm.output_messages, input.value, and output.value.

SpanKind Values

Value	Description
LLM	A call to a language model
AGENT	An autonomous agent step
TOOL	A tool or function call
RETRIEVER	A vector search or retrieval step
RERANKER	A reranking step
CHAIN	A chain or orchestration step
EVALUATOR	An evaluation step
EMBEDDING	An embedding generation step

Common Span Attributes

Attribute key	Description
llm.input_messages	List of input message dicts (message.role, message.content)
llm.output_messages	List of output message dicts
llm.token_count.prompt	Prompt token count
llm.token_count.completion	Completion token count
input.value	Raw input string (fallback when structured messages are absent)
output.value	Raw output string
metadata	Arbitrary metadata dict

Rubrics and Dimensions

A Rubric defines what "good" looks like. It contains one or more Dimensions, each measuring a specific quality aspect.

Each dimension has:

• Scale: how the dimension expresses a score. The types are integer (for example, 1–5), continuous (for example, 0.0–1.0), and binary (pass/fail, 0 or 1).
• Weight: how much this dimension contributes to the overall score.
• Passing threshold (min_passing_score): the minimum score to pass in the dimension's native scale. It defaults to the scale minimum.
• Inversion (is_inverted): applies when lower is better, such as Toxicity. Evalm8 flips inverted scores during normalisation.

Rubric Pass Rules

The rubric aggregates dimension scores into an overall result using a weighted average normalised to [0, 1]. It applies two pass rules, in priority order:

1. passing_expression is an optional CEL expression. When set, it is authoritative and Evalm8 ignores the threshold.
2. passing_score_threshold defaults to 0.7 when no expression is set. The rubric passes only if the weighted average meets this threshold and every individual dimension passes its own min_passing_score.

Without a passing_expression, a single failing dimension fails the entire rubric regardless of the overall score.

Passing Expression Context

When Evalm8 evaluates a passing_expression, these variables are available in the CEL scope:

Variable	Type	Description
total_score	float	The weighted-average normalised score across all dimensions.
dimensions	dict	Per-dimension detail. Each key is a dimension name, value is an object with is_passing (bool), score (float), rank (list), error (string or null), and type (result type name).
<dimension_name>	bool	Shorthand. Each dimension name is also a top-level boolean (true if passing).

Example expressions:

total_score > 0.7 && safety == true

total_score > 0.8 && dimensions["correctness"]["score"] >= 4.0

Dimension Passing

Evalm8 decides each dimension's pass/fail using min_passing_score in that dimension's native scale (not normalised):

• Normal dimensions: the score must be ≥ min_passing_score to pass.
• Inverted dimensions (is_inverted: true): the score must be ≤ min_passing_score to pass (lower is better, e.g. Toxicity).
• If you leave min_passing_score unset, it defaults to the scale minimum, so any valid score passes.

Score Normalisation

Judges can score on their own scale, which may differ from the dimension's scale. Evalm8 normalises each score in three steps:

1. Scale mapping: if the judge's scale differs from the dimension's scale, Evalm8 maps the score to the same relative position on the dimension's scale. It clamps values to the target scale's bounds and coerces them by type. Integer scales round values and binary scales threshold them at the midpoint.
2. Universal normalisation: Evalm8 maps the native-scale score to [0, 1] using (score - min) / (max - min).
3. Inversion: for an inverted dimension, Evalm8 flips the normalised score to 1.0 - normalised.

For an integer 1–5 scale, the normalisation formula is (raw - 1) / 4:

Normalised	Raw score	Meaning
n/a	n/a	Pipeline guard exited before the judge ran (score absent)
0.00	1	Lowest score
0.25	2	Below average
0.50	3	Mid-range
0.75	4	Good
1.00	5	Excellent

For a binary 0–1 scale: 0.0 = fail, 1.0 = pass.

Evaluation Status

The rubric result includes a status field reflecting the health of the evaluation run:

Status	Meaning
success	All dimensions produced valid scores (no system errors).
partial_failure	At least one dimension errored, but the others produced scores. Evalm8 computes the overall score from the dimensions that succeeded.
failed	All dimensions errored, so Evalm8 could not compute a score.

status is distinct from is_passing, which reflects quality. A rubric can have status: success (everything ran) but is_passing: false (the scores were too low).

Judges

Judges produce the scores in Evalm8. Each judge scores one dimension, and there are two types:

• LLM Judge: uses a prompt template and an LLM to reason about quality, such as hallucination detection or coherence scoring. It can be bespoke with a fixed prompt or fine_tuned, which learns from annotated data over refinement cycles.
• Deterministic Judge: uses a Python snippet for checks such as JSON schema validation or regex matching.

Judges declare their inputs, the named variables they expect to receive. A pipeline or the default heuristic extraction populates them from the trace.

→ See Creating Judges for the full guide.

Pipelines

A Pipeline defines how data flows from a raw trace to a judge's inputs. Step types include guards (precondition checks), extract/transform (data pulling and reshaping), call (judge invocation), switch (conditional branching), map (iteration), and reduce (aggregation).

If you provide no pipeline, Evalm8 synthesises a default pipeline using heuristic extraction.

→ See Authoring Pipelines for the full step reference.

Evals

An Eval binds each rubric dimension to one or more judges, with optional custom pipelines and per-judge configuration. Bindings are positional, so the Nth binding corresponds to the Nth dimension.

→ See Building Evals for bindings, configuration, and multi-judge patterns.

Connections

A Connection is a registered link to an external observability platform from which Evalm8 can pull traces. You manage connections under Integrations → Connections in the UI.

Currently supported connection types:

Connection type	Source	Description
Divyam	Divyam Router	Pulls request/response traces logged by the Divyam routing layer.
Langfuse	Langfuse	Pulls traces from a Langfuse observability project.

Each connection stores the platform's base URL and authentication credentials. Once you register and test a connection, the Create Raw Dataset workflow can use it to ingest traces into Evalm8 for evaluation.

See Workflows: Setting Up a Connection for setup steps.

Model Providers

A Model Provider is a registered LLM endpoint that LLM judges use to run inference. You manage model providers under Integrations → Model Providers in the UI.

Each model provider registration includes:

Field	Description
id	Stable identifier used to reference this model in judge configurations.
provider	Provider name (e.g. google, openai, anthropic).
model	The specific model identifier (e.g. gemini-2.5-flash, gpt-4o).
endpoint	The API base URL for the provider.
api_key	API credentials (encrypted at rest, redacted in responses).
api_type	API format: completions (OpenAI-compatible) or provider-native.
pricing	Input/output token pricing for cost tracking.

In an LLM judge, the model field references a registered model provider by its URI (e.g. google/gemini-2.5-flash). If a judge specifies no model, Evalm8 uses the project's default model provider.

tip

Register model providers before creating LLM judges. A judge that references an unregistered model will fail at evaluation time.

Evaluation flow

1. The Eval resolves its rubric and all referenced judges.
2. Evalm8 evaluates the rubric dimensions in parallel, up to a concurrency limit.
3. For each dimension, the bound pipeline runs against the trace: guards validate, extractors pull data, transforms reshape, call steps invoke judges.
4. The judge returns a score (and reason) in its own scale.
5. Evalm8 maps the score from the judge's scale to the dimension's native scale, then sets the dimension's is_passing by comparing that native score with its min_passing_score.
6. Evalm8 normalises the native score to [0, 1] and weights it for rubric aggregation.
7. The rubric combines all dimension results into a RubricResult: weighted-average score, per-dimension detail, status (success / partial_failure / failed), and the overall is_passing verdict (from the passing_expression if set, otherwise from passing_score_threshold + all dimensions passing).

Observability

Every evaluation is itself traceable. Evalm8 tracks prompt and completion tokens for LLM judges, and records per-step inputs, outputs, latency, and condition results when you enable debug mode. It also versions every rubric, judge, and eval, and keeps their history and lineage for provenance.

Next steps

Guide	Covers
Creating Judges	Defining LLM and deterministic judges, declaring inputs, writing code snippets
Authoring Pipelines	Pipeline steps, resolvers, composition patterns
Building Evals	Judge bindings, configuration, the default pipeline, multi-judge setups
Workflows	Creating datasets, human annotation, judge refinement