Authoring Pipelines

A pipeline defines how data flows from a raw trace to a judge's inputs. It is an execution graph of steps that extract, validate, transform, and route data before a judge scores it.

→ For how pipelines attach to evaluations, see Building Evals.

→ For what judges expect as input, see Creating Judges.

Core Concepts

A pipeline runs against a trace. As it runs, each step places its output into a shared execution context, a flat key-value map. Subsequent steps reference these outputs by the step's id. The final step's output becomes the pipeline result.

Pipeline Structure

{
  "steps": [
    { "id": "my_step", "type": "<step_type>", ... }
  ]
}

Rules:

Every id must be unique across the entire pipeline, including inside nested switch cases and map sub-pipelines.
Steps execute one after another in declaration order unless a guard triggers early exit.
The executor stores each step's output in the context under its id.

Accessing Step Outputs

In CEL expressions, reference a prior output by its step ID, with no context prefix:

my_extract_step["my_field"]

In Python code resolvers, use the context dictionary:

value = context["my_extract_step"]["my_field"]

The trace is always available as trace in both CEL and code contexts.

Predicates and Conditions

Predicates are boolean conditions used in condition fields and stop_condition on map steps.

Type	Description
`cel`	A CEL expression with access to `trace` and all prior step outputs.
`code`	A Python snippet that sets `result` to `True` or `False`.

A plain string is shorthand for CEL: "condition": "size(trace.spans) > 0".

Every step supports an optional condition. If the condition is false, the executor skips the step and stores its on_skip value (default null) instead.

Step Reference

guard

Validates a precondition. If the condition is false, the pipeline ends at the guard with the on_fail result, and no later step runs.

{
  "id": "guard_has_output",
  "type": "guard",
  "condition": "trace.spans.filter(s, s['openinference.span.kind'] == 'LLM').size() > 0",
  "on_fail": {
    "result": { "score": 0, "reason": "No LLM span found in trace." },
    "reason": "Early exit: no LLM span."
  }
}

tip

Chain multiple guards at the top of a pipeline to validate the trace one check at a time before doing expensive work.

static

Injects a fixed value into the context. Useful for constants that downstream steps reference.

{
  "id": "expected_prefix",
  "type": "static",
  "result": "You are an expert product matching system."
}

select

Narrows the trace to a subset of spans. All subsequent steps see only the filtered spans. Extract any data you need from the full trace before a select step.

{
  "id": "llm_only",
  "type": "select",
  "selector": {
    "type": "cel",
    "expression": "trace.spans.filter(s, s['openinference.span.kind'] == 'LLM').map(s, s['span_id'])"
  }
}

extract

Pulls data from trace span attributes into the context.

{
  "id": "raw_output",
  "type": "extract",
  "resolver": {
    "type": "cel",
    "mapping": {
      "content": "trace.spans[0].attributes['llm.output_messages'][0]['message.content']"
    }
  }
}

transform

Reshapes data already in the context.

{
  "id": "parsed",
  "type": "transform",
  "resolver": {
    "type": "code",
    "code": "import json; raw = context['raw_output']['content']; result = {'data': json.loads(raw)}"
  }
}

extract vs transform

Use extract when reading from trace.spans[...]. Use transform when working with values already in context.

call

Invokes a judge. Maps context values to the judge's declared inputs via input_mapping.

{
  "id": "score",
  "type": "call",
  "judge_id": "my_judge",
  "input_mapping": {
    "response": "raw_output.content",
    "reference": "expected_answer.text"
  }
}

Values are dot-separated paths: <step_id>.<field_name>.

Auto-wiring: If a judge input has no mapping entry, the executor looks for a context key with the same name as the input.

Config values: The executor merges judge configuration (set on the eval binding) into the resolution scope before it evaluates input_mapping.

switch

Branches execution based on conditions. Only the first matching case runs.

{
  "id": "routing",
  "type": "switch",
  "cases": [
    { "condition": "raw_message['type'] == 'chat'", "steps": [ ... ] },
    { "condition": "raw_message['type'] == 'completion'", "steps": [ ... ] }
  ],
  "default": [ ... ]
}

Sub-steps write to the top-level context under their own id, accessible downstream regardless of which branch ran.

map

Iterates over a list and runs a sub-pipeline for each item, in sequence or in parallel.

{
  "id": "results",
  "type": "map",
  "iterator_ref": "extracted.candidate_urls",
  "item_alias": "candidate_url",
  "parallelism": 10,
  "stop_condition": {
    "type": "cel",
    "condition": "results_iter_result['score'] > 0.0"
  },
  "on_empty": {
    "result": { "score": 0, "reason": "No candidates." },
    "reason": "Early exit: empty list."
  },
  "steps": [
    {
      "id": "comparison",
      "type": "call",
      "judge_id": "url_match",
      "input_mapping": {
        "first_url": "extracted.input_url",
        "second_url": "candidate_url"
      }
    }
  ]
}

Key details:

iterator_ref is a dot-separated path to the list (e.g. my_step.my_list).
In input_mapping, reference the current item by its item_alias, with no step ID prefix.
stop_condition variables: <map_id>_iter_result and <map_id>_iter_index.
on_empty fires if the list is empty, null, or not a list.
If any iteration returns an error, the map cancels remaining iterations.

reduce

Collapses a map step's list output into a single value.

{
  "id": "final_score",
  "type": "reduce",
  "input_ref": "results",
  "aggregation": { "strategy": "or" }
}

Built-in strategies: average (default), min, max, and (all ≥ 0.5), or (any ≥ 0.5).

Custom strategies: cel is an expression over results. code is Python that assigns to output, not result. class is a registered class.

Resolvers

Resolvers define how extract and transform steps compute output. Every resolver accepts an optional selector that narrows the trace first.

Type	Best for	`trace` form	Requires mapping
`jsonpath`	Simple structural lookups	Dict	Yes
`cel`	Flexible expressions, filtering	Dict	Yes
`code`	Complex parsing, regex, JSON	Trace object	No
`class`	Reusable registered logic	Trace object	No
`heuristic`	Standard OI traces, zero config	Trace object	No

trace type differs by resolver

In CEL resolvers, trace is a plain dictionary. In code resolvers, trace is a typed Trace object, so don't mix up their access patterns.

Code resolver contract: Assign output to result as a dictionary and import every module you use. The available variables are trace and context.

import json, re
raw = context["raw_output"]["content"]
match = re.search(r'URL:\s*(https?://\S+)', raw)
result = {"url": match.group(1) if match else None}

Heuristic resolver: Zero-configuration extraction for standard OpenInference traces. See The Default Pipeline for what it extracts.

Result Auto-Normalisation

When any step returns a plain dictionary, the executor promotes it to a typed result:

Keys in dict	Promoted to
`score`	JudgeScore
`error`	JudgeError
`rank`	JudgeRank

This lets you return scores from static steps or on_fail actions:

{ "score": 0, "reason": "No matching products found." }

Composition Patterns

Linear: Validate → Extract → Score

guard → extract → transform → call

Map-Reduce: Score Each Item

extract (get list) → guard (list not empty) → map (call per item) → reduce

Switch on Content Type

extract → guard → switch (per-type transform) → map → reduce

Early-Stopping Map

stop_condition halts the map on the first positive result, combined with an or reduce:

"stop_condition": { "type": "cel", "condition": "my_map_iter_result['score'] > 0.0" }

Common Mistakes

Mistake	Fix
Duplicate step IDs	Every `id` must be unique across the entire pipeline including nested steps.
Wrong `iterator_ref`	Must be a dot-separated path to the list: `my_step.my_list`, not `my_step`.
Missing `result =` in code resolvers	If your code does not set `result`, the step produces no output.
`context["x"]` in CEL	In CEL, reference step outputs by ID, not via `context`.
`trace` type mismatch	CEL: dict. Code: Trace object.
Missing `on_fail` on guards	Every guard needs an `on_fail`.
`result` vs `output` in reduce	Code resolvers: `result`. Reduce code expressions: `output`.
Extracting after `select`	`select` replaces the trace. Extract what you need first.

Next steps

Guide	Covers
Core Concepts	Traces, Rubrics, Dimensions
Creating Judges	Defining judges and writing code snippets
Building Evals	Wiring pipelines and judges to rubric dimensions
Workflows	Datasets, annotation, and judge refinement

Core Concepts​

Pipeline Structure​

Accessing Step Outputs​

Predicates and Conditions​

Step Reference​

guard​

static​

select​

extract​

transform​

call​

switch​

map​

reduce​

Resolvers​

Result Auto-Normalisation​

Composition Patterns​

Linear: Validate → Extract → Score​

Map-Reduce: Score Each Item​

Switch on Content Type​

Early-Stopping Map​

Common Mistakes​

Next steps​