EvaluationHarness
junjo.evaluation.EvaluationHarness
Kind: Class
SDK version: 0.67.0
Documentation channel: Stable release
Signature
Section titled “Signature”EvaluationHarness(*, application_key: str, service_identity: ExecutionServiceIdentity, targets: tuple[EvaluationTarget, ...], evaluators: tuple[Evaluator, ...], runtime_context: Callable[[], AbstractAsyncContextManager[ResourcesT]])One explicit application declaration consumed by the SDK runner.
Parameters
Section titled “Parameters”| Name | Type | Description | Default |
|---|---|---|---|
application_key |
str |
Stable key matching Studio Datasets for this app. | |
service_identity |
ExecutionServiceIdentity |
The application’s real OpenTelemetry service namespace and name. Evaluation does not replace this identity. |
|
targets |
tuple[EvaluationTarget, ...] |
Explicit Node, Workflow, and Agent declarations. | |
evaluators |
tuple[Evaluator, ...] |
Explicit built-in or application callback evaluators. | |
runtime_context |
Callable[[], AbstractAsyncContextManager[ResourcesT]] |
Application-owned asynchronous context for process-lifetime telemetry, provider clients, and execution resources. It is acquired lazily by an evaluation executor. |
Members
Section titled “Members”runtime
Section titled “runtime”runtime() -> AsyncIterator[ResourcesT]Enter one application-owned evaluation-host runtime.
Target discovery and schema listing never enter this context. An
EvaluationExecutor enters it lazily before the first real
target execution, reuses the yielded providers, clients, and telemetry
runtime across operations, and closes it when the executor exits.
target_descriptors
Section titled “target_descriptors”target_descriptors() -> tuple[TargetDescriptor, ...]Return registered targets in deterministic display order.
evaluator_descriptors
Section titled “evaluator_descriptors”evaluator_descriptors() -> tuple[EvaluatorDescriptor, ...]Return registered evaluators in deterministic display order.
prepare_case
Section titled “prepare_case”prepare_case(case: CaseRead) -> PreparedEvaluationResolve and validate one immutable Studio case before provider work.