Evaluation datasets and runs with your coding agent

Junjo Evaluation is the batteries-included loop for building typed input datasets in Junjo AI Studio, executing them against the real application code in your checkout, and comparing structured results with the exact traces Studio received.

This is the technical lifecycle behind recursive self improvement. Start with that guide for the first improvement journey; use this page for target declarations, dataset operations, evaluator contracts, and run diagnostics.

The application remains the execution host because it owns prompts, provider credentials, Tools, databases, and domain services. The Junjo SDK owns the Studio client, dataset and result DTOs, target lifecycles, evaluator contracts, sequential executor, resume behavior, telemetry context, evidence queries, and the junjo eval CLI.

For a developer with a configured application harness, the intended interaction is a product-quality request to a coding agent:

Use Junjo to create and run a baseline evaluation of our assistant’s ability to recommend authentic local places. Store the dataset and results in Studio, inspect the failed cases and their traces, and report what should improve. Do not change the application yet.

The coding agent discovers the application’s targets and evaluator schemas, authors representative inputs and schema-valid expectations, operates the CLI, tracks Studio identifiers, retrieves evidence, and reports results. With an LLM judge, the expectation is a binary decision rubric rather than an expected prose answer. The developer does not need to create JSON files, copy IDs, choose CLI flags, or poll telemetry evidence.

What belongs where

Owner	Responsibility
Coding agent	Discover targets, curate scenarios, investigate evidence, edit and commit application changes, and orchestrate experiments
Application	Typed inputs, real dependency construction, Node/Workflow/Agent factories, output projection, and domain-specific evaluator meaning
Junjo SDK	`EvaluationHarness`, targets, evaluators, Studio transport, Attempt lifecycle, evidence binding, resume, comparison, and CLI
Junjo AI Studio	Canonical datasets, cases, runs, attempts, results, evidence membership, and received trace evidence

Studio never executes uploaded source code. Complete telemetry still enters Studio through authenticated OTLP; evaluation REST operations store only bounded control records and exact evidence references.

Junjo can support your application’s outer orchestration, model–Tool execution loop, and application state, or be adopted selectively alongside an existing runtime. A native Junjo Agent owns the model–Tool loop; Workflows, Nodes, and Stores structure application execution and state changes.

These components are purposefully designed to produce detailed execution evidence. When orchestration and state updates flow through Junjo, you can follow which operation changed a value, when it changed, and how that change contributed to the final outcome. Using Junjo for outer orchestration and application state extends this visibility across the application’s execution.

For applications handling many concurrent LLM calls, Junjo also provides a state-management foundation you do not have to build yourself. Workflow and Agent executions isolate their run state; Store reads return detached snapshots, and each state update is validated and committed atomically against the current state under the Store lock. Your application defines its domain actions and validation rules while Junjo supplies the update mechanics and recorded change history. See state management for the supported update model and Studio’s state history for investigating those changes.

Developers can adopt these capabilities together or incrementally. Telemetry and evaluation remain available when an existing server, agent framework, or job system continues to own the outer runtime.

The lifecycle is draft cases → lock dataset → execute baseline → commit a change → rerun the same dataset → compare and inspect evidence. Setup happens once per application: connect telemetry, configure a developer access token, declare the harness, and install the matching coding-agent skill.

Credentials stay separate

Sign in to Studio, open Access Tokens, choose the required scopes and expiration, and create a developer access token. Copy it to the environment:

JUNJO_AI_STUDIO_BACKEND_BASE_URL=http://localhost:26154
JUNJO_AI_STUDIO_CLI_TOKEN=jcli_...

The access token has explicit evaluation:read, evaluation:write, and evidence:read scopes. It cannot deliver OTLP telemetry. The existing JUNJO_AI_STUDIO_API_KEY, created from API Keys, remains an application-telemetry-only credential and cannot query or mutate datasets.

Remote Studio origins must use HTTPS. Explicit loopback development may use HTTP.

Give a coding agent the runbook

The installed Junjo wheel ships an Agent Skills-compatible junjo-evaluation skill. Locate the exact skill matching the installed SDK:

junjo eval skill path

The command requires no application harness, Studio connection, or credential. Its versioned JSON response contains the absolute skill directory and SKILL.md path. Point the coding agent’s normal skill installer at that directory once.

The installed CLI also explains its own current interface:

junjo eval explain
junjo eval explain --format json

The Markdown form is the concise command, configuration, authentication, and evidence-level reference for a developer or coding agent. The JSON form uses the CLI’s normal versioned envelope for tooling. Both are generated from the same command definitions used by --help and capability discovery, so the explainer is not a second handwritten command manual. The skill owns workflow judgment; the explainer owns exact CLI mechanics.

Installing the Python package does not silently activate a coding-agent skill. The explicit installation keeps agent configuration under the application developer’s control while the versioned skill, SDK, CLI, and documentation remain owned and released together by Junjo.

After installation, a normal request should contain intent rather than Junjo mechanics:

Use Junjo to evaluate whether this workflow extracts complete invoice data. Establish a baseline in Studio and explain the failures from their traces.

For an improvement iteration:

Improve refund-answer accuracy using our existing Junjo dataset, then compare the candidate with the baseline.

The skill owns discovery, scenario design, temporary schema-valid artifacts, dataset operations, execution, evidence queries, and product-oriented reporting. It asks the developer only for a real missing prerequisite, materially ambiguous product intent, or authority to modify and commit code.

Declare one harness

An application exports exactly one explicit EvaluationHarness object.

The following wiring example assumes your application already defines AnswerInputV1, AnswerState, AnswerStore, CreateAnswerNode, and build_provider. The runtime context should acquire and close the same telemetry and provider resources used by real application execution; it is not a separate implementation of the target.

from contextlib import asynccontextmanager

from junjo.evaluation import (
    EvaluationHarness,
    ExactMatchEvaluator,
    ExecutionServiceIdentity,
    NodeInvocation,
    NodeTarget,
)

@asynccontextmanager
async def evaluation_runtime():
    provider = build_provider()
    try:
        yield provider
    finally:
        await provider.close()

async def create_node(input_value, context, provider):
    return NodeInvocation(
        node=CreateAnswerNode(provider),
        store=AnswerStore(
            initial_state=AnswerState(question=input_value.question),
        ),
    )

def project_answer(result, input_value, context, provider):
    return result.state.answer

harness = EvaluationHarness(
    application_key="my_application",
    service_identity=ExecutionServiceIdentity(
        service_namespace="my_company",
        service_name="my_application",
    ),
    targets=(
        NodeTarget(
            key="answer",
            name="Create Answer Node",
            input_version=1,
            input_type=AnswerInputV1,
            factory=create_node,
            projector=project_answer,
        ),
    ),
    evaluators=(ExactMatchEvaluator(),),
    runtime_context=evaluation_runtime,
)

Configure the import explicitly in the application repository:

[tool.junjo.evaluation]
harness = "my_application.evaluation:harness"

junjo eval targets list returns each stable target key, its human-readable Node, Workflow, or Agent name, and its JSON input schema. The CLI records that name with authored and generated cases so Studio can label historical results without presenting the dispatch key as an entity name. junjo eval evaluators list returns stable evaluator identities, roles, and JSON expectation schemas. Both import the harness without entering evaluation_runtime, so discovery does not create providers, connect to application databases, or start telemetry.

The runner verifies that a case’s stored target name still matches its registered target. Renaming a target therefore requires a new dataset rather than silently relabeling immutable historical tests.

NodeTarget, WorkflowTarget, and AgentTarget own the correct public Junjo lifecycle. Their application factories receive typed input, immutable EvaluationContext, and the executor-owned application runtime. Factories may attach an optional per-invocation cleanup callback; Junjo runs it even when subject execution or projection fails.

The complete provider-free standalone declaration is in examples/evaluation_standalone. AI Chat is the full live-provider reference application.

Author a dataset

Case input and expectation are JSON files or standard input. Secrets are never ordinary command arguments.

junjo eval dataset create \
  --key answer-quality-v1 \
  --name "Answer quality"

junjo eval dataset add \
  --dataset-id DATASET_ID \
  --case-key question-1 \
  --evaluation-name "Answer exact match" \
  --target-kind node \
  --target-key answer \
  --input-version 1 \
  --input input.json \
  --expectation expectation.json \
  --evaluator-key junjo.exact \
  --evaluator-version 1

junjo eval dataset lock --dataset-id DATASET_ID

Locking is explicit and irreversible. Before requesting the lock, the CLI validates every Case against the loaded target and evaluator declarations. Studio preserves exact ordered membership for baseline/candidate comparison.

Built-in evaluators intentionally stay small:

ExactMatchEvaluator compares a projected subject with {"expected": ...}.
StructuredFieldEvaluator checks explicit top-level fields.
BooleanPredicateEvaluator adapts a typed application predicate.
CallbackEvaluator runs a bounded domain judge and requires the common EvaluationResult.

Applications can define domain meaning without owning Attempt transitions or Studio writes.

One evaluator should decide one understandable product claim with explicit, observable binary conditions. Calibrate it against known-good, known-bad, and boundary examples before trusting its result. Use deterministic checks for deterministic facts and for current facts that can be verified from a trustworthy current source; an LLM judge should not be expected to remember whether a venue exists, is open, or is correctly located.

A finite source snapshot supports a closed-set case whose target input exposes the same candidates, or a case about one named fact. It is not an authoritative resolver for open-ended generation. A recommendation absent from that snapshot is an evaluator coverage gap, not proof that the recommendation is false; record it as an error or use a real current-data resolver rather than moving the allowlist after every run.

The evaluator’s reason should identify the observation that determined pass or fail. Deeper trace and source diagnosis belongs to the coding agent, which can combine that reason with exact execution evidence without turning one judge call into an open-ended investigation.

Generate a case through real execution

Dataset generation runs the same declared Node, Workflow, or Agent and records its clean source revision plus exact semantic execution identity:

junjo eval case generate \
  --dataset-id DATASET_ID \
  --case-key generated-question-1 \
  --evaluation-name "Answer quality" \
  --target-kind workflow \
  --target-key answer-flow \
  --input-version 1 \
  --input input.json \
  --expectation expectation.json \
  --evaluator-key answer.quality \
  --evaluator-version 1

The observed subject is evidence only. Junjo never copies it into the expected answer or silently promotes it to truth.

Cases can also be authored directly from reviewed production interactions or synthetically constructed inputs using junjo eval dataset add. Generation through a target is useful when you want the scenario linked to an observed execution. In either path, choose the evaluation criterion independently of what the current implementation happened to produce.

Execute, resume, and compare

Run a locked dataset from a clean committed checkout:

junjo eval run execute \
  --dataset-id DATASET_ID \
  --request-key baseline-20260727 \
  --run-label baseline

junjo eval run resume --run-id RUN_ID

EvaluationExecutor executes sequentially. It lazily enters the application’s runtime context once, reuses that telemetry/provider/client runtime across generated cases and Runs, and cleans each invocation explicitly. It validates before provider work, pre-creates Studio Attempts, binds the exact subject execution before judgment, and writes one terminal result.

Resume skips terminal Attempts. A queued Attempt with an already bound subject is not executed again; it is finalized as interrupted, and a new Run is the explicit subject-retry boundary.

A queued Attempt without bound subject evidence is eligible for execution on resume. If the previous process stopped after application work began but before evidence binding completed, that work may execute again. Resume does not guarantee exactly-once external execution. Application-owned operations retain responsibility for idempotency and recovery from uncertain outcomes. Resume also requires the same clean source revision recorded by the Run.

After a committed prompt or implementation change, execute the same locked Dataset with a new request key, then compare exact Case IDs:

junjo eval run compare \
  --baseline-run-id BASELINE_RUN_ID \
  --candidate-run-id CANDIDATE_RUN_ID \
  --target-kind node \
  --target-key answer \
  --input-version 1 \
  --evaluation-name "Answer quality"

Omit the target and evaluation flags to compare the complete Dataset. Run-list queries accept the same exact Case scope:

junjo eval run list \
  --dataset-id DATASET_ID \
  --target-kind node \
  --target-key answer \
  --input-version 1 \
  --evaluation-name "Answer quality"

Studio and the SDK report pass_rate over judged Attempts only (passed / (passed + failed)) and report coverage separately (judged / total). Comparisons classify each immutable Case as improved, regressed, newly_errored, recovered, unchanged, or changed, while retaining both exact execution links.

Each Run records one clean committed source revision. An uncommitted prompt edit is not an eligible candidate; commit it before execution. Keep a baseline checkout or worktree available if you need to rerun the old implementation against a newly expanded dataset.

For parallel experiments, separate coding agents or worktrees can execute independent Runs against the same locked dataset and Studio service. Give each experiment a distinct request key and meaningful Run label. This does not make one EvaluationExecutor concurrent: each executor still runs its cases sequentially. Scheduling and coordination belong to your coding agent, CI, or application environment, including ownership of external side effects.

Query exact evidence

Evidence reads are intentionally staged so an agent can diagnose a result without downloading every full trace:

read the Run and Attempt summaries;
read a manifest for a failed, errored, surprising, or representative Attempt;
request the exact failed or otherwise relevant span IDs exposed by that manifest; and
request full evidence only when the whole trace is needed for relationship, state-reconstruction, or integrity analysis.

junjo eval run get --run-id RUN_ID
junjo eval attempt get --attempt-id ATTEMPT_ID
junjo eval attempt evidence manifest --attempt-id ATTEMPT_ID
junjo eval attempt evidence spans \
  --attempt-id ATTEMPT_ID \
  --span-id SPAN_ID
junjo eval attempt evidence full --attempt-id ATTEMPT_ID
junjo eval evidence membership \
  --kind junjo_execution \
  --executable-type workflow \
  --runtime-id WORKFLOW_RUN_ID

The manifest includes the evaluated subject, trace shape, failed spans, executable and operation summaries, Store integrity, relationships, and diagnostics. Operation summaries recognize native Junjo operations plus the OpenInference and GenAI model and Tool conventions Studio already uses; an external operation can be present without a Junjo executable owner. Selected-span reads return complete normalized records for the explicit IDs. Full evidence remains lossless. Use junjo eval explain and the individual command help for the current flags and response contracts.

Normal ingestion/indexing delay has a distinct pending-evidence error. An ambiguous semantic execution identity is a conflict, not an arbitrary first match.

The Python API uses the same implementation:

from junjo.evaluation import EvaluationExecutor
from junjo.studio import StudioClient

async with StudioClient(base_url=studio_url, token=studio_token) as studio:
    async with EvaluationExecutor(client=studio, harness=harness) as evaluation:
        baseline = await evaluation.run(
            dataset_id=dataset_id,
            request_key="baseline-1",
            run_label="baseline",
        )
        repeat = await evaluation.run(
            dataset_id=dataset_id,
            request_key="repeat-2",
            run_label="repeat-same-revision",
        )

Both calls above use the same imported application code. To test a committed code change, run from the candidate checkout in a fresh application process; changing a Run label does not change prompts or reload Python modules.

EvaluationExecutor is one application-host lifetime. It acquires the application runtime only before real target execution, reuses process-global telemetry and provider clients across iterative Runs and generated Cases, and closes them once on exit. StudioClient is an async context manager with one bounded HTTP connection pool, explicit timeouts, bounded response sizes, typed errors, and opt-in evidence hydration.

Telemetry classification

Evaluation execution retains the application’s real OpenTelemetry Resource. Junjo adds one bounded orchestration span and role spans:

junjo.evaluation.attempt
junjo.evaluation.dataset_generation
junjo.evaluation.subject
junjo.evaluation.judge
junjo.evaluation.verifier

The spans carry junjo.evaluation.* Dataset, Run, Case, Attempt, source revision, class, and role attributes. Junjo does not copy this metadata onto every descendant model, Tool, Node, or Store span. Studio’s Attempt-to- evidence binding remains the canonical result/evidence join.

Machine contract and low-resource defaults

Every CLI command writes one versioned JSON envelope to standard output. Diagnostics go to standard error. Important exit statuses are:

Status	Meaning
`0`	Command completed and no Attempt failed
`2`	Usage, local schema, or declaration error
`3`	Authentication or authorization failure
`4`	Immutable/idempotency or identity conflict
`5`	Subject or Attempt execution error
`6`	Evaluation completed with a failing judgment
`7`	Exact evidence is not ready
`8`	Studio remained transiently unavailable
`9`	Studio response is incompatible with the SDK contract

Pages and responses are bounded, comparison uses summaries, evidence is explicit, and concurrency defaults to one. Evaluation control adds no query, cache, database lookup, or protocol to the Rust OTLP ingestion hot path.