Task Onboarding

1. Minimal Extension Point

A new task normally does not require changes to LNR, ESTRA, resume, resource management, monitoring, or global merge. Add a task package that declares the artifact and metric contract and provides a system-side evaluator capable of validating one workspace candidate.

tasks/<family>/<task-id>/
├── task.yaml
├── description_lite.md
├── evaluator.py
└── optional problem assets

2. Task Manifest Contract

id: example-task
category: opt_solver
profile: opt_solver
description: description_lite.md
artifact:
  path: artifacts/best_solution.json
  kind: json_solution
metric:
  name: objective
  lower_is_better: false
  type: benchmark
evaluator:
  backend: task_package
  entrypoint: evaluator.py:evaluate
  timeout_sec: 300
  visible_to_agent: false
gate:
  policy: default

provider is optional. profile selects the prompt/artifact profile, while the evaluator and Gate remain domain-neutral. A task may select a trusted alternative Gate policy, but workspace code cannot register arbitrary policies.

3. Evaluator Function

def evaluate(*, artifact_path, workspace_dir, task_dir, dataset_dir, config):
    # Validate schema, instance identity, constraints, and numerical values.
    return {
        "metric": {"name": "objective", "value": 1.23},
        "valid": True,
    }

The entry point must return a JSON object. The task-package backend copies the package to a system-owned runtime directory, verifies its hash, launches it in a subprocess, and normalizes its result. Exceptions, non-zero exit status, timeouts, malformed output, non-finite metrics, and invalid candidates remain explicit evaluator states.

Reject missing or malformed artifacts with actionable errors.
Validate array lengths, identifiers, bounds, feasibility constraints, and finite numeric values.
Compute the metric from the artifact and task data; never trust an agent-reported score.
Keep lower_is_better aligned with the actual objective.
Return valid: true only after the complete task contract passes.

4. Current Packages

Task	Artifact	Metric	Direction
`opt_solver/circle-packing`	`artifacts/best_solution.json`	`radii_sum`	Higher is better
`opt_solver/kttsp`	`artifacts/best_solution.json`	`mission_duration_days`	Lower is better
`opt_solver/ratio-minimization`	`artifacts/best_solution.json`	`inv_ratio_squared`	Higher is better
`opt_solver/uncertainty-ineq`	`artifacts/best_solution.json`	`c4_score`	Higher is better
`sci_modeling_bench/*`	`artifacts/submission.json`	`best_k_mean`, `normalized_enrichment`, or `global_ndcg`	Higher is better

5. Onboarding Sequence

Place task data outside the source tree and expose it through input_data_dir.
Write a concise description_lite.md that names the artifact path and constraints.
Add task.yaml with task identity, profile, artifact, metric, evaluator, and optional Gate policy.
Implement a deterministic system-side evaluator and test valid, invalid, malformed, missing, and boundary cases.
Create a run manifest with budget and CPU/GPU boundaries.
Run a short end-to-end task and inspect evaluator events, Gate decisions, Stage rows, snapshots, and final artifacts.