Resource Management

1. Responsibility Boundary

The resource control plane combines reusable safety primitives in deepbuilder/safety/resource/ with the LNR task-local runtime in solver/lnr/resource_runtime/. It observes Bash jobs, enforces resource boundaries, manages GPU ownership and waiters, reviews execution progress, and returns compact operational feedback.

Note

Resource management decides whether and how an execution should receive more time or scarce resources. It does not select the scientific model, feature set, optimization method, or research hypothesis for the main agent.

---

2. Control Loop

Observeprocess, CPU/GPU, output, metric, artifact

Review Stateheartbeat state machine

Boundarystall, value, progress, pressure, expiry

Advisoryowning agent preference

Arbitervalue and opportunity cost

Strict Gatetarget and evidence fuse

ExecutionKILL, TIMEBOX, NO_ACTION

The default resource_smart_llm profile uses a monitor-first loop. Deterministic code identifies hard boundaries and decides when a review is warranted. The owning-agent advisory and resource arbiter reason over a compact fact packet, and a strict execution gate validates any destructive intervention.

---

3. User-Facing Modes

resource_control_mode	Enabled control surface
off	Resource runtime, monitoring, review state, queue, arbiter, advisory, observation, and sharing are disabled.
resource_smart_policy	Deterministic runtime, queue, review state, and policy arbiter are enabled; admission LLM, owning-agent advisory, and GPU sharing are disabled.
resource_smart_llm	The full system is enabled: admission LLM, LLM resource arbiter, owning-agent advisory, task-local queue, observation, and GPU sharing.

config/resource_modes.py expands the selected mode into low-level switches. Explicit keys in the same LNR mapping take precedence, allowing narrow expert overrides without duplicating a complete profile.

---

4. Task-Local Multi-Worker Control

All workers in one LNR task share a task-local resource runtime, unified event store, GPU lease store, pressure state, history, and waiter queue. No worker is permanently the “main” resource holder. The current lease owner is determined by actual admission and can change as workers explore stronger routes.

Layer	Mechanism	Meaning
Task CPU boundary	Manifest cpu_list and DEEPBUILDER_TASK_CPU_LIST	Constrains the complete task process tree.
Worker CPU slice	DEEPBUILDER_WORKER_CPU_LIST, DEEPBUILDER_CPU_LIST, _DEEPBUILDER_CPU_SET	LNR subdivides the task CPUs across workers and caps OpenMP threads.
GPU visibility	Task gpu_list and configured resource_gpu_pool	Establish the hard set of devices the task may use.
GPU ownership	Task-local lease, queue, sharing, and TTL	Coordinates heavy, test-time, feature, and secondary jobs inside the allowed pool.

---

5. Startup Admission and Monitored Trials

The default startup policy is trial_first. Admission first checks deterministic hard boundaries and atomic lease availability. Uncertain or historical soft blocks can become OBSERVE_THEN_RUN monitored trials, allowing the runtime to gather current evidence before making a value judgment.

Condition	Typical result
GPU visibility/pool violation, unsafe workspace boundary, invalid deliverable preflight, or explicit stop	Hard block before execution.
Stale pressure, soft post-feedback restriction, active-plan concern, or historical queue timeout	Monitored trial when a safe task-local lease is available.
Capacity exhausted, incompatible active class, class limit, or an earlier waiter	Pending queue, managed wait, or sharing review.
Blocked state with several safe candidate devices	Admission arbiter chooses RUN_NOW, OBSERVE_THEN_RUN, PENDING, or REPLAN within the task-local candidate set.

Note

A monitored trial is still resource-governed. It acquires a lease, records trial events, and enters normal heartbeat review; it is not a bypass around admission.

---

6. Observed Execution Facts

BashTool and the observer build a time-consistent snapshot rather than treating a growing log file as sufficient evidence of value.

Fact	Current semantics
Process liveness	Actual process tree, exit status, CPU activity, elapsed time, and target identity.
GPU utilization	Visible device, memory, utilization samples, lease role, and pressure state.
stdout_observation	Distinguishes direct stdout from redirected-log activity; stale redirected logs do not become fresh stdout progress.
Metric history	Comparable metric values, scope, direction, changes, best-so-far relation, and final-score signals.
artifact_updates	Current-run file growth and age. Logs are context, not deliverables; recent non-log artifacts may be value signals.
recoverability	Whether the current job has a current-run, stable artifact that lowers stop cost. Ages are refreshed when each snapshot is built.
Opportunity cost	Active waiter pressure, blocked worker count, queue age, holder role, and estimated unlock conditions.

Artifact, recoverability, and nested update ages share the same elapsed-time basis. This avoids decisions where a top-level artifact appears stale while a nested recoverability record still appears fresh.

---

7. Review State Machine

safety/resource/review_signal.py compresses raw observations into ResourceReviewSignal. review_state.py advances one state per job, and review_boundary.py decides when to request a review.

State	Meaning
WARMUP	Collect evidence before route-value no-progress counters become active.
RUNNING_HEALTHY	Active work and recent useful progress support continued observation.
RUNNING_NO_PROGRESS	Work is active but the configured useful-progress window has not advanced.
STALL_SUSPECT	Repeated inactivity lacks CPU/GPU/output/artifact counter-evidence.
TIMEBOX_ACTIVE	A bounded proof window is running until its clear condition, pressure interrupt, or deadline.
ACTIONED	A terminal control action has been executed.

Review Boundaries

Boundary	Trigger role
stall	Inactive windows support a possible stop review.
route_value	Active execution has not produced useful progress for the configured value window.
progress_window	Useful progress exists, but a low-frequency value review is due.
resource_pressure	New contention can interrupt a long proof window.
timebox_expired	The previous bounded continuation must prove its named clear condition.

---

8. Advisory, Arbiter, and Strict Gate

When the owning main agent is blocked on a Bash result, the control plane can ask a narrow no-tools question through a side channel. The advisory reports a preference such as safe_to_stop, timebox_continue, continue, or unknown, plus confidence and status. The raw exchange is audited but not inserted into the live research transcript.

The resource arbiter then evaluates progress, metric history, remaining budget, ETA, recoverability, waiter pressure, and prior review commitments. review_outcome.py normalizes the value-review execution surface to:

• KILL — request termination and replan feedback.
• TIMEBOX — continue for a bounded interval with a concrete clear_on condition.
• NO_ACTION — continue observation, normally without main-agent feedback.

resource_runtime/review/arbiter_gate.py is the execution fuse. It validates action class, target job, lease ownership, advisory/evidence support, confidence, deliverable state, and secondary-only targeting before an actual kill is allowed.

Advisory vs. Authorization

An advisory is evidence, not authorization. A discretionary kill normally requires owning-agent advisory support, high-confidence arbiter output, and strict-gate approval. Hard safety boundaries remain separate.

---

9. Timeboxes and Proof Windows

A timebox names a measurable clear condition such as metric update, artifact growth, structured progress advance, active-work recovery, or cleared opportunity cost. compute_timebox_sec combines observed cadence, heartbeat, configured minimum/maximum, remaining-budget fraction, and waiter pressure.

During TIMEBOX_ACTIVE, ordinary route-value retriggers are frozen. If the clear condition is met, intervention pressure falls. If a proof window expires without useful progress, its failure is recorded. After resource_review_max_proof_windows failures, the proposal carries resource_budget_escalation.kind=final_resource_review; the resource arbiter makes one final budget decision from structured facts without repeating owning-agent advisory.

---

10. GPU Queue, Sharing, and Release

Mechanism	Behavior
Exclusive lease	A worker atomically acquires a task-local device and BashTool receives the corresponding visible-device environment.
Waiter queue	Blocked requests retain queue position, holder, ETA/unlock facts, priority, and expiry rather than busy-waiting.
Managed wait	The agent may use resource_wait with a runtime-provided token and wake when the capability state changes.
Shared lease	When utilization and memory headroom permit, a waiter may receive a revocable secondary lease.
Primary protection	If sharing threatens the primary holder, the secondary is revoked first.
Idle release	An idle lease can be released after warmup and sample thresholds, then conservative admission accounts for residual memory.

---

11. Feedback and Audit Hygiene

Complete control-plane events are appended to task_logs/resource/resource_events.jsonl. The main agent receives only actionable wait, share, replan, kill, or compact review facts through the current tool result. ResourceFeedbackMemoryDeduper folds repeated capability states and does not create a continuously changing live-summary slot.

Channel	Content
Resource event log	Admission, trial, lease/share/wait, review signal/state/boundary, advisory audit, arbiter decision, strict gate, and execution.
Main-agent memory	Only the compact operational consequence needed for the next action; repeated state is folded.
Monitor	Pending/wait/share state, timeboxes, active jobs, kill_req, and actual kill_exec.

---

12. Current File Ownership

deepbuilder/
├── safety/resource/
│   ├── review_signal.py      # Stable observed-fact summary
│   ├── review_state.py       # Per-job state machine and timebox sizing
│   ├── review_boundary.py    # Review trigger types
│   ├── review_outcome.py     # KILL / TIMEBOX / NO_ACTION normalization
│   └── ...                   # Process, completion, artifact, lifecycle helpers
└── solver/lnr/resource_runtime/
    ├── runtime.py            # Task-local coordination
    ├── observer/controller.py# Bash-job observation and event production
    ├── admission.py          # Startup admission arbiter
    ├── gpu_lease_store.py    # Lease persistence
    ├── gpu_sharing.py        # Sharing policy and configuration
    ├── unified_store.py      # Canonical resource event store
    └── review/
        ├── arbiter.py        # Resource-value decision
        ├── arbiter_gate.py   # Pre-execution strict gate
        ├── execution_facts.py
        ├── progress.py
        ├── research_cadence.py
        └── state_generation.py

---

13. Important Defaults

Setting	Default	Role
resource_startup_policy	trial_first	Convert eligible soft startup blocks into monitored trials.
resource_review_heartbeat_sec	60	Review-state heartbeat.
resource_review_warmup_windows	10	Warmup before route-value no-progress counting.
resource_review_inactive_windows	3	Inactive windows before stall suspicion.
resource_review_value_windows	5	Active-work windows without useful progress before route-value review.
resource_review_progress_event_min_windows	5	Minimum spacing between progress-based reviews.
resource_review_timebox_windows	10	Fallback timebox length in heartbeat windows.
resource_review_min_timebox_sec / max_timebox_sec	60 / 1800	Explicit timebox bounds.
resource_review_timebox_budget_fraction	0.10	Maximum fraction of remaining budget used by a proof window.
resource_review_max_proof_windows	2	Failed proof windows before final resource review.
resource_gpu_queue_max_wait_sec	1800	Maximum configured GPU queue wait.
resource_gpu_lease_ttl_sec	7200	Lease expiry horizon.