Data Preparation
1. Boundary
Section titled “1. Boundary”Solver code reads a single workspace-relative root: workspace/dataset/. Raw task data is treated as read-only. Preparation may create metadata and derived splits inside the workspace, while large unchanged media directories are linked rather than copied.
2. Entry Points
Section titled “2. Entry Points”| Mode | Behavior | Output |
|---|---|---|
deepbuilder prep |
Creates an empty workspace dataset directory, loads the repository skill registry, and runs the data-preparation ScienceAgent. | Prepared dataset/, optional split manifest, and a human-readable report. |
deepbuilder prep --link-only |
Skips the agent and symlinks every top-level input entry into the flat dataset view. | A fast, non-semantic workspace view. |
deepbuilder repl |
Uses the same flat-symlink fast path when the workspace has no dataset view yet. | Dataset access for an interactive agent session. |
3. Agent Protocol
Section titled “3. Agent Protocol”solver/data_prep.py requires the agent to list data-preparation skills, read data_prep, inspect the input root, and write only within the workspace. It uses the standard ScienceAgent with a stable system prompt, bash-write tools, repository skills, and a dedicated memory directory.
4. Validation Policy
Section titled “4. Validation Policy”- Group-aware splitting: Prefer group-aware splitting over row-level random splitting when stable patient, study, source, sequence, or paired-item identifiers exist.
- Task-like validation: Prefer task-like validation when test examples are constructed differently from raw training rows.
- Stratification: Use stratification or coverage balancing for imbalanced, multilabel, medical, detection, segmentation, and sparse-positive tasks when feasible.
- Strict isolation: Keep held-out groups and rows completely outside the remaining training files.
- No data leakage: Do not use private answers, test labels, or final-test keys for tuning.
- Honest reporting: If a safe split cannot be inferred, preserve the usable data view and document the limitation instead of manufacturing a misleading holdout.
5. Split Artifacts
Section titled “5. Split Artifacts”| Artifact | Contract |
|---|---|
dataset/train* |
Training-side records and labels after any holdout has been removed. |
dataset/validation* |
Optional local validation records in the schema expected by the task. |
dataset/mask_validation_label.* |
Optional hidden labels when validation features must not expose the target. |
dataset/split_manifest.json |
Machine-readable seed, strategy, group key, target columns, counts, distributions, and validity notes. |
dataset/split_report.md |
Short explanation of the resulting layout, validation policy, and known limitations. |
6. Current Skill Library
Section titled “6. Current Skill Library”The repository currently provides data_prep, image-task guidance, BSON image loading, recommendation time splitting, tabular preprocessing speedups, and self-evaluation guidance under .deepbuilder/skills/data_processing/. New dataset knowledge should normally be added as a skill rather than embedded as task-specific branches in the runtime.