Scope
The benchmark contains 225 task fixtures: nominal, variation, boundary, validation, and safety cases for each of 45 operations. Every arm was scheduled for three trials per task, producing 2,700 scored trial records. In the frozen runner goldens, 188 fixtures returned success and 37 returned invalid_request, giving each arm 564 valid calculation trials and 111 invalid-input trials. Terminal failures are retained and scored as nonmatches. Cases exclude weapon, targeting, operational-control, dispatch, and safety-of-life scenarios.
Arms
| Arm | Access |
|---|---|
| Nemotron 3.5 Lightning | No skills, files, runners, or web access. |
| Nemotron 3.5 Lightning + skills | One matching skill read and at most two controlled local runner calls. |
| GPT-5.6 Terra | No skills, files, runners, or web access; descriptive control only. |
| GPT-5.6 Sol | No skills, files, runners, or web access; deterministically split across direct OpenAI and OpenRouter. |
Primary score
For valid calculation tasks, the primary outcome is complete numerical agreement: every numerical value in the local runner result must appear in the model’s final result within max(1e-6, 0.5% relative error). Object nesting, field names, units, and explanatory prose do not gate this score. A returned number can match only one runner number.
We separately report partial numerical coverage (at least one correct runner value), invalid-input classification, and the stricter runner-equivalent metric. The latter also requires the runner result structure and units, so it should not be used as the headline answer-quality result.
The matched Nemotron 3.5 Lightning skills uplift under the primary measure is 11.7% across 564 valid trial pairs (task-stratified 95% CI: 8.0%–15.2%).
Goldens and scoring
Goldens are generated before inference by executing the self-contained local skill runners. This measures skill-assisted task completion against the bundled runtime; it is not an independent validation of aerospace physics. The scorer is deterministic—no LLM judge is used.
The public value-only trial artifact includes model and runner numerical outputs for all trials. It deliberately excludes prompts, inputs, reasoning traces, and response prose.
Execution controls
Nemotron 3.5 Lightning was served through OpenRouter’s provider-specific nvidia/nemotron-3.5-lightning path. The skills arm may read one skill document, issue no more than two runner calls, and use no more than four model turns. Baselines receive the same task wording and output schema without those tools.
Limitations
- Fixed prompts and runner-derived goldens do not establish external physical truth.
- Provider serving versions, model behavior, and costs can drift.
- Terra, Sol, and Nemotron are different model families; only the matched Nemotron comparison isolates the skills condition.
- Numerical agreement does not establish operational suitability, certification compliance, or safety-of-life reliability.
- A condition-blinded qualitative review packet is available separately, but it is not an independent judge-model result.