Preprint · 2026

Elicitation or Expansion?

Demystifying Gains from Agent Harness Optimization

Younghyun Kim^1*, Wooseok Seo^2*, Minseo Kim³, Seunghyuk Oh¹, Minjae Lee¹, Donghoon Kim¹, Hyung Il Koo¹, Jaehyung Kim^2†, Wonjun Kang^1†

¹FuriosaAI ²Yonsei University ³University of California, Berkeley

^*Equal contribution ^†Co-corresponding authors

arXiv Code Dataset

What we did: We bring pass@k analysis to agent harness optimization,
comparing optimized harnesses against larger models and higher reasoning effort across attempt budgets and task difficulty.
What we found: A lead at pass@1 can reverse with more attempts.
Even when optimized harnesses approach or exceed stronger baselines overall, they do not surpass them on the hard subsets we evaluate.

01 · Background

Same weights.
Wider reach?

Harness optimization rewrites the code around a fixed model, including its prompts, context, tool use, and execution logic, and reports large gains. A higher success rate can mean two different things.

Illustrative Per-task success probability for eight hypothetical tasks; gray shows the baseline and red the success added above it. Elicitation makes tasks already within reach more reliable. Expansion adds successes on tasks with no baseline success, broadening coverage.

The difference matters for methods that learn from their own successes, such as self-training and recursive self-improvement. For them, even rare successes on previously unreachable tasks can seed further search or learning.

02 · The lens

A lens from RL.
Now on harness optimization.

RL studies compare models before and after training with pass@k: pass@1 gains can shrink or reverse at larger sampling budgets (Yue et al., 2025). We bring the same analysis to harnesses, with three comparisons.

Step 1

Across attempt budgets

pass@1 measures reliability; pass@k, the expected share of tasks solved at least once in k attempts, measures coverage.

Step 2

Against stronger interventions

The baseline harness with a larger model or higher reasoning effort.

Step 3

By task difficulty

Using the benchmarks' own difficulty labels for HotpotQA and AppWorld.

Text classification GPT-OSS-20B vs. 120B
HotpotQA Qwen3-8B vs. 32B
AppWorld GPT-OSS-120B, medium vs. high effort

The estimator, precisely

pass@k estimator

\[ \widehat{\mathrm{pass@}k} \;=\; \frac{1}{N}\sum_{i=1}^{N}\left[\,1-\binom{n-c_i}{k}\Big/\binom{n}{k}\right], \qquad 1 \le k \le n \]

Each of the \(N\) tasks is attempted \(n\) times, and \(c_i\) counts the successes on task \(x_i\). An attempt is one complete execution, including every model call and tool interaction; \(n\) is 64 for text classification and 128 for HotpotQA and AppWorld. Success is scored by binary correctness for text classification, answer containment confirmed by an LLM judge for HotpotQA, and arity-fixed task goal completion for AppWorld. Harnesses are optimized with Meta-Harness.

03 · Across budgets

A first-try lead
need not be a coverage lead.

On LawBench, the optimized 20B harness leads the 120B model by 16.05 pp at pass@1, but falls behind from \(k = 11\). On all three text benchmarks, it ends below the 120B model at pass@64.

Legend: base solver with H0 (solid black), base solver with H* (red), reference with H0 (dashed). — Measured · text classification · 64 attempts per task Pick a dataset and drag the budget. The red curve is the optimized harness with GPT-OSS-20B and the dashed curve keeps the baseline harness with a larger GPT-OSS-120B.

pass@k curves on Symptom2Disease. — Measured · text classification · 64 attempts per task Pick a dataset and drag the budget. The red curve is the optimized harness with GPT-OSS-20B and the dashed curve keeps the baseline harness with a larger GPT-OSS-120B.

Text classification: mostly elicitation.

Optimization mostly makes existing successes more frequent. On LawBench, it keeps 49 of the 53 tasks the baseline solves within 64 attempts, loses 4, and newly solves 7.

Legend: GPT-OSS-20B with H0 (hatched gray) and with H* (red). — Measured · GPT-OSS-20B · 64 attempts per task Tasks grouped by how many of their 64 attempts succeeded.

Distribution of per-task success fractions on Symptom2Disease. — Measured · GPT-OSS-20B · 64 attempts per task Tasks grouped by how many of their 64 attempts succeeded.

Even among candidates from one run.

The same split appears among the harness candidates that a single optimization run generates on HotpotQA. The candidate with the highest pass@1 is not the one with the highest pass@128, and two candidates can trade places against Qwen3-32B as the budget grows.

Scatter of pass@1 against pass@128 for the Qwen3-8B harness candidates of one optimization run; the pass@1 leader and the pass@128 leader are different candidates. — Measured · HotpotQA · Qwen3-8B · 300 questions · 128 attempts each (a) The 40 candidates generated by one optimization run. (b) The 40 candidates of a separate run with relaxed per-attempt resource limits. One leads Qwen3-32B at pass@1 (74.3% vs. 69.1%) but trails it at pass@128 (85.0% vs. 87.7%); the other trails at pass@1 (62.8%) but leads at pass@128 (88.7%).

pass@k curves for the candidates of one optimization run: candidate A leads Qwen3-32B at pass@1 but falls below it as k grows, while candidate B starts below and ends above it. — Measured · HotpotQA · Qwen3-8B · 300 questions · 128 attempts each (a) The 40 candidates generated by one optimization run. (b) The 40 candidates of a separate run with relaxed per-attempt resource limits. One leads Qwen3-32B at pass@1 (74.3% vs. 69.1%) but trails it at pass@128 (85.0% vs. 87.7%); the other trails at pass@1 (62.8%) but leads at pass@128 (88.7%).

04 · Key finding · Task difficulty

Close in aggregate.
Never ahead on hard tasks.

On HotpotQA and AppWorld, the selected optimized harnesses approach or exceed a larger model or higher reasoning effort across all tasks. On the hard subsets, none exceeds it at any evaluated budget up to 128 attempts.

Legend: Qwen3-8B and Qwen3-32B baselines, and the optimized Qwen3-8B harnesses of three runs with their mean and SD. — Measured · 128 attempts per task · benchmark difficulty labels Pick a benchmark and a difficulty level. pass@k across three optimization runs: the mean ± 1 sample SD (thick red curve and band), the individual runs (thin marked curves), and the baselines (gray). Qwen3-8B vs. Qwen3-32B on HotpotQA; GPT-OSS-120B at medium vs. high reasoning effort on AppWorld.

HotpotQA pass@k across three runs, all questions. — Measured · 128 attempts per task · benchmark difficulty labels Pick a benchmark and a difficulty level. pass@k across three optimization runs: the mean ± 1 sample SD (thick red curve and band), the individual runs (thin marked curves), and the baselines (gray). Qwen3-8B vs. Qwen3-32B on HotpotQA; GPT-OSS-120B at medium vs. high reasoning effort on AppWorld.

Harness gains trail scaling on hard tasks

On hard tasks, harness optimization yields smaller mean pass@1 gains than scaling on all three benchmarks (dataset-level difficulty for text).

Every candidate falls short

All 40 candidates from one HotpotQA optimization run stay below the 32B model on hard questions.

Hard-focused re-optimization did not close the gap

Re-optimizing the HotpotQA harness on a hard-focused task mixture raises hard-task pass@1 (56.88% → 59.28%; 32B: 59.51%) but leaves pass@128 unchanged at 73.17% (32B: 78.05%).

05 · Looking ahead

Measure reliability
and reach separately.

Check a harness gain at the attempt budget you will use, against a larger model or more reasoning effort, and on hard tasks. Achieving consistent capability expansion on hard tasks remains an open challenge for harness optimization, especially when successful executions support recursive self-improvement or evolutionary search. Our results are bounded by the evaluated benchmarks, solver families, and optimization runs.

BibTeX

@article{kim2026elicitation,
  title   = {Elicitation or Expansion? Demystifying Gains from Agent Harness Optimization},
  author  = {Kim, Younghyun and Seo, Wooseok and Kim, Minseo and Oh, Seunghyuk and
             Lee, Minjae and Kim, Donghoon and Koo, Hyung Il and Kim, Jaehyung and Kang, Wonjun},
  journal = {arXiv preprint arXiv:2610.27889},
  year    = {2026}
}