# An Executable Benchmarking Suite for Tool-Using Agents

Preprint · Zhiqing Zhong, Zhijing Ye, Jiamin Wang, Xiaodong Yu

- Archive id: arxiv:2605.11030
- Published: 2026-05-10
- arXiv: https://arxiv.org/abs/2605.11030
- Citations: 0
- Source URL: https://arxiv.org/abs/2605.11030
- PDF: https://arxiv.org/pdf/2605.11030
- Provenance: arxiv

## Abstract

Closed-loop tool-using agents are increasingly evaluated in executable web, code, and micro-task environments, but benchmark reports often conflate workloads, action-generating drivers, and the evidence admitted for systems-facing claims. We present an executable benchmarking suite that makes these objects explicit under a shared evidence-admission contract. The suite connects WebArena Verified, a SWE-Gym slice with SWE-bench-compatible verification, and MiniWoB++ through common workload adapters, task manifests, event schemas, replay/freeze policy, declared drivers, and reporting pipelines. In the canonical release, the gate separates paper-facing evidence from preflight, fixture, smoke, and diagnostic rows while preserving non-admitted artifacts for audit and onboarding. The admitted evidence records latency, invalid-action behavior, patch-generation cost, verifier metadata, replay bindings, and provenance under one auditable contract. The gate is decision-relevant rather than merely clerical: in a separate WebArena Verified controller study, clean-baseline and medium live-stressed evaluation select different fixed controller variants under the same workload and admission contract. The release is scoped as a benchmarking suite and admitted evidence, not a new agent policy, model leaderboard, backend comparison, or autonomous SWE-bench solver.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

Topics: [LLM Agents & Tool Use](https://space-kitty.com/topics/ai-agents)

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.