Resistant Task Construction

Key Highlights

Tencent launched WorkBuddy Bench, a coding agent benchmark suite covering four work domains—Code, Web, Office, and Security. Its most rigorous aspect is that every task is reverse-engineered from real commits, PRs, or business scenarios and then rewritten as colloquial role-play requests, resisting data contamination by construction. Put simply, it tests not "memorized problems" but the agent's real ability to get the job done in a true workflow. The design directly attacks the core weakness of modern benchmarks: that a model's training data and the test set are no longer cleanly separated, so a high score may reflect memory rather than skill, and WorkBuddy Bench closes that loophole at the source by grounding every item in proprietary, messy, real-world history. By rooting tasks in real repositories, the benchmark also ages better than frozen datasets, because as tools evolve the same real-world scenarios stay relevant while static test sets quietly go stale.

Capabilities and What Happened

WorkBuddy Bench pulls evaluation back from "solving problems" to "doing work." Traditional benchmarks often use public datasets where the model may have seen the answers during training; here, real tasks from enterprises—development, web operations, office automation, security handling—are repackaged as natural-language requests, so the agent executes them as if receiving a colleague's demand. The four domains cover most of an engineer's daily actions: writing code, running web pages, processing documents, and doing security response, thus better reflecting the agent's real performance after deployment. By spanning these four domains in one suite, the benchmark captures the cross-cutting nature of real jobs, where a single assignment might require writing a script, opening a dashboard, editing a spreadsheet, and flagging a vulnerability in the same breath—something single-domain tests systematically miss. The role-play framing also surfaces a model's ability to interpret vague human requests, a skill that matters more in production than solving precisely specified puzzles that rarely occur outside benchmarks.

Technical Details

The key to contamination resistance is "reverse engineering plus role-play rewriting": task sources are real repositories and business records, the answer space is not fixed, and the model cannot cram by memory. The benchmark runs on CodeBuddy Code and Claude Code, covering mainstream coding agents to ensure cross-comparability. All task directories, environment images, scoring tools, and reference solutions are fully open-sourced, meaning any team can reproduce results and align standards rather than only trusting vendor self-reported scores. This "reproducible evaluation" is exactly the credibility the industry lacks most. The open-sourcing is not a courtesy; it is the point, because a benchmark whose numbers cannot be independently rerun is just marketing, and by handing over the environments and scorers, Tencent invites the community to challenge and improve the suite rather than merely admire it. Running on both CodeBuddy Code and Claude Code gives the suite a neutral footing, so the score reflects the agent, not the harness, and lets teams compare across ecosystems without rebuilding the environment.

Comparison With Competitors

Compared with benchmarks that only test algorithm problems or single-turn generation, WorkBuddy Bench wins on "complete work domains" and "real sources." It does not compete with academic leaderboards on who scores higher, but provides a unified ruler so different coding agents can fairly contend under the same set of real tasks. For purchasers, such open-source, reproducible benchmarks are more informative than marketing rhetoric. Where a vendor demo can cherry-pick the flattering case, a shared, contamination-resistant suite removes the staging; the result is a number that means the same thing to a startup evaluator and a Fortune 500 procurement team, which is the prerequisite for agents ever being bought on evidence instead of on slides. Where single-domain coding tests reward narrow trickery, WorkBuddy Bench's four domains punish agents that cannot generalize, which is the property buyers actually need before trusting an agent with a live codebase.

Industry Impact and Use Cases

To put it bluntly, coding agents have entered the stage of competing on "real work," and benchmarks like WorkBuddy Bench will become a necessity for selection and iteration. It suits enterprises evaluating agents' competence in R&D, office, and security scenarios, and also helps developers locate weaknesses against scores. The fully open-source approach can push the entire coding-agent track from talking to itself toward public comparability, accelerating the overall rise of capability. The strategic effect is a rising tide: as every agent is measured against the same honest yardstick, the weak ones are exposed and the strong ones are proven, and the whole category earns the trust it needs to move from pilot projects to production payrolls. If the suite becomes a default yardstick, it could do for coding agents what standardized benchmarks did for image models: convert vague claims into comparable numbers and accelerate the whole field by making progress measurable and contestable. Until then, buyers should treat any unverified agent claim with the skepticism this suite is designed to extinguish.