· 3 min read
Why "not checked" is never "clean"
A clean verdict is a claim about what was tested. Why Horizon Audit never counts an unchecked benchmark task as clean, and what it shows instead.
Pre-release audits for benchmark tasks
Audit benchmark tasks before release.
Horizon Audit holds each task to a published intake standard, audits it, and returns a verdict with its proof.
Every part of the task is read, exactly as it was sent.
The problem
A flawed task gives a score that does not measure the work.
A benchmark is only as good as its tasks. One task that pays for no work, or fails a right answer, moves every score built on it. Most of these flaws stay hidden until someone runs the task against its own grader.
In a scan of public benchmark datasets, the company behind this product found:
- public datasets scanned
- 20
- tasks scanned
- 5,241
- tasks confirmed broken
- 29
What was wrong, by type
- Incomplete tests17
- Answer contamination14
- Specification-to-test mismatch11
- Gameable grading8
- Oracle or flaky grading4
A task can have more than one type of flaw.
Source: Benchmarks are more broken than we could have imagined. Horizon Analytics Labs LLC, .
The verdict
What a clean verdict proves.
No reward without the work.
An agent that does nothing, or only looks busy, earns nothing.
The task can be solved as written.
The task's own solution earns full reward in the task's own environment.
A wrong answer is refused.
An answer that is close but wrong does not pass.
A right answer is accepted.
A correct answer passes, also when it does not look like the author's.
The answer cannot be found without the work.
The answer is not there to be read from the environment.
The verdict
Clean or flawed. Each with its proof.
A verdict rests on what was measured, or on what an engineer has confirmed. An engineer confirms every flawed verdict before you see it. You never see a draft.
Clean
Every promise of a clean verdict held. The verdict holds for the exact files that were audited.
Flawed
A flaw that can change a realistic score was proved. You get the flaw, where it is, and the proof.
Needs from you
The task cannot be audited until you supply something. This is not a verdict, and it is never counted as clean.
What the verdict rests on
- Not checked is never clean.
- Code owns the verdict.
- Clean is a claim about what was tested.
- Tasks are never edited to make them run.
- You never see a draft.
How it works
From your tasks to a verdict.
01
Send your tasks
Send your Harbor tasks as one archive. Each task is pinned to a digest of its files, and nothing runs until you confirm the list.
02
Supply what is missing, if anything
A task that misses a rule of the intake standard comes back as "Needs from you", with one line for each thing that is missing. Supply it, and the task goes on.
03
Get a verdict with its proof
Clean, or flawed with the flaw and its proof. The verdict holds for the exact files that were audited. A changed task is sent again and audited again.
The intake standard
Nine rules a task meets before it is audited.
The standard is public. A task that misses a rule is not judged. It comes back with what is missing.
- IS-1An instruction
- IS-2A verifier
- IS-3A configuration that parses
- IS-4An environment
- IS-5A reference solution
- IS-6An environment that builds
- IS-7Every secret declared and supplied
- IS-8No access to the host
- IS-9Resources within the limits
From the blog
Notes on auditing benchmark tasks.
Questions
What people ask first.
What is Horizon Audit?
Horizon Audit audits benchmark tasks before they are released. You submit tasks, and each comes back with a verdict: clean, for the exact files that were audited, or flawed, with the flaw and its proof.
Who is it for?
Dataset vendors who prepare tasks for model labs, benchmark publishers, and teams that turn workflows into environments for agents. Model labs use the verdicts to decide which tasks to trust.
What makes a benchmark task flawed?
A task is flawed when its score does not measure the work. The common cases: an agent that does nothing passes, the reference solution fails, the tests check less than the instruction asks, the answer can be read from the environment, or the grading depends on something that can break.
Which task format do you accept?
Harbor tasks. A task is an instruction, a task.toml, an environment, a reference solution and a verifier. You send your tasks as one archive.
What is the intake standard?
The published list of nine rules a task must meet before it can be audited, IS-1 to IS-9. A task that misses a rule is not judged. It comes back as "Needs from you", with what is missing.
Know what a task measures before you ship it.
Access is by invitation. Tell us what you are building.
A product of Horizon Analytics Labs LLC