Harness Costs Vary 17x
Tasks cost between $3.24 and $18.34 across 9 harnesses with the same model
A recent evaluation by FrontierHarness Eval has revealed a significant variation in costs across 9 harnesses with the same model. The cost per pass varied 17 times, with costs ranging from $3.24 to $18.34 per task.
The evaluation was conducted on Runta agent runtimes, with all harnesses and the task environment prepared once as a golden checkpoint. Every run was a fresh restore with identical vCPU, memory, disk size, disk contents, and memory state. This ensured that the results were consistent and comparable across all 9 harnesses.
The cost per task was calculated excluding failed attempts. However, when failed attempts are included, the cost per task increases to $3.24. This highlights the importance of considering failed attempts when evaluating the cost-effectiveness of a harness.
Interestingly, the cache hit rate was found not to be a direct factor in cost. This means that a cached task can still incur high costs, even if it fails. Additionally, the evaluation found that quality and cost can diverge, with some tasks costing $18.34 despite passing 19 tasks.
FrontierHarness v1.0, which was used in the evaluation, focuses on software engineering contexts and terminal-based tasks. The results of the evaluation may not generalize to other areas of knowledge work.
The significant variation in costs across the 9 harnesses highlights the need for careful evaluation and selection of harnesses to ensure cost-effectiveness. With the cost per task ranging from $3.24 to $18.34, the choice of harness can have a significant impact on overall costs.
## What happens next The evaluation provides valuable insights for developers and organizations looking to optimize their harnesses and reduce costs. By understanding the factors that affect cost and selecting the most cost-effective harness, developers can improve the efficiency and effectiveness of their software engineering workflows.
