(I'm the founder of Snyk and Tessl, apply what biases you wish)
I think we're mixing three optimizations: higher success rate, higher consistency and higher efficiency.
Success rate is about building confidence the agent can succeed. The hardest part here is defining what success is in the first place, and creating evals that help you measure it.
Consistency is about guardrails. The harness shines here, as it takes control away from the model, moving it to hooks that force certain paths, or require the use of deterministic tools for certain steps.
Given consistency and success rate, you can work on efficiency. You can save cognition (tokens) by moving work to tools, try to use a cheaper model, etc.
These three build on one another, at least if you want to scale them.
You can't improve consistency if you don't define success, and you can't add efficiency if you're not consistent.
I think we're mixing three optimizations: higher success rate, higher consistency and higher efficiency.
Success rate is about building confidence the agent can succeed. The hardest part here is defining what success is in the first place, and creating evals that help you measure it. Consistency is about guardrails. The harness shines here, as it takes control away from the model, moving it to hooks that force certain paths, or require the use of deterministic tools for certain steps. Given consistency and success rate, you can work on efficiency. You can save cognition (tokens) by moving work to tools, try to use a cheaper model, etc.
These three build on one another, at least if you want to scale them. You can't improve consistency if you don't define success, and you can't add efficiency if you're not consistent.