Jev vs Claude: which picks the right tool?

—

—

What we learned

Tap a card to see a real example from the run.

Top-1 accuracy

—Jev
—Claude, first pick
—Claude, final action

First pick, look-ups counted

—Jev
—Claude

Median latency

—Jev
—Claude, first
—Claude, final

Claude cost per 1,000 picks (Jev uses TypeSafe credit)

—Claude, first
—Claude, final

Accuracy by request type

First pick: Claude's first tool call. Final action: searches allowed, the action after them recorded.

Latency per pick

Every call, log scale.

Why Claude's first pick missed

First-pick run: Claude's wrong picks, by kind.

Can you trust the confidence?

On the diagonal = honest. Jev: its own score. Claude: share of 5 runs agreeing.

Jev first, Claude as backup

Hand over to Claude when Jev's confidence is below τ.

Accuracy by language

Share right.

Code, dataset and raw results: github.com/TubaAli/jev-vs-claude. Synthetic requests only; tools were picked, never run.