Same Scores, Different Decisions: Evaluating JEV and Language Models for Legal Document Understanding
A new arXiv paper and a community benchmark both tested Jev against conventional LLMs, while the tooling around it kept growing.
TL;DR
- An arXiv paper titled 'Same Scores, Different Decisions' evaluated TypeSafe's Jev against language models on legal document understanding.
- A Hacker News submission compared Jev with LLMs on 770 'Am I the Asshole?' posts.
- GitHub projects offered to turn any LLM into a Jev-style decision model and catalogued 805 verified Jev examples, while a reply assistant from the jev-chat project passed 5,000 stars.
An arXiv paper, 'Same Scores, Different Decisions', evaluated JEV and language models for legal document understanding. Its title frames the result as matching scores but diverging decisions. [1]
A separate Hacker News submission ran Jev against LLMs on 770 'Am I the Asshole?' posts, applying both to crowd-judged social dilemmas rather than factual questions. [2]
Tooling kept accumulating. A repository under the nokia-applied-research account said it can turn any LLM into a Jev-style decision model with 'typed decisions, real probabilities, no training', and another list indexed 805 verified examples of Jev by the decision each one makes. [3] [4]
Consumer-facing experiments appeared too: jev-chat-jarvis, a phone-based conversation co-pilot for apps including WeChat, QQ, X and Feishu that reads the screen and suggests candidate replies, had about 5,250 GitHub stars at collection time. [5]
Why it matters
Matching benchmark scores can hide different decisions, which matters for anyone putting typed-decision layers into legal or compliance workflows: evaluation has to compare outputs case by case, not just aggregate accuracy.
Editor's note
The arXiv paper is cited at the title level and its results were not reproduced here; the AITA comparison and the GitHub projects are community work whose claims were not independently verified.