Jev and the System One Model: Typed Decisions with Calibrated Probabilities

Introduction: When the Output Is Not Supposed to Be Text

Most AI systems in production are asked to make decisions, not write essays. Is this ticket billing or technical? Does this request need human review? How urgent is this message? Should the agent act now or escalate? Each of those has a small, known set of valid answers, and the software around the model needs to branch on the result.

The default way to build this today is to ask a large language model for JSON and hope. It works often enough to ship, and fails in ways that are annoying to debug: an extra sentence before the JSON, a category that was never in the list, or a "confidence: 0.99" that turns out to mean very little.

Jev is built around a different premise: if the answer is a decision, the model's native output should be a decision. This article explains what that means from first principles — what goes in, what comes out, why it differs from an LLM, what System One refers to, and why calibration is the part that matters most. It also separates what is publicly documented from what is not, because that distinction is easy to lose in a topic this new.

This is the model-level companion to the agent-level posts in this blog. If you want the surrounding system — loops, guardrails, sandboxes — see The Agent Harness. Here the focus is the model itself.

What Is Jev?

Jev is a model specialized for making decisions, not for generating text. You give it two things:

1. State — the information to judge: text, conversation context, events, retrieved documents, tool results.
2. Typed questions — questions where the shape of a valid answer is fixed in advance.

It returns a typed value with a probability attached, in one of three forms: a Choice (pick from a fixed set), a Score (a level on an ordered scale), or a Noul (the probability that a statement is true). The result is data your code can use directly, not prose it has to parse.

LLM:   prompt                    →  token sequence
Jev:   (state, question, type)   →  (decision, probability)

                ┌────────────────────┐
                │       State        │
                │  text · context ·  │
                │  events · retrieved│
                └─────────┬──────────┘
                          ▼
                ┌────────────────────┐
                │        Jev         │
                │   Decision model   │
                └─────────┬──────────┘
             ┌────────────┼────────────┐
             ▼            ▼            ▼
          Choice        Score         Noul
         option +      level +      P(statement
         probs         probs          is true)

Question	Jev	A typical LLM
What is the output?	A typed decision plus probabilities	A token sequence
Can it invent an option?	No — the answer space is defined by the question type	Yes, unless you validate and retry
Where does confidence come from?	A decision probability, designed to be calibrated	Generated text, or token probabilities
Does it explain itself?	Not the goal	Yes, if asked

Why It Is Called System One

The name borrows from the System 1 / System 2 distinction in cognitive psychology. System 1 is fast, direct, and suited to frequent, bounded judgments. System 2 is slower and deliberate: multi-step reasoning, analysis, writing out a solution.

                  AI SYSTEM
                     │
          ┌──────────┴──────────┐
       Decision / control    Reasoning / generation
          │                     │
         Jev                   LLM
          │                     │
       fast, typed          slow, free-form

Examples of System 1 work: “is this spam?”, “which department owns this?”, “does this need review?” Examples of System 2 work: “derive this result and show each step”, “draft a migration plan.” Jev is positioned as the first kind. It is not presented as a replacement for an LLM, and that framing is the point: the two are meant to be composed.

Why an LLM Is an Awkward Decision Maker

A standard LLM is trained to model the next token given the previous ones:

P(x_t | x_1, ..., x_{t-1})

Everything it outputs, including a JSON object, is a sequence of tokens produced one at a time. Asking it for a decision means asking it to write down a decision. That indirection produces four recurring problems:

1. Parsing. “Sure! Here is the JSON:” followed by the JSON.
2. Schema violations. "needs_review": "probably" where a boolean was required.
3. Invented categories. You list A, B, C and receive D.
4. Unreliable confidence. A model can print 0.99 next to a wrong answer. Token probabilities describe how likely a token was, not how likely the decision is to be correct.

You can mitigate the first three with constrained decoding, schema validation and retries. The fourth is harder, because it is not a formatting problem. It is a question of whether the number means anything.

The Answer Space, Made Explicit

LLM output space:        all token sequences over the vocabulary   (astronomically large)
Jev Choice output space: the K options you supplied                (for example, K = 5)

In a Choice question the model is not free to say anything. The set of valid answers is part of the question. That is a systems-engineering property, not just a modeling one: downstream code can rely on the output being one of K known values.

The Three Primitives

1. Choice: Pick One From a Fixed Set

Given options A = {a1, ..., aK}, the model produces a probability distribution over them:

P(a_i | state, options),   with   sum over i of P(a_i) = 1

decision = argmax_i P(a_i)

An illustrative result for a support ticket where the customer cannot access their account:

{
  "choice": "account",
  "probabilities": { "billing": 0.03, "account": 0.94, "technical": 0.03 }
}

The whole distribution is the useful part. 0.97 / 0.02 / 0.01 and 0.36 / 0.33 / 0.31 both have an argmax, but only one of them should be acted on automatically.

2. Score: A Level on an Ordered Scale

Score differs from Choice because the options have an order. “Urgent” is closer to “high” than to “low.” Over levels 0 to 4:

P(0)=0.01   P(1)=0.04   P(2)=0.20   P(3)=0.60   P(4)=0.15

Expected score = 0(0.01) + 1(0.04) + 2(0.20) + 3(0.60) + 4(0.15) = 2.84

The expected value of 2.84 says more than the label “high” does: most of the mass sits on level 3, but there is real weight on 2 and 4. Ordered outputs also let you threshold sensibly (“escalate at 3 or above”), which a flat category list does not.

3. Noul: The Probability a Statement Is True

Noul is the binary form. It returns a single number in [0, 1]:

Noul = P(statement is true | state)

State:     "This article contains medical misinformation."
Question:  Is the statement true?
Result:    0.91

This fits gating, verification, moderation, grounding checks and escalation triggers: any place where code needs a probability to compare against a threshold.

Primitive	Answer space	Output	Typical use
Choice	Unordered set of K options	Option + distribution	Routing, categorization, next-action selection
Score	Ordered levels	Level + distribution	Urgency, severity, rubric grading
Noul	True / false	A probability in [0, 1]	Verification, policy checks, escalation gates

State, and Many Questions on One State

State is not limited to a single user message. It can bundle the message, conversation history, retrieved documents, the previous action and tool results. A useful consequence is that one state can be asked several typed questions in a single request, and the questions can be evaluated in parallel:

                 State
                   │
       ┌───────────┼───────────┐
       ▼           ▼           ▼
    Choice        Noul        Score
    category?   needs_human?  urgency?

The snippet below shows the shape of such a request. It is an illustration of the concept, not the actual Jev API — check the official documentation linked in the references for the real interface.

# Conceptual shape only. NOT the real Jev SDK.
request = {
    "state": {
        "user_message": "I can't log in and I need access before a 9am meeting.",
        "conversation": "...",
        "retrieved_documents": "...",
    },
    "questions": [
        {"type": "choice", "id": "category",
         "options": ["billing", "account", "technical"]},
        {"type": "noul",   "id": "needs_human"},
        {"type": "score",  "id": "urgency", "levels": [0, 1, 2, 3, 4]},
    ],
}

What Is Public, and What Is Not

This is the part most explanations skip. Be precise about it, especially if you plan to cite anything below in a design review.

Publicly documented	Not publicly documented
Specialized for typed decisions	Number of layers and parameters
Choice, Score and Noul primitives	Hidden dimension and attention design
Returns probabilities rather than prose	Tokenizer details
Calibration is a core design goal	Full training corpus
Described by its developer as trained with RLCD	The exact RLCD loss, reward model and optimizer
Evaluated in an independent benchmark paper	Any official internal architecture diagram

If you read a claim such as “Jev is a Transformer with X layers and Y parameters,” treat it as unverified unless it points to a primary source from the developer or a technical paper. Everything in the next section is therefore a mental model, not a description of Jev's internals.

A Mental Model: One Representation, Typed Heads

To reason about what a decision model could look like, picture a shared representation of the input feeding one head per answer type. The code below is a generic sketch of a bounded-decision model. It is not Jev's architecture.

import torch
import torch.nn as nn

class DecisionHeads(nn.Module):
    """Generic sketch of typed decision heads. NOT Jev's published architecture."""
    def __init__(self, d_model: int, n_choices: int, n_levels: int):
        super().__init__()
        self.choice_head = nn.Linear(d_model, n_choices)  # Choice
        self.score_head  = nn.Linear(d_model, n_levels)   # Score
        self.noul_head   = nn.Linear(d_model, 1)          # Noul

    def forward(self, h):                                 # h: (batch, d_model)
        choice_p = torch.softmax(self.choice_head(h), dim=-1)  # sums to 1
        score_p  = torch.softmax(self.score_head(h),  dim=-1)  # distribution over levels
        noul_p   = torch.sigmoid(self.noul_head(h)).squeeze(-1) # in [0, 1]
        return choice_p, score_p, noul_p

The mathematics behind each head is standard. A Choice head is a softmax over K logits:

z = W h + b          (W has shape K x d)
p_i = exp(z_i) / sum_j exp(z_j)

A Noul head is a sigmoid over one logit:

z = w . h + b
p = 1 / (1 + exp(-z))

A production ordinal Score head would usually respect the ordering of levels (for example via cumulative probabilities), which a plain softmax does not. The sketch trades that for brevity.

Calibration: The Part That Makes Probabilities Usable

A probability is only useful if it means what it says. Take 1,000 predictions that each report 0.9 confidence. If the model is calibrated, roughly 900 of them should be correct:

P(correct | confidence = 0.9)  ≈  0.9

Accuracy Is Not Calibration

Two models can both reach 95% accuracy and still behave very differently when wrapped in code that uses their confidence:

	Model A	Model B
Accuracy	95%	95%
Confidence when right	~0.99	~0.99
Confidence when wrong	~0.52	~0.99
Can software trust the confidence?	Yes — errors are visible	No — errors look identical to successes

For an agent that routes on confidence, Model A is far more useful. This is the practical reason calibration is the headline property of a decision model rather than a footnote.

Measuring It: Expected Calibration Error

Bucket predictions by confidence, then compare each bucket's average confidence with its actual accuracy. ECE is the weighted average of that gap:

ECE = sum over bins m of ( |B_m| / n ) * | accuracy(B_m) - confidence(B_m) |

import numpy as np

def expected_calibration_error(confidences, correct, n_bins=10):
    """confidences: predicted probability of the chosen option.
       correct:     1 if the decision was right, else 0."""
    confidences = np.asarray(confidences, dtype=float)
    correct = np.asarray(correct, dtype=float)
    edges = np.linspace(0.0, 1.0, n_bins + 1)
    ece = 0.0
    for lo, hi in zip(edges[:-1], edges[1:]):
        in_bin = (confidences > lo) & (confidences <= hi)
        if in_bin.any():
            gap = abs(correct[in_bin].mean() - confidences[in_bin].mean())
            ece += in_bin.mean() * gap
    return ece

Run this on your own labeled data, per question type. A published number from someone else's datasets is a hint, not a guarantee for your distribution.

Selective Prediction: Let the Model Abstain

Once confidence is trustworthy, you can stop forcing the model to answer everything. Define a confidence function c(x) and a threshold τ:

f_tau(x) = f(x)       if c(x) >= tau
         = abstain    if c(x) <  tau

The two quantities that describe the trade-off:

Coverage(tau) = P(c >= tau)              how many cases the model takes
Risk(tau)     = P(error | c >= tau)      how often it is wrong on those cases

Raising the threshold lowers coverage and lowers risk. Plot risk against coverage across all thresholds and you get the risk-coverage curve; the area under it is AURC. A model that keeps risk low while holding coverage high is better at selective prediction, independent of its raw accuracy.

def risk_coverage(confidences, correct, thresholds):
    confidences = np.asarray(confidences, dtype=float)
    correct = np.asarray(correct, dtype=float)
    rows = []
    for tau in thresholds:
        taken = confidences >= tau
        coverage = taken.mean()
        risk = 1.0 - correct[taken].mean() if taken.any() else 0.0
        rows.append((tau, coverage, risk))
    return rows

def route(decision, act_at=0.95, confirm_at=0.70):
    """Thresholds are placeholders. Tune them on validation data."""
    p = decision["confidence"]
    if p >= act_at:
        return "act"
    if p >= confirm_at:
        return "confirm"
    return "escalate_to_llm_or_human"

                Jev
                 │
            probability
       ┌─────────┼─────────┐
       ▼         ▼         ▼
     high      medium      low
       │         │         │
      act     confirm   escalate

This is also the idea behind REFLEX, the paper on using Jev for selective control in LLM agents: let the fast decision model handle the cases it is confident about, and hand the rest to a stronger and more expensive system. The goal is not “make Jev solve everything,” but “make Jev solve what it can solve reliably.”

What RLCD Is Said to Change

The developer describes Jev as trained with RLCD, presented as reinforcement learning for calibrated decisions. Compare that with the more familiar RLHF:

	RLHF	RLCD (as described)
Typical signal	Human preference between responses	Decision correctness and probability quality
What is optimized	Responses people prefer	Right decisions with uncertainty that matches reliability
Output being shaped	Text	Typed decisions with probabilities

The internal details of RLCD, including its loss and reward formulation, are not published in full. It is worth resisting the urge to fill that gap with a plausible-looking objective.

What an Independent Benchmark Found

A separate paper, Evaluating and Benchmarking the System One Model Jev, tested the model across 37 datasets and 346,009 requests. The tasks included classification, routing, natural language inference, reading comprehension, commonsense reasoning, moderation, legal analysis and rubric scoring.

The reported pattern, in plain terms:

• Strong performance on several of those task families.
• Choice probabilities were well calibrated.
• Binary (Noul) probabilities sometimes needed threshold tuning before they behaved well in practice.
• Performance dropped on low-resource languages, fine-grained label sets, noisy labels and subjective rubric-based judgments.

That last list is the sensible part of the result. Telling cats from cars is easy; telling “slightly positive” from “positive” from “strongly positive” is hard for any classifier, because the classes are close in meaning. Treat the benchmark as evidence that the approach is credible, then validate on your own data before relying on it.

Action Space Size Matters

REFLEX reports a related finding: decisions get harder as the number of options grows, especially when there are near-valid alternatives. Consider:

Easy:   A = cat       B = dog
Hard:   A = allowed
        B = almost allowed
        C = allowed under another condition
        D = forbidden

The second case is not harder because the words are longer. The boundaries between options are policy boundaries, and the model has to resolve them, not just match keywords. If your decision set looks like the second case, expect to spend more effort on option design and evaluation.

Where Jev Fits Next to an LLM

A useful way to place it: the LLM answers “what should I say or do, and why?”; Jev answers “which of these should happen, and how sure are we?”

User request
     │
     ▼
Jev: typed decision + probability
     │
     ├── high confidence  → execute the decision directly
     ├── medium           → ask for confirmation
     └── low              → escalate to an LLM or a human

In the terms of the agentic loop, this slots into the reasoning step for the bounded sub-decisions, while open-ended reasoning stays with the LLM. In terms of the agent harness, a calibrated probability is a natural input to guardrails: “require approval when the probability is below a threshold” is a policy you can express in code rather than in a prompt. And in a secure agent runtime, it strengthens the decision layer that sits in front of the sandbox. It does not replace the sandbox, because a confident wrong decision is still wrong.

There is also a cost angle. A short typed decision avoids generating a long response, which matters for latency and spend; the decode-phase mechanics are covered in How LLM Inference Really Works. No exact speed or price figures for Jev are asserted here, since those depend on deployment and are best measured on your own workload.

Is Jev Just a Classifier, a Router, or Prompt Engineering?

• Not just a classifier. Classification is one of three primitives. Score covers ordered judgments and Noul covers probabilistic binary judgments, and the model is meant to be a general decision interface for software.
• Not just a router. A router picks a destination. Jev's output can be a destination, but also a severity, a truth probability or a gate.
• Not “an LLM plus a prompt that asks for a probability.” The claim is that the model itself is exposed as a decision interface, with training and evaluation aimed at structured, calibrated decisions. If it were only a prompt, there would be no need for a separate model class.
• Not a small ChatGPT. Its parameter count is not publicly stated, and size is the wrong axis anyway: the difference is the task it is specialized for.
• Not a reasoner that shows its work. It can handle rich natural-language state, but its job is to return a decision, not a chain of reasoning.

Limitations and Honest Caveats

• No explanation by default. If you need a justification for audit, you need another component, or you log the state, question and probabilities as the audit trail.
• Calibration is distribution-dependent. A model calibrated on benchmark datasets can drift on your data. Measure ECE and risk-coverage on a labeled sample from production.
• Thresholds are not free. The 0.95 and 0.70 in the example above are placeholders. The benchmark noting that binary outputs can need tuning is a reminder to set them empirically.
• Hard option sets stay hard. Fine-grained labels, noisy labels and subjective rubrics were weaker areas in the independent evaluation.
• Language coverage. Low-resource languages were a reported weakness, which matters for multilingual deployments.
• Containment is separate. Calibrated decisions reduce bad actions; they do not make arbitrary code safe to run. Keep the isolation described in Sandboxing AI Agents.
• Young ecosystem. Public detail is limited and some claims are vendor-originated. Prefer the two papers and the official documentation over secondary summaries, including this one.

A Practical Adoption Checklist

1. List the decisions. Write down every place your system branches on a model output. Mark the ones with a fixed, known answer set.
2. Pick the primitive. Unordered options → Choice. Ordered levels → Score. Yes/no with a threshold → Noul.
3. Design the options carefully. Make them mutually exclusive and clearly separated. Near-valid alternatives are where accuracy falls.
4. Build a labeled evaluation set from real traffic, including awkward and low-quality inputs.
5. Measure calibration, not just accuracy. Compute ECE and the risk-coverage curve per question.
6. Set thresholds from the curve, based on the cost of an error versus the cost of escalating.
7. Keep the fallback. Route low-confidence cases to an LLM or a person, and log them. They are your best source of new evaluation data.

Key Takeaways

• Jev outputs decisions, not text. The unit of output is a typed value plus a probability.
• Choice, Score and Noul cover unordered selection, ordered grading and true/false probability.
• The answer space is constrained by the question type, so invented categories and schema drift are not the failure mode.
• Calibration is the differentiator. Equal accuracy does not mean equal usefulness once software acts on confidence.
• Selective prediction turns probabilities into policy: act, confirm or escalate, measured by coverage and risk.
• Internals are not public. Architecture, size and the exact RLCD recipe are unpublished; don't fill the gaps with guesses.
• Validate on your own data. Independent results are encouraging, with known weak spots in fine-grained, noisy, subjective and low-resource settings.

Final Thought

The interesting idea in System One is not that a model can be fast. It is a change in the interface between AI and software. When a model returns a paragraph, the surrounding code has to interpret it. When a model returns a typed decision and a probability you have verified is meaningful, the surrounding code can use it: branch on it, threshold it, audit it, and measure it. That shift, from generated text to calibrated decisions, is what the name System One is really pointing at. Whether it fits your system depends on one thing you can test: does its confidence, on your data, mean what it says?

References

• REFLEX with Jev for Efficient Selective Control in LLM Agents — arXiv
• Evaluating and Benchmarking the System One Model Jev — arXiv
• System One / Jev Documentation — TypeSafe
• System One Concepts — TypeSafe
• OpenJev — GitHub
• OpenJev (alternative implementation) — GitHub

Further Reading

• The Agent Harness: The System Around the LLM — where decision probabilities plug into guardrails
• The Agentic Loop — the reasoning step that a decision model can serve
• Building a Secure Agent Runtime — decision layer plus sandbox
• Transformer Model Families — the generation-versus-understanding split behind model design
• Entropy in Decision Trees — uncertainty as a measurable quantity