How can I detect LLM hallucinations without extra API calls?

Run a cheaper specialist model in parallel and measure how far its score diverges from the LLM's. On Financial PhraseBank this separates correct from incorrect LLM outputs at ROC AUC 0.898, with Welch t = 46.8. Mean divergence is 0.168 when the LLM is right and 0.709 when it is wrong, a four-fold gap. Because the specialist output is already computed for routing, the trust score costs nothing additional.

How does this compare to SelfCheckGPT?

SelfCheckGPT and self-consistency detect errors by re-sampling the same model several times, so detection cost scales linearly with the number of samples. Cross-model divergence compares against a different model the pipeline already ran for another purpose, so the marginal cost is zero. The trade-off is that it requires you to already be running a second heterogeneous model; SelfCheckGPT works on a single model in isolation.

Does any pair of models work as a hallucination detector?

No. Only the cross-granularity pairing carries the signal. The encoder-versus-reasoner divergence reaches AUC 0.898, while lexicon-versus-encoder reaches 0.620 and lexicon-versus-reasoner only 0.575. Disagreement in general is a weak signal; disagreement across contextual granularities is a strong one.

Does divergence also detect adversarial attacks on LLM inputs?

Only partially. It reaches AUC approximately 0.71 on negation-insertion attacks but roughly 0.50, which is chance, on synonym swaps, numeric flips, and character-level typos. It is a strong hallucination detector and a weak adversarial detector, and should not be deployed as a substitute for purpose-built adversarial defences.

TriAgent → Hallucination detection

Detecting LLM hallucinations for free

If you already run a cheaper specialist alongside your LLM, the gap between their scores tells you when the LLM is wrong — at AUC 0.898, with no extra model call, no labels, and no training.

CIKM 2026 arXiv:2607.19794 Code

The measurement

Stratify Financial PhraseBank by whether the LLM got the answer right, then look at SDI_ER — the absolute difference between the sentence-level encoder's score and the LLM's:

Condition	Mean `SDI_ER`
LLM is correct	0.168
LLM is wrong	0.709
Separation	4.2×

Thresholded as a binary "the LLM is hallucinating" flag, that reaches ROC AUC = 0.898, with Welch t = 46.8 and p ≈ 6 × 10⁻²⁵⁵.

ROC curve plot with four lines. The SDI_ER curve bows sharply toward the top-left corner reaching AUC 0.898. SDI_max reaches 0.782. SDI_LE reaches 0.620 and SDI_LR 0.575, both close to the diagonal chance line. — **ROC curves for each SDI variant as an "LLM is wrong" detector.** Only the encoder-versus-reasoner pairing (`SDI_ER`) is a strong detector. The pairings that involve the word-level lexicon hug the diagonal.

Why it costs nothing

In a tier-routed pipeline the encoder's score is already computed — it is what decides whether the query needs to escalate in the first place. Reading it a second time as a trust signal adds one subtraction.

Method	Marginal cost per query	Requires
Cross-model divergence	zero — reuses an existing output	a second heterogeneous model in the pipeline
SelfCheckGPT	n extra LLM samples	nothing beyond the LLM itself
Self-consistency	n extra LLM samples	nothing beyond the LLM itself

The honest trade-off: if you run exactly one model, SelfCheckGPT applies and this does not. There is nothing to diverge from.

Only cross-granularity disagreement works

This is the result that identifies the mechanism. Evaluated as detectors on the same data:

Variant	Pairing	Mean (right)	Mean (wrong)	Welch t	AUC
`SDI_LE`	lexicon vs. encoder	0.394	0.499	8.66	0.620
`SDI_LR`	lexicon vs. reasoner	0.324	0.386	5.27	0.575
`SDI_ER`	encoder vs. reasoner	0.168	0.709	46.82	0.898
`SDI_max`	max of the three	0.443	0.797	32.21	0.782

The lexicon-based pairings barely beat chance. What makes SDI_ER work is that the encoder and the reasoner read genuinely different amounts of context, so their errors are structurally different rather than correlated.

Where it does not work

The signal is a hallucination detector, not an adversarial detector, and the paper reports the negative result rather than omitting it:

Attack type	Detection AUC	Verdict
Negation insertion	0.71	moderate
Synonym swap	≈ 0.50	chance
Numeric flip	≈ 0.50	chance
Character drop / typo	≈ 0.50	chance

It catches negation attacks and essentially nothing else. Do not deploy it as an adversarial defence — that needs purpose-built detectors, and the paper says so explicitly.

Wiring it up

If you are already running a specialist and an LLM in parallel, the whole implementation is:

s_specialist = encoder.score(query)     # already computed for routing
s_llm        = reasoner.score(query)    # already computed for the answer

trust = 1.0 - abs(s_specialist - s_llm) / 2.0

if trust < threshold:
    flag_for_review(query)              # or fall back to the specialist

Threshold selection follows the same grid-search recipe as the routing thresholds: sweep on a held-out validation split and pick the point matching your tolerance for false flags.