ElephantBench

A closed-book knowledge probe

ElephantBench

Blind Men and the Elephant: Probing the Epistemic Myopia of LLMs under Long-Tail Divergent Knowledge

Can a language model remember a long-tail fact—and recover all of its different verified accounts?

Zhuoshi Pan1,2*, Junru Lu2*, Yan Qian3*, H. Vicky Zhao1†, Di Yin2, Xing Sun2†

1Tsinghua University  ·  2Tencent Youtu Lab  ·  3University of Warwick

* Equal contribution   † Corresponding authors

Code Dataset Source corpus
Benchmark at a glance Open full resolution ↗
ElephantBench construction, conflict example, and closed-book evaluation overview
4,127
conflict subgraphs
1,094
questions
22
knowledge fields
32
model configurations

The problem

Knowing one answer is not the same as knowing the whole fact.

Factual QA usually assumes one canonical answer. Long-tail web knowledge is different: evidence is sparse, sources may preserve incompatible accounts, and a model can recall the prevailing view while silently omitting another verified account.

01

Long-tail QA

Was the rare fact recalled?

Useful for retention, but usually scores against a single canonical answer.

02

ElephantBench

Do different accounts coexist in memory?

Sources are withheld. The model must recover every verified account from parametric memory.

03

Open-book conflict benchmarks

Can supplied evidence be reconciled?

Useful for reading and reasoning, but typically exposes the conflict at inference time.

Blindfolded people examining different parts of an elephant

Epistemic myopia

A response may be locally correct yet globally incomplete: the model remembers one documented account and presents it as the whole story.

Benchmark construction

From a low-exposure corpus to auditable knowledge probes.

A graph-based pipeline discovers naturally occurring disagreements, gathers support on both sides, and converts each verified conflict into a matched question pair.

  1. 1

    Discover

    Start from Dlow

    Mine the filtered remainder of a web corpus, where low-exposure facts are less saturated by repetition.

  2. 2

    Link

    Retrieve related documents

    Knowledge-point tags form precise local clusters; normalized entities recover related documents across clusters.

  3. 3

    Classify

    Build the document graph

    An LLM reads full document pairs and induces support and conflict edges for the same subject–attribute pair.

  4. 4

    Synthesize

    Sample subgraphs and generate QA

    Each sampled conflict edge is expanded with support neighbors. Its local subgraph yields one named-entity and one clue-based question.

  5. 5

    Verify

    Validate every account

    An independent LLM checks the full seed documents, a web agent seeks external evidence, and human reviewers conduct the final audit.

  6. 6

    Probe

    Ask without showing the sources

    Each conflict yields a named-entity question and a clue-based question with the same verified answer set.

Discovery graph
Hidden at evaluation time
Question only

What one item looks like

Two sources. Two accounts. One closed-book test.

Source documents establish the reference answers during construction, but are concealed from every evaluated model.

Source AIMDb

“Mother Teresa was born on August 26, 1910 … as Agnes Gonxha Bojaxhiu.”

≠
Source BPoem of Quotes

“Mother Teresa, Agnes Gonxha Bojaxhiu, was born in Skopje, Macedonia on August 27, 1910.”

Named-entity formulation

What birth date was reported for Mother Teresa?

Clue-based formulation

What birth date was reported for the Albanian-born Roman Catholic nun who founded the Missionaries of Charity and received the 1979 Nobel Peace Prize?

Verified answer set August 26, 1910 and August 27, 1910 sources withheld

Diagnostic scoring

Separate accessibility from completeness.

Instead of collapsing behavior into one accuracy number, ElephantBench partitions every response into complete, partial, or failed recall, with C + P + F = 1.

C

Complete recall

Every verified account is recovered without a material contradiction.

Higher is better
P

Partial recall

At least one, but not every, verified account is recovered.

Lower is better
F

Failed recall

No verified account is recovered, or the response materially contradicts the references.

Lower is better
K

Conditional completeness

K = C / (C + P) measures the share of complete recall among questions where at least one verified account is remembered.

Higher is better

Model leaderboard

Complete recall remains below 54% for every single model.

Main configurations reported in the paper, evaluated on all 1,094 questions. Higher C and K and lower P and F indicate better performance.

Sorted by complete recall

Rank Model Type C ↑ P ↓ F ↓ K ↑

Values are percentages. C/P/F denote complete, partial, and failed recall; K = C / (C + P) is conditional completeness.

Main findings

Stronger recall still leaves much of the elephant unseen.

Across 32 open-weight and proprietary model configurations, incomplete recall remains the dominant failure mode once a long-tail fact becomes accessible.

Strongest single model

Kimi-K3

52.4% complete
C 52.4 P 45.2 F
CompletePartialFailed

Nearly all remaining questions are answered with only part of the verified account set.

52.4%→81.2%

Models are complementary, but share a blind spot.

A greedy oracle over all configurations raises complete recall substantially, yet 18.8% of questions remain partial for every model.

Scale

Scaling improves recall, but incompleteness persists.

Larger models generally reduce complete failure, yet many questions still recover only part of the verified account set.

Reasoning

More deliberation is not a uniform remedy.

Reasoning improves complete recall for some frontier models, but can suppress the less salient account in smaller models.

Exposure asymmetry

The rare account determines whether memory is complete.

Exposure to the prevailing view helps a model remember the fact. Exposure to its less frequently reported counterpart is more closely associated with recalling the full account set.

Use ElephantBench

Data, evaluation, and construction in one repository.

Run target models closed-book, grade responses with the published rubric, compute C/P/F/K, or reconstruct the benchmark from the source corpus.

Evaluation code ↗ Benchmark data ↗ Source corpus ↗
Quick start
# Install the evaluation toolkit
git clone https://github.com/Tencent/ElephantBench.git
cd ElephantBench
python -m pip install -e .

# Query a model without sources or tools
elephantbench-run \
  --input data/elephantbench.jsonl \
  --output outputs/my-model.jsonl \
  --model my-model --resume

# Judge and compute C / P / F / K
elephantbench-judge --benchmark data/elephantbench.jsonl \
  --responses outputs/my-model.jsonl \
  --output outputs/my-model.judged.jsonl \
  --judge-model my-judge-model

elephantbench-score --benchmark data/elephantbench.jsonl \
  --results outputs/my-model.judged.jsonl

Citation

Build on the probe.

@misc{pan2026elephantbench,
  title  = {Blind Men and the Elephant: Probing the Epistemic
            Myopia of LLMs under Long-Tail Divergent Knowledge},
  author = {Pan, Zhuoshi and Lu, Junru and Qian, Yan and
            Zhao, H. Vicky and Yin, Di and Sun, Xing},
  year   = {2026},
  url    = {https://tencent.github.io/ElephantBench/}
}