SciValidityBench

A benchmark for evaluating whether scientific agents use remembered evidence with the right validity state.

Yuexi Pan1,2·Yongchao Chen1,*

1 Tsinghua University   2 University of Science and Technology of China

* Corresponding author

Paper Code Video Benchmark

Watch the benchmark

See the distinction in motion.

01 / The idea

A fact can be true.
Its use can still be wrong.

1

Correct record

Retrieve the right result.

Where retrieval stopsMemory correctness is local.
2

Governing state

Recover the conditions that give it meaning.

3

Valid decision

Use evidence only when its state permits.

What SciValidityBench testsScientific validity is global.

02 / The benchmark

One executable contract.
Five ways to remember.

Five memory views. One state-aware contract.

Validity modules

Live contract preview
TaskSpec / seed 7
SCIENTIFIC STATE

Inferential Ledger

Outcome selection and stopping rules determine which claim a reported result can support.

state-awareoracle-scoredcounterfactual
factp = 0.03
governing stateplanned / selected
legal actionconfirm / explore
# Inferential Ledger — schematic example Reported result: p = 0.03 The outcome-selection history and stopping rule determine which scientific claim is licensed.
Can this result support a confirmatory claim? Compare the same result under the two histories in the State Difference tab.
State APlan fixed in advance
  • Outcome preregistered
  • Fixed stopping rule
  • No selection after inspection
Confirmatory use
State BOutcome selected after inspection
  • Twenty results inspected
  • Outcome selected afterward
  • Same reported p-value
Exploratory use only
Five evaluated memory views
Raw EventsBM25 EventsEvent GraphFlat SSGTyped SVL

03 / Evidence

Expose the governing state.
Improve the decision.

Four model families. Ten unseen seeds. One consistent gain.

+9.2–18.8pp

Compiled state beats a generic Event Graph

Every model family shows a positive TaskSpec-balanced gain. Retrieval and generic organization do not reproduce the same cross-family pattern.

4 / 4model families
positive intervals
Figure 01Same fact, different scientific use↗
Two histories share a reported result but lead to different scientific uses
Retrieval alone cannot distinguish a planned analysis from a result selected after inspection.
Figure 02SciStateForge pipeline↗
SciStateForge compiler pipeline
One contract connects generated histories, compiled state and scoring.
Figure 03Main effect and replications↗
Main effect and independent replications
Primary gains remain positive across four model families.
Figure 04State use diagnosis↗
State use and failure diagnosis
Module effects and repairs reveal reader-specific boundaries.

04 / Reference implementation

Start with one world.
See what the reader sees.

Deterministic generators, five views and strict scoring.

quick start
git clone https://github.com/SciValidityBench/SciValidityBench.git
cd SciValidityBench
python -m pip install -e ".[dev]"
pytest -q

Runnable by design

Compile one TaskSpec, inspect every memory view, then exercise counterfactual pairing and repair.

View the reference code ↗
Citation

Cite the preprint when you build on it.

@misc{pan2026rightfactwrongdecision,
  title={The Right Fact, the Wrong Decision:
         SciValidityBench for Scientific Agent Memory},
  author={Yuexi Pan and Yongchao Chen},
  year={2026},
  eprint={2610.34767},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  url={https://arxiv.org/abs/2610.34767}
}