RewardLens · 实验样例RewardLens · Experiment example
只改一处，答案该不该变？Change one detail. Should the answer change?
问题与候选答案始终不变。切换三张已保存的实验图，比较正确答案与模型当时的输出。The question and candidate answers stay fixed. Compare three saved experiment images, their expected answers, and the model’s recorded outputs.
固定问题：Fixed question: 立方体是什么颜色？What color is the cube?
固定候选：Fixed candidates: A · 蓝色（blue）blueB · 红色（red）red
原图Original
立方体是红色。先看模型能否回答原始问题。The cube is red. First, check the model’s answer to the original question.
- 预期答案Expected answer
- B · 红色red
- Molmo-7B-D 已保存的输出recorded output
- B · 红色red
与预期一致Matches the expected answer
改立方体Edit the cube
只把立方体从红色改成蓝色。正确答案应该随证据改变。Only the cube changes from red to blue. The correct answer should change with the evidence.
- 预期答案Expected answer
- A · 蓝色blue
- Molmo-7B-D 已保存的输出recorded output
- A · 蓝色blue
与预期一致Matches the expected answer
改旁边的球Edit the sphere
只把右侧球体从红色改成蓝色，立方体仍是红色。正确答案应该保持不变。Only the sphere on the right changes from red to blue; the cube stays red. The correct answer should stay the same.
- 预期答案Expected answer
- B · 红色red
- Molmo-7B-D 已保存的输出recorded output
- A · 蓝色blue
与预期不符Does not match the expected answer
这个例子里，模型跟随了立方体的颜色变化，却也被与问题无关的球体颜色变化带偏。RewardLens 分别检查这两种行为。In this example, the model follows the cube’s color change, but also changes its answer when only an irrelevant sphere changes. RewardLens checks these two behaviors separately.
一个已保存的仿真样例 · 离线输出，不代表总体准确率。One saved simulation example · Offline outputs, not overall accuracy.
来源与适用范围Sources and scope
这是一个已保存的仿真实验样例及离线模型输出，不是实时推理，也不代表模型的总体准确率。模型接收的原始问题与候选均为英文；中文是页面释义。These are saved simulation images and offline model outputs, not live inference or an estimate of overall model accuracy. The original question and candidates were in English; Chinese text is a page translation.
样例 attr_000000；三幅图由公开附录拼图按原像素提取，与同一条目编号的标签及输出对应。Example attr_000000. The three images are exact pixel crops from the public appendix montage, matched to labels and outputs with the same item IDs.
两个多模态评判模型取得相同的静态偏好准确率，是否意味着它们都会正确响应视觉证据的变化？RewardLens 用独立静态评测和受控图像干预，测量这个分数背后仍可能存在的行为差异。论文
研究状态（作者更新于 2026 年 10 月）：第一作者；已投稿 ICLR 2027，审稿中。 OpenReview 投稿页面
这里关注的是评估的行为分辨率：当一个分数不能区分两个模型时，受控干预能否揭示它们使用视觉证据的差异。以下结果保留公开仓库的固定版本与具体评测口径。
关键决策：固定语言输入，改变视觉证据
每组审计样本包含原图、相关编辑图和无关编辑图。问题、候选回答及候选顺序保持固定；相关编辑被构造成改变正确选项，无关编辑则保持正确选项。程序在渲染后重新检查场景事实与标签关系。构造与验证
在模型原图判断正确的子集上，分别测量：
- 相关适应 RA：相关视觉事实改变后，能否选对新的答案。
- 无关不变 II：无关视觉事实改变后，能否继续保持正确判断。
这两个量使用各模型自身的原图正确子集作为分母；独立静态准确率在另一组样本上测量。指标定义
实验覆盖 8 个多模态评判模型、4 类视觉因素：数量、属性、存在性和空间关系。独立静态集包含 800 个条目；CLEVR 风格审计包含 800 组三元组、2,400 张渲染图像。实验设置
一个直观结果
在 Attribute 因素上，两模型的实测静态准确率均为 80.5%，但相关编辑后的表现不同：结果表与分析
| 模型 | 独立静态准确率 | 相关适应 RA |
|---|---|---|
| Phi-3.5-Vision | 80.5% | 84.95% |
| LLaVA-OneVision-7B | 80.5% | 100.00% |
RA 相差 15.05 个百分点，配对 bootstrap 的 95% 区间为 10.21–20.43 个百分点。按同一因素内静态分数差不超过 1 个百分点的规则，全部 7 组比较的 RA 差异中位数同样为 15.05 个百分点。匹配比较数据
另一例来自 Qwen3-VL-4B 的 Count 审计：200 个原图正确案例中，166 个在相关编辑后仍保留原选择，因正确选项已改变而变成错误。这说明在该分支中，保持不变也可能是失败。对应结果
审计原图都答对之后，差异仍然存在
另一种对照直接看同一审计的原图。Spatial（空间关系）因素上，以下五个模型均答对全部 200 张原图，但相关编辑后的 RA 从 55.0% 到 98.5%，相差 43.5 个百分点。原始指标表
| 模型 | 空间关系审计原图准确率 | 相关适应 RA |
|---|---|---|
| Qwen3-VL-4B | 100% | 98.5% |
| Skywork-VL-Reward-7B | 100% | 86.0% |
| Idefics3-8B | 100% | 55.0% |
| LLaVA-OneVision-7B | 100% | 87.0% |
| InternVL3-8B | 100% | 91.5% |
这里的 100% 是审计原图准确率，与前节单独采样的独立静态准确率不同。Count 因素也呈现类似分离：Qwen3-VL-4B 与 Skywork-VL-Reward-7B 的审计原图准确率分别为 100% 和 99.5%，RA 分别为 17.0% 和 62.81%，差 45.81 个百分点。这些差值由同一公开指标表复算，不能替代独立静态集上的匹配比较。
这项工作带来的判断
当应用依赖模型随证据变化而调整判断时，静态偏好分数之外还需要检查具体干预行为。RewardLens 为评估增加了这个维度，并保留完整匹配比较、指标定义和结果工件，便于追溯结论。
当前结果限于四类受控因素和冻结的候选顺序。静态分数相等指有限样本中的实测相等；研究没有由此推断模型内部推理，也没有验证部署系统的实际安全性。讨论与限制
查看研究仓库 · 阅读论文 · OpenReview
If two multimodal judges have the same static preference accuracy, will they respond equally well when visual evidence changes? RewardLens combines an independent static evaluation with controlled image interventions to measure behavioral differences that a shared score can leave unresolved. Paper
Research status (author update, October 2026): first author; submitted to ICLR 2027, under review. OpenReview submission
The question is one of behavioral resolution: when a score cannot distinguish two judges, can controlled interventions reveal how differently they respond to visual evidence? The results below retain the public repository’s pinned version and explicit evaluation definitions.
Key decision: fix the language inputs and edit visual evidence
Each audit triplet contains a base image, a relevant edit, and an irrelevant edit. The question, candidate responses, and candidate order stay fixed. Relevant edits are constructed to change the correct choice; irrelevant edits preserve it. Post-render programmatic checks verify the scene facts and label relationships. Construction and validation
Among cases answered correctly on the base image, the audit measures:
- Relevant Adaptation (RA): correctness after a relevant visual fact changes.
- Irrelevant Invariance (II): correctness after an irrelevant visual fact changes.
Both use each judge’s own base-correct subset as the denominator. Independent static accuracy is measured on a separate set. Metric definitions
The experiment covers eight multimodal judges and four visual factors: Count, Attribute, Presence, and Spatial. The independent static set has 800 items; the CLEVR-style audit has 800 triplets and 2,400 rendered images. Experimental setup
A concrete result
On Attribute, two judges both achieve 80.5% measured static accuracy, yet differ after relevant edits. Results and analysis
| Judge | Independent static accuracy | Relevant Adaptation |
|---|---|---|
| Phi-3.5-Vision | 80.5% | 84.95% |
| LLaVA-OneVision-7B | 80.5% | 100.00% |
The RA gap is 15.05 percentage points, with a paired-bootstrap 95% interval of 10.21–20.43 points. Across all seven within-factor comparisons whose measured static scores differ by at most one percentage point, the median RA gap is also 15.05 points. Matched comparison data
Qwen3-VL-4B’s Count audit provides another example: in 166 of 200 base-correct cases, it retained its original choice after a relevant edit changed the correct answer. Prediction stability was a failure in that branch. Corresponding result
Perfect audit-base accuracy still leaves behavioral differences
A separate comparison uses the original images from the intervention audit itself. On Spatial, the following five judges answer all 200 base images correctly, yet their RA ranges from 55.0% to 98.5%: a 43.5 percentage-point gap. Recorded metric table
| Judge | Spatial audit-base accuracy | Relevant Adaptation |
|---|---|---|
| Qwen3-VL-4B | 100% | 98.5% |
| Skywork-VL-Reward-7B | 100% | 86.0% |
| Idefics3-8B | 100% | 55.0% |
| LLaVA-OneVision-7B | 100% | 87.0% |
| InternVL3-8B | 100% | 91.5% |
The 100% figure is audit-base accuracy, distinct from the separately sampled independent static accuracy used above. Count provides another separation: Qwen3-VL-4B and Skywork-VL-Reward-7B have audit-base accuracies of 100% and 99.5%, with RA of 17.0% and 62.81% respectively, a 45.81 percentage-point gap. These gaps are recalculated from the same public metric table and do not replace the independent-static matched comparisons.
What the work adds
When an application depends on a judge adapting to changing evidence, static preference accuracy can be complemented with direct intervention measurements. RewardLens supplies this additional axis and preserves the complete matched comparison set, metric definitions, and result artifacts.
The findings apply to four controlled factors and a frozen candidate order. An equal static score means equal observed accuracy on a finite sample. The study does not identify internal reasoning or establish deployed-system safety. Discussion and limitations