DIGITAL AI 研究实验 Research experiment

Reward Modeling Lab：8B 奖励模型训练与捷径审计 Reward Modeling Lab: 8B training and shortcut audits

在单张 A10 上完成 8B 奖励模型 QLoRA 训练，再用长度匹配、反长度挑战与排序评估检验高分来自哪里。 Train an 8B reward model on one A10, then test its gains with length-controlled challenges and ranking audits.

↓ 向下看：关键结果、架构、技术难点与实验记录 ↓ Scroll for results, architecture, challenges and experiments GitHub 源码GitHub source ↗

AT A GLANCE / Reward Modeling Lab

关键结果与实验条件Key results and experimental conditions

1,0001,000

单卡训练更新步数Single-GPU updates

NVIDIA A10 23GB 的 V1 冻结实验，约 3 小时 18 分钟。Frozen V1 experiment on one A10 23GB, about 3h18m.

91.35%91.35%

微调后成对判断准确率Pairwise accuracy after training

451 个独立问题、4510 对冻结测试；仅一次种子实验。451 held-out questions / 4,510 pairs in a single-seed evaluation.

94.61%94.61%

简单长度规则准确率Length-heuristic accuracy

“总选更长回答”规则高于原分布上的奖励模型分数，揭示潜在捷径。Always choosing the longer answer beats the model on the original distribution.

每项结果下方保留数据范围与实验条件；仿真、汇总快照与实际模型实验不可混同为业务收益。Scope and conditions are listed with each result. Simulations, aggregate snapshots and actual model tests are not business-impact claims.

SYSTEM DESIGN

系统架构与执行流程Architecture and workflow

以下是基于现有实现与研究流程归纳的模块关系，不是对所有模块已通过生产验证的声明。An overview derived from the documented implementation and research workflow, not a claim that every component has been production-validated.

01
偏好数据治理Preference data

问题级隔离、去重、分层Isolate questions, deduplicate and stratify
02
8B QLoRA 训练8B QLoRA training

单张 A10、4-bit NF4、1000 步One A10, NF4 4-bit, 1,000 updates
03
冻结测试Held-out evaluation

独立问题上的成对比较Pairwise judgments on held-out questions
04
偏差审计Shortcut audit

长度匹配与优质短答案挑战Length-matched and shorter-better challenges
05
排序与复盘Ranking & review

多候选排序、检查点比较Multi-candidate ranking and checkpoint review

ENGINEERING & RESEARCH DECISIONS

核心技术难点与解决取舍Technical challenges and tradeoffs

准确率不是最终答案A high score is not enough

“总选较长回答”的规则超过了微调模型原始准确率，因此主动补做困难测试。A pick-the-longer rule beat the model on the original test, motivating harder controls.

单卡可复现后训练Reproducible single-GPU post-training

基于冻结训练配置保留步数、硬件与评测工件；V2 GPU 仍未运行。Preserve frozen training configs, hardware and results; V2 GPU training has not run.

检查成对分数之外的能力Check beyond pairwise scores

引入 NDCG@5、长度控制及 Checkpoint 对照，明确结论适用范围。Use NDCG@5, length-controlled tests and checkpoint comparisons.

EVIDENCE & SCOPE

我的贡献、交付物与验证边界My contribution, deliverables and validation limits

我的贡献My contribution: 构建偏好数据与 QLoRA 训练流程，完成单卡训练、排序评测和长度偏差诊断。Built the preference-data and QLoRA workflow, completing single-GPU training, ranking evaluation, and length-bias analysis.
项目范围Scope: 8B 奖励模型的单卡 QLoRA 后训练，以及冻结测试、长度挑战、整体排序和检查点审计。Single-GPU QLoRA post-training of an 8B reward model, with frozen tests, length challenges, ranking evaluation, and checkpoint audits.
关键取舍Key decision: 将高准确率与长度启发式一起检查，用长度匹配和反长度挑战区分偏好能力与捷径依赖。Audit high accuracy against a length heuristic, using length-matched and reversed-length challenges to separate preference discrimination from shortcut reliance.
已完成的产物Available artifacts: 训练与评估代码、冻结配置、结果 JSON、排序和检查点对照，以及 V2 实验协议；原始偏好数据不随仓库公开。Training and evaluation code, frozen configuration, result JSON, ranking and checkpoint comparisons, and a V2 protocol. Raw preference data is not distributed.
验证与边界Validation & limits: V1 已在单张 A10 上完成 1,000 步训练与单种子评估；V2 已实现协议与代码，但 GPU 训练尚未执行，无 V2 性能结论。V1 completed 1,000 training steps and single-seed evaluation on one A10. V2 protocol and code exist, but GPU training has not run and no V2 performance is reported. 核查记录 ↗Evidence ↗

8B 奖励模型 QLoRA 训练与长度捷径审计的概念示意图。Concept diagram of 8B reward-model QLoRA training and length-shortcut audits.

DEEP DIVE

完整技术与实验记录Full technical & experimental record

以下保留原始案例中的实验口径、设计取舍、复核链接以及未完成的部分。The full account below retains methodology, design decisions, source links and unfinished work.

奖励模型需要给较好的回答更高分，但回答长度、格式和数据生成方式也可能与偏好标签相关。这个项目围绕 Skywork Reward Llama 3.1 8B，把训练与审计放在同一条实验链路中：完成后训练，检查冻结测试集，再用受控挑战和多候选排序分析模型学到了什么。

关键决策：高分需要通过捷径审计

冻结测试集包含 451 个问题、4,510 对比较。微调后准确率达到 91.35%，但“总选更长回答”的规则达到 94.61%。因此，原始分数需要结合长度控制实验一起解释。结果与挑战集口径

评测	基础模型	微调模型	样本与条件
原始冻结测试	50.42%	91.35%	4,510 对；长度启发式为 94.61%
长度匹配挑战	49.56%	77.78%	450 对；两回答相对长度差不超过 10%
反长度挑战	47.70%	74.90%	239 对；较优回答更短

模型在长度匹配和反长度样本上仍优于基础模型，原始分布的高分也明显受到长度相关性的影响。这促使后续评估同时追踪偏好区分能力和捷径依赖。

训练与实验设计

基础模型为 Skywork Reward Llama 3.1 8B。正式实验在单张 NVIDIA A10 23GB 上使用 4-bit NF4 QLoRA 与 BF16 计算，运行 1,000 个优化步骤，耗时约 3 小时 18 分钟。训练池包含 3,599 个问题、35,990 对偏好样本；这次固定步数实验实际处理约 8,000 个样本对实例，未遍历全部可用训练对。数据规模 · 冻结配置 · 实验记录

训练目标优化 chosen 与 rejected 回答的奖励差；LoRA 覆盖注意力和前馈层，并保存奖励 score head。评测同时观察成对准确率、五候选排序、长度相关性和检查点差异。训练实现

排序与检查点

同一批测试问题还被重构为五候选排序：记录的 Kendall τ 为 0.8828、NDCG@5 为 0.9474，完全正确排序比例为 57.87%。这些指标补充了成对准确率无法覆盖的整体顺序信息。排序记录

检查点对比中，最低验证损失与更好的下游排序指标没有完全一致。实验也记录到，512-token 限制下，2,255 条去重测试回答中约 98.54% 被截断。这两点都进入了下一轮实验设计。检查点对照 · 结果说明

当前进展

V1 是单随机种子的金融问答偏好实验，原始偏好数据不随公开仓库分发。V2 已实现长度平衡采样、反长度样本增曝和 1,024-token 上下文配置，并将原始分布、长度匹配与反长度挑战纳入同一晋级协议；真实 GPU 训练尚未执行，因此没有 V2 性能结论。V2 实验协议

查看源码与复现实验入口 · 完整结果工件

A reward model should give better answers higher scores, yet response length, format, and data-generation choices may correlate with preference labels. This project connects training and auditing around Skywork Reward Llama 3.1 8B: fine-tune the model, evaluate a frozen test set, then examine controlled challenges and multi-candidate ranking.

Key decision: audit the shortcut behind a high score

The frozen test set contains 451 questions and 4,510 comparisons. Fine-tuned accuracy reached 91.35%, while a rule that always chooses the longer response reached 94.61%. The original score therefore needs to be read alongside length-controlled evaluations. Results and challenge definitions

Evaluation	Base	Fine-tuned	Sample and condition
Original frozen test	50.42%	91.35%	4,510 pairs; longer-answer heuristic: 94.61%
Length-matched challenge	49.56%	77.78%	450 pairs; relative length difference at most 10%
Reversed-length challenge	47.70%	74.90%	239 pairs; the preferred answer is shorter

The fine-tuned model retains gains over the base model on both challenges, while the original high score is also affected by length correlation. This motivated an evaluation process that tracks preference discrimination and shortcut dependence together.

Training and experiment design

The base model is Skywork Reward Llama 3.1 8B. The formal run used 4-bit NF4 QLoRA with BF16 compute on one NVIDIA A10 23GB for 1,000 optimizer steps, taking approximately 3 hours 18 minutes. The training pool contained 3,599 questions and 35,990 preference pairs; the fixed-step run processed approximately 8,000 pair instances, rather than traversing the full pool. Data scope · Frozen configuration · Experiment record

The objective optimizes the reward difference between chosen and rejected responses. LoRA targets attention and feed-forward layers, and the reward score head is saved with the adapter. Evaluation covers pairwise accuracy, five-way ranking, length correlations, and checkpoint differences. Training implementation

Ranking and checkpoint selection

The same questions were reconstructed as five-candidate rankings. Recorded results include Kendall τ of 0.8828, NDCG@5 of 0.9474, and 57.87% perfectly ordered rankings. These measurements expose global ordering behavior beyond pairwise accuracy. Ranking record

The checkpoint with the lowest validation loss did not lead on every downstream ranking metric. A context audit also recorded truncation for approximately 98.54% of unique test responses at the 512-token limit. Both observations informed the next experiment. Checkpoint comparison · Result notes

Current progress

V1 is a single-seed financial-QA preference experiment; raw preference data is not distributed with the public repository. V2 implements length-balanced sampling, greater exposure to natural reversed-length pairs, and a 1,024-token configuration, with a joint promotion protocol for IID, length-matched, and reversed-length evaluations. Its GPU training has not run, so no V2 performance result is reported. V2 protocol

Source and experiment entry points · Result artifacts