DIGITAL AI RL 工程原型 RL engineering prototype

Agentic-RL：面向 Text-to-SQL 的可验证 GRPO 训练链路 Agentic-RL: a verifiable GRPO workflow for Text-to-SQL

实现无答案泄漏的 SQL 环境、Agent Lightning／veRL GRPO 接口，以及权重变化与固定预算对照验收。 Implement a Gold-isolated SQL environment, Agent Lightning/veRL GRPO integration, and checks for weight changes and fixed-budget controls.

↓ 向下看：关键结果、架构、技术难点与实验记录 ↓ Scroll for results, architecture, challenges and experiments GitHub 源码GitHub source ↗

AT A GLANCE / Agentic-RL

关键结果与实验条件Key results and experimental conditions

CPU CICPU CI

逻辑与契约验证Logic and contract verification

SQL 环境、泄漏审计和上游框架契约已有固定 CI。SQL runner, leakage checks and upstream-framework contracts have CI evidence.

GRPOGRPO

策略训练接口Policy-training integration

Agent Lightning 与 veRL 的框架接入代码已存在。Agent Lightning and veRL integration code is present.

待验证Pending

真实 GPU 策略收益Actual GPU policy gain

无可声明的训练前后准确率改善；应完成更新和盲对照后报告。No verified pre/post-training gain; a blind controlled GPU run is needed.

每项结果下方保留数据范围与实验条件；仿真、汇总快照与实际模型实验不可混同为业务收益。Scope and conditions are listed with each result. Simulations, aggregate snapshots and actual model tests are not business-impact claims.

SYSTEM DESIGN

系统架构与执行流程Architecture and workflow

以下是基于现有实现与研究流程归纳的模块关系，不是对所有模块已通过生产验证的声明。An overview derived from the documented implementation and research workflow, not a claim that every component has been production-validated.

01
盲任务输入Blind task input

问题、Schema 与受控观察Question, schema and bounded context
02
策略生成 SQLPolicy rollout

单次或有限轮查询提议Generate SQL under a fixed attempt budget
03
只读执行反馈Read-only execution

结构化错误、结果与安全限制Structured errors and safe results
04
离线计算奖励Post-trajectory reward

在轨迹完成后才访问 GoldGold answer only after the trajectory
05
GRPO 与对照GRPO & controls

No-update、参数变化与同预算对照No-update, tensor deltas and equal-budget controls

ENGINEERING & RESEARCH DECISIONS

核心技术难点与解决取舍Technical challenges and tradeoffs

防止 Gold 答案泄漏Isolate Gold answers

策略运行时只允许可见任务类型，标准答案独立保存在评分器。The rollout accepts only public task types; Gold stays in the evaluator.

证明权重更新不是空跑Prove updates changed weights

计划比较 Base、GRPO、No-update 张量差异，并审计恢复后的权重身份。Prepare Base, GRPO and No-update tensor checks and reloaded checkpoint identities.

同预算比较，才有学习结论Equal budgets for causal attribution

固定上下文和执行次数；当前 GPU 训练及对照尚未运行。Hold context and execution attempts fixed; GPU updates and comparisons are pending.

EVIDENCE & SCOPE

我的贡献、交付物与验证边界My contribution, deliverables and validation limits

我的贡献My contribution: 实现 SQL 执行环境、Gold 与策略隔离、GRPO 训练接口及权重／对照证据检查。Implemented the SQL environment, Gold/policy isolation, GRPO training integration, and weight/control evidence checks.
项目范围Scope: Qwen2.5-Coder-3B 的 Text-to-SQL RLVR／GRPO 工程链路，覆盖 Spider 与 BIRD 数据及盲评测入口。A Text-to-SQL RLVR/GRPO engineering workflow for Qwen2.5-Coder-3B, with Spider/BIRD data and blind-evaluation entry points.
关键取舍Key decision: 轨迹结束后才使用 Gold 评分；在固定观察与预算下设置 No-update 和 reward 消融对照。Use Gold for scoring only after a trajectory ends; compare No-update and reward ablations under fixed observations and budgets.
已完成的产物Available artifacts: 只读 SQLite 环境、真实框架配置接口、Checkpoint 导出、张量变化检查、配对统计及证据门禁。A read-only SQLite environment, upstream-framework configuration integration, checkpoint export, tensor-delta checks, paired statistics, and an evidence gate.
验证与边界Validation & limits: CPU CI、框架契约和泄漏审计通过；真实 Qwen GPU 更新、重载及 Base／GRPO 对照评测待完成。CPU CI, framework-contract checks, and leakage audits pass. Real Qwen GPU updates, reload, and Base/GRPO evaluation remain pending. 核查记录 ↗Evidence ↗

DEEP DIVE

完整技术与实验记录Full technical & experimental record

以下保留原始案例中的实验口径、设计取舍、复核链接以及未完成的部分。The full account below retains methodology, design decisions, source links and unfinished work.

一个 SQL Agent 多试几次、看到更多上下文，成功率也可能上升。真正需要回答的是：在相同观察信息、轮次和计算预算下，强化学习是否让策略本身作出了更好的决策？

Agentic-RL 围绕这个问题实现可审计的 Text-to-SQL 训练与评测链路：以 Qwen2.5-Coder-3B-Instruct 为目标策略，连接 SQLite 执行环境、Agent Lightning 和 veRL GRPO，并把原始轨迹、模型身份、权重变化和统计对照纳入验收。项目说明

首先保证 Agent 没有偷看答案

我把带标准 SQL 的私有任务与模型可见任务拆成不同类型。策略只接收问题、Schema、允许的额外证据与执行反馈；运行器拒绝直接接收包含 Gold SQL 的私有任务。标准答案在整条轨迹结束后才进入独立评估器，计算奖励与测试指标。类型隔离

这也约束了多轮纠错：是否继续、是否检查、何时结束，不能由 Gold 是否匹配来决定。模型输出 final 或 inspect，环境提供真实 SQLite 结果或错误。交换 Gold SQL 的变形测试要求策略输入、SQL、轮次与终止行为不变，评分则可以改变。策略运行器 · 泄漏审计

从执行奖励到可追溯的策略更新

已实现的链路包括：

执行环境：SQL 安全解析、只读查询、超时、结果预览和可观察错误反馈。
训练入口：Agent Lightning 记录模型请求与奖励，veRL 执行 GRPO；配置覆盖 FSDP、梯度检查点、vLLM rollout 和上下文预算。
工件记录：保存数据哈希、训练配置、步级指标、GPU 记录和 Checkpoint 清单，并提供 FSDP 到 Hugging Face 权重的导出工具。
权重验收：直接比较 Base、GRPO 和 No-update 的张量，要求 GRPO 权重发生变化、No-update 权重保持不变。

训练入口 · 权重变化检查

这些工具把“命令返回成功”和“训练确实发生”分开。报告门禁要求真实优化步骤、权重、原始评测轨迹及对照记录齐备，缺项时停止发布实验结论。证据校验器

关键取舍：把学习收益与推理预算分开

实验矩阵分别控制上下文长度、单轮／多轮、可选检查器，并设置 No-update（学习率为 0） 与 validity-only reward 对照。比较时锁定任务 ID、数据哈希、Prompt、轮次、Token 和 SQL 执行预算，提供配对 bootstrap 区间与 McNemar 检验。评测协议

数据协议同样区分三条用途：从 Spider 官方训练集按数据库划分训练与内部验证，把官方 Dev 留作最终测试；不可评分的 Gold SQL 单独审计并披露覆盖率。BIRD 作为外部分布评测入口，不能与 Spider 混报一个准确率。

当前已经验证到哪里

最新公开 CI 已通过 CPU 单元与集成测试、实际 Agent Lightning 1.0.2／veRL 0.8.0 配置契约验证、Gold 泄漏审计及离线就绪检查。固定提交 CI

CPU 测试还验证了一个小型线性策略在真实 AdamW 步骤后参数发生变化，证明本地 GRPO 目标函数可以驱动梯度更新。它验证的是算法实现，不是 Qwen 的 GPU 训练结果。优化器测试

项目当前处于工程链路已实现、GPU 实验待验证的阶段。下一步是完成真实 Qwen GRPO 更新、导出重载，以及固定预算下的 Base／GRPO／No-update 盲评测。仓库将既有准确率表明确标为参考实验快照，本案例不把它当作已完成的训练收益。当前执行计划 · 参考表的来源说明

这项工作的价值在于让一次 Agent 强化学习实验能够回答三个具体问题：模型看到什么、参数是否真的更新、收益是否来自学习本身。它为后续 GPU 运行准备了可执行的验证路径。

查看项目源码

A SQL agent may solve more tasks simply by receiving more context or making more attempts. The harder question is: with the same observations, turns, and compute budget, does reinforcement learning improve the policy’s decisions?

Agentic-RL implements an auditable Text-to-SQL training and evaluation workflow around that question. It targets Qwen2.5-Coder-3B-Instruct, connects SQLite execution to Agent Lightning and veRL GRPO, and treats raw trajectories, model identities, weight changes, and statistical controls as part of experimental acceptance. Project overview

Keep the answer out of the policy

I separated private tasks containing Gold SQL from the public task type passed to the model. The policy receives the question, schema, permitted evidence, and execution feedback; the runner rejects the private task type. Only after the full trajectory ends does an independent evaluator use Gold SQL to calculate reward and test metrics. Type boundary

This also constrains correction and stopping. Gold correctness cannot decide whether to retry, invoke a checker, or stop. The model chooses final or inspect, while the environment returns actual SQLite results or errors. A Gold-mutation audit requires policy inputs, SQL, turns, and stopping behavior to remain unchanged when the answer is swapped; scores may change. Policy runner · Leakage audit

Connect execution rewards to traceable updates

The implemented workflow includes:

Execution environment: SQL safety parsing, read-only queries, timeouts, result previews, and observable error feedback.
Training integration: Agent Lightning records model requests and rewards; veRL performs GRPO, with configuration for FSDP, gradient checkpointing, vLLM rollouts, and context budgets.
Run artifacts: dataset hashes, training configuration, step metrics, GPU telemetry, and checkpoint manifests, plus tools to export FSDP weights into Hugging Face format.
Weight checks: direct tensor comparisons across Base, GRPO, and No-update, requiring a changed GRPO policy and an unchanged No-update control.

Training entry point · Weight-change audit

The evidence gate separates a command returning successfully from training having actually occurred. Measured optimization steps, weights, evaluation trajectories, and controls must be present before a report can claim experimental completion. Evidence verifier

Separate learning gains from inference budget

The experiment matrix controls context length, single versus multiple turns, and an optional checker. It includes No-update (zero learning rate) and validity-only reward controls. Comparisons lock task IDs, data hashes, prompts, turns, token limits, and SQL execution budgets, with paired-bootstrap intervals and McNemar tests. Evaluation protocol

Spider’s official training set is split by database into training and internal validation; official Dev is reserved for final testing. Unscorable Gold SQL is audited separately with coverage disclosed. BIRD is an external-distribution evaluation path and must not be mixed into a single Spider score.

What has been verified

The latest public CI run passed CPU unit and integration tests, configuration-contract checks against Agent Lightning 1.0.2 and veRL 0.8.0, Gold-leakage auditing, and an offline-readiness gate. CI for the pinned commit

A CPU test also verifies that a small linear policy changes after a real AdamW step using the local GRPO objective. This validates the implementation’s gradient path; it is not evidence of GPU training or benchmark gains for Qwen. Optimizer test

The current status is an implemented engineering workflow awaiting GPU experimental validation. The next milestone is a real Qwen GRPO update, export and reload, followed by blind Base/GRPO/No-update evaluation under fixed budgets. The repository explicitly labels its existing accuracy table as a reference experiment snapshot; this case does not present it as a completed training gain. Current work plan · Reference-table provenance

The contribution is an executable path for answering three concrete questions about agent RL: what the model observed, whether its parameters changed, and whether any improvement came from learning. The GPU experiments can then be evaluated against that protocol.

Project source