Unsupervised Intrinsic Rewards on Vision-OPD

Code for studying three label-free intrinsic-reward RL methods — TTRL (majority-vote self-labeling), Intuitor (self-certainty), RENT (negative entropy) — as replacements for the self-distillation objective of Vision-OPD, on Qwen3.5-4B/9B with the Vision-OPD-6K dataset. Built on the upstream Vision-OPD codebase (verl).

Finding: none of the three improves the model. All six runs (3 methods × 2 sizes) peak at step 5–20 of a 65-step epoch and degrade monotonically afterwards. TTRL degrades an order of magnitude slower than the continuous rewards, and a KL penalty (β=0.01) does not prevent collapse.

Mean over V*Bench, ZoomBench, HR-Bench 4K/8K, MME-RealWorld(-CN); selection split; random ≈ 22–25. Full tables: docs/RESULTS_select.md.

• Best checkpoints (ready to eval): q1716523669/visionopd-best-ckpts
• Full archive (13 ckpts/run incl. optimizer, train/eval logs, raw eval outputs): q1716523669/visionopd-archive

verl/            upstream Vision-OPD (verl) + our patch — see RECORD.md for the 7-file diff
verl_ttrl/       TTRL reward manager, intrinsic-reward run scripts, data prep (out-of-tree package)
jobs/            the exact training / evaluation job scripts used for all runs
eval/            evaluation kit: vLLM inference → rule grading → LLM judge → accuracy
env/             environment entry script, vLLM patch, snapshot restore
docs/            experiment record, results, root-cause / incident reports
Dockerfile       the environment as code (CUDA 12.8, torch 2.10, vLLM 0.18, flash-attn from source)
RECORD.md        what we changed, what broke, and how it was fixed

Docker (recommended):

docker build -t vision-odp .

Or conda: create a Python 3.12 env and follow the pinned install order in the Dockerfile (torch 2.10.0+cu128 → flash-attn 2.8.3 from source → vLLM 0.18.0 → pip install -e .). The order matters; see RECORD.md § environment pitfalls.

Two runtime settings are mandatory (baked into env/env-verlopd.sh and the Dockerfile):

# cuBLAS/cuBLASLt must come from the same wheel set, or every bf16 GEMM fails
export LD_LIBRARY_PATH="$PY_SITE/nvidia/cublas/lib:$PY_SITE/nvidia/cuda_runtime/lib:$PY_SITE/nvidia/cudnn/lib:$LD_LIBRARY_PATH"
# vLLM 0.18 symmetric-memory allreduce is incompatible with torch 2.10
export VLLM_DISABLE_CUSTOM_ALL_REDUCE=1
export VLLM_ALLREDUCE_USE_SYMM_MEM=0

python verl_ttrl/make_verl_parquet.py   # Vision-OPD-6K → verl parquet, uses clean original_images

We train on the clean, unannotated images (original_images column) — not the red-box images nor the cropped teacher_images that Vision-OPD's privileged teacher sees. Rationale in verl_ttrl/README.md.

One entry script per method family, model size passed as an argument:

bash verl_ttrl/run_ttrl_qwen35_vopdhp.sh      4b   # TTRL     (majority-vote reward manager)
bash verl_ttrl/run_intrinsic_qwen35_vopdhp.sh 4b entropy         # RENT
bash verl_ttrl/run_intrinsic_qwen35_vopdhp.sh 9b self_certainty  # Intuitor

*_vopdhp.sh = Vision-OPD's original hyper-parameters (no KL); run_intrinsic_qwen35.sh / run_ttrl_qwen35_{4b,9b}.sh = each method's own published config (KL β=0.01). The jobs/ directory contains the wrapped, watchdog-supervised versions actually used on the cluster.

bash eval/orchestrate_mmeval.sh   # merge FSDP ckpt → vLLM (T=0, seed 0) → grade → judge → scores

Grading is layered: answer extraction → mathruler → first-letter match → Qwen2.5-72B-Instruct judge only for the remainder. Benchmark JSONs are in eval/; images are rebuilt with eval/prepare_data.py. Checkpoint selection uses a fixed 10% split (eval/selection_manifest.json, seed 20260802), disjoint from the reporting split.

Code builds on Vision-OPD and verl. Methods reproduced: TTRL (arXiv:2504.16084), Intuitor (arXiv:2505.19590), RENT (arXiv:2505.22660).