Relative Value Learning

In reinforcement learning (RL), critics traditionally learn absolute state values, estimating how good a particular situation is in isolation. Adding any constant to V(s) leaves action preferences unchanged; thus only value differences are relevant for decision making. Motivated by this fact, we propose Relative Value Learning (RV), a framework that learns antisymmetric value differences directly. We define a pairwise Bellman operator with a unique fixed point equal to true value differences, derive well-posed return targets and reconstruct generalized advantage estimation (R-GAE), resulting in an unbiased policy-gradient estimator. Empirically, integrating RV with PPO gives competitive performance on the Atari benchmark compared to standard PPO, indicating that learning relative value differences is a viable alternative to absolute critics.