We consider the problem of policy optimization with noisy rewards, where the policy is constrained to be in the neighborhood of a reference policy. We study the risk of the policy optimization process, defined to be the chance of obtaining a policy worse off than the reference policy. In an idealized multivariate Gaussian model for the noisy reward function, we show that the regularizer achieving optimal risk in a suitably local neighborhood of the reference policy is the quadratic norm weighted by the reward covariance matrix. This result is used to develop a new policy optimization algorithm based on regularization via reward variance estimates. Experiments across synthetic bandit models, large language models and reward models show that this algorithm yields more robust alignment outcomes than existing approaches.
Debangshu Banerjee is a Research Scientist at HP Inc., where he has been working since 2025. He completed his B.Tech. from Jadavpur University (2012–2016) and M.Tech. from the Indian Institute of Science (IISc) (2016–2018), under the guidance of Prof. Shalabh Bhatnagar. He completed his Ph.D. at IISc (2019–2025) under the guidance of Prof. Aditya Gopalan. Prior to joining HP Inc., he worked as an AI Engineer at AlphaIC (2018–2019). He has also held research internships with IBM Research in 2023 and Google DeepMind during 2024–2025.