Environment friendly Algorithms for Empirical Group Distributional Strong Optimization and Past
Authors: Dingzhi Yu, Yunuo Cai, Wei Jiang, Lijun Zhang
Summary: We examine the empirical counterpart of group distributionally sturdy optimization (GDRO), which goals to attenuate the maximal empirical danger throughout m distinct teams. We formulate empirical GDRO as a two-level finite-sum convex-concave minimax optimization drawback and develop a stochastic variance diminished mirror prox algorithm. Not like current strategies, we assemble the stochastic gradient by per-group sampling approach and carry out variance discount for all teams, which absolutely exploits the two-level finite-sum construction of empirical GDRO. Moreover, we compute the snapshot and mirror snapshot level by a one-index-shifted weighted common, which distinguishes us from the naive ergodic common. Our algorithm additionally helps non-constant studying charges, which is totally different from current literature. We set up convergence ensures each in expectation and with excessive likelihood, demonstrating a complexity of O(mn¯lnm√ε), the place n¯ is the common variety of samples amongst m teams. Remarkably, our method outperforms the state-of-the-art methodology by an element of m−−√. Moreover, we prolong our methodology to cope with the empirical minimax extra danger optimization (MERO) drawback and handle to present the expectation sure and the excessive likelihood sure, accordingly. The complexity of our empirical MERO algorithm matches that of empirical GDRO at O(mn¯lnm√ε), considerably surpassing the bounds of current strategies
