Cautious Weight Decay

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

• AdaGrad Meets Muon: Adaptive Stepsizes for Orthogonal Updates (2025)
• ANO : Faster is Better in Noisy Landscape (2025)
• REG: A Regularization Optimizer for Robust Training Dynamics (2025)
• Adaptive Memory Momentum via a Model-Based Framework for Deep Learning Optimization (2025)
• Conda: Column-Normalized Adam for Training Large Language Models Faster (2025)
• Gradient Shaping Beyond Clipping: A Functional Perspective on Update Magnitude Control (2025)
• Muon: Training and Trade-offs with Latent Attention and MoE (2025)

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend

One question, are you planning to release the code for reproduction?

Hi, thank you for the very interesting work on Cautious Weight Decay and its integration with LION-K and MUON. I have a question specifically about the MUON + CWD variant (Algorithm 6).

In Algorithm 6, the MUON update is based on a momentum-like matrix M_t (from the stochastic gradient G_t), and then you apply the Newton–Schulz iteration to obtain O_t. As I understand it, O_t is an approximation to the matrix sign (or a spectrally normalized / orthogonalized transform of M_t). Because of this, the entries of O_t, and especially the product O_t * X_t, are not necessarily coordinate-wise aligned with the original gradient G_t or the momentum M_t. In particular, the sign of (O_t * X_t)_ij does not have to match the sign of G_t,ij or M_t,ij.

In the MUON + CWD update, the cautious weight decay mask is defined using the condition “O_t * X_t >= 0” (elementwise). This raises a few questions:
1. Conceptually, is this mask meant to approximate the more intuitive condition “gradient and parameter have the same sign”, i.e. something like a mask based on G_t and X_t? Or is it intentionally defined purely in terms of the post–Newton–Schulz update direction (O_t * X_t), regardless of the raw gradient signs?
2. Have you measured in practice how often the signs of (O_t * X_t)_ij and G_t,ij (or M_t,ij) disagree? In other words, how frequently does MUON + CWD disable or enable weight decay on coordinates where the raw gradient would suggest the opposite?
3. Do you see any theoretical or practical drawbacks in defining a “grad-aware” variant of MUON + CWD (for example, using a mask based on G_t and X_t, or a hybrid mask that combines information from G_t and O_t * X_t)? Would that conflict with your analysis for LION-K / MUON, or do you mainly view the current mask definition as a pragmatic design choice?

I am trying to understand whether, in the MUON setting, the cautious condition should be interpreted as “not opposing the actual optimizer update geometry” (given by O_t), rather than “not opposing the raw gradient direction”, and whether the potential mismatch between O_t and G_t at the coordinate level matters in practice.