From 0dd2b89cf07be98492933c26ceb80b7d6cbe690f Mon Sep 17 00:00:00 2001 From: Jens Ahrensfeld Date: Mon, 27 Jul 2026 15:21:51 +0200 Subject: [PATCH] Decouple weight decay from momentum in Rbm::train inc_whv = momentum*inc_whv + lr*(dwhv/numcases - weightDecay*m_whv) folded the weight-decay term into the momentum-accumulated velocity, so its steady-state effective strength was amplified by roughly weightDecay/(1-momentum) rather than the configured value -- e.g. ~2x at momentum=0.5, ~10x at momentum=0.9. This is the same L2-regularization- vs-momentum interaction that motivated decoupled weight decay (AdamW) in the broader literature; pyRBM's state_adjust() already applies its regularization terms directly to the weights outside the momentum recursion, at full undiscounted strength every step. Apply weightDecay directly to m_whv after the momentum-accumulated CD update instead. weightDecay defaults to 0.0 in every .prj in this repo, so this is currently inert in practice -- confirmed via unchanged poet.elf output and unchanged test-suite results (9 passed, 1 known issue, 2 failed) -- but anyone who sets it will now get the strength they actually configured. L1 regularization has no equivalent to fix: there's no l1Lambda field in Rbm::Params at all, Status::L1 is a dead field always -1, and the GUI's "Lambda" control is explicitly tooltipped "Unused" with an empty change-handler stub -- it was scaffolded and never implemented. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_016K8Gu7Qejd11JbdiHZqYAs --- source/Rbm.cpp | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/source/Rbm.cpp b/source/Rbm.cpp index 7e5fe87..a8f566e 100644 --- a/source/Rbm.cpp +++ b/source/Rbm.cpp @@ -361,12 +361,21 @@ void Rbm::train(arma::mat const &batch, IListener* pListener) // Adjust weight and biases inc_bv = m_params.momentum*inc_bv + m_params.learningRate/numcases*dbv; inc_bh = m_params.momentum*inc_bh + m_params.learningRate/numcases*dbh; - inc_whv = m_params.momentum*inc_whv + m_params.learningRate*(dwhv/numcases - m_params.weightDecay*m_whv); + inc_whv = m_params.momentum*inc_whv + m_params.learningRate*dwhv/numcases; m_bv += inc_bv; m_bh += inc_bh; m_whv += inc_whv; + // Weight decay applied directly to the weights, outside the + // momentum recursion above -- folding it into inc_whv would let + // momentum geometrically amplify its effective strength to + // roughly weightDecay/(1-momentum), well past the configured value. + if (m_params.weightDecay != 0.0) + { + m_whv -= m_params.learningRate*m_params.weightDecay*m_whv; + } + progress += dProgress*miniBatchSizeActual; status.progress = (int)(progress + 0.5);