jens/Rbm
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Decouple weight decay from momentum in Rbm::train

Browse Source 
inc_whv = momentum*inc_whv + lr*(dwhv/numcases - weightDecay*m_whv) folded
the weight-decay term into the momentum-accumulated velocity, so its
steady-state effective strength was amplified by roughly
weightDecay/(1-momentum) rather than the configured value -- e.g. ~2x at
momentum=0.5, ~10x at momentum=0.9. This is the same L2-regularization-
vs-momentum interaction that motivated decoupled weight decay (AdamW) in
the broader literature; pyRBM's state_adjust() already applies its
regularization terms directly to the weights outside the momentum
recursion, at full undiscounted strength every step.

Apply weightDecay directly to m_whv after the momentum-accumulated CD
update instead. weightDecay defaults to 0.0 in every .prj in this repo,
so this is currently inert in practice -- confirmed via unchanged
poet.elf output and unchanged test-suite results (9 passed, 1 known
issue, 2 failed) -- but anyone who sets it will now get the strength
they actually configured.

L1 regularization has no equivalent to fix: there's no l1Lambda field in
Rbm::Params at all, Status::L1 is a dead field always -1, and the GUI's
"Lambda" control is explicitly tooltipped "Unused" with an empty
change-handler stub -- it was scaffolded and never implemented.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016K8Gu7Qejd11JbdiHZqYAs
This commit is contained in:
jens
2026-07-27 15:21:51 +02:00
co-authored by Claude Sonnet 5
parent 5736541744
commit 0dd2b89cf0
1 changed files with 10 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
source/Rbm.cpp
+10 -1
View File
@@ -361,12 +361,21 @@ void Rbm::train(arma::mat const &batch, IListener* pListener)
// Adjust weight and biases
inc_bv = m_params.momentum*inc_bv + m_params.learningRate/numcases*dbv;
inc_bh = m_params.momentum*inc_bh + m_params.learningRate/numcases*dbh;
inc_whv = m_params.momentum*inc_whv + m_params.learningRate*(dwhv/numcases - m_params.weightDecay*m_whv);
inc_whv = m_params.momentum*inc_whv + m_params.learningRate*dwhv/numcases;
m_bv += inc_bv;
m_bh += inc_bh;
m_whv += inc_whv;
// Weight decay applied directly to the weights, outside the
// momentum recursion above -- folding it into inc_whv would let
// momentum geometrically amplify its effective strength to
// roughly weightDecay/(1-momentum), well past the configured value.
if (m_params.weightDecay != 0.0)
{
m_whv -= m_params.learningRate*m_params.weightDecay*m_whv;
}
progress += dProgress*miniBatchSizeActual;
status.progress = (int)(progress + 0.5);