inc_whv = momentum*inc_whv + lr*(dwhv/numcases - weightDecay*m_whv) folded
the weight-decay term into the momentum-accumulated velocity, so its
steady-state effective strength was amplified by roughly
weightDecay/(1-momentum) rather than the configured value -- e.g. ~2x at
momentum=0.5, ~10x at momentum=0.9. This is the same L2-regularization-
vs-momentum interaction that motivated decoupled weight decay (AdamW) in
the broader literature; pyRBM's state_adjust() already applies its
regularization terms directly to the weights outside the momentum
recursion, at full undiscounted strength every step.
Apply weightDecay directly to m_whv after the momentum-accumulated CD
update instead. weightDecay defaults to 0.0 in every .prj in this repo,
so this is currently inert in practice -- confirmed via unchanged
poet.elf output and unchanged test-suite results (9 passed, 1 known
issue, 2 failed) -- but anyone who sets it will now get the strength
they actually configured.
L1 regularization has no equivalent to fix: there's no l1Lambda field in
Rbm::Params at all, Status::L1 is a dead field always -1, and the GUI's
"Lambda" control is explicitly tooltipped "Unused" with an empty
change-handler stub -- it was scaffolded and never implemented.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016K8Gu7Qejd11JbdiHZqYAs
jensandClaude Sonnet 5
0dd2b89cf0