jens/Rbm
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Implement L1 regularization

Browse Source 
Rbm::Params had no l1Lambda field at all -- L1 didn't exist anywhere in
the C++ codebase. Status::L1 was a dead field always -1 (never assigned
outside its constructor), and the GUI's "Lambda" control was explicitly
tooltipped "Unused" with an empty change-handler stub. Scaffolded, never
implemented.

Add Params::l1Lambda (default 0.0, inert unless set) with full toJson/
fromJson round-trip. Apply its subgradient (lambda*sign(W), matching
pyRBM's l1_lambda) directly to m_whv in Rbm::train, in the same place
and same decoupled-from-momentum manner as the weightDecay fix from the
previous commit -- folding it into inc_whv's momentum recursion would
cause the same momentum-amplification bug. Doesn't touch the biases,
matching pyRBM's state_adjust().

Also wire up Status::L1 to actually report something (the current L1
norm of the weights, sum(abs(W))) instead of permanently printing -1,
computed alongside status.err/err_total.

Inert by default (l1Lambda=0.0 in every existing .prj): confirmed via
unchanged poet.elf output and unchanged test-suite results (9 passed,
1 known issue, 2 failed).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016K8Gu7Qejd11JbdiHZqYAs
This commit is contained in:
jens
2026-07-27 15:31:59 +02:00
co-authored by Claude Sonnet 5
parent 0dd2b89cf0
commit cb20060732
2 changed files with 16 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
source/Rbm.cpp
+12 -4
View File
@@ -367,14 +367,20 @@ void Rbm::train(arma::mat const &batch, IListener* pListener)
m_bh += inc_bh;
m_whv += inc_whv;
// Weight decay applied directly to the weights, outside the
// momentum recursion above -- folding it into inc_whv would let
// momentum geometrically amplify its effective strength to
// roughly weightDecay/(1-momentum), well past the configured value.
// Weight decay and L1 applied directly to the weights, outside the
// momentum recursion above -- folding either into inc_whv would let
// momentum geometrically amplify its effective strength to roughly
// lambda/(1-momentum), well past the configured value. Neither
// touches the biases, matching pyRBM's state_adjust().
if (m_params.weightDecay != 0.0)
{
m_whv -= m_params.learningRate*m_params.weightDecay*m_whv;
}
if (m_params.l1Lambda != 0.0)
{
// Subgradient of lambda*||W||_1 is lambda*sign(W) (sign(0)=0).
m_whv -= m_params.learningRate*m_params.l1Lambda*arma::sign(m_whv);
}
progress += dProgress*miniBatchSizeActual;
status.progress = (int)(progress + 0.5);
@@ -386,6 +392,7 @@ void Rbm::train(arma::mat const &batch, IListener* pListener)
// Calculate error
status.err = rms_error_accu(miniBatch - toVisibleProbs(toHiddenProbs(v_states)));
status.L1 = arma::accu(arma::abs(m_whv));
if (pListener)
{
if(!pListener->onProgress(this, status))
@@ -402,6 +409,7 @@ void Rbm::train(arma::mat const &batch, IListener* pListener)
// Update final status
status.err_total = rms_error_accu(batch - toVisibleProbs(toHiddenProbs(batch)));
status.L1 = arma::accu(arma::abs(m_whv));
if (pListener)
{
source/Rbm.hpp
+4
View File
@@ -28,6 +28,7 @@ public:
Params()
: learningRate(0.1)
, weightDecay(0.0)
, l1Lambda(0.0)
, momentum(0.9)
, doGaussianHidden(false)
, doGaussianVisible(false)
@@ -46,6 +47,7 @@ public:
std::cout << "Exporting Rbm::Params" << std::endl;
Json::Value params;
params["weightDecay"] = weightDecay;
params["l1Lambda"] = l1Lambda;
params["learningRate"] = learningRate;
params["momentum"] = momentum;
params["doGaussianVisible"] = (int)doGaussianVisible;
@@ -65,6 +67,7 @@ public:
{
std::cout << "Importing Rbm::Params" << std::endl;
weightDecay = params.get("weightDecay", weightDecay).asDouble();
l1Lambda = params.get("l1Lambda", l1Lambda).asDouble();
learningRate = params.get("learningRate", learningRate).asDouble();
momentum = params.get("momentum", momentum).asDouble();
doGaussianVisible = params.get("doGaussianVisible", doGaussianVisible) == 1;
@@ -79,6 +82,7 @@ public:
}
double weightDecay;
double l1Lambda;
double learningRate;
double momentum;
bool doGaussianVisible;