Rbm::Params had no l1Lambda field at all -- L1 didn't exist anywhere in
the C++ codebase. Status::L1 was a dead field always -1 (never assigned
outside its constructor), and the GUI's "Lambda" control was explicitly
tooltipped "Unused" with an empty change-handler stub. Scaffolded, never
implemented.
Add Params::l1Lambda (default 0.0, inert unless set) with full toJson/
fromJson round-trip. Apply its subgradient (lambda*sign(W), matching
pyRBM's l1_lambda) directly to m_whv in Rbm::train, in the same place
and same decoupled-from-momentum manner as the weightDecay fix from the
previous commit -- folding it into inc_whv's momentum recursion would
cause the same momentum-amplification bug. Doesn't touch the biases,
matching pyRBM's state_adjust().
Also wire up Status::L1 to actually report something (the current L1
norm of the weights, sum(abs(W))) instead of permanently printing -1,
computed alongside status.err/err_total.
Inert by default (l1Lambda=0.0 in every existing .prj): confirmed via
unchanged poet.elf output and unchanged test-suite results (9 passed,
1 known issue, 2 failed).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016K8Gu7Qejd11JbdiHZqYAs
inc_whv = momentum*inc_whv + lr*(dwhv/numcases - weightDecay*m_whv) folded
the weight-decay term into the momentum-accumulated velocity, so its
steady-state effective strength was amplified by roughly
weightDecay/(1-momentum) rather than the configured value -- e.g. ~2x at
momentum=0.5, ~10x at momentum=0.9. This is the same L2-regularization-
vs-momentum interaction that motivated decoupled weight decay (AdamW) in
the broader literature; pyRBM's state_adjust() already applies its
regularization terms directly to the weights outside the momentum
recursion, at full undiscounted strength every step.
Apply weightDecay directly to m_whv after the momentum-accumulated CD
update instead. weightDecay defaults to 0.0 in every .prj in this repo,
so this is currently inert in practice -- confirmed via unchanged
poet.elf output and unchanged test-suite results (9 passed, 1 known
issue, 2 failed) -- but anyone who sets it will now get the strength
they actually configured.
L1 regularization has no equivalent to fix: there's no l1Lambda field in
Rbm::Params at all, Status::L1 is a dead field always -1, and the GUI's
"Lambda" control is explicitly tooltipped "Unused" with an empty
change-handler stub -- it was scaffolded and never implemented.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016K8Gu7Qejd11JbdiHZqYAs
doGaussianHidden took priority over doRaoBlackwell when deciding h_states
for the positive-phase gradient, so a Gaussian-hidden RBM always got the
noisy sample (h_probs + randn) for dw/dbh even with doRaoBlackwell set --
ignoring the flag and adding avoidable gradient variance. Standard CD
practice (and pyRBM's cd_gaussian_gaussian) uses the mean for the
weight/bias gradient when Rao-Blackwellizing, regardless of unit type.
Reordered so doRaoBlackwell is checked first (mean, any unit type) and
only samples otherwise, dispatching via the sampleHidden() helper added
in the previous commit so Gaussian/binary hidden units are each sampled
correctly. Behavior-preserving for poet (doRaoBlackwell=true,
doGaussianHidden=false already took the mean branch); only changes
behavior for a Gaussian-hidden RBM trained with doRaoBlackwell enabled.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016K8Gu7Qejd11JbdiHZqYAs
Matutils::sample() always binary-thresholds (src > uniform(src)), but it
was the only sampler in the codebase and was called unconditionally on
h_probs/v_probs/miniBatch in several CD Gibbs-loop branches regardless
of doGaussianVisible/doGaussianHidden. Binary-thresholding a Gaussian
unit's continuous activation is meaningless -- it would corrupt any
Gaussian-visible/hidden RBM (image-domain experiments via the GUI or
TEST target); doesn't affect poet's plain BB-RBM path since both flags
are false there.
Add sample_gaussian() (mean + N(0,1) noise) alongside the existing
Bernoulli sample() in matutils.hpp, plus Rbm::sampleVisible/sampleHidden
helpers that dispatch to the right one per the RBM's configured type.
Replace every visible/hidden Gibbs-step sample() call in cd_jens (the
active path) and cd_hinton (compiled but currently unused, behind
USE_CD_HINTON) with the appropriate dispatch helper, and fix the same
issue in Rbm::train's doSampleBatch path.
Behavior is unchanged for any RBM with doGaussianVisible/doGaussianHidden
both false (confirmed: poet.elf f output identical before/after).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016K8Gu7Qejd11JbdiHZqYAs
Rbm::train had mini-batch chunks as the outer loop and epochs as the
inner loop: each fixed, never-reshuffled slice of the data got all
numEpochs gradient steps back-to-back before ever being revisited, so
"numEpochs" didn't mean "passes over the whole dataset" and training
was biased toward whatever data came last. Invert the nesting (epochs
outer, mini-batches inner, batch reshuffled via arma::shuffle at the
start of each epoch) so every epoch is an actual full pass over the
data in a fresh random order.
Also drop a redundant toHiddenProbs(v_states) call in cd_jens: the
positive-phase hidden probabilities were computed once unconditionally
and then discarded, recomputed a second time with identical input in
two of the three sampling branches. Same result, half the cost, on
every mini-batch of every epoch of every layer.
Both only affect training; inference (step_forward/generation) is
unchanged, confirmed by identical poet.elf f output before and after.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016K8Gu7Qejd11JbdiHZqYAs
- enable linear hidden : instead of prob(v_to_h()): use toHiddenProbs()
- enable linear visible: instead of prob(h_to_v()): use toVisibleProbs()
- make v_to_h() and h_to_v() private and force to use toHiddenProbs() and toVisibleProbs()
- use cd_jens or cd_hinton. cd_hinton_hid_lineaer is not of use anymre, since it is not capable of CDn
- load / store training batch with context
- on load: add context part to legacy training batches
- removed Rbm::setBatch()
git-svn-id: http://moon:8086/svn/software/trunk/projects/Rbm@790 b431acfa-c32f-4a4a-93f1-934dc6c82436
jensandClaude Sonnet 5