Rbm::Params::weightDecay is now applied directly to the weights outside
the momentum recursion (previous commit), exactly like the newly-added
l1Lambda and matching pyRBM's l2_lambda -- same lambda*W gradient form,
same decoupled-from-momentum treatment. The two are functionally the
same mechanism in this codebase, so name it accordingly. Renamed the
Params field, its JSON key, and the two GUI references in
MainComponent.cpp that read/write it (the "weightDecayLabel" widget
identifier and its JUCE-generated marker comment are left as-is --
cosmetic, not part of the actual API).
Existing .prj files still have a "weightDecay" JSON key; fromJson()
no longer reads it, so it's silently ignored on load. Harmless today
since every current project has it set to 0.0 (confirmed: unchanged
poet.elf output, unchanged test-suite results). Not migrating the .prj
files themselves in this change.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016K8Gu7Qejd11JbdiHZqYAs
Rbm::Params had no l1Lambda field at all -- L1 didn't exist anywhere in
the C++ codebase. Status::L1 was a dead field always -1 (never assigned
outside its constructor), and the GUI's "Lambda" control was explicitly
tooltipped "Unused" with an empty change-handler stub. Scaffolded, never
implemented.
Add Params::l1Lambda (default 0.0, inert unless set) with full toJson/
fromJson round-trip. Apply its subgradient (lambda*sign(W), matching
pyRBM's l1_lambda) directly to m_whv in Rbm::train, in the same place
and same decoupled-from-momentum manner as the weightDecay fix from the
previous commit -- folding it into inc_whv's momentum recursion would
cause the same momentum-amplification bug. Doesn't touch the biases,
matching pyRBM's state_adjust().
Also wire up Status::L1 to actually report something (the current L1
norm of the weights, sum(abs(W))) instead of permanently printing -1,
computed alongside status.err/err_total.
Inert by default (l1Lambda=0.0 in every existing .prj): confirmed via
unchanged poet.elf output and unchanged test-suite results (9 passed,
1 known issue, 2 failed).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016K8Gu7Qejd11JbdiHZqYAs
inc_whv = momentum*inc_whv + lr*(dwhv/numcases - weightDecay*m_whv) folded
the weight-decay term into the momentum-accumulated velocity, so its
steady-state effective strength was amplified by roughly
weightDecay/(1-momentum) rather than the configured value -- e.g. ~2x at
momentum=0.5, ~10x at momentum=0.9. This is the same L2-regularization-
vs-momentum interaction that motivated decoupled weight decay (AdamW) in
the broader literature; pyRBM's state_adjust() already applies its
regularization terms directly to the weights outside the momentum
recursion, at full undiscounted strength every step.
Apply weightDecay directly to m_whv after the momentum-accumulated CD
update instead. weightDecay defaults to 0.0 in every .prj in this repo,
so this is currently inert in practice -- confirmed via unchanged
poet.elf output and unchanged test-suite results (9 passed, 1 known
issue, 2 failed) -- but anyone who sets it will now get the strength
they actually configured.
L1 regularization has no equivalent to fix: there's no l1Lambda field in
Rbm::Params at all, Status::L1 is a dead field always -1, and the GUI's
"Lambda" control is explicitly tooltipped "Unused" with an empty
change-handler stub -- it was scaffolded and never implemented.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016K8Gu7Qejd11JbdiHZqYAs
doGaussianHidden took priority over doRaoBlackwell when deciding h_states
for the positive-phase gradient, so a Gaussian-hidden RBM always got the
noisy sample (h_probs + randn) for dw/dbh even with doRaoBlackwell set --
ignoring the flag and adding avoidable gradient variance. Standard CD
practice (and pyRBM's cd_gaussian_gaussian) uses the mean for the
weight/bias gradient when Rao-Blackwellizing, regardless of unit type.
Reordered so doRaoBlackwell is checked first (mean, any unit type) and
only samples otherwise, dispatching via the sampleHidden() helper added
in the previous commit so Gaussian/binary hidden units are each sampled
correctly. Behavior-preserving for poet (doRaoBlackwell=true,
doGaussianHidden=false already took the mean branch); only changes
behavior for a Gaussian-hidden RBM trained with doRaoBlackwell enabled.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016K8Gu7Qejd11JbdiHZqYAs
Matutils::sample() always binary-thresholds (src > uniform(src)), but it
was the only sampler in the codebase and was called unconditionally on
h_probs/v_probs/miniBatch in several CD Gibbs-loop branches regardless
of doGaussianVisible/doGaussianHidden. Binary-thresholding a Gaussian
unit's continuous activation is meaningless -- it would corrupt any
Gaussian-visible/hidden RBM (image-domain experiments via the GUI or
TEST target); doesn't affect poet's plain BB-RBM path since both flags
are false there.
Add sample_gaussian() (mean + N(0,1) noise) alongside the existing
Bernoulli sample() in matutils.hpp, plus Rbm::sampleVisible/sampleHidden
helpers that dispatch to the right one per the RBM's configured type.
Replace every visible/hidden Gibbs-step sample() call in cd_jens (the
active path) and cd_hinton (compiled but currently unused, behind
USE_CD_HINTON) with the appropriate dispatch helper, and fix the same
issue in Rbm::train's doSampleBatch path.
Behavior is unchanged for any RBM with doGaussianVisible/doGaussianHidden
both false (confirmed: poet.elf f output identical before/after).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016K8Gu7Qejd11JbdiHZqYAs
Rbm::train had mini-batch chunks as the outer loop and epochs as the
inner loop: each fixed, never-reshuffled slice of the data got all
numEpochs gradient steps back-to-back before ever being revisited, so
"numEpochs" didn't mean "passes over the whole dataset" and training
was biased toward whatever data came last. Invert the nesting (epochs
outer, mini-batches inner, batch reshuffled via arma::shuffle at the
start of each epoch) so every epoch is an actual full pass over the
data in a fresh random order.
Also drop a redundant toHiddenProbs(v_states) call in cd_jens: the
positive-phase hidden probabilities were computed once unconditionally
and then discarded, recomputed a second time with identical input in
two of the three sampling branches. Same result, half the cost, on
every mini-batch of every epoch of every layer.
Both only affect training; inference (step_forward/generation) is
unchanged, confirmed by identical poet.elf f output before and after.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016K8Gu7Qejd11JbdiHZqYAs
- enable linear hidden : instead of prob(v_to_h()): use toHiddenProbs()
- enable linear visible: instead of prob(h_to_v()): use toVisibleProbs()
- make v_to_h() and h_to_v() private and force to use toHiddenProbs() and toVisibleProbs()
- use cd_jens or cd_hinton. cd_hinton_hid_lineaer is not of use anymre, since it is not capable of CDn
- load / store training batch with context
- on load: add context part to legacy training batches
- removed Rbm::setBatch()
git-svn-id: http://moon:8086/svn/software/trunk/projects/Rbm@790 b431acfa-c32f-4a4a-93f1-934dc6c82436
jensandClaude Sonnet 5