jens/Rbm
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Wire up temperature-based sampling for poet text generation

Browse Source 
Greedy argmax decoding in RbmListener::forward always produces the
exact same character sequence and quickly falls into short repeating
loops once the 5-character lookback state revisits a prior cycle.

Add an optional temperature argument (poet f <seed> [temperature]) that
switches decoding to RnnStack::sample_one_hot, which now does proper
categorical sampling (temperature-scaled, renormalized draw) instead of
the old per-code Bernoulli approach that could leave the result as a
non-one-hot probability vector. Also seed Armadillo's RNG in main(),
since it otherwise defaults to a fixed seed and every run would sample
identically.

Add docs/RNN_ARCHITECTURE.md documenting how the RnnStack/Layer stack
implements the RNN (context chaining across layers, training/generation
data flow, and the decoding behavior above).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016K8Gu7Qejd11JbdiHZqYAs
This commit is contained in:
jens
2026-07-27 12:34:44 +02:00
co-authored by Claude Sonnet 5
parent 33a0647a51
commit 656a0252d3
4 changed files with 247 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files
source/RnnStack.cpp
+26 -13
View File
@@ -92,25 +92,38 @@ void RnnStack::clamp_one_hot(arma::mat& srcDst)
srcDst[index] = 1;
}
void RnnStack::sample_one_hot(arma::mat& srcDst)
void RnnStack::sample_one_hot(arma::mat& srcDst, double temperature)
{
double k = arma::accu(srcDst);
if (k != 0)
// srcDst holds independent per-code sigmoid activations, not a normalized
// distribution. Raising to 1/temperature before renormalizing sharpens
// (temperature < 1) or flattens (temperature > 1) the resulting
// categorical distribution, then we draw one index from it directly.
arma::mat p = arma::pow(arma::clamp(srcDst, 1e-9, 1.0), 1.0 / temperature);
double sum = arma::accu(p);
if (sum > 0)
{
srcDst = srcDst / k;
p /= sum;
}
else
{
p = arma::ones(arma::size(srcDst)) / srcDst.n_elem;
}
arma::mat ps = Matutils::sample(srcDst);
if (arma::accu(ps) > 1)
double r = arma::randu(1)[0];
double cumulative = 0.0;
size_t chosen = p.n_elem - 1;
for (size_t i = 0; i < p.n_elem; i++)
{
arma::uvec q1 = find(ps > 0);
int winner = (q1.n_elem - 1) * arma::randu(1)[0];
int wix = q1(winner);
srcDst = zeros(arma::size(srcDst));
srcDst[wix] = 1;
cumulative += p[i];
if (r <= cumulative)
{
chosen = i;
break;
}
}
srcDst = arma::zeros(arma::size(srcDst));
srcDst[chosen] = 1;
}
arma::mat RnnStack::to_curr(const arma::mat& v)