jens/pyRBM
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[StackRnn] - temporal-shift padding, joint training, vocab reduction, TEMPORAL_DEPTH=64

Browse Source 
stack_rnn.py:
  - _train_unrolled: C++-style temporal-shift padding — flatten (num_seq, T, s)
    → (N, s), append T-1 zero rows, layer t trains on batch[t:N+t]; joint
    training replaces greedy sequential (all layers update each epoch)
  - Reverted sequential/greedy training path (poor next-step prediction)

moby_rnn.ipynb:
  - Vocabulary reduced 85 → 40 chars (A-Z, 0-9, space, .!?)
  - T renamed to TEMPORAL_DEPTH throughout
  - TEMPORAL_DEPTH increased 5 → 64; CONTEXT_SIZE 128
  - SEED "Call me Ishmael." now falls within training data coverage

state.py:
  - Remove per-file load/save print messages (too noisy with 64-layer checkpoints)

README_moby_rnn.md:
  - Update vocab, constants, parameter count, checkpoint listing
  - Add temporal-shift padding and joint training sections
  - Clarify reconstruction accuracy vs next-step prediction as honest metric

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
jens
2026-05-31 14:37:08 +02:00
co-authored by Claude Sonnet 4.6
parent 3698c85ed4
commit 952045b9f1
4 changed files with 611 additions and 289 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README_moby_rnn.md
+105 -52
View File
@@ -12,7 +12,7 @@ of a **context** vector (the previous hidden state) and the current sensory inpu
### Single time step
```
Sensory input x_t (one-hot, vocab_size = 85)
Sensory input x_t (one-hot, vocab_size = 40)
│
▼
┌──────────────────────────────────────────┐
@@ -22,7 +22,7 @@ of a **context** vector (the previous hidden state) and the current sensory inpu
│ │ (CONTEXT_SIZE) │ (vocab_size) │ │
│ └─────────────────┴────────────────┘ │
│ │ │
│ W (shared across time) │
│ W_t, b_v_t, b_h_t │
│ │ │
│ ┌───────────────────────────────────┐ │
│ │ hidden layer │ │
@@ -38,7 +38,7 @@ of a **context** vector (the previous hidden state) and the current sensory inpu
(next time step) (predict char)
```
### Temporal unrolling (unrolled / own-weights mode)
### Temporal unrolling — own weights per position
```
x_{t-1} x_t x_{t+1}
@@ -54,37 +54,92 @@ of a **context** vector (the previous hidden state) and the current sensory inpu
└─────────────────►┘ ──────────────────►┘ ──────► ...
```
Each time step has its **own weight matrix** W_t, b_v_t, b_h_t — the model is
**unrolled**: N positions in a sequence → N separate RBMs. This lets each
position specialise for the statistical patterns that occur at that offset in a
sequence. The sequence length T must be fixed and equal to N at training time;
during generation, position indices wrap modulo N.
Each time step has its **own weight matrix** W_t, b_v_t, b_h_t.
`TEMPORAL_DEPTH` positions → `TEMPORAL_DEPTH` separate RBMs, each specialising
for the statistical patterns at that offset in a sequence.
For comparison, the shared-weights variant (1 entity, `make_layer`) uses a
single W across all time steps — the RTRBM concatenation variant of Sutskever &
Hinton (2007).
During generation, position indices wrap modulo `TEMPORAL_DEPTH`.
For the shared-weights variant (1 entity via `make_layer`), a single W is reused
at every time step — the RTRBM concatenation variant of Sutskever & Hinton (2007).
---
## Vocabulary
The raw text is preprocessed to a 40-character vocabulary:
```
uppercase letters A–Z (26)
digits 0–9 (10)
punctuation . ! ? (3)
separator space (1)
─────
40
```
Lowercase is folded to uppercase; all other characters collapse to a single space;
consecutive spaces are merged.
---
## Configuration
All hyperparameters live in the **Build model** cell of `moby_rnn.ipynb`:
All constants live in the **Configuration** cell of `moby_rnn.ipynb`:
| Constant | Default | Description |
| Constant | Current | Description |
|---|---|---|
| `T` | 100 | Characters per training sequence |
| `NUM_SEQ` | 2000 | Number of training sequences (first 200k chars) |
| `CONTEXT_SIZE` | 512 | Recurrent hidden / context state size |
| `TEMPORAL_DEPTH` | 64 | Sequence length = number of RBMs |
| `NUM_SEQ` | 2000 | Training sequences (covers first ~128k chars) |
| `CONTEXT_SIZE` | 128 | Recurrent hidden / context state size |
| `PRJ_NAME` | `"moby_rnn"` | Checkpoint file prefix |
| `WORK_DIR` | `"results"` | Directory for saved weights |
| `EVAL_CHARS` | 2000 | Characters used for reconstruction accuracy |
| `PRED_CHARS` | 500 | Characters used for next-step prediction |
| `N_GIBBS` | 10 | Gibbs steps in clamped-Gibbs generation |
| `EVAL_CHARS` | 2000 | Characters for reconstruction accuracy |
| `PRED_CHARS` | 500 | Characters for next-step prediction |
| `N_GIBBS` | 3 | Gibbs steps in clamped-Gibbs generation |
| `SEED` | `"Call me Ishmael."` | Seed text for free generation |
| `SEQ_IDX` | 42 | Sequence index for the hidden-state trace plot |
| `SEQ_IDX` | 42 | Sequence index for hidden-state trace plot |
Model size with defaults: **(CONTEXT_SIZE + 85) × CONTEXT_SIZE × T parameters** —
e.g. T=100, CONTEXT_SIZE=256 → 100 × 27,136 = **2,713,600 parameters** total.
**Model size:** `(CONTEXT_SIZE + vocab_size) × CONTEXT_SIZE × TEMPORAL_DEPTH`
— with defaults: `(128 + 40) × 128 × 64 = `**1,376,256 parameters** total.
---
## Training
### Temporal-shift padding
Sequences are flattened from `(NUM_SEQ, TEMPORAL_DEPTH, vocab_size)` to a
flat batch of `N = NUM_SEQ × TEMPORAL_DEPTH` rows. `TEMPORAL_DEPTH − 1` zero
rows are appended, then **layer t trains on `batch[t : N+t]`** — a one-step
temporal delay matching the C++ `RnnStack` implementation:
```
Layer 0: batch[0 .. N-1] context = zeros
Layer 1: batch[1 .. N] context = h from layer 0 on rows 0..N-1
Layer 2: batch[2 .. N+1] context = h from layer 1 on rows 1..N
...
```
Context from layer t at row j feeds layer t+1 at row j, whose sensory input is
the original row j+1 — a one-step look-ahead.
### Joint training
All `TEMPORAL_DEPTH` layers are updated **together each epoch**:
the context chain is live during training, so each layer sees realistic context
from the layers below rather than frozen approximations.
### Resuming
Cell 5 saves a checkpoint at every progress report. Re-run it to continue:
```
results/moby_rnn-0-state.npz ← RBM at position 0
results/moby_rnn-1-state.npz ← RBM at position 1
...
results/moby_rnn-63-state.npz ← RBM at position TEMPORAL_DEPTH-1
```
---
@@ -93,48 +148,46 @@ e.g. T=100, CONTEXT_SIZE=256 → 100 × 27,136 = **2,713,600 parameters** total.
| Cell | Title | What it does |
|---|---|---|
| 1 | Imports | Standard + rbm imports |
| 2 | Load text | Reads `data/moby.txt`, builds `char_to_idx` / `idx_to_char` |
| 3 | Encode sequences | One-hot encodes text → `(NUM_SEQ, T, vocab_size)` array |
| 4 | **Build model** | All constants; constructs `StackRnn`; loads checkpoint |
| 5 | Train | Runs CD training; checkpoints every 5% progress |
| 6 | Generation helpers | `predict_next` (clamped Gibbs) and `generate_text` |
| 7 | Reconstruction accuracy | Teacher-forced: how well does `h_t` remember `x_t`? |
| 8 | Next-step prediction | Given `context_{t-1}`, predict `x_t` before seeing it |
| 9 | Free generation | Generate text at temperatures 0.5 / 1.0 / 1.5 |
| 10 | Hidden state trace | Heatmap of context activations over one sequence |
| 11 | Character distribution | Data vs model character frequency comparison |
| 2 | **Configuration** | All constants in one place |
| 3 | Load text | Read and preprocess `data/moby.txt`, build vocabulary |
| 4 | Encode sequences | One-hot encode → `(NUM_SEQ, TEMPORAL_DEPTH, vocab_size)` |
| 5 | Build model | Construct `StackRnn` with `TEMPORAL_DEPTH` layers; load checkpoint |
| 6 | Train | Joint CD training with temporal-shift padding; checkpoint every 5% |
| 7 | Generation helpers | `predict_next` (clamped Gibbs) and `generate_text` |
| 8 | Reconstruction accuracy | Teacher-forced: how well does `h_t` encode `x_t`? |
| 9 | Next-step prediction | Given `context_{t-1}`, predict `x_t` before seeing it |
| 10 | Free generation | Generate text at temperatures 0.5 / 1.0 / 1.5 |
| 11 | Hidden state trace | Heatmap of context activations over one sequence |
| 12 | Character distribution | Data vs model character frequency comparison |
---
### Evaluation metrics
## Training and resuming
**Reconstruction accuracy** (cell 8) is teacher-forced — the model sees `x_t`
as part of its visible input, so high accuracy just means the RBM is a decent
autoencoder. It does not measure sequence modeling ability.
Cell 5 saves a checkpoint at every progress report. Re-run it at any time to
continue training from the last saved state — `model.state_load()` in cell 4
picks it up automatically.
```
results/moby_rnn-0-state.npz ← RBM at position 0
results/moby_rnn-1-state.npz ← RBM at position 1
...
results/moby_rnn-99-state.npz ← RBM at position T-1
```
**Next-step prediction accuracy** (cell 9) is the honest metric: given only
`context_{t-1}` (no `x_t`), predict the next character via clamped Gibbs.
Random baseline is `100 / vocab_size = 2.5%`.
---
## Text generation
`predict_next` uses **clamped Gibbs sampling**: the context vector is held fixed
while the sensory (character) part of the visible layer runs free for `N_GIBBS`
steps. The resulting Bernoulli probabilities are temperature-scaled and
normalised to a categorical distribution.
`predict_next` uses **clamped Gibbs sampling**: context is held fixed while
the sensory part of the visible layer iterates for `N_GIBBS` steps.
The Bernoulli outputs are temperature-scaled and normalised to a categorical
distribution.
```python
# Generate 500 chars from a seed at temperature 0.8
text = generate_text(rnn, "Call me Ishmael.", length=500, temperature=0.8)
text = generate_text(rnn, "CALL ME ISHMAEL.", length=500, temperature=0.8)
```
Lower temperature → more conservative / repetitive output.
Higher temperature → more diverse / noisier output.
Note: the seed is preprocessed to the reduced vocabulary (uppercase, allowed
chars only) before priming the hidden state.
Lower temperature → more conservative / repetitive.
Higher temperature → more diverse / noisier.
---