jens/pyRBM
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[StackRnn] - add unrolled (own-weights) mode; each position gets its own RBM

Browse Source 
Previously a single shared-weight Entity processed every time step.  Now:
- Shared mode  (1 layer via make_layer):  original behaviour unchanged
- Unrolled mode (N layers via make_unrolled): layers[t] owns W_t, b_v_t, b_h_t

New API:
  make_unrolled(T, sensory_size, h_size, ...)  → list[Layer]
  next_entity()   → Entity for the upcoming step() call
  current_entity() → Entity from the most recent step() call
  is_shared        → bool

moby_rnn.ipynb: switch Build model cell to make_unrolled(T=100); update
  predict_next() to use rnn.next_entity() for position-correct Gibbs sampling
README_moby_rnn.md: redraw temporal-unrolling ASCII art showing per-position
  weights W_t; update parameter count and checkpoint file listing

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
jens
2026-05-31 11:52:01 +02:00
co-authored by Claude Sonnet 4.6
parent 0b302a9c12
commit 3698c85ed4
4 changed files with 229 additions and 200 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README_moby_rnn.md
+27 -18
View File
@@ -38,26 +38,31 @@ of a **context** vector (the previous hidden state) and the current sensory inpu
(next time step) (predict char)
```
### Temporal unrolling
### Temporal unrolling (unrolled / own-weights mode)
```
x_{t-1} x_t x_{t+1}
│ │ │
┌──────┴──────┐ ┌──────┴──────┐ ┌──────┴──────┐
│ ctx │x_{t-1} │ ctx │ x_t │ │ ctx │x_{t+1}│
│ ╠═══════╣ │ ╠═══════╣ │ ╠═══════╣
│ W (shared) │ W (shared) │ W (shared) │
│ ╠═══════╣ │ ╠═══════╣ │ ╠═══════╣
│ hidden_t-1 │ │ hidden_t │ │ hidden_t+1 │
└───────┬───────┘ └───────┬───────┘ └───────┬───────┘
│ context_t-1 │ context_t │ context_t+1
└────────────────►┘ ────────────────►┘ ──────► ...
x_{t-1} x_t x_{t+1}
│ │ │
┌──────┴──────┐ ┌───────┴──────┐ ┌───────┴──────┐
│ ctx │x_{t-1} │ ctx │ x_t │ │ ctx │x_{t+1}│
│ ╠═══════╣ │ ╠════════╣ │ ╠════════╣
│ W_{t-1} │ │ W_t │ │ W_{t+1} │
│ ╠═══════╣ │ ╠════════╣ │ ╠════════╣
│ hidden_{t-1} │ │ hidden_t │ │ hidden_{t+1} │
└───────┬───────┘ └───────┬────────┘ └───────┬────────┘
│ context_{t-1} │ context_t │ context_{t+1}
└─────────────────►┘ ──────────────────►┘ ──────► ...
```
Weights **W**, **b_v**, **b_h** are shared across all time steps — the same RBM
processes every character. This is the RTRBM concatenation variant: instead of
modulating the hidden biases (Sutskever & Hinton, 2007), the previous hidden
state is directly concatenated to the visible layer.
Each time step has its **own weight matrix** W_t, b_v_t, b_h_t — the model is
**unrolled**: N positions in a sequence → N separate RBMs. This lets each
position specialise for the statistical patterns that occur at that offset in a
sequence. The sequence length T must be fixed and equal to N at training time;
during generation, position indices wrap modulo N.
For comparison, the shared-weights variant (1 entity, `make_layer`) uses a
single W across all time steps — the RTRBM concatenation variant of Sutskever &
Hinton (2007).
---
@@ -78,7 +83,8 @@ All hyperparameters live in the **Build model** cell of `moby_rnn.ipynb`:
| `SEED` | `"Call me Ishmael."` | Seed text for free generation |
| `SEQ_IDX` | 42 | Sequence index for the hidden-state trace plot |
Model size with defaults: **(512 + 85) × 512 = 306,176 parameters**.
Model size with defaults: **(CONTEXT_SIZE + 85) × CONTEXT_SIZE × T parameters** —
e.g. T=100, CONTEXT_SIZE=256 → 100 × 27,136 = **2,713,600 parameters** total.
---
@@ -107,7 +113,10 @@ continue training from the last saved state — `model.state_load()` in cell 4
picks it up automatically.
```
results/moby_rnn-0-state.npz ← single-layer checkpoint
results/moby_rnn-0-state.npz ← RBM at position 0
results/moby_rnn-1-state.npz ← RBM at position 1
...
results/moby_rnn-99-state.npz ← RBM at position T-1
```
---