jensandClaude Sonnet 4.6
3698c85ed4
Previously a single shared-weight Entity processed every time step. Now: - Shared mode (1 layer via make_layer): original behaviour unchanged - Unrolled mode (N layers via make_unrolled): layers[t] owns W_t, b_v_t, b_h_t New API: make_unrolled(T, sensory_size, h_size, ...) → list[Layer] next_entity() → Entity for the upcoming step() call current_entity() → Entity from the most recent step() call is_shared → bool moby_rnn.ipynb: switch Build model cell to make_unrolled(T=100); update predict_next() to use rnn.next_entity() for position-correct Gibbs sampling README_moby_rnn.md: redraw temporal-unrolling ASCII art showing per-position weights W_t; update parameter count and checkpoint file listing Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
147 lines
6.6 KiB
Markdown
147 lines
6.6 KiB
Markdown
# Moby RNN — Character-Level Language Model with Recurrent RBM
|
||
|
||
A character-level language model built on `StackRnn`, a recurrent Restricted
|
||
Boltzmann Machine where the visible layer at each time step is the concatenation
|
||
of a **context** vector (the previous hidden state) and the current sensory input
|
||
(one-hot encoded character).
|
||
|
||
---
|
||
|
||
## Architecture
|
||
|
||
### Single time step
|
||
|
||
```
|
||
Sensory input x_t (one-hot, vocab_size = 85)
|
||
│
|
||
▼
|
||
┌──────────────────────────────────────────┐
|
||
│ visible layer │
|
||
│ ┌─────────────────┬────────────────┐ │
|
||
│ │ context_t-1 │ x_t │ │
|
||
│ │ (CONTEXT_SIZE) │ (vocab_size) │ │
|
||
│ └─────────────────┴────────────────┘ │
|
||
│ │ │
|
||
│ W (shared across time) │
|
||
│ │ │
|
||
│ ┌───────────────────────────────────┐ │
|
||
│ │ hidden layer │ │
|
||
│ │ context_t │ │
|
||
│ │ (CONTEXT_SIZE) │ │
|
||
│ └───────────────────────────────────┘ │
|
||
└──────────────────┬───────────────────────┘
|
||
│
|
||
┌──────────┴──────────┐
|
||
│ │
|
||
▼ ▼
|
||
context_{t+1} reconstruct x_t
|
||
(next time step) (predict char)
|
||
```
|
||
|
||
### Temporal unrolling (unrolled / own-weights mode)
|
||
|
||
```
|
||
x_{t-1} x_t x_{t+1}
|
||
│ │ │
|
||
┌──────┴──────┐ ┌───────┴──────┐ ┌───────┴──────┐
|
||
│ ctx │x_{t-1} │ ctx │ x_t │ │ ctx │x_{t+1}│
|
||
│ ╠═══════╣ │ ╠════════╣ │ ╠════════╣
|
||
│ W_{t-1} │ │ W_t │ │ W_{t+1} │
|
||
│ ╠═══════╣ │ ╠════════╣ │ ╠════════╣
|
||
│ hidden_{t-1} │ │ hidden_t │ │ hidden_{t+1} │
|
||
└───────┬───────┘ └───────┬────────┘ └───────┬────────┘
|
||
│ context_{t-1} │ context_t │ context_{t+1}
|
||
└─────────────────►┘ ──────────────────►┘ ──────► ...
|
||
```
|
||
|
||
Each time step has its **own weight matrix** W_t, b_v_t, b_h_t — the model is
|
||
**unrolled**: N positions in a sequence → N separate RBMs. This lets each
|
||
position specialise for the statistical patterns that occur at that offset in a
|
||
sequence. The sequence length T must be fixed and equal to N at training time;
|
||
during generation, position indices wrap modulo N.
|
||
|
||
For comparison, the shared-weights variant (1 entity, `make_layer`) uses a
|
||
single W across all time steps — the RTRBM concatenation variant of Sutskever &
|
||
Hinton (2007).
|
||
|
||
---
|
||
|
||
## Configuration
|
||
|
||
All hyperparameters live in the **Build model** cell of `moby_rnn.ipynb`:
|
||
|
||
| Constant | Default | Description |
|
||
|---|---|---|
|
||
| `T` | 100 | Characters per training sequence |
|
||
| `NUM_SEQ` | 2000 | Number of training sequences (first 200k chars) |
|
||
| `CONTEXT_SIZE` | 512 | Recurrent hidden / context state size |
|
||
| `PRJ_NAME` | `"moby_rnn"` | Checkpoint file prefix |
|
||
| `WORK_DIR` | `"results"` | Directory for saved weights |
|
||
| `EVAL_CHARS` | 2000 | Characters used for reconstruction accuracy |
|
||
| `PRED_CHARS` | 500 | Characters used for next-step prediction |
|
||
| `N_GIBBS` | 10 | Gibbs steps in clamped-Gibbs generation |
|
||
| `SEED` | `"Call me Ishmael."` | Seed text for free generation |
|
||
| `SEQ_IDX` | 42 | Sequence index for the hidden-state trace plot |
|
||
|
||
Model size with defaults: **(CONTEXT_SIZE + 85) × CONTEXT_SIZE × T parameters** —
|
||
e.g. T=100, CONTEXT_SIZE=256 → 100 × 27,136 = **2,713,600 parameters** total.
|
||
|
||
---
|
||
|
||
## Notebook walkthrough
|
||
|
||
| Cell | Title | What it does |
|
||
|---|---|---|
|
||
| 1 | Imports | Standard + rbm imports |
|
||
| 2 | Load text | Reads `data/moby.txt`, builds `char_to_idx` / `idx_to_char` |
|
||
| 3 | Encode sequences | One-hot encodes text → `(NUM_SEQ, T, vocab_size)` array |
|
||
| 4 | **Build model** | All constants; constructs `StackRnn`; loads checkpoint |
|
||
| 5 | Train | Runs CD training; checkpoints every 5% progress |
|
||
| 6 | Generation helpers | `predict_next` (clamped Gibbs) and `generate_text` |
|
||
| 7 | Reconstruction accuracy | Teacher-forced: how well does `h_t` remember `x_t`? |
|
||
| 8 | Next-step prediction | Given `context_{t-1}`, predict `x_t` before seeing it |
|
||
| 9 | Free generation | Generate text at temperatures 0.5 / 1.0 / 1.5 |
|
||
| 10 | Hidden state trace | Heatmap of context activations over one sequence |
|
||
| 11 | Character distribution | Data vs model character frequency comparison |
|
||
|
||
---
|
||
|
||
## Training and resuming
|
||
|
||
Cell 5 saves a checkpoint at every progress report. Re-run it at any time to
|
||
continue training from the last saved state — `model.state_load()` in cell 4
|
||
picks it up automatically.
|
||
|
||
```
|
||
results/moby_rnn-0-state.npz ← RBM at position 0
|
||
results/moby_rnn-1-state.npz ← RBM at position 1
|
||
...
|
||
results/moby_rnn-99-state.npz ← RBM at position T-1
|
||
```
|
||
|
||
---
|
||
|
||
## Text generation
|
||
|
||
`predict_next` uses **clamped Gibbs sampling**: the context vector is held fixed
|
||
while the sensory (character) part of the visible layer runs free for `N_GIBBS`
|
||
steps. The resulting Bernoulli probabilities are temperature-scaled and
|
||
normalised to a categorical distribution.
|
||
|
||
```python
|
||
# Generate 500 chars from a seed at temperature 0.8
|
||
text = generate_text(rnn, "Call me Ishmael.", length=500, temperature=0.8)
|
||
```
|
||
|
||
Lower temperature → more conservative / repetitive output.
|
||
Higher temperature → more diverse / noisier output.
|
||
|
||
---
|
||
|
||
## References
|
||
|
||
- Sutskever & Hinton (2007) — *Learning Multilevel Distributed Representations
|
||
for High-Dimensional Sequences*
|
||
- Boulanger-Lewandowski et al. (2012) — *Modeling Temporal Dependencies in
|
||
High-Dimensional Sequences: Application to Polyphonic Music Generation*
|