jensandClaude Sonnet 4.6
952045b9f1
stack_rnn.py:
- _train_unrolled: C++-style temporal-shift padding — flatten (num_seq, T, s)
→ (N, s), append T-1 zero rows, layer t trains on batch[t:N+t]; joint
training replaces greedy sequential (all layers update each epoch)
- Reverted sequential/greedy training path (poor next-step prediction)
moby_rnn.ipynb:
- Vocabulary reduced 85 → 40 chars (A-Z, 0-9, space, .!?)
- T renamed to TEMPORAL_DEPTH throughout
- TEMPORAL_DEPTH increased 5 → 64; CONTEXT_SIZE 128
- SEED "Call me Ishmael." now falls within training data coverage
state.py:
- Remove per-file load/save print messages (too noisy with 64-layer checkpoints)
README_moby_rnn.md:
- Update vocab, constants, parameter count, checkpoint listing
- Add temporal-shift padding and joint training sections
- Clarify reconstruction accuracy vs next-step prediction as honest metric
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
200 lines
8.2 KiB
Markdown
200 lines
8.2 KiB
Markdown
# Moby RNN — Character-Level Language Model with Recurrent RBM
|
||
|
||
A character-level language model built on `StackRnn`, a recurrent Restricted
|
||
Boltzmann Machine where the visible layer at each time step is the concatenation
|
||
of a **context** vector (the previous hidden state) and the current sensory input
|
||
(one-hot encoded character).
|
||
|
||
---
|
||
|
||
## Architecture
|
||
|
||
### Single time step
|
||
|
||
```
|
||
Sensory input x_t (one-hot, vocab_size = 40)
|
||
│
|
||
▼
|
||
┌──────────────────────────────────────────┐
|
||
│ visible layer │
|
||
│ ┌─────────────────┬────────────────┐ │
|
||
│ │ context_t-1 │ x_t │ │
|
||
│ │ (CONTEXT_SIZE) │ (vocab_size) │ │
|
||
│ └─────────────────┴────────────────┘ │
|
||
│ │ │
|
||
│ W_t, b_v_t, b_h_t │
|
||
│ │ │
|
||
│ ┌───────────────────────────────────┐ │
|
||
│ │ hidden layer │ │
|
||
│ │ context_t │ │
|
||
│ │ (CONTEXT_SIZE) │ │
|
||
│ └───────────────────────────────────┘ │
|
||
└──────────────────┬───────────────────────┘
|
||
│
|
||
┌──────────┴──────────┐
|
||
│ │
|
||
▼ ▼
|
||
context_{t+1} reconstruct x_t
|
||
(next time step) (predict char)
|
||
```
|
||
|
||
### Temporal unrolling — own weights per position
|
||
|
||
```
|
||
x_{t-1} x_t x_{t+1}
|
||
│ │ │
|
||
┌──────┴──────┐ ┌───────┴──────┐ ┌───────┴──────┐
|
||
│ ctx │x_{t-1} │ ctx │ x_t │ │ ctx │x_{t+1}│
|
||
│ ╠═══════╣ │ ╠════════╣ │ ╠════════╣
|
||
│ W_{t-1} │ │ W_t │ │ W_{t+1} │
|
||
│ ╠═══════╣ │ ╠════════╣ │ ╠════════╣
|
||
│ hidden_{t-1} │ │ hidden_t │ │ hidden_{t+1} │
|
||
└───────┬───────┘ └───────┬────────┘ └───────┬────────┘
|
||
│ context_{t-1} │ context_t │ context_{t+1}
|
||
└─────────────────►┘ ──────────────────►┘ ──────► ...
|
||
```
|
||
|
||
Each time step has its **own weight matrix** W_t, b_v_t, b_h_t.
|
||
`TEMPORAL_DEPTH` positions → `TEMPORAL_DEPTH` separate RBMs, each specialising
|
||
for the statistical patterns at that offset in a sequence.
|
||
|
||
During generation, position indices wrap modulo `TEMPORAL_DEPTH`.
|
||
|
||
For the shared-weights variant (1 entity via `make_layer`), a single W is reused
|
||
at every time step — the RTRBM concatenation variant of Sutskever & Hinton (2007).
|
||
|
||
---
|
||
|
||
## Vocabulary
|
||
|
||
The raw text is preprocessed to a 40-character vocabulary:
|
||
|
||
```
|
||
uppercase letters A–Z (26)
|
||
digits 0–9 (10)
|
||
punctuation . ! ? (3)
|
||
separator space (1)
|
||
─────
|
||
40
|
||
```
|
||
|
||
Lowercase is folded to uppercase; all other characters collapse to a single space;
|
||
consecutive spaces are merged.
|
||
|
||
---
|
||
|
||
## Configuration
|
||
|
||
All constants live in the **Configuration** cell of `moby_rnn.ipynb`:
|
||
|
||
| Constant | Current | Description |
|
||
|---|---|---|
|
||
| `TEMPORAL_DEPTH` | 64 | Sequence length = number of RBMs |
|
||
| `NUM_SEQ` | 2000 | Training sequences (covers first ~128k chars) |
|
||
| `CONTEXT_SIZE` | 128 | Recurrent hidden / context state size |
|
||
| `PRJ_NAME` | `"moby_rnn"` | Checkpoint file prefix |
|
||
| `WORK_DIR` | `"results"` | Directory for saved weights |
|
||
| `EVAL_CHARS` | 2000 | Characters for reconstruction accuracy |
|
||
| `PRED_CHARS` | 500 | Characters for next-step prediction |
|
||
| `N_GIBBS` | 3 | Gibbs steps in clamped-Gibbs generation |
|
||
| `SEED` | `"Call me Ishmael."` | Seed text for free generation |
|
||
| `SEQ_IDX` | 42 | Sequence index for hidden-state trace plot |
|
||
|
||
**Model size:** `(CONTEXT_SIZE + vocab_size) × CONTEXT_SIZE × TEMPORAL_DEPTH`
|
||
— with defaults: `(128 + 40) × 128 × 64 = `**1,376,256 parameters** total.
|
||
|
||
---
|
||
|
||
## Training
|
||
|
||
### Temporal-shift padding
|
||
|
||
Sequences are flattened from `(NUM_SEQ, TEMPORAL_DEPTH, vocab_size)` to a
|
||
flat batch of `N = NUM_SEQ × TEMPORAL_DEPTH` rows. `TEMPORAL_DEPTH − 1` zero
|
||
rows are appended, then **layer t trains on `batch[t : N+t]`** — a one-step
|
||
temporal delay matching the C++ `RnnStack` implementation:
|
||
|
||
```
|
||
Layer 0: batch[0 .. N-1] context = zeros
|
||
Layer 1: batch[1 .. N] context = h from layer 0 on rows 0..N-1
|
||
Layer 2: batch[2 .. N+1] context = h from layer 1 on rows 1..N
|
||
...
|
||
```
|
||
|
||
Context from layer t at row j feeds layer t+1 at row j, whose sensory input is
|
||
the original row j+1 — a one-step look-ahead.
|
||
|
||
### Joint training
|
||
|
||
All `TEMPORAL_DEPTH` layers are updated **together each epoch**:
|
||
the context chain is live during training, so each layer sees realistic context
|
||
from the layers below rather than frozen approximations.
|
||
|
||
### Resuming
|
||
|
||
Cell 5 saves a checkpoint at every progress report. Re-run it to continue:
|
||
|
||
```
|
||
results/moby_rnn-0-state.npz ← RBM at position 0
|
||
results/moby_rnn-1-state.npz ← RBM at position 1
|
||
...
|
||
results/moby_rnn-63-state.npz ← RBM at position TEMPORAL_DEPTH-1
|
||
```
|
||
|
||
---
|
||
|
||
## Notebook walkthrough
|
||
|
||
| Cell | Title | What it does |
|
||
|---|---|---|
|
||
| 1 | Imports | Standard + rbm imports |
|
||
| 2 | **Configuration** | All constants in one place |
|
||
| 3 | Load text | Read and preprocess `data/moby.txt`, build vocabulary |
|
||
| 4 | Encode sequences | One-hot encode → `(NUM_SEQ, TEMPORAL_DEPTH, vocab_size)` |
|
||
| 5 | Build model | Construct `StackRnn` with `TEMPORAL_DEPTH` layers; load checkpoint |
|
||
| 6 | Train | Joint CD training with temporal-shift padding; checkpoint every 5% |
|
||
| 7 | Generation helpers | `predict_next` (clamped Gibbs) and `generate_text` |
|
||
| 8 | Reconstruction accuracy | Teacher-forced: how well does `h_t` encode `x_t`? |
|
||
| 9 | Next-step prediction | Given `context_{t-1}`, predict `x_t` before seeing it |
|
||
| 10 | Free generation | Generate text at temperatures 0.5 / 1.0 / 1.5 |
|
||
| 11 | Hidden state trace | Heatmap of context activations over one sequence |
|
||
| 12 | Character distribution | Data vs model character frequency comparison |
|
||
|
||
### Evaluation metrics
|
||
|
||
**Reconstruction accuracy** (cell 8) is teacher-forced — the model sees `x_t`
|
||
as part of its visible input, so high accuracy just means the RBM is a decent
|
||
autoencoder. It does not measure sequence modeling ability.
|
||
|
||
**Next-step prediction accuracy** (cell 9) is the honest metric: given only
|
||
`context_{t-1}` (no `x_t`), predict the next character via clamped Gibbs.
|
||
Random baseline is `100 / vocab_size = 2.5%`.
|
||
|
||
---
|
||
|
||
## Text generation
|
||
|
||
`predict_next` uses **clamped Gibbs sampling**: context is held fixed while
|
||
the sensory part of the visible layer iterates for `N_GIBBS` steps.
|
||
The Bernoulli outputs are temperature-scaled and normalised to a categorical
|
||
distribution.
|
||
|
||
```python
|
||
text = generate_text(rnn, "CALL ME ISHMAEL.", length=500, temperature=0.8)
|
||
```
|
||
|
||
Note: the seed is preprocessed to the reduced vocabulary (uppercase, allowed
|
||
chars only) before priming the hidden state.
|
||
|
||
Lower temperature → more conservative / repetitive.
|
||
Higher temperature → more diverse / noisier.
|
||
|
||
---
|
||
|
||
## References
|
||
|
||
- Sutskever & Hinton (2007) — *Learning Multilevel Distributed Representations
|
||
for High-Dimensional Sequences*
|
||
- Boulanger-Lewandowski et al. (2012) — *Modeling Temporal Dependencies in
|
||
High-Dimensional Sequences: Application to Polyphonic Music Generation*
|