jensandClaude Sonnet 4.6
0b302a9c12
- README_moby_rnn.md: single-step and temporally-unrolled ASCII diagrams, configuration table, notebook cell guide, generation API docs, references - moby_rnn.ipynb: move all constants (T, NUM_SEQ, EVAL_CHARS, PRED_CHARS, N_GIBBS, SEED, SEQ_IDX) into the Build model cell under a Configuration header; rename H_SIZE → CONTEXT_SIZE throughout Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
138 lines
6.1 KiB
Markdown
138 lines
6.1 KiB
Markdown
# Moby RNN — Character-Level Language Model with Recurrent RBM
|
||
|
||
A character-level language model built on `StackRnn`, a recurrent Restricted
|
||
Boltzmann Machine where the visible layer at each time step is the concatenation
|
||
of a **context** vector (the previous hidden state) and the current sensory input
|
||
(one-hot encoded character).
|
||
|
||
---
|
||
|
||
## Architecture
|
||
|
||
### Single time step
|
||
|
||
```
|
||
Sensory input x_t (one-hot, vocab_size = 85)
|
||
│
|
||
▼
|
||
┌──────────────────────────────────────────┐
|
||
│ visible layer │
|
||
│ ┌─────────────────┬────────────────┐ │
|
||
│ │ context_t-1 │ x_t │ │
|
||
│ │ (CONTEXT_SIZE) │ (vocab_size) │ │
|
||
│ └─────────────────┴────────────────┘ │
|
||
│ │ │
|
||
│ W (shared across time) │
|
||
│ │ │
|
||
│ ┌───────────────────────────────────┐ │
|
||
│ │ hidden layer │ │
|
||
│ │ context_t │ │
|
||
│ │ (CONTEXT_SIZE) │ │
|
||
│ └───────────────────────────────────┘ │
|
||
└──────────────────┬───────────────────────┘
|
||
│
|
||
┌──────────┴──────────┐
|
||
│ │
|
||
▼ ▼
|
||
context_{t+1} reconstruct x_t
|
||
(next time step) (predict char)
|
||
```
|
||
|
||
### Temporal unrolling
|
||
|
||
```
|
||
x_{t-1} x_t x_{t+1}
|
||
│ │ │
|
||
┌──────┴──────┐ ┌──────┴──────┐ ┌──────┴──────┐
|
||
│ ctx │x_{t-1} │ ctx │ x_t │ │ ctx │x_{t+1}│
|
||
│ ╠═══════╣ │ ╠═══════╣ │ ╠═══════╣
|
||
│ W (shared) │ W (shared) │ W (shared) │
|
||
│ ╠═══════╣ │ ╠═══════╣ │ ╠═══════╣
|
||
│ hidden_t-1 │ │ hidden_t │ │ hidden_t+1 │
|
||
└───────┬───────┘ └───────┬───────┘ └───────┬───────┘
|
||
│ context_t-1 │ context_t │ context_t+1
|
||
└────────────────►┘ ────────────────►┘ ──────► ...
|
||
```
|
||
|
||
Weights **W**, **b_v**, **b_h** are shared across all time steps — the same RBM
|
||
processes every character. This is the RTRBM concatenation variant: instead of
|
||
modulating the hidden biases (Sutskever & Hinton, 2007), the previous hidden
|
||
state is directly concatenated to the visible layer.
|
||
|
||
---
|
||
|
||
## Configuration
|
||
|
||
All hyperparameters live in the **Build model** cell of `moby_rnn.ipynb`:
|
||
|
||
| Constant | Default | Description |
|
||
|---|---|---|
|
||
| `T` | 100 | Characters per training sequence |
|
||
| `NUM_SEQ` | 2000 | Number of training sequences (first 200k chars) |
|
||
| `CONTEXT_SIZE` | 512 | Recurrent hidden / context state size |
|
||
| `PRJ_NAME` | `"moby_rnn"` | Checkpoint file prefix |
|
||
| `WORK_DIR` | `"results"` | Directory for saved weights |
|
||
| `EVAL_CHARS` | 2000 | Characters used for reconstruction accuracy |
|
||
| `PRED_CHARS` | 500 | Characters used for next-step prediction |
|
||
| `N_GIBBS` | 10 | Gibbs steps in clamped-Gibbs generation |
|
||
| `SEED` | `"Call me Ishmael."` | Seed text for free generation |
|
||
| `SEQ_IDX` | 42 | Sequence index for the hidden-state trace plot |
|
||
|
||
Model size with defaults: **(512 + 85) × 512 = 306,176 parameters**.
|
||
|
||
---
|
||
|
||
## Notebook walkthrough
|
||
|
||
| Cell | Title | What it does |
|
||
|---|---|---|
|
||
| 1 | Imports | Standard + rbm imports |
|
||
| 2 | Load text | Reads `data/moby.txt`, builds `char_to_idx` / `idx_to_char` |
|
||
| 3 | Encode sequences | One-hot encodes text → `(NUM_SEQ, T, vocab_size)` array |
|
||
| 4 | **Build model** | All constants; constructs `StackRnn`; loads checkpoint |
|
||
| 5 | Train | Runs CD training; checkpoints every 5% progress |
|
||
| 6 | Generation helpers | `predict_next` (clamped Gibbs) and `generate_text` |
|
||
| 7 | Reconstruction accuracy | Teacher-forced: how well does `h_t` remember `x_t`? |
|
||
| 8 | Next-step prediction | Given `context_{t-1}`, predict `x_t` before seeing it |
|
||
| 9 | Free generation | Generate text at temperatures 0.5 / 1.0 / 1.5 |
|
||
| 10 | Hidden state trace | Heatmap of context activations over one sequence |
|
||
| 11 | Character distribution | Data vs model character frequency comparison |
|
||
|
||
---
|
||
|
||
## Training and resuming
|
||
|
||
Cell 5 saves a checkpoint at every progress report. Re-run it at any time to
|
||
continue training from the last saved state — `model.state_load()` in cell 4
|
||
picks it up automatically.
|
||
|
||
```
|
||
results/moby_rnn-0-state.npz ← single-layer checkpoint
|
||
```
|
||
|
||
---
|
||
|
||
## Text generation
|
||
|
||
`predict_next` uses **clamped Gibbs sampling**: the context vector is held fixed
|
||
while the sensory (character) part of the visible layer runs free for `N_GIBBS`
|
||
steps. The resulting Bernoulli probabilities are temperature-scaled and
|
||
normalised to a categorical distribution.
|
||
|
||
```python
|
||
# Generate 500 chars from a seed at temperature 0.8
|
||
text = generate_text(rnn, "Call me Ishmael.", length=500, temperature=0.8)
|
||
```
|
||
|
||
Lower temperature → more conservative / repetitive output.
|
||
Higher temperature → more diverse / noisier output.
|
||
|
||
---
|
||
|
||
## References
|
||
|
||
- Sutskever & Hinton (2007) — *Learning Multilevel Distributed Representations
|
||
for High-Dimensional Sequences*
|
||
- Boulanger-Lewandowski et al. (2012) — *Modeling Temporal Dependencies in
|
||
High-Dimensional Sequences: Application to Polyphonic Music Generation*
|