# Moby RNN — Character-Level Language Model with Recurrent RBM A character-level language model built on `StackRnn`, a recurrent Restricted Boltzmann Machine where the visible layer at each time step is the concatenation of a **context** vector (the previous hidden state) and the current sensory input (one-hot encoded character). --- ## Architecture ### Single time step ``` Sensory input x_t (one-hot, vocab_size = 40) │ ▼ ┌──────────────────────────────────────────┐ │ visible layer │ │ ┌─────────────────┬────────────────┐ │ │ │ context_t-1 │ x_t │ │ │ │ (CONTEXT_SIZE) │ (vocab_size) │ │ │ └─────────────────┴────────────────┘ │ │ │ │ │ W_t, b_v_t, b_h_t │ │ │ │ │ ┌───────────────────────────────────┐ │ │ │ hidden layer │ │ │ │ context_t │ │ │ │ (CONTEXT_SIZE) │ │ │ └───────────────────────────────────┘ │ └──────────────────┬───────────────────────┘ │ ┌──────────┴──────────┐ │ │ ▼ ▼ context_{t+1} reconstruct x_t (next time step) (predict char) ``` ### Temporal unrolling — own weights per position ``` x_{t-1} x_t x_{t+1} │ │ │ ┌──────┴──────┐ ┌───────┴──────┐ ┌───────┴──────┐ │ ctx │x_{t-1} │ ctx │ x_t │ │ ctx │x_{t+1}│ │ ╠═══════╣ │ ╠════════╣ │ ╠════════╣ │ W_{t-1} │ │ W_t │ │ W_{t+1} │ │ ╠═══════╣ │ ╠════════╣ │ ╠════════╣ │ hidden_{t-1} │ │ hidden_t │ │ hidden_{t+1} │ └───────┬───────┘ └───────┬────────┘ └───────┬────────┘ │ context_{t-1} │ context_t │ context_{t+1} └─────────────────►┘ ──────────────────►┘ ──────► ... ``` Each time step has its **own weight matrix** W_t, b_v_t, b_h_t. `TEMPORAL_DEPTH` positions → `TEMPORAL_DEPTH` separate RBMs, each specialising for the statistical patterns at that offset in a sequence. During generation, position indices wrap modulo `TEMPORAL_DEPTH`. For the shared-weights variant (1 entity via `make_layer`), a single W is reused at every time step — the RTRBM concatenation variant of Sutskever & Hinton (2007). --- ## Vocabulary The raw text is preprocessed to a 40-character vocabulary: ``` uppercase letters A–Z (26) digits 0–9 (10) punctuation . ! ? (3) separator space (1) ───── 40 ``` Lowercase is folded to uppercase; all other characters collapse to a single space; consecutive spaces are merged. --- ## Configuration All constants live in the **Configuration** cell of `moby_rnn.ipynb`: | Constant | Current | Description | |---|---|---| | `TEMPORAL_DEPTH` | 64 | Sequence length = number of RBMs | | `NUM_SEQ` | 2000 | Training sequences (covers first ~128k chars) | | `CONTEXT_SIZE` | 128 | Recurrent hidden / context state size | | `PRJ_NAME` | `"moby_rnn"` | Checkpoint file prefix | | `WORK_DIR` | `"results"` | Directory for saved weights | | `EVAL_CHARS` | 2000 | Characters for reconstruction accuracy | | `PRED_CHARS` | 500 | Characters for next-step prediction | | `N_GIBBS` | 3 | Gibbs steps in clamped-Gibbs generation | | `SEED` | `"Call me Ishmael."` | Seed text for free generation | | `SEQ_IDX` | 42 | Sequence index for hidden-state trace plot | **Model size:** `(CONTEXT_SIZE + vocab_size) × CONTEXT_SIZE × TEMPORAL_DEPTH` — with defaults: `(128 + 40) × 128 × 64 = `**1,376,256 parameters** total. --- ## Training ### Temporal-shift padding Sequences are flattened from `(NUM_SEQ, TEMPORAL_DEPTH, vocab_size)` to a flat batch of `N = NUM_SEQ × TEMPORAL_DEPTH` rows. `TEMPORAL_DEPTH − 1` zero rows are appended, then **layer t trains on `batch[t : N+t]`** — a one-step temporal delay matching the C++ `RnnStack` implementation: ``` Layer 0: batch[0 .. N-1] context = zeros Layer 1: batch[1 .. N] context = h from layer 0 on rows 0..N-1 Layer 2: batch[2 .. N+1] context = h from layer 1 on rows 1..N ... ``` Context from layer t at row j feeds layer t+1 at row j, whose sensory input is the original row j+1 — a one-step look-ahead. ### Joint training All `TEMPORAL_DEPTH` layers are updated **together each epoch**: the context chain is live during training, so each layer sees realistic context from the layers below rather than frozen approximations. ### Resuming Cell 5 saves a checkpoint at every progress report. Re-run it to continue: ``` results/moby_rnn-0-state.npz ← RBM at position 0 results/moby_rnn-1-state.npz ← RBM at position 1 ... results/moby_rnn-63-state.npz ← RBM at position TEMPORAL_DEPTH-1 ``` --- ## Notebook walkthrough | Cell | Title | What it does | |---|---|---| | 1 | Imports | Standard + rbm imports | | 2 | **Configuration** | All constants in one place | | 3 | Load text | Read and preprocess `data/moby.txt`, build vocabulary | | 4 | Encode sequences | One-hot encode → `(NUM_SEQ, TEMPORAL_DEPTH, vocab_size)` | | 5 | Build model | Construct `StackRnn` with `TEMPORAL_DEPTH` layers; load checkpoint | | 6 | Train | Joint CD training with temporal-shift padding; checkpoint every 5% | | 7 | Generation helpers | `predict_next` (clamped Gibbs) and `generate_text` | | 8 | Reconstruction accuracy | Teacher-forced: how well does `h_t` encode `x_t`? | | 9 | Next-step prediction | Given `context_{t-1}`, predict `x_t` before seeing it | | 10 | Free generation | Generate text at temperatures 0.5 / 1.0 / 1.5 | | 11 | Hidden state trace | Heatmap of context activations over one sequence | | 12 | Character distribution | Data vs model character frequency comparison | ### Evaluation metrics **Reconstruction accuracy** (cell 8) is teacher-forced — the model sees `x_t` as part of its visible input, so high accuracy just means the RBM is a decent autoencoder. It does not measure sequence modeling ability. **Next-step prediction accuracy** (cell 9) is the honest metric: given only `context_{t-1}` (no `x_t`), predict the next character via clamped Gibbs. Random baseline is `100 / vocab_size = 2.5%`. --- ## Text generation `predict_next` uses **clamped Gibbs sampling**: context is held fixed while the sensory part of the visible layer iterates for `N_GIBBS` steps. The Bernoulli outputs are temperature-scaled and normalised to a categorical distribution. ```python text = generate_text(rnn, "CALL ME ISHMAEL.", length=500, temperature=0.8) ``` Note: the seed is preprocessed to the reduced vocabulary (uppercase, allowed chars only) before priming the hidden state. Lower temperature → more conservative / repetitive. Higher temperature → more diverse / noisier. --- ## References - Sutskever & Hinton (2007) — *Learning Multilevel Distributed Representations for High-Dimensional Sequences* - Boulanger-Lewandowski et al. (2012) — *Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation*