pyRBM/README_moby_rnn.md

Moby RNN — Character-Level Language Model with Recurrent RBM

A character-level language model built on StackRnn, a recurrent Restricted Boltzmann Machine where the visible layer at each time step is the concatenation of a context vector (the previous hidden state) and the current sensory input (one-hot encoded character).

---

Architecture

Single time step

  Sensory input x_t  (one-hot, vocab_size = 85)
         │
         ▼
  ┌──────────────────────────────────────────┐
  │           visible layer                  │
  │  ┌─────────────────┬────────────────┐   │
  │  │    context_t-1  │      x_t       │   │
  │  │  (CONTEXT_SIZE) │  (vocab_size)  │   │
  │  └─────────────────┴────────────────┘   │
  │                  │                       │
  │        W  (shared across time)           │
  │                  │                       │
  │  ┌───────────────────────────────────┐  │
  │  │           hidden layer            │  │
  │  │           context_t               │  │
  │  │         (CONTEXT_SIZE)            │  │
  │  └───────────────────────────────────┘  │
  └──────────────────┬───────────────────────┘
                     │
          ┌──────────┴──────────┐
          │                     │
          ▼                     ▼
   context_{t+1}        reconstruct x_t
   (next time step)     (predict char)

Temporal unrolling (unrolled / own-weights mode)

      x_{t-1}              x_t               x_{t+1}
         │                   │                   │
  ┌──────┴──────┐    ┌───────┴──────┐    ┌───────┴──────┐
  │  ctx  │x_{t-1}  │  ctx  │  x_t  │   │  ctx  │x_{t+1}│
  │       ╠═══════╣  │       ╠════════╣   │       ╠════════╣
  │    W_{t-1}    │  │      W_t      │   │     W_{t+1}   │
  │       ╠═══════╣  │       ╠════════╣   │       ╠════════╣
  │  hidden_{t-1} │  │   hidden_t    │   │  hidden_{t+1} │
  └───────┬───────┘  └───────┬────────┘  └───────┬────────┘
          │  context_{t-1}   │  context_t         │  context_{t+1}
          └─────────────────►┘ ──────────────────►┘ ──────► ...

Each time step has its own weight matrix W_t, b_v_t, b_h_t — the model is unrolled: N positions in a sequence → N separate RBMs. This lets each position specialise for the statistical patterns that occur at that offset in a sequence. The sequence length T must be fixed and equal to N at training time; during generation, position indices wrap modulo N.

For comparison, the shared-weights variant (1 entity, make_layer) uses a single W across all time steps — the RTRBM concatenation variant of Sutskever & Hinton (2007).

---

Configuration

All hyperparameters live in the Build model cell of moby_rnn.ipynb:

Constant	Default	Description
T	100	Characters per training sequence
NUM_SEQ	2000	Number of training sequences (first 200k chars)
CONTEXT_SIZE	512	Recurrent hidden / context state size
PRJ_NAME	"moby_rnn"	Checkpoint file prefix
WORK_DIR	"results"	Directory for saved weights
EVAL_CHARS	2000	Characters used for reconstruction accuracy
PRED_CHARS	500	Characters used for next-step prediction
N_GIBBS	10	Gibbs steps in clamped-Gibbs generation
SEED	"Call me Ishmael."	Seed text for free generation
SEQ_IDX	42	Sequence index for the hidden-state trace plot

Model size with defaults: (CONTEXT_SIZE + 85) × CONTEXT_SIZE × T parameters — e.g. T=100, CONTEXT_SIZE=256 → 100 × 27,136 = 2,713,600 parameters total.

---

Notebook walkthrough

Cell	Title	What it does
1	Imports	Standard + rbm imports
2	Load text	Reads data/moby.txt, builds char_to_idx / idx_to_char
3	Encode sequences	One-hot encodes text → (NUM_SEQ, T, vocab_size) array
4	Build model	All constants; constructs StackRnn; loads checkpoint
5	Train	Runs CD training; checkpoints every 5% progress
6	Generation helpers	predict_next (clamped Gibbs) and generate_text
7	Reconstruction accuracy	Teacher-forced: how well does h_t remember x_t?
8	Next-step prediction	Given context_{t-1}, predict x_t before seeing it
9	Free generation	Generate text at temperatures 0.5 / 1.0 / 1.5
10	Hidden state trace	Heatmap of context activations over one sequence
11	Character distribution	Data vs model character frequency comparison

---

Training and resuming

Cell 5 saves a checkpoint at every progress report. Re-run it at any time to continue training from the last saved state — model.state_load() in cell 4 picks it up automatically.

results/moby_rnn-0-state.npz      ← RBM at position 0
results/moby_rnn-1-state.npz      ← RBM at position 1
...
results/moby_rnn-99-state.npz     ← RBM at position T-1

---

Text generation

predict_next uses clamped Gibbs sampling: the context vector is held fixed while the sensory (character) part of the visible layer runs free for N_GIBBS steps. The resulting Bernoulli probabilities are temperature-scaled and normalised to a categorical distribution.

# Generate 500 chars from a seed at temperature 0.8
text = generate_text(rnn, "Call me Ishmael.", length=500, temperature=0.8)

Lower temperature → more conservative / repetitive output. Higher temperature → more diverse / noisier output.

---

References

• Sutskever & Hinton (2007) — Learning Multilevel Distributed Representations for High-Dimensional Sequences
• Boulanger-Lewandowski et al. (2012) — Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation