pyRBM/README_moby_rnn.md

Moby RNN — Character-Level Language Model with Recurrent RBM

A character-level language model built on StackRnn, a recurrent Restricted Boltzmann Machine where the visible layer at each time step is the concatenation of a context vector (the previous hidden state) and the current sensory input (one-hot encoded character).

---

Architecture

Single time step

  Sensory input x_t  (one-hot, vocab_size = 85)
         │
         ▼
  ┌──────────────────────────────────────────┐
  │           visible layer                  │
  │  ┌─────────────────┬────────────────┐   │
  │  │    context_t-1  │      x_t       │   │
  │  │  (CONTEXT_SIZE) │  (vocab_size)  │   │
  │  └─────────────────┴────────────────┘   │
  │                  │                       │
  │        W  (shared across time)           │
  │                  │                       │
  │  ┌───────────────────────────────────┐  │
  │  │           hidden layer            │  │
  │  │           context_t               │  │
  │  │         (CONTEXT_SIZE)            │  │
  │  └───────────────────────────────────┘  │
  └──────────────────┬───────────────────────┘
                     │
          ┌──────────┴──────────┐
          │                     │
          ▼                     ▼
   context_{t+1}        reconstruct x_t
   (next time step)     (predict char)

Temporal unrolling

      x_{t-1}           x_t            x_{t+1}
         │                │                │
  ┌──────┴──────┐  ┌──────┴──────┐  ┌──────┴──────┐
  │  ctx  │x_{t-1}  │  ctx  │ x_t  │  │  ctx  │x_{t+1}│
  │       ╠═══════╣ │       ╠═══════╣ │       ╠═══════╣
  │    W (shared)   │    W (shared)   │    W (shared)   │
  │       ╠═══════╣ │       ╠═══════╣ │       ╠═══════╣
  │   hidden_t-1  │ │    hidden_t   │ │   hidden_t+1  │
  └───────┬───────┘ └───────┬───────┘ └───────┬───────┘
          │   context_t-1   │   context_t      │   context_t+1
          └────────────────►┘ ────────────────►┘ ──────► ...

Weights W, b_v, b_h are shared across all time steps — the same RBM processes every character. This is the RTRBM concatenation variant: instead of modulating the hidden biases (Sutskever & Hinton, 2007), the previous hidden state is directly concatenated to the visible layer.

---

Configuration

All hyperparameters live in the Build model cell of moby_rnn.ipynb:

Constant	Default	Description
T	100	Characters per training sequence
NUM_SEQ	2000	Number of training sequences (first 200k chars)
CONTEXT_SIZE	512	Recurrent hidden / context state size
PRJ_NAME	"moby_rnn"	Checkpoint file prefix
WORK_DIR	"results"	Directory for saved weights
EVAL_CHARS	2000	Characters used for reconstruction accuracy
PRED_CHARS	500	Characters used for next-step prediction
N_GIBBS	10	Gibbs steps in clamped-Gibbs generation
SEED	"Call me Ishmael."	Seed text for free generation
SEQ_IDX	42	Sequence index for the hidden-state trace plot

Model size with defaults: (512 + 85) × 512 = 306,176 parameters.

---

Notebook walkthrough

Cell	Title	What it does
1	Imports	Standard + rbm imports
2	Load text	Reads data/moby.txt, builds char_to_idx / idx_to_char
3	Encode sequences	One-hot encodes text → (NUM_SEQ, T, vocab_size) array
4	Build model	All constants; constructs StackRnn; loads checkpoint
5	Train	Runs CD training; checkpoints every 5% progress
6	Generation helpers	predict_next (clamped Gibbs) and generate_text
7	Reconstruction accuracy	Teacher-forced: how well does h_t remember x_t?
8	Next-step prediction	Given context_{t-1}, predict x_t before seeing it
9	Free generation	Generate text at temperatures 0.5 / 1.0 / 1.5
10	Hidden state trace	Heatmap of context activations over one sequence
11	Character distribution	Data vs model character frequency comparison

---

Training and resuming

Cell 5 saves a checkpoint at every progress report. Re-run it at any time to continue training from the last saved state — model.state_load() in cell 4 picks it up automatically.

results/moby_rnn-0-state.npz   ← single-layer checkpoint

---

Text generation

predict_next uses clamped Gibbs sampling: the context vector is held fixed while the sensory (character) part of the visible layer runs free for N_GIBBS steps. The resulting Bernoulli probabilities are temperature-scaled and normalised to a categorical distribution.

# Generate 500 chars from a seed at temperature 0.8
text = generate_text(rnn, "Call me Ishmael.", length=500, temperature=0.8)

Lower temperature → more conservative / repetitive output. Higher temperature → more diverse / noisier output.

---

References

• Sutskever & Hinton (2007) — Learning Multilevel Distributed Representations for High-Dimensional Sequences
• Boulanger-Lewandowski et al. (2012) — Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation