pyRBM/README_moby_rnn.md

Moby RNN — Character-Level Language Model with Recurrent RBM

A character-level language model built on StackRnn, a recurrent Restricted Boltzmann Machine where the visible layer at each time step is the concatenation of a context vector (the previous hidden state) and the current sensory input (one-hot encoded character).

---

Architecture

Single time step

  Sensory input x_t  (one-hot, vocab_size = 40)
         │
         ▼
  ┌──────────────────────────────────────────┐
  │           visible layer                  │
  │  ┌─────────────────┬────────────────┐   │
  │  │    context_t-1  │      x_t       │   │
  │  │  (CONTEXT_SIZE) │  (vocab_size)  │   │
  │  └─────────────────┴────────────────┘   │
  │                  │                       │
  │              W_t, b_v_t, b_h_t           │
  │                  │                       │
  │  ┌───────────────────────────────────┐  │
  │  │           hidden layer            │  │
  │  │           context_t               │  │
  │  │         (CONTEXT_SIZE)            │  │
  │  └───────────────────────────────────┘  │
  └──────────────────┬───────────────────────┘
                     │
          ┌──────────┴──────────┐
          │                     │
          ▼                     ▼
   context_{t+1}        reconstruct x_t
   (next time step)     (predict char)

Temporal unrolling — own weights per position

      x_{t-1}              x_t               x_{t+1}
         │                   │                   │
  ┌──────┴──────┐    ┌───────┴──────┐    ┌───────┴──────┐
  │  ctx  │x_{t-1}  │  ctx  │  x_t  │   │  ctx  │x_{t+1}│
  │       ╠═══════╣  │       ╠════════╣   │       ╠════════╣
  │    W_{t-1}    │  │      W_t      │   │     W_{t+1}   │
  │       ╠═══════╣  │       ╠════════╣   │       ╠════════╣
  │  hidden_{t-1} │  │   hidden_t    │   │  hidden_{t+1} │
  └───────┬───────┘  └───────┬────────┘  └───────┬────────┘
          │  context_{t-1}   │  context_t         │  context_{t+1}
          └─────────────────►┘ ──────────────────►┘ ──────► ...

Each time step has its own weight matrix W_t, b_v_t, b_h_t. TEMPORAL_DEPTH positions → TEMPORAL_DEPTH separate RBMs, each specialising for the statistical patterns at that offset in a sequence.

During generation, position indices wrap modulo TEMPORAL_DEPTH.

For the shared-weights variant (1 entity via make_layer), a single W is reused at every time step — the RTRBM concatenation variant of Sutskever & Hinton (2007).

---

Vocabulary

The raw text is preprocessed to a 40-character vocabulary:

uppercase letters  A–Z   (26)
digits             0–9   (10)
punctuation        . ! ?  (3)
separator          space  (1)
                  ─────
                    40

Lowercase is folded to uppercase; all other characters collapse to a single space; consecutive spaces are merged.

---

Configuration

All constants live in the Configuration cell of moby_rnn.ipynb:

Constant	Current	Description
TEMPORAL_DEPTH	64	Sequence length = number of RBMs
NUM_SEQ	2000	Training sequences (covers first ~128k chars)
CONTEXT_SIZE	128	Recurrent hidden / context state size
PRJ_NAME	"moby_rnn"	Checkpoint file prefix
WORK_DIR	"results"	Directory for saved weights
EVAL_CHARS	2000	Characters for reconstruction accuracy
PRED_CHARS	500	Characters for next-step prediction
N_GIBBS	3	Gibbs steps in clamped-Gibbs generation
SEED	"Call me Ishmael."	Seed text for free generation
SEQ_IDX	42	Sequence index for hidden-state trace plot

Model size: (CONTEXT_SIZE + vocab_size) × CONTEXT_SIZE × TEMPORAL_DEPTH — with defaults: (128 + 40) × 128 × 64 = 1,376,256 parameters total.

---

Training

Temporal-shift padding

Sequences are flattened from (NUM_SEQ, TEMPORAL_DEPTH, vocab_size) to a flat batch of N = NUM_SEQ × TEMPORAL_DEPTH rows. TEMPORAL_DEPTH − 1 zero rows are appended, then layer t trains on batch[t : N+t] — a one-step temporal delay matching the C++ RnnStack implementation:

Layer 0: batch[0 .. N-1]          context = zeros
Layer 1: batch[1 .. N]            context = h from layer 0 on rows 0..N-1
Layer 2: batch[2 .. N+1]          context = h from layer 1 on rows 1..N
  ...

Context from layer t at row j feeds layer t+1 at row j, whose sensory input is the original row j+1 — a one-step look-ahead.

Joint training

All TEMPORAL_DEPTH layers are updated together each epoch: the context chain is live during training, so each layer sees realistic context from the layers below rather than frozen approximations.

Resuming

Cell 5 saves a checkpoint at every progress report. Re-run it to continue:

results/moby_rnn-0-state.npz      ← RBM at position 0
results/moby_rnn-1-state.npz      ← RBM at position 1
...
results/moby_rnn-63-state.npz     ← RBM at position TEMPORAL_DEPTH-1

---

Notebook walkthrough

Cell	Title	What it does
1	Imports	Standard + rbm imports
2	Configuration	All constants in one place
3	Load text	Read and preprocess data/moby.txt, build vocabulary
4	Encode sequences	One-hot encode → (NUM_SEQ, TEMPORAL_DEPTH, vocab_size)
5	Build model	Construct StackRnn with TEMPORAL_DEPTH layers; load checkpoint
6	Train	Joint CD training with temporal-shift padding; checkpoint every 5%
7	Generation helpers	predict_next (clamped Gibbs) and generate_text
8	Reconstruction accuracy	Teacher-forced: how well does h_t encode x_t?
9	Next-step prediction	Given context_{t-1}, predict x_t before seeing it
10	Free generation	Generate text at temperatures 0.5 / 1.0 / 1.5
11	Hidden state trace	Heatmap of context activations over one sequence
12	Character distribution	Data vs model character frequency comparison

Evaluation metrics

Reconstruction accuracy (cell 8) is teacher-forced — the model sees x_t as part of its visible input, so high accuracy just means the RBM is a decent autoencoder. It does not measure sequence modeling ability.

Next-step prediction accuracy (cell 9) is the honest metric: given only context_{t-1} (no x_t), predict the next character via clamped Gibbs. Random baseline is 100 / vocab_size = 2.5%.

---

Text generation

predict_next uses clamped Gibbs sampling: context is held fixed while the sensory part of the visible layer iterates for N_GIBBS steps. The Bernoulli outputs are temperature-scaled and normalised to a categorical distribution.

text = generate_text(rnn, "CALL ME ISHMAEL.", length=500, temperature=0.8)

Note: the seed is preprocessed to the reduced vocabulary (uppercase, allowed chars only) before priming the hidden state.

Lower temperature → more conservative / repetitive. Higher temperature → more diverse / noisier.

---

References

• Sutskever & Hinton (2007) — Learning Multilevel Distributed Representations for High-Dimensional Sequences
• Boulanger-Lewandowski et al. (2012) — Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation