pyRBM/README_JayRnn.md

JayRnn — Character-Level Language Model with Recurrent RBM

JayRnn is a character-level sequence model built on a Recurrent Restricted Boltzmann Machine (RTRBM). It learns to predict the next character in a string by threading a recurrent context vector through an RBM that also sees a sliding window of the most recent characters.

---

Algorithm

Core idea

A Restricted Boltzmann Machine (RBM) has two layers — visible and hidden — connected by a weight matrix W. The forward pass computes hidden activations from visible input via h = σ(v · W^T + b_h); reconstruction maps back via v' = σ(h · W + b_v).

In JayRnn the visible layer has two concatenated parts:

Portion	Size	Content
Sensory window	WIN × vocab_size	One-hot encoding of the last WIN characters
Context	H_SIZE	Previous hidden state h_{t-1} (recurrent memory)

The hidden layer (H_SIZE units) encodes the current input jointly with temporal history. Its output becomes the context for the next time step.

Vocabulary

41 characters: ^, space, .!?, A–Z, 0–9.
Each character is one-hot encoded into 41 bits; a window of WIN characters becomes a flat WIN × 41 vector.

Training

1. Windowing — the training text is padded with spaces and split into len(TEXT) overlapping windows of width WIN (stride 1).
2. Batch construction — each window is one-hot encoded and stacked row-wise → shape (len(TEXT), WIN × vocab_size).
3. Context init — the context column block starts as all-zeros.
4. CD update — Contrastive Divergence (CD-1, optional Rao-Blackwell estimator) updates W, b_v, b_h.
5. Context propagation — after each CD step the hidden activations at row t are written as context for row t+1: vc[1:, WIN×vocab_size:] = h[:-1, :].

Steps 4–5 repeat for NUM_EPOCHS epochs per iteration, and the whole loop runs NUM_ITERATIONS times, with weights checkpointed after each.

Inference

Given a seed string of WIN characters and an initial context vector:

1. Concatenate window v (one-hot, WIN × vocab_size) with context c (H_SIZE) → visible vc.
2. Forward: h = entity.forward(vc) — new context.
3. Reconstruct: vc' = entity.reconstruct(h) — predicted visible.
4. Split vc' into (v', c') where v' is the predicted window.
5. Clamp — argmax over each character slot of v' → hard one-hot.
6. Slide — shift window left by one slot and append the last predicted character.
7. Repeat from step 1 using the new h as context.

---

Architecture

Single time step

  Text position t  →  window = text[t .. t+WIN-1]

  ┌────────────────────────────────────────────────────────────────────────────┐
  │                   Visible  v_t   (WIN × vocab_size + H_SIZE)              │
  │                                                                            │
  │  ┌──────────┬──────────┬──────────┬──────────┬──────────┬──────────────┐  │
  │  │  char[0] │  char[1] │  char[2] │  char[3] │  char[4] │  context     │  │
  │  │ one-hot  │ one-hot  │ one-hot  │ one-hot  │ one-hot  │  h_{t-1}     │  │
  │  │ (41 bit) │ (41 bit) │ (41 bit) │ (41 bit) │ (41 bit) │  (H_SIZE)    │  │
  │  └──────────┴──────────┴──────────┴──────────┴──────────┴──────────────┘  │
  └───────────────────────────────────────┬────────────────────────────────────┘
                                          │
                                  W  (visible × H_SIZE)
                                          │
  ┌───────────────────────────────────────▼────────────────────────────────────┐
  │                    Hidden  h_t   (H_SIZE)                                  │
  │                    σ( v_t · W^T + b_h )                                   │
  └──────────────────────────────┬────────────────────────┬────────────────────┘
                                 │                        │
                                 ▼                        ▼
                          context_{t+1}          W + b_v  →  reconstruct  v_t'
                       (fed to next step)                        │
                                                   ┌────────────┴────────────┐
                                                   │                         │
                                              window'  (WIN × vocab_size)    ctx'
                                                   │                    (discarded)
                                             argmax per slot
                                                   │
                                            predicted chars

Temporal unrolling (shared weights W)

  t = 0              t = 1              t = 2              t = T-1
  ─────────          ─────────          ─────────          ──────────

ctx = zeros        ctx = h_0          ctx = h_1        ctx = h_{T-2}
     │                  │                  │                  │
  "  HAL"            " HALL"            "HALLO"             ...
     │                  │                  │                  │
  ┌──┴──────────┐    ┌──┴──────────┐    ┌──┴──────────┐    ┌──┴──────────┐
  │ [win | 0  ] │    │ [win | h_0] │    │ [win | h_1] │    │ [win |h_{T-2}]
  │             │    │             │    │             │    │             │
  │      W      │    │      W      │    │      W      │    │      W      │
  │  (shared)   │    │  (shared)   │    │  (shared)   │    │  (shared)   │
  └──────┬──────┘    └──────┬──────┘    └──────┬──────┘    └──────┬──────┘
         │                  │                  │                  │
         ▼                  ▼                  ▼                  ▼
        h_0 ─────────►    h_1 ─────────►    h_2 ─────────►  ...  h_{T-1}

A single weight matrix W is shared across all time steps.
Context flows left-to-right through the batch within each training epoch.

---

Configuration

All constants are at the top of JayRnn.py:

Constant	Default	Description
TEXT	" HALLO SUPER JENS UND SUPER MAUSI!"	Training sequence
WIN	5	Sliding-window width (characters)
STRIDE	1	Window step size
UNITS	1	Number of stacked RBM units
H_SIZE	180	Hidden units (= context size)
NUM_EPOCHS	100	CD epochs per training iteration
NUM_ITERATIONS	10	Outer training loop repetitions

Visible layer size: WIN × vocab_size + H_SIZE = 5 × 41 + 180 = 385
Weight matrix W: 385 × 180 = 69 300 parameters

---

Running

python JayRnn.py

Weights are saved to results/JayRnn-* after each iteration. On subsequent runs the model is loaded and training resumes from the last checkpoint.

Generation output prints each forward step and the full predicted string at the end:

Forward 00:  HA^
Predict 00:  ALLO
...