# JayRnn — Character-Level Language Model with Recurrent RBM JayRnn is a character-level sequence model built on a **Recurrent Restricted Boltzmann Machine (RTRBM)**. It learns to predict the next character in a string by threading a recurrent context vector through an RBM that also sees a sliding window of the most recent characters. --- ## Algorithm ### Core idea A Restricted Boltzmann Machine (RBM) has two layers — visible and hidden — connected by a weight matrix W. The forward pass computes hidden activations from visible input via `h = σ(v · W^T + b_h)`; reconstruction maps back via `v' = σ(h · W + b_v)`. In JayRnn the visible layer has two concatenated parts: | Portion | Size | Content | |---|---|---| | Sensory window | `WIN × vocab_size` | One-hot encoding of the last WIN characters | | Context | `H_SIZE` | Previous hidden state h_{t-1} (recurrent memory) | The hidden layer (`H_SIZE` units) encodes the current input jointly with temporal history. Its output becomes the context for the next time step. ### Vocabulary 41 characters: `^`, space, `.!?`, `A–Z`, `0–9`. Each character is one-hot encoded into 41 bits; a window of WIN characters becomes a flat `WIN × 41` vector. ### Training 1. **Windowing** — the training text is padded with spaces and split into `len(TEXT)` overlapping windows of width WIN (stride 1). 2. **Batch construction** — each window is one-hot encoded and stacked row-wise → shape `(len(TEXT), WIN × vocab_size)`. 3. **Context init** — the context column block starts as all-zeros. 4. **CD update** — Contrastive Divergence (CD-1, optional Rao-Blackwell estimator) updates W, b_v, b_h. 5. **Context propagation** — after each CD step the hidden activations at row t are written as context for row t+1: `vc[1:, WIN×vocab_size:] = h[:-1, :]`. Steps 4–5 repeat for `NUM_EPOCHS` epochs per iteration, and the whole loop runs `NUM_ITERATIONS` times, with weights checkpointed after each. ### Inference Given a seed string of WIN characters and an initial context vector: 1. Concatenate window `v` (one-hot, `WIN × vocab_size`) with context `c` (H_SIZE) → visible `vc`. 2. **Forward**: `h = entity.forward(vc)` — new context. 3. **Reconstruct**: `vc' = entity.reconstruct(h)` — predicted visible. 4. **Split** `vc'` into `(v', c')` where `v'` is the predicted window. 5. **Clamp** — argmax over each character slot of `v'` → hard one-hot. 6. **Slide** — shift window left by one slot and append the last predicted character. 7. Repeat from step 1 using the new `h` as context. --- ## Architecture ### Single time step ``` Text position t → window = text[t .. t+WIN-1] ┌────────────────────────────────────────────────────────────────────────────┐ │ Visible v_t (WIN × vocab_size + H_SIZE) │ │ │ │ ┌──────────┬──────────┬──────────┬──────────┬──────────┬──────────────┐ │ │ │ char[0] │ char[1] │ char[2] │ char[3] │ char[4] │ context │ │ │ │ one-hot │ one-hot │ one-hot │ one-hot │ one-hot │ h_{t-1} │ │ │ │ (41 bit) │ (41 bit) │ (41 bit) │ (41 bit) │ (41 bit) │ (H_SIZE) │ │ │ └──────────┴──────────┴──────────┴──────────┴──────────┴──────────────┘ │ └───────────────────────────────────────┬────────────────────────────────────┘ │ W (visible × H_SIZE) │ ┌───────────────────────────────────────▼────────────────────────────────────┐ │ Hidden h_t (H_SIZE) │ │ σ( v_t · W^T + b_h ) │ └──────────────────────────────┬────────────────────────┬────────────────────┘ │ │ ▼ ▼ context_{t+1} W + b_v → reconstruct v_t' (fed to next step) │ ┌────────────┴────────────┐ │ │ window' (WIN × vocab_size) ctx' │ (discarded) argmax per slot │ predicted chars ``` ### Temporal unrolling (shared weights W) ``` t = 0 t = 1 t = 2 t = T-1 ───────── ───────── ───────── ────────── ctx = zeros ctx = h_0 ctx = h_1 ctx = h_{T-2} │ │ │ │ " HAL" " HALL" "HALLO" ... │ │ │ │ ┌──┴──────────┐ ┌──┴──────────┐ ┌──┴──────────┐ ┌──┴──────────┐ │ [win | 0 ] │ │ [win | h_0] │ │ [win | h_1] │ │ [win |h_{T-2}] │ │ │ │ │ │ │ │ │ W │ │ W │ │ W │ │ W │ │ (shared) │ │ (shared) │ │ (shared) │ │ (shared) │ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │ │ │ │ ▼ ▼ ▼ ▼ h_0 ─────────► h_1 ─────────► h_2 ─────────► ... h_{T-1} ``` A single weight matrix W is shared across all time steps. Context flows left-to-right through the batch within each training epoch. --- ## Configuration All constants are at the top of `JayRnn.py`: | Constant | Default | Description | |---|---|---| | `TEXT` | `" HALLO SUPER JENS UND SUPER MAUSI!"` | Training sequence | | `WIN` | `5` | Sliding-window width (characters) | | `STRIDE` | `1` | Window step size | | `UNITS` | `1` | Number of stacked RBM units | | `H_SIZE` | `180` | Hidden units (= context size) | | `NUM_EPOCHS` | `100` | CD epochs per training iteration | | `NUM_ITERATIONS` | `10` | Outer training loop repetitions | **Visible layer size:** `WIN × vocab_size + H_SIZE = 5 × 41 + 180 = 385` **Weight matrix W:** `385 × 180 = 69 300 parameters` --- ## Running ```bash python JayRnn.py ``` Weights are saved to `results/JayRnn-*` after each iteration. On subsequent runs the model is loaded and training resumes from the last checkpoint. Generation output prints each forward step and the full predicted string at the end: ``` Forward 00: HA^ Predict 00: ALLO ... ```