jensandClaude Sonnet 4.6
248ae4f199
- reduce WIN to 3 and H_SIZE to 32 for faster iteration - increase NUM_EPOCHS to 1000 and collapse NUM_ITERATIONS to a single pass - add README_JayRnn.md with algorithm description and ASCII architecture diagrams Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
146 lines
8.2 KiB
Markdown
146 lines
8.2 KiB
Markdown
# JayRnn — Character-Level Language Model with Recurrent RBM
|
||
|
||
JayRnn is a character-level sequence model built on a **Recurrent Restricted Boltzmann Machine (RTRBM)**. It learns to predict the next character in a string by threading a recurrent context vector through an RBM that also sees a sliding window of the most recent characters.
|
||
|
||
---
|
||
|
||
## Algorithm
|
||
|
||
### Core idea
|
||
|
||
A Restricted Boltzmann Machine (RBM) has two layers — visible and hidden — connected by a weight matrix W. The forward pass computes hidden activations from visible input via `h = σ(v · W^T + b_h)`; reconstruction maps back via `v' = σ(h · W + b_v)`.
|
||
|
||
In JayRnn the visible layer has two concatenated parts:
|
||
|
||
| Portion | Size | Content |
|
||
|---|---|---|
|
||
| Sensory window | `WIN × vocab_size` | One-hot encoding of the last WIN characters |
|
||
| Context | `H_SIZE` | Previous hidden state h_{t-1} (recurrent memory) |
|
||
|
||
The hidden layer (`H_SIZE` units) encodes the current input jointly with temporal history. Its output becomes the context for the next time step.
|
||
|
||
### Vocabulary
|
||
|
||
41 characters: `^`, space, `.!?`, `A–Z`, `0–9`.
|
||
Each character is one-hot encoded into 41 bits; a window of WIN characters becomes a flat `WIN × 41` vector.
|
||
|
||
### Training
|
||
|
||
1. **Windowing** — the training text is padded with spaces and split into `len(TEXT)` overlapping windows of width WIN (stride 1).
|
||
2. **Batch construction** — each window is one-hot encoded and stacked row-wise → shape `(len(TEXT), WIN × vocab_size)`.
|
||
3. **Context init** — the context column block starts as all-zeros.
|
||
4. **CD update** — Contrastive Divergence (CD-1, optional Rao-Blackwell estimator) updates W, b_v, b_h.
|
||
5. **Context propagation** — after each CD step the hidden activations at row t are written as context for row t+1: `vc[1:, WIN×vocab_size:] = h[:-1, :]`.
|
||
|
||
Steps 4–5 repeat for `NUM_EPOCHS` epochs per iteration, and the whole loop runs `NUM_ITERATIONS` times, with weights checkpointed after each.
|
||
|
||
### Inference
|
||
|
||
Given a seed string of WIN characters and an initial context vector:
|
||
|
||
1. Concatenate window `v` (one-hot, `WIN × vocab_size`) with context `c` (H_SIZE) → visible `vc`.
|
||
2. **Forward**: `h = entity.forward(vc)` — new context.
|
||
3. **Reconstruct**: `vc' = entity.reconstruct(h)` — predicted visible.
|
||
4. **Split** `vc'` into `(v', c')` where `v'` is the predicted window.
|
||
5. **Clamp** — argmax over each character slot of `v'` → hard one-hot.
|
||
6. **Slide** — shift window left by one slot and append the last predicted character.
|
||
7. Repeat from step 1 using the new `h` as context.
|
||
|
||
---
|
||
|
||
## Architecture
|
||
|
||
### Single time step
|
||
|
||
```
|
||
Text position t → window = text[t .. t+WIN-1]
|
||
|
||
┌────────────────────────────────────────────────────────────────────────────┐
|
||
│ Visible v_t (WIN × vocab_size + H_SIZE) │
|
||
│ │
|
||
│ ┌──────────┬──────────┬──────────┬──────────┬──────────┬──────────────┐ │
|
||
│ │ char[0] │ char[1] │ char[2] │ char[3] │ char[4] │ context │ │
|
||
│ │ one-hot │ one-hot │ one-hot │ one-hot │ one-hot │ h_{t-1} │ │
|
||
│ │ (41 bit) │ (41 bit) │ (41 bit) │ (41 bit) │ (41 bit) │ (H_SIZE) │ │
|
||
│ └──────────┴──────────┴──────────┴──────────┴──────────┴──────────────┘ │
|
||
└───────────────────────────────────────┬────────────────────────────────────┘
|
||
│
|
||
W (visible × H_SIZE)
|
||
│
|
||
┌───────────────────────────────────────▼────────────────────────────────────┐
|
||
│ Hidden h_t (H_SIZE) │
|
||
│ σ( v_t · W^T + b_h ) │
|
||
└──────────────────────────────┬────────────────────────┬────────────────────┘
|
||
│ │
|
||
▼ ▼
|
||
context_{t+1} W + b_v → reconstruct v_t'
|
||
(fed to next step) │
|
||
┌────────────┴────────────┐
|
||
│ │
|
||
window' (WIN × vocab_size) ctx'
|
||
│ (discarded)
|
||
argmax per slot
|
||
│
|
||
predicted chars
|
||
```
|
||
|
||
### Temporal unrolling (shared weights W)
|
||
|
||
```
|
||
t = 0 t = 1 t = 2 t = T-1
|
||
───────── ───────── ───────── ──────────
|
||
|
||
ctx = zeros ctx = h_0 ctx = h_1 ctx = h_{T-2}
|
||
│ │ │ │
|
||
" HAL" " HALL" "HALLO" ...
|
||
│ │ │ │
|
||
┌──┴──────────┐ ┌──┴──────────┐ ┌──┴──────────┐ ┌──┴──────────┐
|
||
│ [win | 0 ] │ │ [win | h_0] │ │ [win | h_1] │ │ [win |h_{T-2}]
|
||
│ │ │ │ │ │ │ │
|
||
│ W │ │ W │ │ W │ │ W │
|
||
│ (shared) │ │ (shared) │ │ (shared) │ │ (shared) │
|
||
└──────┬──────┘ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘
|
||
│ │ │ │
|
||
▼ ▼ ▼ ▼
|
||
h_0 ─────────► h_1 ─────────► h_2 ─────────► ... h_{T-1}
|
||
```
|
||
|
||
A single weight matrix W is shared across all time steps.
|
||
Context flows left-to-right through the batch within each training epoch.
|
||
|
||
---
|
||
|
||
## Configuration
|
||
|
||
All constants are at the top of `JayRnn.py`:
|
||
|
||
| Constant | Default | Description |
|
||
|---|---|---|
|
||
| `TEXT` | `" HALLO SUPER JENS UND SUPER MAUSI!"` | Training sequence |
|
||
| `WIN` | `5` | Sliding-window width (characters) |
|
||
| `STRIDE` | `1` | Window step size |
|
||
| `UNITS` | `1` | Number of stacked RBM units |
|
||
| `H_SIZE` | `180` | Hidden units (= context size) |
|
||
| `NUM_EPOCHS` | `100` | CD epochs per training iteration |
|
||
| `NUM_ITERATIONS` | `10` | Outer training loop repetitions |
|
||
|
||
**Visible layer size:** `WIN × vocab_size + H_SIZE = 5 × 41 + 180 = 385`
|
||
**Weight matrix W:** `385 × 180 = 69 300 parameters`
|
||
|
||
---
|
||
|
||
## Running
|
||
|
||
```bash
|
||
python JayRnn.py
|
||
```
|
||
|
||
Weights are saved to `results/JayRnn-*` after each iteration. On subsequent runs the model is loaded and training resumes from the last checkpoint.
|
||
|
||
Generation output prints each forward step and the full predicted string at the end:
|
||
|
||
```
|
||
Forward 00: HA^
|
||
Predict 00: ALLO
|
||
...
|
||
``` |