pyRBM/README_moby_rnn.md

# Moby RNN — Character-Level Language Model with Recurrent RBM

A character-level language model built on `StackRnn`, a recurrent Restricted
Boltzmann Machine where the visible layer at each time step is the concatenation
of a **context** vector (the previous hidden state) and the current sensory input
(one-hot encoded character).

---

## Architecture

### Single time step

```
  Sensory input x_t  (one-hot, vocab_size = 40)
         │
         ▼
  ┌──────────────────────────────────────────┐
  │           visible layer                  │
  │  ┌─────────────────┬────────────────┐   │
  │  │    context_t-1  │      x_t       │   │
  │  │  (CONTEXT_SIZE) │  (vocab_size)  │   │
  │  └─────────────────┴────────────────┘   │
  │                  │                       │
  │              W_t, b_v_t, b_h_t           │
  │                  │                       │
  │  ┌───────────────────────────────────┐  │
  │  │           hidden layer            │  │
  │  │           context_t               │  │
  │  │         (CONTEXT_SIZE)            │  │
  │  └───────────────────────────────────┘  │
  └──────────────────┬───────────────────────┘
                     │
          ┌──────────┴──────────┐
          │                     │
          ▼                     ▼
   context_{t+1}        reconstruct x_t
   (next time step)     (predict char)
```

### Temporal unrolling — own weights per position

```
      x_{t-1}              x_t               x_{t+1}
         │                   │                   │
  ┌──────┴──────┐    ┌───────┴──────┐    ┌───────┴──────┐
  │  ctx  │x_{t-1}  │  ctx  │  x_t  │   │  ctx  │x_{t+1}│
  │       ╠═══════╣  │       ╠════════╣   │       ╠════════╣
  │    W_{t-1}    │  │      W_t      │   │     W_{t+1}   │
  │       ╠═══════╣  │       ╠════════╣   │       ╠════════╣
  │  hidden_{t-1} │  │   hidden_t    │   │  hidden_{t+1} │
  └───────┬───────┘  └───────┬────────┘  └───────┬────────┘
          │  context_{t-1}   │  context_t         │  context_{t+1}
          └─────────────────►┘ ──────────────────►┘ ──────► ...
```

Each time step has its **own weight matrix** W_t, b_v_t, b_h_t.
`TEMPORAL_DEPTH` positions → `TEMPORAL_DEPTH` separate RBMs, each specialising
for the statistical patterns at that offset in a sequence.

During generation, position indices wrap modulo `TEMPORAL_DEPTH`.

For the shared-weights variant (1 entity via `make_layer`), a single W is reused
at every time step — the RTRBM concatenation variant of Sutskever & Hinton (2007).

---

## Vocabulary

The raw text is preprocessed to a 40-character vocabulary:

```
uppercase letters  A–Z   (26)
digits             0–9   (10)
punctuation        . ! ?  (3)
separator          space  (1)
                  ─────
                    40
```

Lowercase is folded to uppercase; all other characters collapse to a single space;
consecutive spaces are merged.

---

## Configuration

All constants live in the **Configuration** cell of `moby_rnn.ipynb`:

| Constant | Current | Description |
|---|---|---|
| `TEMPORAL_DEPTH` | 64 | Sequence length = number of RBMs |
| `NUM_SEQ` | 2000 | Training sequences (covers first ~128k chars) |
| `CONTEXT_SIZE` | 128 | Recurrent hidden / context state size |
| `PRJ_NAME` | `"moby_rnn"` | Checkpoint file prefix |
| `WORK_DIR` | `"results"` | Directory for saved weights |
| `EVAL_CHARS` | 2000 | Characters for reconstruction accuracy |
| `PRED_CHARS` | 500 | Characters for next-step prediction |
| `N_GIBBS` | 3 | Gibbs steps in clamped-Gibbs generation |
| `SEED` | `"Call me Ishmael."` | Seed text for free generation |
| `SEQ_IDX` | 42 | Sequence index for hidden-state trace plot |

**Model size:** `(CONTEXT_SIZE + vocab_size) × CONTEXT_SIZE × TEMPORAL_DEPTH`
— with defaults: `(128 + 40) × 128 × 64 = `**1,376,256 parameters** total.

---

## Training

### Temporal-shift padding

Sequences are flattened from `(NUM_SEQ, TEMPORAL_DEPTH, vocab_size)` to a
flat batch of `N = NUM_SEQ × TEMPORAL_DEPTH` rows.  `TEMPORAL_DEPTH − 1` zero
rows are appended, then **layer t trains on `batch[t : N+t]`** — a one-step
temporal delay matching the C++ `RnnStack` implementation:

```
Layer 0: batch[0 .. N-1]          context = zeros
Layer 1: batch[1 .. N]            context = h from layer 0 on rows 0..N-1
Layer 2: batch[2 .. N+1]          context = h from layer 1 on rows 1..N
  ...
```

Context from layer t at row j feeds layer t+1 at row j, whose sensory input is
the original row j+1 — a one-step look-ahead.

### Joint training

All `TEMPORAL_DEPTH` layers are updated **together each epoch**:
the context chain is live during training, so each layer sees realistic context
from the layers below rather than frozen approximations.

### Resuming

Cell 5 saves a checkpoint at every progress report.  Re-run it to continue:

```
results/moby_rnn-0-state.npz      ← RBM at position 0
results/moby_rnn-1-state.npz      ← RBM at position 1
...
results/moby_rnn-63-state.npz     ← RBM at position TEMPORAL_DEPTH-1
```

---

## Notebook walkthrough

| Cell | Title | What it does |
|---|---|---|
| 1 | Imports | Standard + rbm imports |
| 2 | **Configuration** | All constants in one place |
| 3 | Load text | Read and preprocess `data/moby.txt`, build vocabulary |
| 4 | Encode sequences | One-hot encode → `(NUM_SEQ, TEMPORAL_DEPTH, vocab_size)` |
| 5 | Build model | Construct `StackRnn` with `TEMPORAL_DEPTH` layers; load checkpoint |
| 6 | Train | Joint CD training with temporal-shift padding; checkpoint every 5% |
| 7 | Generation helpers | `predict_next` (clamped Gibbs) and `generate_text` |
| 8 | Reconstruction accuracy | Teacher-forced: how well does `h_t` encode `x_t`? |
| 9 | Next-step prediction | Given `context_{t-1}`, predict `x_t` before seeing it |
| 10 | Free generation | Generate text at temperatures 0.5 / 1.0 / 1.5 |
| 11 | Hidden state trace | Heatmap of context activations over one sequence |
| 12 | Character distribution | Data vs model character frequency comparison |

### Evaluation metrics

**Reconstruction accuracy** (cell 8) is teacher-forced — the model sees `x_t`
as part of its visible input, so high accuracy just means the RBM is a decent
autoencoder.  It does not measure sequence modeling ability.

**Next-step prediction accuracy** (cell 9) is the honest metric: given only
`context_{t-1}` (no `x_t`), predict the next character via clamped Gibbs.
Random baseline is `100 / vocab_size = 2.5%`.

---

## Text generation

`predict_next` uses **clamped Gibbs sampling**: context is held fixed while
the sensory part of the visible layer iterates for `N_GIBBS` steps.
The Bernoulli outputs are temperature-scaled and normalised to a categorical
distribution.

```python
text = generate_text(rnn, "CALL ME ISHMAEL.", length=500, temperature=0.8)
```

Note: the seed is preprocessed to the reduced vocabulary (uppercase, allowed
chars only) before priming the hidden state.

Lower temperature → more conservative / repetitive.
Higher temperature → more diverse / noisier.

---

## References

- Sutskever & Hinton (2007) — *Learning Multilevel Distributed Representations
  for High-Dimensional Sequences*
- Boulanger-Lewandowski et al. (2012) — *Modeling Temporal Dependencies in
  High-Dimensional Sequences: Application to Polyphonic Music Generation*