# Moby RNN — Character-Level Language Model with Recurrent RBM

A character-level language model built on `StackRnn`, a recurrent Restricted
Boltzmann Machine where the visible layer at each time step is the concatenation
of a **context** vector (the previous hidden state) and the current sensory input
(one-hot encoded character).

---

## Architecture

### Single time step

```
  Sensory input x_t  (one-hot, vocab_size = 85)
         │
         ▼
  ┌──────────────────────────────────────────┐
  │           visible layer                  │
  │  ┌─────────────────┬────────────────┐   │
  │  │    context_t-1  │      x_t       │   │
  │  │  (CONTEXT_SIZE) │  (vocab_size)  │   │
  │  └─────────────────┴────────────────┘   │
  │                  │                       │
  │        W  (shared across time)           │
  │                  │                       │
  │  ┌───────────────────────────────────┐  │
  │  │           hidden layer            │  │
  │  │           context_t               │  │
  │  │         (CONTEXT_SIZE)            │  │
  │  └───────────────────────────────────┘  │
  └──────────────────┬───────────────────────┘
                     │
          ┌──────────┴──────────┐
          │                     │
          ▼                     ▼
   context_{t+1}        reconstruct x_t
   (next time step)     (predict char)
```

### Temporal unrolling (unrolled / own-weights mode)

```
      x_{t-1}              x_t               x_{t+1}
         │                   │                   │
  ┌──────┴──────┐    ┌───────┴──────┐    ┌───────┴──────┐
  │  ctx  │x_{t-1}  │  ctx  │  x_t  │   │  ctx  │x_{t+1}│
  │       ╠═══════╣  │       ╠════════╣   │       ╠════════╣
  │    W_{t-1}    │  │      W_t      │   │     W_{t+1}   │
  │       ╠═══════╣  │       ╠════════╣   │       ╠════════╣
  │  hidden_{t-1} │  │   hidden_t    │   │  hidden_{t+1} │
  └───────┬───────┘  └───────┬────────┘  └───────┬────────┘
          │  context_{t-1}   │  context_t         │  context_{t+1}
          └─────────────────►┘ ──────────────────►┘ ──────► ...
```

Each time step has its **own weight matrix** W_t, b_v_t, b_h_t — the model is
**unrolled**: N positions in a sequence → N separate RBMs.  This lets each
position specialise for the statistical patterns that occur at that offset in a
sequence.  The sequence length T must be fixed and equal to N at training time;
during generation, position indices wrap modulo N.

For comparison, the shared-weights variant (1 entity, `make_layer`) uses a
single W across all time steps — the RTRBM concatenation variant of Sutskever &
Hinton (2007).

---

## Configuration

All hyperparameters live in the **Build model** cell of `moby_rnn.ipynb`:

| Constant | Default | Description |
|---|---|---|
| `T` | 100 | Characters per training sequence |
| `NUM_SEQ` | 2000 | Number of training sequences (first 200k chars) |
| `CONTEXT_SIZE` | 512 | Recurrent hidden / context state size |
| `PRJ_NAME` | `"moby_rnn"` | Checkpoint file prefix |
| `WORK_DIR` | `"results"` | Directory for saved weights |
| `EVAL_CHARS` | 2000 | Characters used for reconstruction accuracy |
| `PRED_CHARS` | 500 | Characters used for next-step prediction |
| `N_GIBBS` | 10 | Gibbs steps in clamped-Gibbs generation |
| `SEED` | `"Call me Ishmael."` | Seed text for free generation |
| `SEQ_IDX` | 42 | Sequence index for the hidden-state trace plot |

Model size with defaults: **(CONTEXT_SIZE + 85) × CONTEXT_SIZE × T parameters** —
e.g. T=100, CONTEXT_SIZE=256 → 100 × 27,136 = **2,713,600 parameters** total.

---

## Notebook walkthrough

| Cell | Title | What it does |
|---|---|---|
| 1 | Imports | Standard + rbm imports |
| 2 | Load text | Reads `data/moby.txt`, builds `char_to_idx` / `idx_to_char` |
| 3 | Encode sequences | One-hot encodes text → `(NUM_SEQ, T, vocab_size)` array |
| 4 | **Build model** | All constants; constructs `StackRnn`; loads checkpoint |
| 5 | Train | Runs CD training; checkpoints every 5% progress |
| 6 | Generation helpers | `predict_next` (clamped Gibbs) and `generate_text` |
| 7 | Reconstruction accuracy | Teacher-forced: how well does `h_t` remember `x_t`? |
| 8 | Next-step prediction | Given `context_{t-1}`, predict `x_t` before seeing it |
| 9 | Free generation | Generate text at temperatures 0.5 / 1.0 / 1.5 |
| 10 | Hidden state trace | Heatmap of context activations over one sequence |
| 11 | Character distribution | Data vs model character frequency comparison |

---

## Training and resuming

Cell 5 saves a checkpoint at every progress report.  Re-run it at any time to
continue training from the last saved state — `model.state_load()` in cell 4
picks it up automatically.

```
results/moby_rnn-0-state.npz      ← RBM at position 0
results/moby_rnn-1-state.npz      ← RBM at position 1
...
results/moby_rnn-99-state.npz     ← RBM at position T-1
```

---

## Text generation

`predict_next` uses **clamped Gibbs sampling**: the context vector is held fixed
while the sensory (character) part of the visible layer runs free for `N_GIBBS`
steps.  The resulting Bernoulli probabilities are temperature-scaled and
normalised to a categorical distribution.

```python
# Generate 500 chars from a seed at temperature 0.8
text = generate_text(rnn, "Call me Ishmael.", length=500, temperature=0.8)
```

Lower temperature → more conservative / repetitive output.
Higher temperature → more diverse / noisier output.

---

## References

- Sutskever & Hinton (2007) — *Learning Multilevel Distributed Representations
  for High-Dimensional Sequences*
- Boulanger-Lewandowski et al. (2012) — *Modeling Temporal Dependencies in
  High-Dimensional Sequences: Application to Polyphonic Music Generation*