Files

T

jensandClaude Sonnet 4.6 cce3943b27 [RNN-RBM] rename to RNN-RBM, add unrolled depth-2 mode, update README

- Rename context33.ipynb → RNN-RBM.ipynb
- Add UNROLL_DEPTH parameter (1 = shared weights, N = alternating entities)
- Training loop uses t % n_layers instead of hardcoded t % 2
- README: RNN-RBM section with signal-flow ASCII art, modes table, parameter table, papers

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

2026-05-31 23:22:00 +02:00

2.9 KiB

Raw Permalink Blame History

RNN-RBM

A Recurrent Temporal RBM in which the visible layer at each time step is [h_{t-1} | x_t] — the previous hidden state (context) concatenated with the current input. Training uses Contrastive Divergence on [h_t | x_{t+1}] to learn next-step prediction.

Signal flow (UNROLL_DEPTH = 2)

        x[0]              x[1]              x[2]              x[3]
          │                 │                 │                 │
          ▼                 ▼                 ▼                 ▼
      ┌───────┐         ┌───────┐         ┌───────┐         ┌───────┐
 0 ──►│       ├─ h[0] ─►│       ├─ h[1] ─►│       ├─ h[2] ─►│       ├─ h[3] ─►
      │  W0   │         │  W1   │         │  W0   │         │  W1   │
      └───────┘         └───────┘         └───────┘         └───────┘
        t=0               t=1               t=2               t=3

  vis[t]  =  [ h[t-1] | x[t] ]    (context ‖ sensory  →  input to W)
  h[t]    =  sigmoid( W · vis[t] + b_h )   (new context  →  passed right)

  Training:  CD on [ h[t] | x[t+1] ]  to predict the next sensory from h[t]

Modes

`UNROLL_DEPTH`	Mode	Behaviour
1	shared weights	one entity reused at every time step; sequences may have any length
N > 1	unrolled	N entities rotate as `layer[t % N]`; each position learns its own W

With UNROLL_DEPTH = 2 the two entities specialise for even and odd positions respectively, doubling parameter count while keeping inference identical to the shared case.

Key parameters (`RNN-RBM.ipynb`)

Parameter	Default	Meaning
`SENSORY_SIZE`	40	vocabulary size (one-hot)
`CONTEXT_SIZE`	256	hidden / context dimension
`UNROLL_DEPTH`	2	number of alternating entities
`NUM_EPOCHS`	100	training epochs

Papers

https://proceedings.mlr.press/v5/sutskever09a.html — The Recurrent Temporal RBM (Sutskever, Hinton, Taylor 2009) https://arxiv.org/abs/1206.6392 — Modeling temporal dependencies with RNN-RBM (Boulanger-Lewandowski et al. 2012)