jensandClaude Sonnet 4.6
cce3943b27
- Rename context33.ipynb → RNN-RBM.ipynb - Add UNROLL_DEPTH parameter (1 = shared weights, N = alternating entities) - Training loop uses t % n_layers instead of hardcoded t % 2 - README: RNN-RBM section with signal-flow ASCII art, modes table, parameter table, papers Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
68 lines
2.9 KiB
Markdown
68 lines
2.9 KiB
Markdown
# RNN-RBM
|
|
|
|
A Recurrent Temporal RBM in which the visible layer at each time step is
|
|
`[h_{t-1} | x_t]` — the previous hidden state (context) concatenated with the
|
|
current input. Training uses Contrastive Divergence on `[h_t | x_{t+1}]` to
|
|
learn next-step prediction.
|
|
|
|
## Signal flow (UNROLL_DEPTH = 2)
|
|
|
|
```
|
|
x[0] x[1] x[2] x[3]
|
|
│ │ │ │
|
|
▼ ▼ ▼ ▼
|
|
┌───────┐ ┌───────┐ ┌───────┐ ┌───────┐
|
|
0 ──►│ ├─ h[0] ─►│ ├─ h[1] ─►│ ├─ h[2] ─►│ ├─ h[3] ─►
|
|
│ W0 │ │ W1 │ │ W0 │ │ W1 │
|
|
└───────┘ └───────┘ └───────┘ └───────┘
|
|
t=0 t=1 t=2 t=3
|
|
|
|
vis[t] = [ h[t-1] | x[t] ] (context ‖ sensory → input to W)
|
|
h[t] = sigmoid( W · vis[t] + b_h ) (new context → passed right)
|
|
|
|
Training: CD on [ h[t] | x[t+1] ] to predict the next sensory from h[t]
|
|
```
|
|
|
|
## Modes
|
|
|
|
| `UNROLL_DEPTH` | Mode | Behaviour |
|
|
|---|---|---|
|
|
| 1 | shared weights | one entity reused at every time step; sequences may have any length |
|
|
| N > 1 | unrolled | N entities rotate as `layer[t % N]`; each position learns its own W |
|
|
|
|
With `UNROLL_DEPTH = 2` the two entities specialise for even and odd positions
|
|
respectively, doubling parameter count while keeping inference identical to the
|
|
shared case.
|
|
|
|
## Key parameters (`RNN-RBM.ipynb`)
|
|
|
|
| Parameter | Default | Meaning |
|
|
|---|---|---|
|
|
| `SENSORY_SIZE` | 40 | vocabulary size (one-hot) |
|
|
| `CONTEXT_SIZE` | 256 | hidden / context dimension |
|
|
| `UNROLL_DEPTH` | 2 | number of alternating entities |
|
|
| `NUM_EPOCHS` | 100 | training epochs |
|
|
|
|
## Papers
|
|
https://proceedings.mlr.press/v5/sutskever09a.html — The Recurrent Temporal RBM (Sutskever, Hinton, Taylor 2009)
|
|
https://arxiv.org/abs/1206.6392 — Modeling temporal dependencies with RNN-RBM (Boulanger-Lewandowski et al. 2012)
|
|
|
|
---
|
|
|
|
# GRBM
|
|
## Paper
|
|
https://medium.com/@rtdcunha/gaussian-bernoulli-restricted-boltzmann-machines-4a68b8765485
|
|
https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0171015&type=printable
|
|
|
|
## Code
|
|
https://github.com/DSL-Lab/GRBM/tree/main
|
|
|
|
# CRBM
|
|
## Paper
|
|
https://www.ee.nthu.edu.tw/hchen/pubs/iee2003.pdf
|
|
|
|
# Regularization
|
|
https://benihime91.github.io/blog/machinelearning/deeplearning/python3.x/tensorflow2.x/2020/10/08/adamW.html
|
|
https://medium.com/analytics-vidhya/l1-vs-l2-regularization-which-is-better-d01068e6658c
|
|
https://jamesmccaffreyblog.com/2019/05/09/the-difference-between-neural-network-l2-regularization-and-weight-decay
|