# RNN-RBM

A Recurrent Temporal RBM in which the visible layer at each time step is
`[h_{t-1} | x_t]` — the previous hidden state (context) concatenated with the
current input.  Training uses Contrastive Divergence on `[h_t | x_{t+1}]` to
learn next-step prediction.

## Signal flow (UNROLL_DEPTH = 2)

```
        x[0]              x[1]              x[2]              x[3]
          │                 │                 │                 │
          ▼                 ▼                 ▼                 ▼
      ┌───────┐         ┌───────┐         ┌───────┐         ┌───────┐
 0 ──►│       ├─ h[0] ─►│       ├─ h[1] ─►│       ├─ h[2] ─►│       ├─ h[3] ─►
      │  W0   │         │  W1   │         │  W0   │         │  W1   │
      └───────┘         └───────┘         └───────┘         └───────┘
        t=0               t=1               t=2               t=3

  vis[t]  =  [ h[t-1] | x[t] ]    (context ‖ sensory  →  input to W)
  h[t]    =  sigmoid( W · vis[t] + b_h )   (new context  →  passed right)

  Training:  CD on [ h[t] | x[t+1] ]  to predict the next sensory from h[t]
```

## Modes

| `UNROLL_DEPTH` | Mode | Behaviour |
|---|---|---|
| 1 | shared weights | one entity reused at every time step; sequences may have any length |
| N > 1 | unrolled | N entities rotate as `layer[t % N]`; each position learns its own W |

With `UNROLL_DEPTH = 2` the two entities specialise for even and odd positions
respectively, doubling parameter count while keeping inference identical to the
shared case.

## Key parameters (`RNN-RBM.ipynb`)

| Parameter | Default | Meaning |
|---|---|---|
| `SENSORY_SIZE` | 40 | vocabulary size (one-hot) |
| `CONTEXT_SIZE` | 256 | hidden / context dimension |
| `UNROLL_DEPTH` | 2 | number of alternating entities |
| `NUM_EPOCHS` | 100 | training epochs |

## Papers
https://proceedings.mlr.press/v5/sutskever09a.html — The Recurrent Temporal RBM (Sutskever, Hinton, Taylor 2009)
https://arxiv.org/abs/1206.6392 — Modeling temporal dependencies with RNN-RBM (Boulanger-Lewandowski et al. 2012)

---

# GRBM
## Paper
https://medium.com/@rtdcunha/gaussian-bernoulli-restricted-boltzmann-machines-4a68b8765485
https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0171015&type=printable

## Code
https://github.com/DSL-Lab/GRBM/tree/main

# CRBM 
## Paper
https://www.ee.nthu.edu.tw/hchen/pubs/iee2003.pdf

# Regularization
https://benihime91.github.io/blog/machinelearning/deeplearning/python3.x/tensorflow2.x/2020/10/08/adamW.html
https://medium.com/analytics-vidhya/l1-vs-l2-regularization-which-is-better-d01068e6658c
https://jamesmccaffreyblog.com/2019/05/09/the-difference-between-neural-network-l2-regularization-and-weight-decay