From ae81e3e340c2b215ebaed7473226e45a7c5ebc01 Mon Sep 17 00:00:00 2001 From: Jens Ahrensfeld Date: Wed, 3 Jun 2026 11:05:48 +0200 Subject: [PATCH] [add] README.rbm.md: Q&A on RBMs, VAEs, Transformers, and autodiff Co-Authored-By: Claude Sonnet 4.6 --- README.rbm.md | 190 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 190 insertions(+) create mode 100644 README.rbm.md diff --git a/README.rbm.md b/README.rbm.md new file mode 100644 index 0000000..0ed6049 --- /dev/null +++ b/README.rbm.md @@ -0,0 +1,190 @@ +# RBM Discussion + +A Q&A covering the design of `Rnn.py` and the broader context of RBMs. + +--- + +**Q: look at the Rnn.png in docs. Implement it as ./Rnn.py** + +The diagram shows an unrolled RNN-RBM with M time steps: + +- `v[0]` = recurrent context (previous hidden state `h`, or zeros at `t=0`) +- `v[1:N]` = sliding window of N one-hot-encoded characters +- `W[t]` = separate RBM weight matrix per time step (unrolled mode) + +Implemented as `Rnn.py` using `StackRnn` in unrolled mode. For `TEXT=" JAY"`, `N_WIN=3`: + +| Step | v[0] | v[1:N] | Weight | +|---|---|---|---| +| t=0 | {0} | [' ',' ','J'] | W[0] | +| t=1 | h₀ | [' ','J','A'] | W[1] | +| t=M-1 | h_{M-2} | ['J','A','Y'] | W[M-1] | + +--- + +**Q: What are the benefits of RBMs compared to other networks?** + +A few key advantages: + +**Unsupervised / generative** — RBMs learn the joint distribution P(v, h) over data, so they can generate new samples and work without labels. Most discriminative networks (CNNs, MLPs) only learn P(label | input). + +**Energy-based probabilistic foundation** — the energy function gives a principled way to reason about uncertainty and to compare how "likely" different inputs are under the model. + +**Efficient training with CD** — Contrastive Divergence avoids backpropagating through time or deep chains. Each layer trains locally, which is biologically more plausible and avoids vanishing gradients. + +**Good feature initializers for deep nets** — historically, stacking RBMs (DBNs) gave much better weight initializations than random init, especially before BatchNorm and ReLU were widespread. Less relevant today. + +**Compact hidden representations** — the binary hidden units act as a learned feature detector; sparsity regularization (L1) produces interpretable, localized filters. + +**The main weaknesses** in comparison: training is slower and less stable than backprop, scaling to high-dimensional data (large images, long sequences) is hard, and modern VAEs/diffusion models have largely replaced them for generative tasks while Transformers dominate sequence modeling. RBMs remain interesting as a research tool and for understanding energy-based learning. + +--- + +**Q: What are VAEs/diffusion models?** + +**VAEs (Variational Autoencoders)** + +An encoder compresses input → latent vector z, a decoder reconstructs input from z. The trick is forcing z to follow a known distribution (usually Gaussian) via a KL-divergence penalty. You can then sample z ~ N(0,1) and decode to generate new data. Training is end-to-end backprop, which makes them faster and easier to scale than RBMs. + +**Diffusion Models** + +Learn to reverse a noise process. Forward: gradually add Gaussian noise to data over T steps until it's pure noise. Reverse: train a neural net to predict and remove the noise step by step. At generation time, start from random noise and denoise T times. Current state of the art for image/audio/video generation (Stable Diffusion, DALL-E 3, Sora). + +**Compared to RBMs** + +| | RBM | VAE | Diffusion | +|---|---|---|---| +| Training | CD (local, approximate) | Backprop + KL loss | Backprop + noise prediction | +| Generation | Gibbs sampling (slow) | Single decoder pass (fast) | T denoising steps (slow-ish) | +| Quality | Moderate | Tends to blur | State of the art | +| Theory | Energy-based | Variational Bayes | Score matching / SDE | + +--- + +**Q: How does pyRBM compare to those?** + +Honestly, pyRBM is not competitive with VAEs or diffusion models for any practical generative task. It's a research/learning codebase, not a production tool. + +**What pyRBM does well** +- Clean, readable implementation of the RBM family (BB, GB, GG, BG variants) +- Stacking into deep nets and the RTRBM sequence model +- CuPy support for GPU acceleration +- Good for understanding energy-based learning from first principles + +**Where it falls short** + +| Capability | pyRBM | VAE/Diffusion | +|---|---|---| +| Image quality | Low — blurry reconstructions | High — sharp, diverse samples | +| Sequence modeling | Short sequences only (RTRBM) | Long-range (Transformers) | +| Scalability | Struggles beyond ~1024 visible units | Scales to megapixel images | +| Training stability | CD is noisy, sensitive to hyperparams | Backprop is well-understood | +| Ecosystem | Standalone, no autodiff | PyTorch/JAX — huge community | + +**The honest summary** + +pyRBM is best thought of as an educational implementation and a testbed for RBM research ideas. If the goal were to build a competitive character-level language model, a small Transformer (even a 1-layer one) would outperform the RTRBM in `Rnn.py` by a wide margin with far less tuning effort. The value here is in understanding the mechanics, not beating benchmarks. + +--- + +**Q: Is the Transformer architecture possible to implement with pyRBM? Or is it too dependent on the basic learning algorithm?** + +**Short answer: no, not meaningfully.** The incompatibility is fundamental, not incidental. + +**Why Transformers don't fit** + +The core Transformer operations — `softmax(QK^T/√d)V`, layer norm, residuals — are deterministic and differentiable. They are trained end-to-end with backpropagation, which pyRBM has no support for at all. You'd need to add autodiff, and at that point you're rebuilding PyTorch inside pyRBM. + +CD (Contrastive Divergence) has no equivalent for these operations. It only works for the specific energy-based forward/backward pass of an RBM. + +**The interesting theoretical bridge** + +The closest legitimate connection is **modern Hopfield networks** (Ramsauer et al., 2020 — "Hopfield Networks is All You Need"). They showed that the Transformer attention mechanism is mathematically equivalent to the update rule of a continuous Hopfield network, which is itself an energy-based model closely related to RBMs. So the *theory* connects, but the *implementation* path still goes through backprop. + +**What you could realistically add to pyRBM** + +If the goal is better sequence modeling, more tractable extensions would be: + +| Extension | Effort | Benefit | +|---|---|---| +| Deeper RTRBM stacking | Low — fits existing `Stack` | More expressive hidden states | +| Larger hidden layer + sparse regularization | Trivial | Better feature quality | +| Hybrid: RBM features → linear classifier | Medium | Supervised tasks | + +Adding a full Transformer would mean replacing the learning algorithm entirely — at which point you no longer have an RBM codebase. + +--- + +**Q: What is autodiff?** + +**Autodiff (automatic differentiation)** is a technique for computing exact derivatives of any computation expressed as code, automatically. + +**The core idea** + +Every computation is a chain of primitive operations (add, multiply, exp, etc.). Each primitive has a known derivative. Autodiff tracks these operations and applies the chain rule mechanically to compute the gradient of any output with respect to any input — exactly, not approximately. + +**Two modes** + +- **Forward mode** — propagates derivatives forward through the computation alongside the values. Efficient when inputs are few. +- **Reverse mode** — propagates derivatives backward from the output. This is what neural networks use. You may know it as **backpropagation** — backprop is just reverse-mode autodiff applied to a loss function. + +**What it looks like in practice** + +```python +# PyTorch — autodiff is implicit +x = torch.tensor(2.0, requires_grad=True) +y = x ** 3 + 2 * x # forward pass builds a computation graph +y.backward() # reverse pass computes dy/dx automatically +print(x.grad) # → 14.0 (3x² + 2 at x=2) +``` + +pyRBM has none of this. Every gradient in pyRBM is derived analytically by hand (the CD update rules in `train.py`) and hard-coded for the specific RBM energy function. That's why you can't just plug in an attention layer — there's no machinery to differentiate through it. + +**Why it matters for Transformers** + +A Transformer has millions of parameters spread across many non-trivial operations. Deriving and coding all those gradients by hand would be essentially impossible. Autodiff frameworks (PyTorch, JAX, TensorFlow) make it trivial — you just define the forward pass and get gradients for free. + +--- + +**Q: Can pyRBM be extended with autodiff support?** + +Yes, and it's actually a reasonable extension. The cleanest approach would be to replace the numpy/cupy arrays with PyTorch tensors, since PyTorch tensors carry gradient information transparently. + +**What would change** + +`rbm/matrix.py` is the central switching point — it currently picks numpy or cupy: + +```python +# current +import cupy as np + +# with autodiff +import torch +``` + +Then `RbmState` (the weight matrices `w_hv`, `b_v`, `b_h`) would become `nn.Parameter` tensors, and the forward/backward passes in `entity.py` would be differentiable automatically. + +**What you'd keep** + +The CD training in `train.py` could stay as-is — CD is still a valid training signal, just now computed through differentiable ops. You'd gain the *option* to use backprop in addition, not a forced replacement. + +**What you'd gain** + +- Attention layers, layer norm, residuals — anything PyTorch supports — could be added and trained +- Gradient flow through stacked layers (currently each layer trains independently) +- Access to PyTorch optimizers (Adam, etc.) instead of the hand-coded momentum in `entity.py` + +**The compatibility shim already exists** + +`src/compat/torch.py` — pyRBM already has a PyTorch bridge. It's worth reading what's already there before designing anything new. + +**Rough effort estimate** + +| Task | Effort | +|---|---| +| Replace matrix backend with torch tensors | Medium | +| Refactor `RbmState` → `nn.Module` | Medium | +| Keep CD working on top of torch | Low | +| Add new differentiable components (attention etc.) | Low once the above is done | + +It's a meaningful but tractable refactor — not a rewrite from scratch.