258 lines
13 KiB
Markdown
258 lines
13 KiB
Markdown
# RBM Discussion
|
||
|
||
A Q&A covering the design of `Rnn.py` and the broader context of RBMs.
|
||
|
||
---
|
||
|
||
**Q: look at the Rnn.png in docs. Implement it as ./Rnn.py**
|
||
|
||
The diagram shows an unrolled RNN-RBM with M time steps:
|
||
|
||
- `v[0]` = recurrent context (previous hidden state `h`, or zeros at `t=0`)
|
||
- `v[1:N]` = sliding window of N one-hot-encoded characters
|
||
- `W[t]` = separate RBM weight matrix per time step (unrolled mode)
|
||
|
||
Implemented as `Rnn.py` using `StackRnn` in unrolled mode. For `TEXT=" JAY"`, `N_WIN=3`:
|
||
|
||
| Step | v[0] | v[1:N] | Weight |
|
||
|---|---|---|---|
|
||
| t=0 | {0} | [' ',' ','J'] | W[0] |
|
||
| t=1 | h₀ | [' ','J','A'] | W[1] |
|
||
| t=M-1 | h_{M-2} | ['J','A','Y'] | W[M-1] |
|
||
|
||
---
|
||
|
||
**Q: What are the benefits of RBMs compared to other networks?**
|
||
|
||
A few key advantages:
|
||
|
||
**Unsupervised / generative** — RBMs learn the joint distribution P(v, h) over data, so they can generate new samples and work without labels. Most discriminative networks (CNNs, MLPs) only learn P(label | input).
|
||
|
||
**Energy-based probabilistic foundation** — the energy function gives a principled way to reason about uncertainty and to compare how "likely" different inputs are under the model.
|
||
|
||
**Efficient training with CD** — Contrastive Divergence avoids backpropagating through time or deep chains. Each layer trains locally, which is biologically more plausible and avoids vanishing gradients.
|
||
|
||
**Good feature initializers for deep nets** — historically, stacking RBMs (DBNs) gave much better weight initializations than random init, especially before BatchNorm and ReLU were widespread. Less relevant today.
|
||
|
||
**Compact hidden representations** — the binary hidden units act as a learned feature detector; sparsity regularization (L1) produces interpretable, localized filters.
|
||
|
||
**The main weaknesses** in comparison: training is slower and less stable than backprop, scaling to high-dimensional data (large images, long sequences) is hard, and modern VAEs/diffusion models have largely replaced them for generative tasks while Transformers dominate sequence modeling. RBMs remain interesting as a research tool and for understanding energy-based learning.
|
||
|
||
---
|
||
|
||
**Q: What are VAEs/diffusion models?**
|
||
|
||
**VAEs (Variational Autoencoders)**
|
||
|
||
An encoder compresses input → latent vector z, a decoder reconstructs input from z. The trick is forcing z to follow a known distribution (usually Gaussian) via a KL-divergence penalty. You can then sample z ~ N(0,1) and decode to generate new data. Training is end-to-end backprop, which makes them faster and easier to scale than RBMs.
|
||
|
||
**Diffusion Models**
|
||
|
||
Learn to reverse a noise process. Forward: gradually add Gaussian noise to data over T steps until it's pure noise. Reverse: train a neural net to predict and remove the noise step by step. At generation time, start from random noise and denoise T times. Current state of the art for image/audio/video generation (Stable Diffusion, DALL-E 3, Sora).
|
||
|
||
**Compared to RBMs**
|
||
|
||
| | RBM | VAE | Diffusion |
|
||
|---|---|---|---|
|
||
| Training | CD (local, approximate) | Backprop + KL loss | Backprop + noise prediction |
|
||
| Generation | Gibbs sampling (slow) | Single decoder pass (fast) | T denoising steps (slow-ish) |
|
||
| Quality | Moderate | Tends to blur | State of the art |
|
||
| Theory | Energy-based | Variational Bayes | Score matching / SDE |
|
||
|
||
---
|
||
|
||
**Q: How does pyRBM compare to those?**
|
||
|
||
Honestly, pyRBM is not competitive with VAEs or diffusion models for any practical generative task. It's a research/learning codebase, not a production tool.
|
||
|
||
**What pyRBM does well**
|
||
- Clean, readable implementation of the RBM family (BB, GB, GG, BG variants)
|
||
- Stacking into deep nets and the RTRBM sequence model
|
||
- CuPy support for GPU acceleration
|
||
- Good for understanding energy-based learning from first principles
|
||
|
||
**Where it falls short**
|
||
|
||
| Capability | pyRBM | VAE/Diffusion |
|
||
|---|---|---|
|
||
| Image quality | Low — blurry reconstructions | High — sharp, diverse samples |
|
||
| Sequence modeling | Short sequences only (RTRBM) | Long-range (Transformers) |
|
||
| Scalability | Struggles beyond ~1024 visible units | Scales to megapixel images |
|
||
| Training stability | CD is noisy, sensitive to hyperparams | Backprop is well-understood |
|
||
| Ecosystem | Standalone, no autodiff | PyTorch/JAX — huge community |
|
||
|
||
**The honest summary**
|
||
|
||
pyRBM is best thought of as an educational implementation and a testbed for RBM research ideas. If the goal were to build a competitive character-level language model, a small Transformer (even a 1-layer one) would outperform the RTRBM in `Rnn.py` by a wide margin with far less tuning effort. The value here is in understanding the mechanics, not beating benchmarks.
|
||
|
||
---
|
||
|
||
**Q: Is the Transformer architecture possible to implement with pyRBM? Or is it too dependent on the basic learning algorithm?**
|
||
|
||
**Short answer: no, not meaningfully.** The incompatibility is fundamental, not incidental.
|
||
|
||
**Why Transformers don't fit**
|
||
|
||
The core Transformer operations — `softmax(QK^T/√d)V`, layer norm, residuals — are deterministic and differentiable. They are trained end-to-end with backpropagation, which pyRBM has no support for at all. You'd need to add autodiff, and at that point you're rebuilding PyTorch inside pyRBM.
|
||
|
||
CD (Contrastive Divergence) has no equivalent for these operations. It only works for the specific energy-based forward/backward pass of an RBM.
|
||
|
||
**The interesting theoretical bridge**
|
||
|
||
The closest legitimate connection is **modern Hopfield networks** (Ramsauer et al., 2020 — "Hopfield Networks is All You Need"). They showed that the Transformer attention mechanism is mathematically equivalent to the update rule of a continuous Hopfield network, which is itself an energy-based model closely related to RBMs. So the *theory* connects, but the *implementation* path still goes through backprop.
|
||
|
||
**What you could realistically add to pyRBM**
|
||
|
||
If the goal is better sequence modeling, more tractable extensions would be:
|
||
|
||
| Extension | Effort | Benefit |
|
||
|---|---|---|
|
||
| Deeper RTRBM stacking | Low — fits existing `Stack` | More expressive hidden states |
|
||
| Larger hidden layer + sparse regularization | Trivial | Better feature quality |
|
||
| Hybrid: RBM features → linear classifier | Medium | Supervised tasks |
|
||
|
||
Adding a full Transformer would mean replacing the learning algorithm entirely — at which point you no longer have an RBM codebase.
|
||
|
||
---
|
||
|
||
**Q: What is autodiff?**
|
||
|
||
**Autodiff (automatic differentiation)** is a technique for computing exact derivatives of any computation expressed as code, automatically.
|
||
|
||
**The core idea**
|
||
|
||
Every computation is a chain of primitive operations (add, multiply, exp, etc.). Each primitive has a known derivative. Autodiff tracks these operations and applies the chain rule mechanically to compute the gradient of any output with respect to any input — exactly, not approximately.
|
||
|
||
**Two modes**
|
||
|
||
- **Forward mode** — propagates derivatives forward through the computation alongside the values. Efficient when inputs are few.
|
||
- **Reverse mode** — propagates derivatives backward from the output. This is what neural networks use. You may know it as **backpropagation** — backprop is just reverse-mode autodiff applied to a loss function.
|
||
|
||
**What it looks like in practice**
|
||
|
||
```python
|
||
# PyTorch — autodiff is implicit
|
||
x = torch.tensor(2.0, requires_grad=True)
|
||
y = x ** 3 + 2 * x # forward pass builds a computation graph
|
||
y.backward() # reverse pass computes dy/dx automatically
|
||
print(x.grad) # → 14.0 (3x² + 2 at x=2)
|
||
```
|
||
|
||
pyRBM has none of this. Every gradient in pyRBM is derived analytically by hand (the CD update rules in `train.py`) and hard-coded for the specific RBM energy function. That's why you can't just plug in an attention layer — there's no machinery to differentiate through it.
|
||
|
||
**Why it matters for Transformers**
|
||
|
||
A Transformer has millions of parameters spread across many non-trivial operations. Deriving and coding all those gradients by hand would be essentially impossible. Autodiff frameworks (PyTorch, JAX, TensorFlow) make it trivial — you just define the forward pass and get gradients for free.
|
||
|
||
---
|
||
|
||
**Q: Can pyRBM be extended with autodiff support?**
|
||
|
||
Yes, and it's actually a reasonable extension. The cleanest approach would be to replace the numpy/cupy arrays with PyTorch tensors, since PyTorch tensors carry gradient information transparently.
|
||
|
||
**What would change**
|
||
|
||
`rbm/matrix.py` is the central switching point — it currently picks numpy or cupy:
|
||
|
||
```python
|
||
# current
|
||
import cupy as np
|
||
|
||
# with autodiff
|
||
import torch
|
||
```
|
||
|
||
Then `RbmState` (the weight matrices `w_hv`, `b_v`, `b_h`) would become `nn.Parameter` tensors, and the forward/backward passes in `entity.py` would be differentiable automatically.
|
||
|
||
**What you'd keep**
|
||
|
||
The CD training in `train.py` could stay as-is — CD is still a valid training signal, just now computed through differentiable ops. You'd gain the *option* to use backprop in addition, not a forced replacement.
|
||
|
||
**What you'd gain**
|
||
|
||
- Attention layers, layer norm, residuals — anything PyTorch supports — could be added and trained
|
||
- Gradient flow through stacked layers (currently each layer trains independently)
|
||
- Access to PyTorch optimizers (Adam, etc.) instead of the hand-coded momentum in `entity.py`
|
||
|
||
**The compatibility shim already exists**
|
||
|
||
`src/compat/torch.py` — pyRBM already has a PyTorch bridge. It's worth reading what's already there before designing anything new.
|
||
|
||
**Rough effort estimate**
|
||
|
||
| Task | Effort |
|
||
|---|---|
|
||
| Replace matrix backend with torch tensors | Medium |
|
||
| Refactor `RbmState` → `nn.Module` | Medium |
|
||
| Keep CD working on top of torch | Low |
|
||
| Add new differentiable components (attention etc.) | Low once the above is done |
|
||
|
||
It's a meaningful but tractable refactor — not a rewrite from scratch.
|
||
|
||
---
|
||
|
||
**Q: Which network architecture is commonly used for high-res image/video generation?**
|
||
|
||
**Diffusion models with a U-Net or Transformer backbone** are the current standard.
|
||
|
||
**U-Net (older, still widely used)**
|
||
|
||
The original diffusion backbone. An encoder downsamples the image through convolutional blocks, a bottleneck processes the compressed representation, then a decoder upsamples back — with skip connections between matching encoder/decoder levels. Attention layers are injected at lower resolutions. Used in the original Stable Diffusion (v1, v2).
|
||
|
||
**DiT — Diffusion Transformer (newer, becoming dominant)**
|
||
|
||
Replaces the U-Net with a pure Transformer operating on image patches. Each patch is a token; the full attention mechanism can model long-range dependencies across the whole image. Sora (video), Stable Diffusion 3, and FLUX all use this architecture. Scales better with compute than U-Net.
|
||
|
||
**For video specifically**
|
||
|
||
The same DiT idea extended to space + time: treat each patch at each frame as a token, apply 3D attention. Sora's key insight was training a single DiT on videos of varying resolution and duration with no architectural changes — just more tokens.
|
||
|
||
**Why not GANs?**
|
||
|
||
GANs (Generator + Discriminator) dominated image generation around 2018–2021 (StyleGAN, BigGAN). They produce sharp images fast but suffer from training instability and mode collapse. Diffusion models surpassed them in quality and diversity around 2022 and have largely taken over.
|
||
|
||
**Summary**
|
||
|
||
| Architecture | Example models | Status |
|
||
|---|---|---|
|
||
| U-Net diffusion | SD 1.x, DALL-E 2 | Mature, widely deployed |
|
||
| DiT diffusion | SD3, FLUX, Sora | Current frontier |
|
||
| GAN | StyleGAN3 | Largely superseded |
|
||
|
||
---
|
||
|
||
**Q: Which network architecture is used for code generators, like Claude Code?**
|
||
|
||
**Decoder-only Transformer** — the same architecture behind GPT, Claude, Gemini, and essentially every large language model used for code generation today.
|
||
|
||
**How it works**
|
||
|
||
Tokens (subword pieces of text/code) are fed in left to right. Each token attends to all previous tokens via **causal self-attention** (masked so it can't peek ahead). The model predicts the next token at each position. That's it — code generation is just next-token prediction applied to source code.
|
||
|
||
**Why decoder-only wins for generation**
|
||
|
||
- Encoder-decoder (like the original Transformer, T5) was designed for translation: encode a full input, then decode an output. Good for fixed input→output tasks.
|
||
- Decoder-only generates autoregressively with no separate encoding step, which is simpler and scales better to very long contexts (entire files, repos).
|
||
|
||
**What makes code models specifically good at code**
|
||
|
||
Not the architecture — the training data. Models trained heavily on GitHub, Stack Overflow, and documentation learn syntax, APIs, idioms, and common patterns. The architecture is identical to a text model.
|
||
|
||
**At scale**
|
||
|
||
Claude, GPT-4, Gemini, and similar models are decoder-only Transformers with:
|
||
- Billions to trillions of parameters
|
||
- Context windows of 100k–1M+ tokens
|
||
- Trained on mixed text + code corpora
|
||
- Fine-tuned with RLHF / constitutional AI for instruction following
|
||
|
||
**Summary relative to this conversation**
|
||
|
||
| Architecture | Used for |
|
||
|---|---|
|
||
| Decoder-only Transformer | LLMs, code generation (Claude, GPT) |
|
||
| DiT (Diffusion Transformer) | Image/video generation (Sora, FLUX) |
|
||
| U-Net Diffusion | Image generation (SD 1.x) |
|
||
| RBM / RTRBM | Energy-based learning, research |
|
||
|