pyRBM/README.rbm.md

RBM Discussion

A Q&A covering the design of Rnn.py and the broader context of RBMs.

---

Q: look at the Rnn.png in docs. Implement it as ./Rnn.py

The diagram shows an unrolled RNN-RBM with M time steps:

• v[0] = recurrent context (previous hidden state h, or zeros at t=0)
• v[1:N] = sliding window of N one-hot-encoded characters
• W[t] = separate RBM weight matrix per time step (unrolled mode)

Implemented as Rnn.py using StackRnn in unrolled mode. For TEXT=" JAY", N_WIN=3:

Step	v[0]	v[1:N]	Weight
t=0	{0}	[' ',' ','J']	W[0]
t=1	h₀	[' ','J','A']	W[1]
t=M-1	h_{M-2}	['J','A','Y']	W[M-1]

---

Q: What are the benefits of RBMs compared to other networks?

A few key advantages:

Unsupervised / generative — RBMs learn the joint distribution P(v, h) over data, so they can generate new samples and work without labels. Most discriminative networks (CNNs, MLPs) only learn P(label | input).

Energy-based probabilistic foundation — the energy function gives a principled way to reason about uncertainty and to compare how "likely" different inputs are under the model.

Efficient training with CD — Contrastive Divergence avoids backpropagating through time or deep chains. Each layer trains locally, which is biologically more plausible and avoids vanishing gradients.

Good feature initializers for deep nets — historically, stacking RBMs (DBNs) gave much better weight initializations than random init, especially before BatchNorm and ReLU were widespread. Less relevant today.

Compact hidden representations — the binary hidden units act as a learned feature detector; sparsity regularization (L1) produces interpretable, localized filters.

The main weaknesses in comparison: training is slower and less stable than backprop, scaling to high-dimensional data (large images, long sequences) is hard, and modern VAEs/diffusion models have largely replaced them for generative tasks while Transformers dominate sequence modeling. RBMs remain interesting as a research tool and for understanding energy-based learning.

---

Q: What are VAEs/diffusion models?

VAEs (Variational Autoencoders)

An encoder compresses input → latent vector z, a decoder reconstructs input from z. The trick is forcing z to follow a known distribution (usually Gaussian) via a KL-divergence penalty. You can then sample z ~ N(0,1) and decode to generate new data. Training is end-to-end backprop, which makes them faster and easier to scale than RBMs.

Diffusion Models

Learn to reverse a noise process. Forward: gradually add Gaussian noise to data over T steps until it's pure noise. Reverse: train a neural net to predict and remove the noise step by step. At generation time, start from random noise and denoise T times. Current state of the art for image/audio/video generation (Stable Diffusion, DALL-E 3, Sora).

Compared to RBMs

	RBM	VAE	Diffusion
Training	CD (local, approximate)	Backprop + KL loss	Backprop + noise prediction
Generation	Gibbs sampling (slow)	Single decoder pass (fast)	T denoising steps (slow-ish)
Quality	Moderate	Tends to blur	State of the art
Theory	Energy-based	Variational Bayes	Score matching / SDE

---

Q: How does pyRBM compare to those?

Honestly, pyRBM is not competitive with VAEs or diffusion models for any practical generative task. It's a research/learning codebase, not a production tool.

What pyRBM does well

• Clean, readable implementation of the RBM family (BB, GB, GG, BG variants)
• Stacking into deep nets and the RTRBM sequence model
• CuPy support for GPU acceleration
• Good for understanding energy-based learning from first principles

Where it falls short

Capability	pyRBM	VAE/Diffusion
Image quality	Low — blurry reconstructions	High — sharp, diverse samples
Sequence modeling	Short sequences only (RTRBM)	Long-range (Transformers)
Scalability	Struggles beyond ~1024 visible units	Scales to megapixel images
Training stability	CD is noisy, sensitive to hyperparams	Backprop is well-understood
Ecosystem	Standalone, no autodiff	PyTorch/JAX — huge community

The honest summary

pyRBM is best thought of as an educational implementation and a testbed for RBM research ideas. If the goal were to build a competitive character-level language model, a small Transformer (even a 1-layer one) would outperform the RTRBM in Rnn.py by a wide margin with far less tuning effort. The value here is in understanding the mechanics, not beating benchmarks.

---

Q: Is the Transformer architecture possible to implement with pyRBM? Or is it too dependent on the basic learning algorithm?

Short answer: no, not meaningfully. The incompatibility is fundamental, not incidental.

Why Transformers don't fit

The core Transformer operations — softmax(QK^T/√d)V, layer norm, residuals — are deterministic and differentiable. They are trained end-to-end with backpropagation, which pyRBM has no support for at all. You'd need to add autodiff, and at that point you're rebuilding PyTorch inside pyRBM.

CD (Contrastive Divergence) has no equivalent for these operations. It only works for the specific energy-based forward/backward pass of an RBM.

The interesting theoretical bridge

The closest legitimate connection is modern Hopfield networks (Ramsauer et al., 2020 — "Hopfield Networks is All You Need"). They showed that the Transformer attention mechanism is mathematically equivalent to the update rule of a continuous Hopfield network, which is itself an energy-based model closely related to RBMs. So the theory connects, but the implementation path still goes through backprop.

What you could realistically add to pyRBM

If the goal is better sequence modeling, more tractable extensions would be:

Extension	Effort	Benefit
Deeper RTRBM stacking	Low — fits existing Stack	More expressive hidden states
Larger hidden layer + sparse regularization	Trivial	Better feature quality
Hybrid: RBM features → linear classifier	Medium	Supervised tasks

Adding a full Transformer would mean replacing the learning algorithm entirely — at which point you no longer have an RBM codebase.

---

Q: What is autodiff?

Autodiff (automatic differentiation) is a technique for computing exact derivatives of any computation expressed as code, automatically.

The core idea

Every computation is a chain of primitive operations (add, multiply, exp, etc.). Each primitive has a known derivative. Autodiff tracks these operations and applies the chain rule mechanically to compute the gradient of any output with respect to any input — exactly, not approximately.

Two modes

• Forward mode — propagates derivatives forward through the computation alongside the values. Efficient when inputs are few.
• Reverse mode — propagates derivatives backward from the output. This is what neural networks use. You may know it as backpropagation — backprop is just reverse-mode autodiff applied to a loss function.

What it looks like in practice

# PyTorch — autodiff is implicit
x = torch.tensor(2.0, requires_grad=True)
y = x ** 3 + 2 * x        # forward pass builds a computation graph
y.backward()               # reverse pass computes dy/dx automatically
print(x.grad)              # → 14.0  (3x² + 2 at x=2)

pyRBM has none of this. Every gradient in pyRBM is derived analytically by hand (the CD update rules in train.py) and hard-coded for the specific RBM energy function. That's why you can't just plug in an attention layer — there's no machinery to differentiate through it.

Why it matters for Transformers

A Transformer has millions of parameters spread across many non-trivial operations. Deriving and coding all those gradients by hand would be essentially impossible. Autodiff frameworks (PyTorch, JAX, TensorFlow) make it trivial — you just define the forward pass and get gradients for free.

---

Q: Can pyRBM be extended with autodiff support?

Yes, and it's actually a reasonable extension. The cleanest approach would be to replace the numpy/cupy arrays with PyTorch tensors, since PyTorch tensors carry gradient information transparently.

What would change

rbm/matrix.py is the central switching point — it currently picks numpy or cupy:

# current
import cupy as np

# with autodiff
import torch

Then RbmState (the weight matrices w_hv, b_v, b_h) would become nn.Parameter tensors, and the forward/backward passes in entity.py would be differentiable automatically.

What you'd keep

The CD training in train.py could stay as-is — CD is still a valid training signal, just now computed through differentiable ops. You'd gain the option to use backprop in addition, not a forced replacement.

What you'd gain

• Attention layers, layer norm, residuals — anything PyTorch supports — could be added and trained
• Gradient flow through stacked layers (currently each layer trains independently)
• Access to PyTorch optimizers (Adam, etc.) instead of the hand-coded momentum in entity.py

The compatibility shim already exists

src/compat/torch.py — pyRBM already has a PyTorch bridge. It's worth reading what's already there before designing anything new.

Rough effort estimate

Task	Effort
Replace matrix backend with torch tensors	Medium
Refactor RbmState → nn.Module	Medium
Keep CD working on top of torch	Low
Add new differentiable components (attention etc.)	Low once the above is done

It's a meaningful but tractable refactor — not a rewrite from scratch.

---

Q: Which network architecture is commonly used for high-res image/video generation?

Diffusion models with a U-Net or Transformer backbone are the current standard.

U-Net (older, still widely used)

The original diffusion backbone. An encoder downsamples the image through convolutional blocks, a bottleneck processes the compressed representation, then a decoder upsamples back — with skip connections between matching encoder/decoder levels. Attention layers are injected at lower resolutions. Used in the original Stable Diffusion (v1, v2).

DiT — Diffusion Transformer (newer, becoming dominant)

Replaces the U-Net with a pure Transformer operating on image patches. Each patch is a token; the full attention mechanism can model long-range dependencies across the whole image. Sora (video), Stable Diffusion 3, and FLUX all use this architecture. Scales better with compute than U-Net.

For video specifically

The same DiT idea extended to space + time: treat each patch at each frame as a token, apply 3D attention. Sora's key insight was training a single DiT on videos of varying resolution and duration with no architectural changes — just more tokens.

Why not GANs?

GANs (Generator + Discriminator) dominated image generation around 2018–2021 (StyleGAN, BigGAN). They produce sharp images fast but suffer from training instability and mode collapse. Diffusion models surpassed them in quality and diversity around 2022 and have largely taken over.

Summary

Architecture	Example models	Status
U-Net diffusion	SD 1.x, DALL-E 2	Mature, widely deployed
DiT diffusion	SD3, FLUX, Sora	Current frontier
GAN	StyleGAN3	Largely superseded

---

Q: Which network architecture is used for code generators, like Claude Code?

Decoder-only Transformer — the same architecture behind GPT, Claude, Gemini, and essentially every large language model used for code generation today.

How it works

Tokens (subword pieces of text/code) are fed in left to right. Each token attends to all previous tokens via causal self-attention (masked so it can't peek ahead). The model predicts the next token at each position. That's it — code generation is just next-token prediction applied to source code.

Why decoder-only wins for generation

• Encoder-decoder (like the original Transformer, T5) was designed for translation: encode a full input, then decode an output. Good for fixed input→output tasks.
• Decoder-only generates autoregressively with no separate encoding step, which is simpler and scales better to very long contexts (entire files, repos).

What makes code models specifically good at code

Not the architecture — the training data. Models trained heavily on GitHub, Stack Overflow, and documentation learn syntax, APIs, idioms, and common patterns. The architecture is identical to a text model.

At scale

Claude, GPT-4, Gemini, and similar models are decoder-only Transformers with:

• Billions to trillions of parameters
• Context windows of 100k–1M+ tokens
• Trained on mixed text + code corpora
• Fine-tuned with RLHF / constitutional AI for instruction following

Summary relative to this conversation

Architecture	Used for
Decoder-only Transformer	LLMs, code generation (Claude, GPT)
DiT (Diffusion Transformer)	Image/video generation (Sora, FLUX)
U-Net Diffusion	Image generation (SD 1.x)
RBM / RTRBM	Energy-based learning, research