[StackRnn] - temporal-shift padding, joint training, vocab reduction, TEMPORAL_DEPTH=64

stack_rnn.py:
  - _train_unrolled: C++-style temporal-shift padding — flatten (num_seq, T, s)
    → (N, s), append T-1 zero rows, layer t trains on batch[t:N+t]; joint
    training replaces greedy sequential (all layers update each epoch)
  - Reverted sequential/greedy training path (poor next-step prediction)

moby_rnn.ipynb:
  - Vocabulary reduced 85 → 40 chars (A-Z, 0-9, space, .!?)
  - T renamed to TEMPORAL_DEPTH throughout
  - TEMPORAL_DEPTH increased 5 → 64; CONTEXT_SIZE 128
  - SEED "Call me Ishmael." now falls within training data coverage

state.py:
  - Remove per-file load/save print messages (too noisy with 64-layer checkpoints)

README_moby_rnn.md:
  - Update vocab, constants, parameter count, checkpoint listing
  - Add temporal-shift padding and joint training sections
  - Clarify reconstruction accuracy vs next-step prediction as honest metric

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

README_moby_rnn.md

@@ -12,7 +12,7 @@ of a **context** vector (the previous hidden state) and the current sensory inpu
 ### Single time step
 
 ```
-  Sensory input x_t  (one-hot, vocab_size = 85)
+  Sensory input x_t  (one-hot, vocab_size = 40)
          │
          ▼
   ┌──────────────────────────────────────────┐
@@ -22,7 +22,7 @@ of a **context** vector (the previous hidden state) and the current sensory inpu
   │  │  (CONTEXT_SIZE) │  (vocab_size)  │   │
   │  └─────────────────┴────────────────┘   │
   │                  │                       │
-  │        W  (shared across time)           │
+  │              W_t, b_v_t, b_h_t           │
   │                  │                       │
   │  ┌───────────────────────────────────┐  │
   │  │           hidden layer            │  │
@@ -38,7 +38,7 @@ of a **context** vector (the previous hidden state) and the current sensory inpu
    (next time step)     (predict char)
 ```
 
-### Temporal unrolling (unrolled / own-weights mode)
+### Temporal unrolling — own weights per position
 
 ```
       x_{t-1}              x_t               x_{t+1}
@@ -54,37 +54,92 @@ of a **context** vector (the previous hidden state) and the current sensory inpu
           └─────────────────►┘ ──────────────────►┘ ──────► ...
 ```
 
-Each time step has its **own weight matrix** W_t, b_v_t, b_h_t — the model is
+Each time step has its **own weight matrix** W_t, b_v_t, b_h_t.
-**unrolled**: N positions in a sequence → N separate RBMs.  This lets each
+`TEMPORAL_DEPTH` positions → `TEMPORAL_DEPTH` separate RBMs, each specialising
-position specialise for the statistical patterns that occur at that offset in a
+for the statistical patterns at that offset in a sequence.
-sequence.  The sequence length T must be fixed and equal to N at training time;
-during generation, position indices wrap modulo N.
 
-For comparison, the shared-weights variant (1 entity, `make_layer`) uses a
+During generation, position indices wrap modulo `TEMPORAL_DEPTH`.
-single W across all time steps — the RTRBM concatenation variant of Sutskever &
+
-Hinton (2007).
+For the shared-weights variant (1 entity via `make_layer`), a single W is reused
+at every time step — the RTRBM concatenation variant of Sutskever & Hinton (2007).
+
+---
+
+## Vocabulary
+
+The raw text is preprocessed to a 40-character vocabulary:
+
+```
+uppercase letters  A–Z   (26)
+digits             0–9   (10)
+punctuation        . ! ?  (3)
+separator          space  (1)
+                  ─────
+                    40
+```
+
+Lowercase is folded to uppercase; all other characters collapse to a single space;
+consecutive spaces are merged.
 
 ---
 
 ## Configuration
 
-All hyperparameters live in the **Build model** cell of `moby_rnn.ipynb`:
+All constants live in the **Configuration** cell of `moby_rnn.ipynb`:
 
-| Constant | Default | Description |
+| Constant | Current | Description |
 |---|---|---|
-| `T` | 100 | Characters per training sequence |
+| `TEMPORAL_DEPTH` | 64 | Sequence length = number of RBMs |
-| `NUM_SEQ` | 2000 | Number of training sequences (first 200k chars) |
+| `NUM_SEQ` | 2000 | Training sequences (covers first ~128k chars) |
-| `CONTEXT_SIZE` | 512 | Recurrent hidden / context state size |
+| `CONTEXT_SIZE` | 128 | Recurrent hidden / context state size |
 | `PRJ_NAME` | `"moby_rnn"` | Checkpoint file prefix |
 | `WORK_DIR` | `"results"` | Directory for saved weights |
-| `EVAL_CHARS` | 2000 | Characters used for reconstruction accuracy |
+| `EVAL_CHARS` | 2000 | Characters for reconstruction accuracy |
-| `PRED_CHARS` | 500 | Characters used for next-step prediction |
+| `PRED_CHARS` | 500 | Characters for next-step prediction |
-| `N_GIBBS` | 10 | Gibbs steps in clamped-Gibbs generation |
+| `N_GIBBS` | 3 | Gibbs steps in clamped-Gibbs generation |
 | `SEED` | `"Call me Ishmael."` | Seed text for free generation |
-| `SEQ_IDX` | 42 | Sequence index for the hidden-state trace plot |
+| `SEQ_IDX` | 42 | Sequence index for hidden-state trace plot |
 
-Model size with defaults: **(CONTEXT_SIZE + 85) × CONTEXT_SIZE × T parameters** —
+**Model size:** `(CONTEXT_SIZE + vocab_size) × CONTEXT_SIZE × TEMPORAL_DEPTH`
-e.g. T=100, CONTEXT_SIZE=256 → 100 × 27,136 = **2,713,600 parameters** total.
+— with defaults: `(128 + 40) × 128 × 64 = `**1,376,256 parameters** total.
+
+---
+
+## Training
+
+### Temporal-shift padding
+
+Sequences are flattened from `(NUM_SEQ, TEMPORAL_DEPTH, vocab_size)` to a
+flat batch of `N = NUM_SEQ × TEMPORAL_DEPTH` rows.  `TEMPORAL_DEPTH − 1` zero
+rows are appended, then **layer t trains on `batch[t : N+t]`** — a one-step
+temporal delay matching the C++ `RnnStack` implementation:
+
+```
+Layer 0: batch[0 .. N-1]          context = zeros
+Layer 1: batch[1 .. N]            context = h from layer 0 on rows 0..N-1
+Layer 2: batch[2 .. N+1]          context = h from layer 1 on rows 1..N
+  ...
+```
+
+Context from layer t at row j feeds layer t+1 at row j, whose sensory input is
+the original row j+1 — a one-step look-ahead.
+
+### Joint training
+
+All `TEMPORAL_DEPTH` layers are updated **together each epoch**:
+the context chain is live during training, so each layer sees realistic context
+from the layers below rather than frozen approximations.
+
+### Resuming
+
+Cell 5 saves a checkpoint at every progress report.  Re-run it to continue:
+
+```
+results/moby_rnn-0-state.npz      ← RBM at position 0
+results/moby_rnn-1-state.npz      ← RBM at position 1
+...
+results/moby_rnn-63-state.npz     ← RBM at position TEMPORAL_DEPTH-1
+```
 
 ---
 
@@ -93,48 +148,46 @@ e.g. T=100, CONTEXT_SIZE=256 → 100 × 27,136 = **2,713,600 parameters** total.
 | Cell | Title | What it does |
 |---|---|---|
 | 1 | Imports | Standard + rbm imports |
-| 2 | Load text | Reads `data/moby.txt`, builds `char_to_idx` / `idx_to_char` |
+| 2 | **Configuration** | All constants in one place |
-| 3 | Encode sequences | One-hot encodes text → `(NUM_SEQ, T, vocab_size)` array |
+| 3 | Load text | Read and preprocess `data/moby.txt`, build vocabulary |
-| 4 | **Build model** | All constants; constructs `StackRnn`; loads checkpoint |
+| 4 | Encode sequences | One-hot encode → `(NUM_SEQ, TEMPORAL_DEPTH, vocab_size)` |
-| 5 | Train | Runs CD training; checkpoints every 5% progress |
+| 5 | Build model | Construct `StackRnn` with `TEMPORAL_DEPTH` layers; load checkpoint |
-| 6 | Generation helpers | `predict_next` (clamped Gibbs) and `generate_text` |
+| 6 | Train | Joint CD training with temporal-shift padding; checkpoint every 5% |
-| 7 | Reconstruction accuracy | Teacher-forced: how well does `h_t` remember `x_t`? |
+| 7 | Generation helpers | `predict_next` (clamped Gibbs) and `generate_text` |
-| 8 | Next-step prediction | Given `context_{t-1}`, predict `x_t` before seeing it |
+| 8 | Reconstruction accuracy | Teacher-forced: how well does `h_t` encode `x_t`? |
-| 9 | Free generation | Generate text at temperatures 0.5 / 1.0 / 1.5 |
+| 9 | Next-step prediction | Given `context_{t-1}`, predict `x_t` before seeing it |
-| 10 | Hidden state trace | Heatmap of context activations over one sequence |
+| 10 | Free generation | Generate text at temperatures 0.5 / 1.0 / 1.5 |
-| 11 | Character distribution | Data vs model character frequency comparison |
+| 11 | Hidden state trace | Heatmap of context activations over one sequence |
+| 12 | Character distribution | Data vs model character frequency comparison |
 
----
+### Evaluation metrics
 
-## Training and resuming
+**Reconstruction accuracy** (cell 8) is teacher-forced — the model sees `x_t`
+as part of its visible input, so high accuracy just means the RBM is a decent
+autoencoder.  It does not measure sequence modeling ability.
 
-Cell 5 saves a checkpoint at every progress report.  Re-run it at any time to
+**Next-step prediction accuracy** (cell 9) is the honest metric: given only
-continue training from the last saved state — `model.state_load()` in cell 4
+`context_{t-1}` (no `x_t`), predict the next character via clamped Gibbs.
-picks it up automatically.
+Random baseline is `100 / vocab_size = 2.5%`.
-
-```
-results/moby_rnn-0-state.npz      ← RBM at position 0
-results/moby_rnn-1-state.npz      ← RBM at position 1
-...
-results/moby_rnn-99-state.npz     ← RBM at position T-1
-```
 
 ---
 
 ## Text generation
 
-`predict_next` uses **clamped Gibbs sampling**: the context vector is held fixed
+`predict_next` uses **clamped Gibbs sampling**: context is held fixed while
-while the sensory (character) part of the visible layer runs free for `N_GIBBS`
+the sensory part of the visible layer iterates for `N_GIBBS` steps.
-steps.  The resulting Bernoulli probabilities are temperature-scaled and
+The Bernoulli outputs are temperature-scaled and normalised to a categorical
-normalised to a categorical distribution.
+distribution.
 
 ```python
-# Generate 500 chars from a seed at temperature 0.8
+text = generate_text(rnn, "CALL ME ISHMAEL.", length=500, temperature=0.8)
-text = generate_text(rnn, "Call me Ishmael.", length=500, temperature=0.8)
 ```
 
-Lower temperature → more conservative / repetitive output.
+Note: the seed is preprocessed to the reduced vocabulary (uppercase, allowed
-Higher temperature → more diverse / noisier output.
+chars only) before priming the hidden state.
+
+Lower temperature → more conservative / repetitive.
+Higher temperature → more diverse / noisier.
 
 ---
 

src/rbm/stack_rnn.py

@@ -209,13 +209,30 @@ class StackRnn(Stack):
         })
 
     def _train_unrolled(self, seqs: Mat, num_seq: int, T: int, status: Status):
-        """N entities, one per time step — each has its own W, b_v, b_h."""
+        """N entities, one per time step — each has its own W, b_v, b_h.
+
+        Uses temporal-shift padding (matching C++ RnnStack):
+          Flatten (num_seq, T, sensory_size) → (N, sensory_size), append T-1 zero
+          rows, then layer t trains on batch_padded[t : N+t] — a one-step delay.
+
+        Joint training: all layers are updated together each epoch.
+          Context c from layer t feeds layer t+1 within the same epoch pass,
+          so gradients propagate through the full temporal chain.
+        """
         params       = self.from_index(0).entity.training_params
         h_sz         = self.from_index(0).entity.shape[1]
+        s_sz         = seqs.shape[2]
         d_progress   = 100.0 / params.num_epochs
         progress     = 0.0
         keep_running = True
 
+        # Flatten to (N, s_sz) keeping sequence-major order, on CPU for slicing
+        flat      = _np_cpu.asarray(convert(seqs) if hasattr(seqs, 'get') else seqs)
+        flat      = flat.reshape(-1, s_sz)
+        N         = flat.shape[0]
+        pad       = _np_cpu.zeros((T - 1, s_sz), dtype=flat.dtype)
+        batch_pad = _np_cpu.concatenate([flat, pad], axis=0)   # (N+T-1, s_sz)
+
         for layer in self.layers:
             layer.entity.grad_zero()
         status.on_change(self.from_index(0).entity)
@@ -224,22 +241,22 @@ class StackRnn(Stack):
             if not keep_running:
                 break
 
-            h = np.zeros((num_seq, h_sz))
+            c = np.zeros((N, h_sz))
             err_total = 0.0
 
             for t, layer in enumerate(self.layers):
                 entity  = layer.entity
                 cd_func = _CD_FUNC[entity.type]
 
-                x_t     = _to_gpu(seqs[:, t, :])
+                x_t     = _to_gpu(batch_pad[t : N + t])   # shifted slice, (N, s_sz)
-                visible = np.concatenate([h, x_t], axis=1)
+                visible = np.concatenate([c, x_t], axis=1)
 
                 dwhv, dbv, dbh = cd_func(entity, visible)
                 grad = entity.grad_compute(dbv, dbh, dwhv)
-                entity.state_adjust(grad, 1.0 / num_seq)
+                entity.state_adjust(grad, 1.0 / N)
 
-                h = entity.forward(visible)
+                c = entity.forward(visible)
-                err_total += rms_error_accu(visible - entity.reconstruct(h))
+                err_total += rms_error_accu(visible - entity.reconstruct(c))
 
             progress += d_progress
             if status.want_report(round(progress)):
@@ -250,14 +267,15 @@ class StackRnn(Stack):
                     keep_running = False
                     break
 
-        h = np.zeros((num_seq, h_sz))
+        # Final report pass
+        c = np.zeros((N, h_sz))
         err_total = 0.0
         for t, layer in enumerate(self.layers):
             entity  = layer.entity
-            x_t     = _to_gpu(seqs[:, t, :])
+            x_t     = _to_gpu(batch_pad[t : N + t])
-            visible = np.concatenate([h, x_t], axis=1)
+            visible = np.concatenate([c, x_t], axis=1)
-            h = entity.forward(visible)
+            c = entity.forward(visible)
-            err_total += rms_error_accu(visible - entity.reconstruct(h))
+            err_total += rms_error_accu(visible - entity.reconstruct(c))
         status.on_change(self.from_index(0).entity, {
             "progress":      {"value": 100,           "unit": "%"},
             "err_rms_total": {"value": err_total / T, "unit": ""},

src/rbm/state.py

@@ -22,7 +22,6 @@ class RbmState:
 			with np.load(filename) as X:
 				w_hv, b_v, b_h = [X[i] for i in ('whv', 'bv', 'bh')]
 				obj = cls(w_hv, b_v, b_h)
-				print(f"{filename} loaded successfully!")
 		except FileNotFoundError:
 			pass
 		except KeyError:
@@ -44,7 +43,6 @@ class RbmState:
 					self.w_hv = w_hv
 					self.b_v = b_v
 					self.b_h = b_h
-				print(f"{filename} loaded successfully!")
 		except FileNotFoundError:
 			pass
 		except KeyError:
@@ -53,7 +51,6 @@ class RbmState:
 
 	def save(self, filename: str):
 		np.savez(filename, whv=self.w_hv, bv=self.b_v, bh=self.b_h)
-		print(f"{filename} saved successfully!")
 
 	def init(self, mu: float = 0.0, std: float = 1.0):
 		self.w_hv = uniform(self.w_hv.shape, mu, std)