[StackRnn] - temporal-shift padding, joint training, vocab reduction, TEMPORAL_DEPTH=64

Browse Source

stack_rnn.py:
  - _train_unrolled: C++-style temporal-shift padding — flatten (num_seq, T, s)
    → (N, s), append T-1 zero rows, layer t trains on batch[t:N+t]; joint
    training replaces greedy sequential (all layers update each epoch)
  - Reverted sequential/greedy training path (poor next-step prediction)

moby_rnn.ipynb:
  - Vocabulary reduced 85 → 40 chars (A-Z, 0-9, space, .!?)
  - T renamed to TEMPORAL_DEPTH throughout
  - TEMPORAL_DEPTH increased 5 → 64; CONTEXT_SIZE 128
  - SEED "Call me Ishmael." now falls within training data coverage

state.py:
  - Remove per-file load/save print messages (too noisy with 64-layer checkpoints)

README_moby_rnn.md:
  - Update vocab, constants, parameter count, checkpoint listing
  - Add temporal-shift padding and joint training sections
  - Clarify reconstruction accuracy vs next-step prediction as honest metric

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

...

This commit is contained in:

jens

2026-05-31 14:37:08 +02:00

co-authored by Claude Sonnet 4.6

parent 3698c85ed4

commit 952045b9f1

4 changed files with 611 additions and 289 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

moby_rnn.ipynb

+476 -222

File diff suppressed because one or more lines are too long