status.py: add CheckpointStatus(save_fn, update_interval) — prints progress
and calls save_fn at every report interval to persist model state mid-training.
model.py: Model.train() now accepts an optional Status instance; defaults to
plain Status() if none provided.
test_faces_sub_image.py: use CheckpointStatus(model.save) during training.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
train.py: add _to_gpu() helper; convert mini-batches from CPU numpy to GPU
CuPy on the fly so large datasets can stay in RAM. Use numpy permutation for
numpy batches during shuffle. Limit status err_rms to first 5000 samples to
avoid GPU OOM on status checks.
model.py: replace np.copy() with np.array() — cupy.copy() does not convert
numpy arrays to CuPy; cupy.array() does. Fixes entity.forward() crash after
training when batch was passed as numpy.
test_faces_sub_image.py: load_patches normalizes on CPU and returns numpy,
avoiding multi-copy GPU OOM for large datasets (500 images ≈ 1.4 GB).
show_reconstructions converts numpy patches to CuPy before inference.
Add --n_images CLI arg.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
jensandClaude Sonnet 4.6