[README] - document L1/L2 batch-size scaling fix

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-30 19:59:34 +02:00
parent d4d3d33931
commit 1d18983192
1 changed files with 4 additions and 0 deletions
@@ -98,6 +98,10 @@ The combined `(l1+l2)·W` term also incorrectly mixed both penalties into one. F
 The `repeat`/`reshape` approach was replaced with `keepdims=True` broadcasting and a `std < 1e-8` guard to prevent NaN on constant patches. The duplicate normalization logic in `test_faces_sub_image.py:load_image_patches` was removed in favour of calling `normalize()` directly.
 #### `entity.py` — L1/L2 regularisation divided by mini-batch size
 `state_adjust(grad, k=1/N)` applied the `1/N` scale to the entire gradient, including the regularisation terms. The CD gradient is a batch sum so `1/N` normalisation is correct; L1/L2/weight-decay penalties are per-weight and batch-size independent. Dividing them by `N` made the effective regularisation strength `lambda / N`, so e.g. `l1_lambda=0.5` with `mini_batch_size=1000` was equivalent to an effective L1 of `0.0005` — too small to have any visible effect on filters. Fixed by moving the regularisation update into `state_adjust` where it is applied at full strength before the `1/N` CD update.
 ---
 ## RBM type reference