From 1d1898319279b997b5b00378f21042081a622e96 Mon Sep 17 00:00:00 2001 From: Jens Ahrensfeld Date: Sat, 30 May 2026 19:59:34 +0200 Subject: [PATCH] [README] - document L1/L2 batch-size scaling fix Co-Authored-By: Claude Sonnet 4.6 --- src/tests/README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/tests/README.md b/src/tests/README.md index 26f8ed1..8954bd8 100644 --- a/src/tests/README.md +++ b/src/tests/README.md @@ -98,6 +98,10 @@ The combined `(l1+l2)·W` term also incorrectly mixed both penalties into one. F The `repeat`/`reshape` approach was replaced with `keepdims=True` broadcasting and a `std < 1e-8` guard to prevent NaN on constant patches. The duplicate normalization logic in `test_faces_sub_image.py:load_image_patches` was removed in favour of calling `normalize()` directly. +#### `entity.py` — L1/L2 regularisation divided by mini-batch size + +`state_adjust(grad, k=1/N)` applied the `1/N` scale to the entire gradient, including the regularisation terms. The CD gradient is a batch sum so `1/N` normalisation is correct; L1/L2/weight-decay penalties are per-weight and batch-size independent. Dividing them by `N` made the effective regularisation strength `lambda / N`, so e.g. `l1_lambda=0.5` with `mini_batch_size=1000` was equivalent to an effective L1 of `0.0005` — too small to have any visible effect on filters. Fixed by moving the regularisation update into `state_adjust` where it is applied at full strength before the `1/N` CD update. + --- ## RBM type reference