[bugfix] - fix L1 and L2 regularisation gradients in grad_compute

Previous code computed scalar norms and multiplied by W, giving gradients
that scaled with total weight magnitude rather than the correct per-element
derivatives:
  L1: gradient is λ·sign(W), not λ·‖W‖₁·W
  L2: gradient is 2λ·W, not λ·‖W‖₂²·W
Also removed the combined (l1+l2)*W term which incorrectly mixed the two.
weight_decay (correct L2 form λW) is kept as a separate term.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-05-30 18:26:06 +02:00
co-authored by Claude Sonnet 4.6
parent e0dcac5c82
commit abee870ada
+5 -9
View File
@@ -97,15 +97,11 @@ class Entity:
self.grad.b_v = params.momentum * self.grad.b_v + lr * d_bv
self.grad.b_h = params.momentum * self.grad.b_h + lr * d_bh
# compute L1 term and penalize cost function (d_whv)
l1_norm = np.sum(np.abs(self.state.w_hv))
l1_term = params.l1_lambda*l1_norm
# compute L2 term and penalize cost function (d_whv)
l2_norm = np.sum(np.square(self.state.w_hv))
l2_term = params.l2_lambda*l2_norm
self.grad.w_hv = params.momentum * self.grad.w_hv + lr * (d_whv - self.state.w_hv*(l1_term+l2_term)) - lr * params.weight_decay * self.state.w_hv
self.grad.w_hv = (params.momentum * self.grad.w_hv
+ lr * d_whv
- lr * params.l1_lambda * np.sign(self.state.w_hv)
- lr * 2 * params.l2_lambda * self.state.w_hv
- lr * params.weight_decay * self.state.w_hv)
return self.grad