From 205e6fd5f3fead3cc9880c79f4a08bdb54f8f98d Mon Sep 17 00:00:00 2001 From: Jens Ahrensfeld Date: Wed, 27 May 2026 14:00:32 +0200 Subject: [PATCH] Remove README.jaydiff.md and now-empty utils/ and tests/ packages jay_diff docs, tests, and data live in jayfield/jaypy; the __init__.py files for the empty local packages are no longer needed. Co-Authored-By: Claude Sonnet 4.6 --- README.jaydiff.md | 195 ---------------------------------------------- tests/__init__.py | 0 utils/__init__.py | 0 3 files changed, 195 deletions(-) delete mode 100644 README.jaydiff.md delete mode 100644 tests/__init__.py delete mode 100644 utils/__init__.py diff --git a/README.jaydiff.md b/README.jaydiff.md deleted file mode 100644 index 2976b5c..0000000 --- a/README.jaydiff.md +++ /dev/null @@ -1,195 +0,0 @@ -# jay_diff — JSON diff / patch library - -`utils/jay_diff.py` is a lightweight, pure-Python library for computing and -applying structural diffs between two JSON-compatible dicts. It is used by -`collect.py` and `gui_client.py` to compress the push-server stream: instead -of retransmitting full snapshots on every poll, the server sends only the -fields that changed since the previous snapshot. - ---- - -## Concepts - -A **diff** is itself a plain dict. It describes what changed from an **old** -state to a **new** state using one or more of these keys: - -| Key | Meaning | -|-----|---------| -| `update_add` | Fields whose value changed *or* that are new in `new` | -| `update` | Fields whose value changed (must already exist in `old`) | -| `add` | Fields that are new in `new` (did not exist in `old`) | -| `delete` | Fields that were removed or whose sub-keys were pruned | - -`update_add` combines `update` and `add` into a single key (the mode used -everywhere in this project). The separate `update` / `add` form is -available via `combine_upd_add=False` for completeness. - -An **empty dict `{}`** returned by `jay_diff_full` means the two states are -identical. - -### Delete-delta encoding - -Values inside the `delete` sub-dict carry a specific meaning: - -| Value in delete delta | Effect when merged | -|-----------------------|--------------------| -| `None` | The **entire key** is deleted from `old` | -| A **non-empty dict** | Recurse into the sub-dict and delete those specific sub-keys; the parent key is kept (possibly as `{}`) | -| A **scalar** (string, int, …) | The key is deleted from `old` | -| `{}` (empty dict) | The key is deleted from `old` | - -This distinction matters when the new state has an empty dict `{}` at a -location where the old state had a populated dict. In that case the delete -delta carries the populated sub-keys as a non-empty dict — after merging, -the parent key is preserved as `{}`. If the key is entirely absent from -the new state, `None` is stored in the delta, and the merge removes the key -completely. - ---- - -## API - -### Diff functions - -```python -jay_diff_full(old, new, combine_upd_add=True) -> dict -``` -The main entry point. Computes a complete diff from `old` to `new`. - -- `combine_upd_add=True` (default) — produces `update_add` + `delete` -- `combine_upd_add=False` — produces `update` + `add` + `delete` - -Returns `{}` when `old == new`. - -```python -jay_diff_add(old, new) -> dict -``` -Returns keys present in `new` but absent from `old` (additions only). - -```python -jay_diff_del(old, new) -> dict -``` -Returns keys present in `old` but absent from `new`, or sub-keys that were -removed from a shared dict. Uses `None` as sentinel for entirely-absent -dict keys. - -```python -jay_diff_upd(old, new) -> dict -``` -Returns keys present in both `old` and `new` whose values differ (updates -only; new keys are ignored). - -```python -jay_diff_upd_add(old, new) -> dict -``` -Like `jay_diff_upd` but also includes keys new in `new` — equivalent to -`jay_diff_full(..., combine_upd_add=True)` minus the delete section. - ---- - -### Merge functions - -```python -jay_merge_full(old, diff) -> dict -``` -The main entry point. Applies a diff produced by `jay_diff_full` to `old` -and returns the reconstructed new state. - -**Bootstrap passthrough:** if `old` is `{}` and `diff` contains none of the -recognised diff keys, `diff` is returned as-is. This is how the first -message from the push server is handled: the server sends -`jay_diff_full({}, record[0])`, which looks like a plain snapshot, and the -client stores it directly. - -Applies operations in order: `update` → `update_add` → `add` → `delete`. - -```python -jay_merge_update(old, new) -> dict # apply an update/update_add delta -jay_merge_add(old, new) -> dict # apply an add delta -jay_merge_delete(old, new) -> dict # apply a delete delta -``` -Low-level helpers; prefer `jay_merge_full` unless you need fine-grained -control. - ---- - -## Round-trip guarantee - -``` -jay_merge_full(old, jay_diff_full(old, new)) == new -``` - -This holds for all inputs where values are JSON-compatible scalars (strings, -numbers, booleans, `None`, lists) or nested dicts thereof. It is verified -by `tests/test_jay_diff_real_data.py` against 674 consecutive pairs of real -vehicle snapshots. - ---- - -## Wire format - -`jay_diff_full` / `jay_merge_full` are used by the push server when -`--diff` is enabled. The diff dict is serialised as a newline-delimited -JSON line. - -- The **first** message to a new client is `jay_diff_full({}, record[0])`, - which always contains `update_add` with every field of the first record. -- Subsequent messages contain only the changed fields. -- The format is self-describing: a message with none of the diff keys is - treated as a full snapshot by `jay_merge_full` (bootstrap passthrough). - -`null` in JSON round-trips to Python `None`, so the delete sentinel survives -serialisation transparently. - ---- - -## Tests - -``` -tests/test_jay_diff.py — 104 unit tests (all diff/merge functions, - corner cases: type changes, empty dicts, - deep nesting, round-trips) -tests/test_jay_diff_real_data.py — parametrised over 675 real VW snapshots - in tests/data/ (674 consecutive pairs): - round-trip, identical diff = {}, and - empty-diff-is-noop checks -``` - -Run with: - -```bash -pytest tests/ -``` - ---- - -## Bug history - -### `jay_merge_delete`: crash on aliased dicts (fixed) - -`jay_diff_rev` (used internally by the old `jay_diff_del`) stored references -to sub-dicts of `_old` directly in the delete delta without copying them. -When `jay_merge_full(old, jay_diff_full(old, new))` was called, some -sub-dicts in the delete delta were the same Python objects as sub-dicts in -`old`. Iterating `_new.items()` while deleting from `_old` (the same -object) raised `RuntimeError: dictionary changed size during iteration`. - -**Fix:** `list(_new.items())` snapshots the items before the loop. Also, -`jay_diff_del` was rewritten as a standalone recursive function that never -stores aliases to its inputs. - -### `jay_merge_delete`: empty dict wrongly pruned (fixed) - -When the new state had an empty dict `{}` at a location where the old state -had a non-empty dict, `jay_merge_delete` would delete all sub-keys and then -prune the now-empty parent key entirely. The result differed from `new`, -which retained the parent key as `{}`. - -Root cause: the old delta format stored the full old sub-dict for both "key -absent from new" and "key present but empty in new" — the two cases were -indistinguishable at merge time. - -**Fix:** `jay_diff_del` now stores `None` for dict keys entirely absent from -`new`, and a non-empty sub-dict for partial deletions. `jay_merge_delete` -no longer prunes: a `None` (or scalar) value in the delta is the explicit -signal to remove the parent key. \ No newline at end of file diff --git a/tests/__init__.py b/tests/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/utils/__init__.py b/utils/__init__.py deleted file mode 100644 index e69de29..0000000