jaypy/diff

jay_diff — JSON diff / patch library

utils/jay_diff.py is a lightweight, pure-Python library for computing and applying structural diffs between two JSON-compatible dicts. It is used by collect.py and gui_client.py to compress the push-server stream: instead of retransmitting full snapshots on every poll, the server sends only the fields that changed since the previous snapshot.

---

Concepts

A diff is itself a plain dict. It describes what changed from an old state to a new state using one or more of these keys:

Key	Meaning
update_add	Fields whose value changed or that are new in new
update	Fields whose value changed (must already exist in old)
add	Fields that are new in new (did not exist in old)
delete	Fields that were removed or whose sub-keys were pruned

update_add combines update and add into a single key (the mode used everywhere in this project). The separate update / add form is available via combine_upd_add=False for completeness.

An empty dict {} returned by jay_diff_full means the two states are identical.

Delete-delta encoding

Values inside the delete sub-dict carry a specific meaning:

Value in delete delta	Effect when merged
None	The entire key is deleted from old
A non-empty dict	Recurse into the sub-dict and delete those specific sub-keys; the parent key is kept (possibly as {})
A scalar (string, int, …)	The key is deleted from old
{} (empty dict)	The key is deleted from old

This distinction matters when the new state has an empty dict {} at a location where the old state had a populated dict. In that case the delete delta carries the populated sub-keys as a non-empty dict — after merging, the parent key is preserved as {}. If the key is entirely absent from the new state, None is stored in the delta, and the merge removes the key completely.

---

API

Diff functions

jay_diff_full(old, new, combine_upd_add=True) -> dict

The main entry point. Computes a complete diff from old to new.

• combine_upd_add=True (default) — produces update_add + delete
• combine_upd_add=False — produces update + add + delete

Returns {} when old == new.

jay_diff_add(old, new) -> dict

Returns keys present in new but absent from old (additions only).

jay_diff_del(old, new) -> dict

Returns keys present in old but absent from new, or sub-keys that were removed from a shared dict. Uses None as sentinel for entirely-absent dict keys.

jay_diff_upd(old, new) -> dict

Returns keys present in both old and new whose values differ (updates only; new keys are ignored).

jay_diff_upd_add(old, new) -> dict

Like jay_diff_upd but also includes keys new in new — equivalent to jay_diff_full(..., combine_upd_add=True) minus the delete section.

---

Merge functions

jay_merge_full(old, diff) -> dict

The main entry point. Applies a diff produced by jay_diff_full to old and returns the reconstructed new state.

Bootstrap passthrough: if old is {} and diff contains none of the recognised diff keys, diff is returned as-is. This is how the first message from the push server is handled: the server sends jay_diff_full({}, record[0]), which looks like a plain snapshot, and the client stores it directly.

Applies operations in order: update → update_add → add → delete.

jay_merge_update(old, new) -> dict   # apply an update/update_add delta
jay_merge_add(old, new)    -> dict   # apply an add delta
jay_merge_delete(old, new) -> dict   # apply a delete delta

Low-level helpers; prefer jay_merge_full unless you need fine-grained control.

---

Round-trip guarantee

jay_merge_full(old, jay_diff_full(old, new)) == new

This holds for all inputs where values are JSON-compatible scalars (strings, numbers, booleans, None, lists) or nested dicts thereof. It is verified by tests/test_jay_diff_real_data.py against 674 consecutive pairs of real vehicle snapshots.

---

Wire format

jay_diff_full / jay_merge_full are used by the push server when --diff is enabled. The diff dict is serialised as a newline-delimited JSON line.

• The first message to a new client is jay_diff_full({}, record[0]), which always contains update_add with every field of the first record.
• Subsequent messages contain only the changed fields.
• The format is self-describing: a message with none of the diff keys is treated as a full snapshot by jay_merge_full (bootstrap passthrough).

null in JSON round-trips to Python None, so the delete sentinel survives serialisation transparently.

---

Tests

tests/test_jay_diff.py           — 104 unit tests (all diff/merge functions,
                                    corner cases: type changes, empty dicts,
                                    deep nesting, round-trips)
tests/test_jay_diff_real_data.py — parametrised over 675 real VW snapshots
                                    in tests/data/ (674 consecutive pairs):
                                    round-trip, identical diff = {}, and
                                    empty-diff-is-noop checks

Run with:

pytest tests/

---

Bug history

jay_merge_delete: crash on aliased dicts (fixed)

jay_diff_rev (used internally by the old jay_diff_del) stored references to sub-dicts of _old directly in the delete delta without copying them. When jay_merge_full(old, jay_diff_full(old, new)) was called, some sub-dicts in the delete delta were the same Python objects as sub-dicts in old. Iterating _new.items() while deleting from _old (the same object) raised RuntimeError: dictionary changed size during iteration.

Fix: list(_new.items()) snapshots the items before the loop. Also, jay_diff_del was rewritten as a standalone recursive function that never stores aliases to its inputs.

jay_merge_delete: empty dict wrongly pruned (fixed)

When the new state had an empty dict {} at a location where the old state had a non-empty dict, jay_merge_delete would delete all sub-keys and then prune the now-empty parent key entirely. The result differed from new, which retained the parent key as {}.

Root cause: the old delta format stored the full old sub-dict for both "key absent from new" and "key present but empty in new" — the two cases were indistinguishable at merge time.

Fix: jay_diff_del now stores None for dict keys entirely absent from new, and a non-empty sub-dict for partial deletions. jay_merge_delete no longer prunes: a None (or scalar) value in the delta is the explicit signal to remove the parent key.