195 lines
6.7 KiB
Markdown
195 lines
6.7 KiB
Markdown
# jay_diff — JSON diff / patch library
|
|
|
|
`utils/jay_diff.py` is a lightweight, pure-Python library for computing and
|
|
applying structural diffs between two JSON-compatible dicts. It is used by
|
|
`collect.py` and `gui_client.py` to compress the push-server stream: instead
|
|
of retransmitting full snapshots on every poll, the server sends only the
|
|
fields that changed since the previous snapshot.
|
|
|
|
---
|
|
|
|
## Concepts
|
|
|
|
A **diff** is itself a plain dict. It describes what changed from an **old**
|
|
state to a **new** state using one or more of these keys:
|
|
|
|
| Key | Meaning |
|
|
|-----|---------|
|
|
| `update_add` | Fields whose value changed *or* that are new in `new` |
|
|
| `update` | Fields whose value changed (must already exist in `old`) |
|
|
| `add` | Fields that are new in `new` (did not exist in `old`) |
|
|
| `delete` | Fields that were removed or whose sub-keys were pruned |
|
|
|
|
`update_add` combines `update` and `add` into a single key (the mode used
|
|
everywhere in this project). The separate `update` / `add` form is
|
|
available via `combine_upd_add=False` for completeness.
|
|
|
|
An **empty dict `{}`** returned by `jay_diff_full` means the two states are
|
|
identical.
|
|
|
|
### Delete-delta encoding
|
|
|
|
Values inside the `delete` sub-dict carry a specific meaning:
|
|
|
|
| Value in delete delta | Effect when merged |
|
|
|-----------------------|--------------------|
|
|
| `None` | The **entire key** is deleted from `old` |
|
|
| A **non-empty dict** | Recurse into the sub-dict and delete those specific sub-keys; the parent key is kept (possibly as `{}`) |
|
|
| A **scalar** (string, int, …) | The key is deleted from `old` |
|
|
| `{}` (empty dict) | The key is deleted from `old` |
|
|
|
|
This distinction matters when the new state has an empty dict `{}` at a
|
|
location where the old state had a populated dict. In that case the delete
|
|
delta carries the populated sub-keys as a non-empty dict — after merging,
|
|
the parent key is preserved as `{}`. If the key is entirely absent from
|
|
the new state, `None` is stored in the delta, and the merge removes the key
|
|
completely.
|
|
|
|
---
|
|
|
|
## API
|
|
|
|
### Diff functions
|
|
|
|
```python
|
|
jay_diff_full(old, new, combine_upd_add=True) -> dict
|
|
```
|
|
The main entry point. Computes a complete diff from `old` to `new`.
|
|
|
|
- `combine_upd_add=True` (default) — produces `update_add` + `delete`
|
|
- `combine_upd_add=False` — produces `update` + `add` + `delete`
|
|
|
|
Returns `{}` when `old == new`.
|
|
|
|
```python
|
|
jay_diff_add(old, new) -> dict
|
|
```
|
|
Returns keys present in `new` but absent from `old` (additions only).
|
|
|
|
```python
|
|
jay_diff_del(old, new) -> dict
|
|
```
|
|
Returns keys present in `old` but absent from `new`, or sub-keys that were
|
|
removed from a shared dict. Uses `None` as sentinel for entirely-absent
|
|
dict keys.
|
|
|
|
```python
|
|
jay_diff_upd(old, new) -> dict
|
|
```
|
|
Returns keys present in both `old` and `new` whose values differ (updates
|
|
only; new keys are ignored).
|
|
|
|
```python
|
|
jay_diff_upd_add(old, new) -> dict
|
|
```
|
|
Like `jay_diff_upd` but also includes keys new in `new` — equivalent to
|
|
`jay_diff_full(..., combine_upd_add=True)` minus the delete section.
|
|
|
|
---
|
|
|
|
### Merge functions
|
|
|
|
```python
|
|
jay_merge_full(old, diff) -> dict
|
|
```
|
|
The main entry point. Applies a diff produced by `jay_diff_full` to `old`
|
|
and returns the reconstructed new state.
|
|
|
|
**Bootstrap passthrough:** if `old` is `{}` and `diff` contains none of the
|
|
recognised diff keys, `diff` is returned as-is. This is how the first
|
|
message from the push server is handled: the server sends
|
|
`jay_diff_full({}, record[0])`, which looks like a plain snapshot, and the
|
|
client stores it directly.
|
|
|
|
Applies operations in order: `update` → `update_add` → `add` → `delete`.
|
|
|
|
```python
|
|
jay_merge_update(old, new) -> dict # apply an update/update_add delta
|
|
jay_merge_add(old, new) -> dict # apply an add delta
|
|
jay_merge_delete(old, new) -> dict # apply a delete delta
|
|
```
|
|
Low-level helpers; prefer `jay_merge_full` unless you need fine-grained
|
|
control.
|
|
|
|
---
|
|
|
|
## Round-trip guarantee
|
|
|
|
```
|
|
jay_merge_full(old, jay_diff_full(old, new)) == new
|
|
```
|
|
|
|
This holds for all inputs where values are JSON-compatible scalars (strings,
|
|
numbers, booleans, `None`, lists) or nested dicts thereof. It is verified
|
|
by `tests/test_jay_diff_real_data.py` against 674 consecutive pairs of real
|
|
vehicle snapshots.
|
|
|
|
---
|
|
|
|
## Wire format
|
|
|
|
`jay_diff_full` / `jay_merge_full` are used by the push server when
|
|
`--diff` is enabled. The diff dict is serialised as a newline-delimited
|
|
JSON line.
|
|
|
|
- The **first** message to a new client is `jay_diff_full({}, record[0])`,
|
|
which always contains `update_add` with every field of the first record.
|
|
- Subsequent messages contain only the changed fields.
|
|
- The format is self-describing: a message with none of the diff keys is
|
|
treated as a full snapshot by `jay_merge_full` (bootstrap passthrough).
|
|
|
|
`null` in JSON round-trips to Python `None`, so the delete sentinel survives
|
|
serialisation transparently.
|
|
|
|
---
|
|
|
|
## Tests
|
|
|
|
```
|
|
tests/test_jay_diff.py — 104 unit tests (all diff/merge functions,
|
|
corner cases: type changes, empty dicts,
|
|
deep nesting, round-trips)
|
|
tests/test_jay_diff_real_data.py — parametrised over 675 real VW snapshots
|
|
in tests/data/ (674 consecutive pairs):
|
|
round-trip, identical diff = {}, and
|
|
empty-diff-is-noop checks
|
|
```
|
|
|
|
Run with:
|
|
|
|
```bash
|
|
pytest tests/
|
|
```
|
|
|
|
---
|
|
|
|
## Bug history
|
|
|
|
### `jay_merge_delete`: crash on aliased dicts (fixed)
|
|
|
|
`jay_diff_rev` (used internally by the old `jay_diff_del`) stored references
|
|
to sub-dicts of `_old` directly in the delete delta without copying them.
|
|
When `jay_merge_full(old, jay_diff_full(old, new))` was called, some
|
|
sub-dicts in the delete delta were the same Python objects as sub-dicts in
|
|
`old`. Iterating `_new.items()` while deleting from `_old` (the same
|
|
object) raised `RuntimeError: dictionary changed size during iteration`.
|
|
|
|
**Fix:** `list(_new.items())` snapshots the items before the loop. Also,
|
|
`jay_diff_del` was rewritten as a standalone recursive function that never
|
|
stores aliases to its inputs.
|
|
|
|
### `jay_merge_delete`: empty dict wrongly pruned (fixed)
|
|
|
|
When the new state had an empty dict `{}` at a location where the old state
|
|
had a non-empty dict, `jay_merge_delete` would delete all sub-keys and then
|
|
prune the now-empty parent key entirely. The result differed from `new`,
|
|
which retained the parent key as `{}`.
|
|
|
|
Root cause: the old delta format stored the full old sub-dict for both "key
|
|
absent from new" and "key present but empty in new" — the two cases were
|
|
indistinguishable at merge time.
|
|
|
|
**Fix:** `jay_diff_del` now stores `None` for dict keys entirely absent from
|
|
`new`, and a non-empty sub-dict for partial deletions. `jay_merge_delete`
|
|
no longer prunes: a `None` (or scalar) value in the delta is the explicit
|
|
signal to remove the parent key. |