jaypy/jaydiff/README.jaydiff.md

# jaypy/jaydiff — JSON diff / patch library

`jaydiff` is a lightweight, pure-Python library for computing and
applying structural diffs between two JSON-compatible dicts.  It is used to 
compress the json data packets with redundant data, by providing a delta instead of the full packet

---

## Concepts

A **diff** is itself a plain dict.  It describes what changed from an **old**
state to a **new** state using one or more of these keys:

| Key | Meaning |
|-----|---------|
| `update_add` | Fields whose value changed *or* that are new in `new` |
| `update` | Fields whose value changed (must already exist in `old`) |
| `add` | Fields that are new in `new` (did not exist in `old`) |
| `delete` | Fields that were removed or whose sub-keys were pruned |

`update_add` combines `update` and `add` into a single key (the mode used
everywhere in this project).  The separate `update` / `add` form is
available via `combine_upd_add=False` for completeness.

An **empty dict `{}`** returned by `diff_full` means the two states are
identical.

### Delete-delta encoding

Values inside the `delete` sub-dict carry a specific meaning:

| Value in delete delta | Effect when merged |
|-----------------------|--------------------|
| `None` | The **entire key** is deleted from `old` |
| A **non-empty dict** | Recurse into the sub-dict and delete those specific sub-keys; the parent key is kept (possibly as `{}`) |
| A **scalar** (string, int, …) | The key is deleted from `old` |
| `{}` (empty dict) | The key is deleted from `old` |

This distinction matters when the new state has an empty dict `{}` at a
location where the old state had a populated dict.  In that case the delete
delta carries the populated sub-keys as a non-empty dict — after merging,
the parent key is preserved as `{}`.  If the key is entirely absent from
the new state, `None` is stored in the delta, and the merge removes the key
completely.

---

## API

### Diff functions

```python
diff_full(old, new, combine_upd_add=True) -> dict
```
The main entry point.  Computes a complete diff from `old` to `new`.

- `combine_upd_add=True` (default) — produces `update_add` + `delete`
- `combine_upd_add=False` — produces `update` + `add` + `delete`

Returns `{}` when `old == new`.

```python
diff_add(old, new) -> dict
```
Returns keys present in `new` but absent from `old` (additions only).

```python
diff_del(old, new) -> dict
```
Returns keys present in `old` but absent from `new`, or sub-keys that were
removed from a shared dict.  Uses `None` as sentinel for entirely-absent
dict keys.

```python
diff_upd(old, new) -> dict
```
Returns keys present in both `old` and `new` whose values differ (updates
only; new keys are ignored).

```python
diff_upd_add(old, new) -> dict
```
Like `diff_upd` but also includes keys new in `new` — equivalent to
`diff_full(..., combine_upd_add=True)` minus the delete section.

---

### Merge functions

```python
merge_full(old, diff) -> dict
```
The main entry point.  Applies a diff produced by `diff_full` to `old`
and returns the reconstructed new state.

**Bootstrap passthrough:** if `old` is `{}` and `diff` contains none of the
recognised diff keys, `diff` is returned as-is.  This is how the first
message from the push server is handled: the server sends
`diff_full({}, record[0])`, which looks like a plain snapshot, and the
client stores it directly.

Applies operations in order: `update` → `update_add` → `add` → `delete`.

```python
merge_update(old, new) -> dict   # apply an update/update_add delta
merge_add(old, new)    -> dict   # apply an add delta
merge_delete(old, new) -> dict   # apply a delete delta
```
Low-level helpers; prefer `merge_full` unless you need fine-grained
control.

---

## Round-trip guarantee

```
merge_full(old, diff_full(old, new)) == new
```

This holds for all inputs where values are JSON-compatible scalars (strings,
numbers, booleans, `None`, lists) or nested dicts thereof.  It is verified
by `tests/test_diff_real_data.py` against 674 consecutive pairs of real
vehicle snapshots.

---

## Wire format

`diff_full` / `merge_full` are used by the push server when
`--diff` is enabled.  The diff dict is serialised as a newline-delimited
JSON line.

- The **first** message to a new client is `diff_full({}, record[0])`,
  which always contains `update_add` with every field of the first record.
- Subsequent messages contain only the changed fields.
- The format is self-describing: a message with none of the diff keys is
  treated as a full snapshot by `merge_full` (bootstrap passthrough).

`null` in JSON round-trips to Python `None`, so the delete sentinel survives
serialisation transparently.

---

## Tests

```
tests/test_diff_.py           — 104 unit tests (all diff/merge functions,
                                    corner cases: type changes, empty dicts,
                                    deep nesting, round-trips)
tests/test_diff_real_data.py — parametrised over 675 real VW snapshots
                                    in tests/data/ (674 consecutive pairs):
                                    round-trip, identical diff = {}, and
                                    empty-diff-is-noop checks
```

Run with:

```bash
pytest tests/
```

---

## Bug history

### `merge_delete`: crash on aliased dicts (fixed)

`diff_rev` (used internally by the old `diff_del`) stored references
to sub-dicts of `_old` directly in the delete delta without copying them.
When `merge_full(old, diff_full(old, new))` was called, some
sub-dicts in the delete delta were the same Python objects as sub-dicts in
`old`.  Iterating `_new.items()` while deleting from `_old` (the same
object) raised `RuntimeError: dictionary changed size during iteration`.

**Fix:** `list(_new.items())` snapshots the items before the loop.  Also,
`diff_del` was rewritten as a standalone recursive function that never
stores aliases to its inputs.

### `merge_delete`: empty dict wrongly pruned (fixed)

When the new state had an empty dict `{}` at a location where the old state
had a non-empty dict, `merge_delete` would delete all sub-keys and then
prune the now-empty parent key entirely.  The result differed from `new`,
which retained the parent key as `{}`.

Root cause: the old delta format stored the full old sub-dict for both "key
absent from new" and "key present but empty in new" — the two cases were
indistinguishable at merge time.

**Fix:** `diff_del` now stores `None` for dict keys entirely absent from
`new`, and a non-empty sub-dict for partial deletions.  `merge_delete`
no longer prunes: a `None` (or scalar) value in the delta is the explicit
signal to remove the parent key.