refactored

This commit is contained in:
2026-05-27 13:11:05 +02:00
parent 519571fe55
commit 84d5e89728
4 changed files with 0 additions and 0 deletions
+195
View File
@@ -0,0 +1,195 @@
# jay_diff — JSON diff / patch library
`utils/jay_diff.py` is a lightweight, pure-Python library for computing and
applying structural diffs between two JSON-compatible dicts. It is used by
`collect.py` and `gui_client.py` to compress the push-server stream: instead
of retransmitting full snapshots on every poll, the server sends only the
fields that changed since the previous snapshot.
---
## Concepts
A **diff** is itself a plain dict. It describes what changed from an **old**
state to a **new** state using one or more of these keys:
| Key | Meaning |
|-----|---------|
| `update_add` | Fields whose value changed *or* that are new in `new` |
| `update` | Fields whose value changed (must already exist in `old`) |
| `add` | Fields that are new in `new` (did not exist in `old`) |
| `delete` | Fields that were removed or whose sub-keys were pruned |
`update_add` combines `update` and `add` into a single key (the mode used
everywhere in this project). The separate `update` / `add` form is
available via `combine_upd_add=False` for completeness.
An **empty dict `{}`** returned by `jay_diff_full` means the two states are
identical.
### Delete-delta encoding
Values inside the `delete` sub-dict carry a specific meaning:
| Value in delete delta | Effect when merged |
|-----------------------|--------------------|
| `None` | The **entire key** is deleted from `old` |
| A **non-empty dict** | Recurse into the sub-dict and delete those specific sub-keys; the parent key is kept (possibly as `{}`) |
| A **scalar** (string, int, …) | The key is deleted from `old` |
| `{}` (empty dict) | The key is deleted from `old` |
This distinction matters when the new state has an empty dict `{}` at a
location where the old state had a populated dict. In that case the delete
delta carries the populated sub-keys as a non-empty dict — after merging,
the parent key is preserved as `{}`. If the key is entirely absent from
the new state, `None` is stored in the delta, and the merge removes the key
completely.
---
## API
### Diff functions
```python
jay_diff_full(old, new, combine_upd_add=True) -> dict
```
The main entry point. Computes a complete diff from `old` to `new`.
- `combine_upd_add=True` (default) — produces `update_add` + `delete`
- `combine_upd_add=False` — produces `update` + `add` + `delete`
Returns `{}` when `old == new`.
```python
jay_diff_add(old, new) -> dict
```
Returns keys present in `new` but absent from `old` (additions only).
```python
jay_diff_del(old, new) -> dict
```
Returns keys present in `old` but absent from `new`, or sub-keys that were
removed from a shared dict. Uses `None` as sentinel for entirely-absent
dict keys.
```python
jay_diff_upd(old, new) -> dict
```
Returns keys present in both `old` and `new` whose values differ (updates
only; new keys are ignored).
```python
jay_diff_upd_add(old, new) -> dict
```
Like `jay_diff_upd` but also includes keys new in `new` — equivalent to
`jay_diff_full(..., combine_upd_add=True)` minus the delete section.
---
### Merge functions
```python
jay_merge_full(old, diff) -> dict
```
The main entry point. Applies a diff produced by `jay_diff_full` to `old`
and returns the reconstructed new state.
**Bootstrap passthrough:** if `old` is `{}` and `diff` contains none of the
recognised diff keys, `diff` is returned as-is. This is how the first
message from the push server is handled: the server sends
`jay_diff_full({}, record[0])`, which looks like a plain snapshot, and the
client stores it directly.
Applies operations in order: `update``update_add``add``delete`.
```python
jay_merge_update(old, new) -> dict # apply an update/update_add delta
jay_merge_add(old, new) -> dict # apply an add delta
jay_merge_delete(old, new) -> dict # apply a delete delta
```
Low-level helpers; prefer `jay_merge_full` unless you need fine-grained
control.
---
## Round-trip guarantee
```
jay_merge_full(old, jay_diff_full(old, new)) == new
```
This holds for all inputs where values are JSON-compatible scalars (strings,
numbers, booleans, `None`, lists) or nested dicts thereof. It is verified
by `tests/test_jay_diff_real_data.py` against 674 consecutive pairs of real
vehicle snapshots.
---
## Wire format
`jay_diff_full` / `jay_merge_full` are used by the push server when
`--diff` is enabled. The diff dict is serialised as a newline-delimited
JSON line.
- The **first** message to a new client is `jay_diff_full({}, record[0])`,
which always contains `update_add` with every field of the first record.
- Subsequent messages contain only the changed fields.
- The format is self-describing: a message with none of the diff keys is
treated as a full snapshot by `jay_merge_full` (bootstrap passthrough).
`null` in JSON round-trips to Python `None`, so the delete sentinel survives
serialisation transparently.
---
## Tests
```
tests/test_jay_diff.py — 104 unit tests (all diff/merge functions,
corner cases: type changes, empty dicts,
deep nesting, round-trips)
tests/test_jay_diff_real_data.py — parametrised over 675 real VW snapshots
in tests/data/ (674 consecutive pairs):
round-trip, identical diff = {}, and
empty-diff-is-noop checks
```
Run with:
```bash
pytest tests/
```
---
## Bug history
### `jay_merge_delete`: crash on aliased dicts (fixed)
`jay_diff_rev` (used internally by the old `jay_diff_del`) stored references
to sub-dicts of `_old` directly in the delete delta without copying them.
When `jay_merge_full(old, jay_diff_full(old, new))` was called, some
sub-dicts in the delete delta were the same Python objects as sub-dicts in
`old`. Iterating `_new.items()` while deleting from `_old` (the same
object) raised `RuntimeError: dictionary changed size during iteration`.
**Fix:** `list(_new.items())` snapshots the items before the loop. Also,
`jay_diff_del` was rewritten as a standalone recursive function that never
stores aliases to its inputs.
### `jay_merge_delete`: empty dict wrongly pruned (fixed)
When the new state had an empty dict `{}` at a location where the old state
had a non-empty dict, `jay_merge_delete` would delete all sub-keys and then
prune the now-empty parent key entirely. The result differed from `new`,
which retained the parent key as `{}`.
Root cause: the old delta format stored the full old sub-dict for both "key
absent from new" and "key present but empty in new" — the two cases were
indistinguishable at merge time.
**Fix:** `jay_diff_del` now stores `None` for dict keys entirely absent from
`new`, and a non-empty sub-dict for partial deletions. `jay_merge_delete`
no longer prunes: a `None` (or scalar) value in the delta is the explicit
signal to remove the parent key.
+4
View File
@@ -0,0 +1,4 @@
from diff.diff import (
jay_diff_add, jay_diff_del, jay_diff_upd, jay_diff_upd_add,
jay_diff_full, jay_merge_add, jay_merge_delete, jay_merge_update, jay_merge_full,
)
+333
View File
@@ -0,0 +1,333 @@
from copy import deepcopy
def diff_rev(_old, _new):
delta = {}
if not isinstance(_old, dict):
return delta
for k in _new.keys():
if k in _old:
if isinstance(_new[k], dict):
res = diff_rev(_old[k], _new[k])
if len(res.keys()):
delta[k] = res
else:
_old[k] = _new[k]
delta[k] = _new[k]
return delta
def diff_add(_old, _new):
__old = deepcopy(_old)
delta = diff_rev(__old, _new)
return delta
def diff_del(_old, _new):
delta = {}
for k in _old:
if k in _new:
if isinstance(_old[k], dict) and isinstance(_new[k], dict):
sub = diff_del(_old[k], _new[k])
if sub:
delta[k] = sub
else:
# key absent from _new: None signals "delete entire key" for dicts
delta[k] = None if isinstance(_old[k], dict) else _old[k]
return delta
def diff_upd(_old, _new, add_if_not_exist=False):
delta = {}
for k in _new.keys():
if k in _old:
if isinstance(_new[k], dict):
res = diff_upd(_old[k], _new[k], add_if_not_exist)
if len(res.keys()):
delta[k] = res
elif _new[k] != _old[k]:
delta[k] = _new[k]
elif add_if_not_exist:
delta[k] = _new[k]
return delta
def diff_upd_add(_old, _new):
return diff_upd(_old, _new, add_if_not_exist=True)
def diff_full(_old, _new, combine_upd_add=True):
diff = {}
if combine_upd_add:
res = diff_upd_add(_old, _new)
if len(res.keys()):
diff['update_add'] = res
res = diff_del(_old, _new)
if len(res.keys()):
diff['delete'] = res
else:
res = diff_upd(_old, _new)
if len(res.keys()):
diff['update'] = res
res = diff_add(_old, _new)
if len(res.keys()):
diff['add'] = res
res = diff_del(_old, _new)
if len(res.keys()):
diff['delete'] = res
return diff
def _legacy_assertions() -> None:
result = diff_add({}, {'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke', 'f': 'coffee'}})
assert result == {'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}
result = diff_add({'a': 'hamburger', 'b': 'fries', 'c': 'pepsi'}, {'a': 'hamburger', 'b': 'fries', 'c': 'coke'})
assert result == {}
result = diff_add({'a': 'hamburger', 'b': 'fries', 'c': 'pepsi', 'd': 'icecream'}, {'a': 'hamburger', 'b': 'fries', 'c': 'pepsi'})
assert result == {}
result = diff_add({'a': 'hamburger', 'b': 'fries', 'c': 'pepsi'}, {'a': 'hamburger', 'd': 'icecream', 'c': 'coke'})
assert result == {'d':'icecream'}
result = diff_add({'a': 'hamburger', 'c': 'pepsi', 'd': 'icecream'}, {'a': 'hamburger', 'b': 'fries', 'c': 'coke'})
assert result == {'b':'fries'}
result = diff_add({'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke'}}, {'c': {'f': 'coffee'}})
assert result == {'c': {'f': 'coffee'}}
result = diff_add({'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'a': 'hamburger', 'b': 'fries', 'c': {'f': 'coffee'}})
assert result == {}
result = diff_del({'a': 'hamburger', 'c': 'pepsi', 'd': 'icecream'}, {'a': 'hamburger', 'b': 'fries', 'c': 'coke'})
assert result == {'d':'icecream'}
result = diff_del({'a': 'hamburger', 'b': 'fries', 'c': 'coke'}, {'a': 'hamburger', 'b': 'fries', 'c': 'coke'})
assert result == {}
result = diff_del({'a': 'hamburger', 'b': 'fries', 'c': 'coke'}, {'a': 'hamburger', 'b': 'fries', 'c': 'pepsi'})
assert result == {}
result = diff_del({'a': 'hamburger', 'b': 'fries', 'c': 'pepsi'}, {'a': 'hamburger', 'b': 'fries'})
assert result == {'c':'pepsi'}
result = diff_del({'a': 'hamburger', 'b': 'fries', 'c': 'pepsi'}, {'a': 'hamburger', 'b': 'fries', 'c': 'coke'})
assert result == {}
result = diff_del({'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'a': 'hamburger', 'b': 'fries'})
assert result == {'c': None}
result = diff_del({'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'c': {'e': 'coke'}})
assert result == {'a':'hamburger', 'b':'fries', 'c': {'f': 'coffee'}}
result = diff_del({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'a':'hamburger', 'b':'fries', 'c': {'e': 'pepsi', 'f': 'coffee'}})
assert result == {}
result = diff_upd({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'a':'hotdog', 'b':'potatoes'})
assert result == {'a':'hotdog', 'b':'potatoes'}
result = diff_upd({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'c': {'e': 'pepsi'}})
assert result == {'c': {'e': 'pepsi'}}
result = diff_upd({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'a':'hotdog', 'b':'fries'})
assert result == {'a':'hotdog'}
result = diff_upd({'a': 'hamburger', 'b': 'fries', 'c': 'pepsi'}, {'a': 'hamburger', 'd': 'icecream', 'c': 'coke'})
assert result == {'c':'coke'}
result = diff_upd({'a': 'hamburger', 'c': 'pepsi', 'd': 'icecream'}, {'a': 'hamburger', 'b': 'fries', 'c': 'coke'})
assert result == {'c':'coke'}
result = diff_upd({'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke'}}, {'c': {'f': 'coffee'}})
assert result == {}
result = diff_upd({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'a':'hamburger', 'b':'fries', 'c': {'e': 'pepsi', 'f': 'coffee'}})
assert result == {'c': {'e': 'pepsi'}}
# Update/add
result = diff_upd_add({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'a':'hotdog', 'b':'potatoes', 'd': 'icecream'})
assert result == {'a':'hotdog', 'b':'potatoes', 'd': 'icecream'}
result = diff_upd_add({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'c': {'e': 'pepsi', 'g': 'tea', 'h': 'cake'}, 'b':'potatoes'})
assert result == {'b':'potatoes', 'c': {'e': 'pepsi', 'g': 'tea', 'h': 'cake'}}
result = diff_upd_add({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'a':'hotdog', 'b':'fries'})
assert result == {'a':'hotdog'}
result = diff_upd_add({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'a':'hamburger', 'b':'fries', 'c': {'e': 'pepsi', 'f': 'coffee'}})
assert result == {'c': {'e': 'pepsi'}}
#####################
result = diff_upd_add({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'a':'hotdog', 'b':'potatoes'})
assert result == {'a':'hotdog', 'b':'potatoes'}
result = diff_upd_add({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'c': {'e': 'pepsi'}})
assert result == {'c': {'e': 'pepsi'}}
result = diff_upd_add({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'a':'hotdog', 'b':'fries'})
assert result == {'a':'hotdog'}
result = diff_upd_add({'a': 'hamburger', 'b': 'fries', 'c': 'pepsi'}, {'a': 'hamburger', 'd': 'icecream', 'c': 'coke'})
assert result == {'c': 'coke', 'd': 'icecream'}
result = diff_upd_add({'a': 'hamburger', 'c': 'pepsi', 'd': 'icecream'}, {'a': 'hamburger', 'b': 'fries', 'c': 'coke'})
assert result == {'b': 'fries', 'c': 'coke'}
result = diff_upd_add({'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke'}}, {'c': {'f': 'coffee'}})
assert result == {'c': {'f': 'coffee'}}
result = diff_upd_add({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'a':'hamburger', 'b':'fries', 'c': {'e': 'pepsi', 'f': 'coffee'}})
assert result == {'c': {'e': 'pepsi'}}
result = diff_upd_add({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'a':'hamburger', 'b':'fries', 'c': {'e': 'pepsi', 'f': 'coffee', 'h': 'cake'}})
assert result == {'c': {'e': 'pepsi', 'h': 'cake'}}
# diff_full
combine_upd_add = False
result = diff_full({}, {'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, combine_upd_add)
assert result == {'add': {'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke', 'f': 'coffee'}}}
result = diff_full({'a': 'hamburger', 'b': 'fries', 'c': 'pepsi'}, {'a': 'hamburger', 'b': 'fries', 'c': 'coke'}, combine_upd_add)
assert result == {'update': {'c': 'coke'}}
result = diff_full({'a': 'hamburger', 'b': 'fries', 'c': 'pepsi', 'd': 'icecream'}, {'a': 'hamburger', 'b': 'fries', 'c': 'pepsi'}, combine_upd_add)
assert result == {'delete': {'d': 'icecream'}}
result = diff_full({'a': 'hamburger', 'b': 'fries', 'c': 'pepsi'}, {'a': 'hamburger', 'd': 'icecream', 'c': 'coke'}, combine_upd_add)
assert result == {'add': {'d': 'icecream'}, 'delete': {'b': 'fries'}, 'update': {'c': 'coke'}}
result = diff_full({'a': 'hamburger', 'c': 'pepsi', 'd': 'icecream'}, {'a': 'hamburger', 'b': 'fries', 'c': 'coke'}, combine_upd_add)
assert result == {'add': {'b': 'fries'}, 'delete': {'d': 'icecream'}, 'update': {'c': 'coke'}}
result = diff_full({'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke'}}, {'c': {'f': 'coffee'}}, combine_upd_add)
assert result == {'add': {'c': {'f': 'coffee'}}, 'delete': {'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke'}}}
result = diff_full({'a': 'hamburger', 'c': 'pepsi', 'd': 'icecream'}, {'a': 'hamburger', 'b': 'fries', 'c': 'coke'}, combine_upd_add)
assert result == {'add': {'b': 'fries'}, 'delete': {'d': 'icecream'}, 'update': {'c': 'coke'}}
result = diff_full({'a': 'hamburger', 'b': 'fries', 'c': 'coke'}, {'a': 'hamburger', 'b': 'fries', 'c': 'coke'}, combine_upd_add)
assert result == {}
result = diff_full({'a': 'hamburger', 'b': 'fries', 'c': 'coke'}, {'a': 'hamburger', 'b': 'fries', 'c': 'pepsi'}, combine_upd_add)
assert result == {'update': {'c': 'pepsi'}}
result = diff_full({'a': 'hamburger', 'b': 'fries', 'c': 'pepsi'}, {'a': 'hamburger', 'b': 'fries'}, combine_upd_add)
assert result == {'delete': {'c': 'pepsi'}}
result = diff_full({'a': 'hamburger', 'b': 'fries', 'c': 'pepsi'}, {'a': 'hamburger', 'b': 'fries', 'c': 'coke'}, combine_upd_add)
assert result == {'update': {'c': 'coke'}}
result = diff_full({'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'a': 'hamburger', 'b': 'fries'}, combine_upd_add)
assert result == {'delete': {'c': None}}
result = diff_full({'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'a': 'hamburger', 'b': 'fries', 'c': {'f': 'coffee'}}, combine_upd_add)
assert result == {'delete': {'c': {'e': 'coke'}}}
result = diff_full({'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'c': {'e': 'coke'}}, combine_upd_add)
assert result == {'delete': {'a': 'hamburger', 'b': 'fries', 'c': {'f': 'coffee'}}}
result = diff_full({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'a':'hotdog', 'b':'potatoes'}, combine_upd_add)
assert result == {'delete': {'c': None}, 'update': {'a': 'hotdog', 'b': 'potatoes'}}
result = diff_full({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'c': {'e': 'pepsi'}}, combine_upd_add)
assert result == {'delete': {'a': 'hamburger', 'b': 'fries', 'c': {'f': 'coffee'}}, 'update': {'c': {'e': 'pepsi'}}}
result = diff_full({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'a':'hotdog', 'b':'fries'}, combine_upd_add)
assert result == {'delete': {'c': None}, 'update': {'a': 'hotdog'}}
result = diff_full({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'a':'hamburger', 'b':'fries', 'c': {'e': 'pepsi', 'f': 'coffee'}}, combine_upd_add)
assert result == {'update': {'c': {'e': 'pepsi'}}}
result = diff_full({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'a':'hotdog', 'b':'potatoes', 'd': 'icecream'}, combine_upd_add)
assert result == {'add': {'d': 'icecream'}, 'delete': {'c': None}, 'update': {'a': 'hotdog', 'b': 'potatoes'}}
result = diff_full({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'c': {'e': 'pepsi', 'g': 'tea', 'h': 'cake'}, 'b':'potatoes'}, combine_upd_add)
assert result == {'add': {'c': {'g': 'tea', 'h': 'cake'}}, 'delete': {'a': 'hamburger', 'c': {'f': 'coffee'}}, 'update': {'b': 'potatoes', 'c': {'e': 'pepsi'}}}
result = diff_full({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'a':'hotdog', 'b':'fries'}, combine_upd_add)
assert result == {'delete': {'c': None}, 'update': {'a': 'hotdog'}}
result = diff_full({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'a':'hamburger', 'b':'fries', 'c': {'e': 'pepsi', 'f': 'coffee'}}, combine_upd_add)
assert result == {'update': {'c': {'e': 'pepsi'}}}
result = diff_full({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'a':'hamburger', 'b':'fries', 'c': {'e': 'pepsi', 'f': 'coffee', 'h': 'cake'}}, combine_upd_add)
assert result == {'add': {'c': {'h': 'cake'}}, 'update': {'c': {'e': 'pepsi'}}}
# ToDo: diff_full
combine_upd_add = True
result = diff_full({}, {'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, combine_upd_add)
assert result == {'update_add': {'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke', 'f': 'coffee'}}}
result = diff_full({'a': 'hamburger', 'b': 'fries', 'c': 'pepsi'}, {'a': 'hamburger', 'b': 'fries', 'c': 'coke'}, combine_upd_add)
assert result == {'update_add': {'c': 'coke'}}
result = diff_full({'a': 'hamburger', 'b': 'fries', 'c': 'pepsi', 'd': 'icecream'}, {'a': 'hamburger', 'b': 'fries', 'c': 'pepsi'}, combine_upd_add)
assert result == {'delete': {'d': 'icecream'}}
result = diff_full({'a': 'hamburger', 'b': 'fries', 'c': 'pepsi'}, {'a': 'hamburger', 'd': 'icecream', 'c': 'coke'}, combine_upd_add)
assert result == {'delete': {'b': 'fries'}, 'update_add': {'c': 'coke', 'd': 'icecream'}}
result = diff_full({'a': 'hamburger', 'c': 'pepsi', 'd': 'icecream'}, {'a': 'hamburger', 'b': 'fries', 'c': 'coke'}, combine_upd_add)
assert result == {'delete': {'d': 'icecream'}, 'update_add': {'b': 'fries', 'c': 'coke'}}
result = diff_full({'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke'}}, {'c': {'f': 'coffee'}}, combine_upd_add)
assert result == {'update_add': {'c': {'f': 'coffee'}}, 'delete': {'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke'}}}
result = diff_full({'a': 'hamburger', 'c': 'pepsi', 'd': 'icecream'}, {'a': 'hamburger', 'b': 'fries', 'c': 'coke'}, combine_upd_add)
assert result == {'delete': {'d': 'icecream'}, 'update_add': {'b': 'fries', 'c': 'coke'}}
result = diff_full({'a': 'hamburger', 'b': 'fries', 'c': 'coke'}, {'a': 'hamburger', 'b': 'fries', 'c': 'coke'}, combine_upd_add)
assert result == {}
result = diff_full({'a': 'hamburger', 'b': 'fries', 'c': 'coke'}, {'a': 'hamburger', 'b': 'fries', 'c': 'pepsi'}, combine_upd_add)
assert result == {'update_add': {'c': 'pepsi'}}
result = diff_full({'a': 'hamburger', 'b': 'fries', 'c': 'pepsi'}, {'a': 'hamburger', 'b': 'fries'}, combine_upd_add)
assert result == {'delete': {'c': 'pepsi'}}
result = diff_full({'a': 'hamburger', 'b': 'fries', 'c': 'pepsi'}, {'a': 'hamburger', 'b': 'fries', 'c': 'coke'}, combine_upd_add)
assert result == {'update_add': {'c': 'coke'}}
result = diff_full({'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'a': 'hamburger', 'b': 'fries'}, combine_upd_add)
assert result == {'delete': {'c': None}}
result = diff_full({'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'a': 'hamburger', 'b': 'fries', 'c': {'f': 'coffee'}}, combine_upd_add)
assert result == {'delete': {'c': {'e': 'coke'}}}
result = diff_full({'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'c': {'e': 'coke'}}, combine_upd_add)
assert result == {'delete': {'a': 'hamburger', 'b': 'fries', 'c': {'f': 'coffee'}}}
result = diff_full({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'a':'hotdog', 'b':'potatoes'}, combine_upd_add)
assert result == {'delete': {'c': None}, 'update_add': {'a': 'hotdog', 'b': 'potatoes'}}
result = diff_full({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'c': {'e': 'pepsi'}}, combine_upd_add)
assert result == {'delete': {'a': 'hamburger', 'b': 'fries', 'c': {'f': 'coffee'}}, 'update_add': {'c': {'e': 'pepsi'}}}
result = diff_full({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'a':'hotdog', 'b':'fries'}, combine_upd_add)
assert result == {'delete': {'c': None}, 'update_add': {'a': 'hotdog'}}
result = diff_full({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'a':'hamburger', 'b':'fries', 'c': {'e': 'pepsi', 'f': 'coffee'}}, combine_upd_add)
assert result == {'update_add': {'c': {'e': 'pepsi'}}}
result = diff_full({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'a':'hotdog', 'b':'potatoes', 'd': 'icecream'}, combine_upd_add)
assert result == {'delete': {'c': None}, 'update_add': {'a': 'hotdog', 'b': 'potatoes', 'd': 'icecream'}}
result = diff_full({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'c': {'e': 'pepsi', 'g': 'tea', 'h': 'cake'}, 'b':'potatoes'}, combine_upd_add)
assert result == {'delete': {'a': 'hamburger', 'c': {'f': 'coffee'}}, 'update_add': {'b': 'potatoes', 'c': {'e': 'pepsi', 'g': 'tea', 'h': 'cake'}}}
result = diff_full({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'a':'hotdog', 'b':'fries'}, combine_upd_add)
assert result == {'delete': {'c': None}, 'update_add': {'a': 'hotdog'}}
result = diff_full({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'a':'hamburger', 'b':'fries', 'c': {'e': 'pepsi', 'f': 'coffee'}}, combine_upd_add)
assert result == {'update_add': {'c': {'e': 'pepsi'}}}
result = diff_full({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'a':'hamburger', 'b':'fries', 'c': {'e': 'pepsi', 'f': 'coffee', 'h': 'cake'}}, combine_upd_add)
assert result == {'update_add': {'c': {'e': 'pepsi', 'h': 'cake'}}}
if __name__ == '__main__':
_legacy_assertions()
+262
View File
@@ -0,0 +1,262 @@
from copy import deepcopy
def merge_update(_old, _new):
for k, v in _new.items():
if isinstance(v, dict):
r = merge_update(_old.get(k, {}), v)
_old[k] = r
else:
if isinstance(_old, dict):
_old[k] = _new[k]
else:
_old = _new
return _old
def merge_add(_old, _new):
for k, v in _new.items():
if isinstance(v, dict):
r = merge_add(_old.get(k, {}), v)
if k in _old:
_old[k].update(r)
else:
_old[k] = r
else:
_old[k] = _new[k]
return _old
def merge_delete(_old, _new):
for k, v in list(_new.items()):
if isinstance(v, dict) and v:
if k in _old and isinstance(_old[k], dict):
merge_delete(_old[k], v)
else:
if k in _old:
del _old[k]
return _old
def merge_full(_old, _diff):
_is_diff = False
res = _old
if "update" in _diff:
res = merge_update(res, _diff['update'])
_is_diff = True
if "update_add" in _diff:
res = merge_update(res, _diff['update_add'])
_is_diff = True
if "add" in _diff:
res = merge_add(res, _diff['add'])
_is_diff = True
if "delete" in _diff:
res = merge_delete(res, _diff['delete'])
_is_diff = True
if not _is_diff and not _old:
res = _diff
return res
def _legacy_assertions() -> None:
# Merge
result = merge_add({}, {'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke', 'f': 'coffee'}})
assert result == {'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}
result = merge_add({'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'d': 'icecream'})
assert result == {'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}, 'd':'icecream'}
result = merge_add({'a': 'hamburger', 'b': 'fries', 'c': {'f': 'coffee'}}, {'d': 'icecream'})
assert result == {'a':'hamburger', 'b':'fries', 'c': {'f': 'coffee'}, 'd':'icecream'}
result = merge_add({'a': 'hamburger', 'b': 'fries', 'c': {'f': 'coffee'}, 'd': 'icecream'}, {'d': 'icecream'})
assert result == {'a':'hamburger', 'b':'fries', 'c': {'f': 'coffee'}, 'd':'icecream'}
result = merge_delete({'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke', 'f': 'coffee'}, 'd': 'icecream'}, {'d': 'icecream'})
assert result == {'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}
result = merge_delete({'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'c': {'f': 'coffee'}})
assert result == {'a':'hamburger', 'b':'fries', 'c': {'e': 'coke'}}
result = merge_delete({'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'c': {'e': 'coke'}})
assert result == {'a':'hamburger', 'b':'fries', 'c': {'f': 'coffee'}}
# merge_full
# the no-jaydiff case
result = merge_full({}, {'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke', 'f': 'coffee'}})
assert result == {'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke', 'f': 'coffee'}}
# combine_upd_add = False
result = merge_full({'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'add': {'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke', 'f': 'coffee'}}})
assert result == {'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke', 'f': 'coffee'}}
result = merge_full({'a': 'hamburger', 'b': 'fries', 'c': 'coke'}, {'update': {'c': 'pepsi'}})
assert result == {'a': 'hamburger', 'b': 'fries', 'c': 'pepsi'}
result = merge_full({'a': 'hamburger', 'b': 'fries', 'c': 'pepsi', 'd': 'icecream'}, {'delete': {'d': 'icecream'}})
assert result == {'a': 'hamburger', 'b': 'fries', 'c': 'pepsi'}
result = merge_full({'a': 'hamburger', 'b': 'fries', 'c': 'pepsi'}, {'add': {'d': 'icecream'}, 'delete': {'b': 'fries'}, 'update': {'c': 'coke'}})
assert result == {'a': 'hamburger', 'd': 'icecream', 'c': 'coke'}
result = merge_full({'a': 'hamburger', 'c': 'pepsi', 'd': 'icecream'}, {'add': {'b': 'fries'}, 'delete': {'d': 'icecream'}, 'update': {'c': 'coke'}})
assert result == {'a': 'hamburger', 'b': 'fries', 'c': 'coke'}
result = merge_full({'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke'}}, {'add': {'c': {'f': 'coffee'}}, 'delete': {'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke'}}})
assert result == {'c': {'f': 'coffee'}}
result = merge_full({'a': 'hamburger', 'c': 'pepsi', 'd': 'icecream'}, {'add': {'b': 'fries'}, 'delete': {'d': 'icecream'}, 'update': {'c': 'coke'}})
assert result == {'a': 'hamburger', 'b': 'fries', 'c': 'coke'}
result = merge_full({'a': 'hamburger', 'b': 'fries', 'c': 'coke'}, {})
assert result == {'a': 'hamburger', 'b': 'fries', 'c': 'coke'}
result = merge_full({'a': 'hamburger', 'b': 'fries', 'c': 'coke'}, {'update': {'c': 'pepsi'}})
assert result == {'a': 'hamburger', 'b': 'fries', 'c': 'pepsi'}
result = merge_full({'a': 'hamburger', 'b': 'fries', 'c': 'pepsi'}, {'delete': {'c': 'pepsi'}})
assert result == {'a': 'hamburger', 'b': 'fries'}
result = merge_full({'a': 'hamburger', 'b': 'fries', 'c': 'pepsi'}, {'update': {'c': 'coke'}})
assert result == {'a': 'hamburger', 'b': 'fries', 'c': 'coke'}
result = merge_full({'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'delete': {'c': None}})
assert result == {'a': 'hamburger', 'b': 'fries'}
result = merge_full({'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'delete': {'c': {'e': 'coke'}}})
assert result == {'a': 'hamburger', 'b': 'fries', 'c': {'f': 'coffee'}}
result = merge_full({'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'delete': {'a': 'hamburger', 'b': 'fries', 'c': {'f': 'coffee'}}})
assert result == {'c': {'e': 'coke'}}
result = merge_full({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'delete': {'c': None}, 'update': {'a': 'hotdog', 'b': 'potatoes'}})
assert result == {'a':'hotdog', 'b':'potatoes'}
result = merge_full({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'delete': {'a': 'hamburger', 'b': 'fries', 'c': {'f': 'coffee'}}, 'update': {'c': {'e': 'pepsi'}}})
assert result == {'c': {'e': 'pepsi'}}
result = merge_full({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'delete': {'c': None}, 'update': {'a': 'hotdog'}})
assert result == {'a':'hotdog', 'b':'fries'}
result = merge_full({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'update': {'c': {'e': 'pepsi'}}})
assert result == {'a':'hamburger', 'b':'fries', 'c': {'e': 'pepsi', 'f': 'coffee'}}
result = merge_full({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'add': {'d': 'icecream'}, 'delete': {'c': None}, 'update': {'a': 'hotdog', 'b': 'potatoes'}})
assert result == {'a':'hotdog', 'b':'potatoes', 'd': 'icecream'}
result = merge_full({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'add': {'c': {'g': 'tea', 'h': 'cake'}}, 'delete': {'a': 'hamburger', 'c': {'f': 'coffee'}}, 'update': {'b': 'potatoes', 'c': {'e': 'pepsi'}}})
assert result == {'c': {'e': 'pepsi', 'g': 'tea', 'h': 'cake'}, 'b':'potatoes'}
result = merge_full({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'delete': {'c': None}, 'update': {'a': 'hotdog'}})
assert result == {'a':'hotdog', 'b':'fries'}
result = merge_full({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'update': {'c': {'e': 'pepsi'}}})
assert result == {'a':'hamburger', 'b':'fries', 'c': {'e': 'pepsi', 'f': 'coffee'}}
result = merge_full({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'add': {'c': {'h': 'cake'}}, 'update': {'c': {'e': 'pepsi'}}})
assert result == {'a':'hamburger', 'b':'fries', 'c': {'e': 'pepsi', 'f': 'coffee', 'h': 'cake'}}
# ToDo: merge_full
# combine_upd_add = True
result = merge_full({}, {'update_add': {'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke', 'f': 'coffee'}}})
assert result == {'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke', 'f': 'coffee'}}
result = merge_full({'a': 'hamburger', 'b': 'fries', 'c': 'pepsi'}, {'update_add': {'c': 'coke'}})
assert result == {'a': 'hamburger', 'b': 'fries', 'c': 'coke'}
result = merge_full({'a': 'hamburger', 'b': 'fries', 'c': 'pepsi', 'd': 'icecream'}, {'delete': {'d': 'icecream'}})
assert result == {'a': 'hamburger', 'b': 'fries', 'c': 'pepsi'}
result = merge_full({'a': 'hamburger', 'b': 'fries', 'c': 'pepsi'}, {'delete': {'b': 'fries'}, 'update_add': {'c': 'coke', 'd': 'icecream'}})
assert result == {'a': 'hamburger', 'd': 'icecream', 'c': 'coke'}
result = merge_full({'a': 'hamburger', 'c': 'pepsi', 'd': 'icecream'}, {'delete': {'d': 'icecream'}, 'update_add': {'b': 'fries', 'c': 'coke'}})
assert result == {'a': 'hamburger', 'b': 'fries', 'c': 'coke'}
result = merge_full({'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke'}}, {'update_add': {'c': {'f': 'coffee'}}, 'delete': {'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke'}}})
assert result == {'c': {'f': 'coffee'}}
result = merge_full({'a': 'hamburger', 'c': 'pepsi', 'd': 'icecream'}, {'delete': {'d': 'icecream'}, 'update_add': {'b': 'fries', 'c': 'coke'}})
assert result == {'a': 'hamburger', 'b': 'fries', 'c': 'coke'}
result = merge_full({'a': 'hamburger', 'b': 'fries', 'c': 'coke'}, {})
assert result == {'a': 'hamburger', 'b': 'fries', 'c': 'coke'}
result = merge_full({'a': 'hamburger', 'b': 'fries', 'c': 'coke'}, {'update_add': {'c': 'pepsi'}})
assert result == {'a': 'hamburger', 'b': 'fries', 'c': 'pepsi'}
result = merge_full({'a': 'hamburger', 'b': 'fries', 'c': 'pepsi'}, {'delete': {'c': 'pepsi'}})
assert result == {'a': 'hamburger', 'b': 'fries'}
result = merge_full({'a': 'hamburger', 'b': 'fries', 'c': 'pepsi'}, {'update_add': {'c': 'coke'}})
assert result == {'a': 'hamburger', 'b': 'fries', 'c': 'coke'}
result = merge_full({'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'delete': {'c': None}})
assert result == {'a': 'hamburger', 'b': 'fries'}
result = merge_full({'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'delete': {'c': {'e': 'coke'}}})
assert result == {'a': 'hamburger', 'b': 'fries', 'c': {'f': 'coffee'}}
result = merge_full({'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'delete': {'a': 'hamburger', 'b': 'fries', 'c': {'f': 'coffee'}}})
assert result == {'c': {'e': 'coke'}}
result = merge_full({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'delete': {'c': None}, 'update_add': {'a': 'hotdog', 'b': 'potatoes'}})
assert result == {'a':'hotdog', 'b':'potatoes'}
result = merge_full({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'delete': {'a': 'hamburger', 'b': 'fries', 'c': {'f': 'coffee'}}, 'update_add': {'c': {'e': 'pepsi'}}})
assert result == {'c': {'e': 'pepsi'}}
result = merge_full({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'delete': {'c': None}, 'update_add': {'a': 'hotdog'}})
assert result == {'a':'hotdog', 'b':'fries'}
result = merge_full({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'update_add': {'c': {'e': 'pepsi'}}})
assert result == {'a':'hamburger', 'b':'fries', 'c': {'e': 'pepsi', 'f': 'coffee'}}
result = merge_full({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'delete': {'c': None}, 'update_add': {'a': 'hotdog', 'b': 'potatoes', 'd': 'icecream'}})
assert result == {'a':'hotdog', 'b':'potatoes', 'd': 'icecream'}
result = merge_full({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'delete': {'a': 'hamburger', 'c': {'f': 'coffee'}}, 'update_add': {'b': 'potatoes', 'c': {'e': 'pepsi', 'g': 'tea', 'h': 'cake'}}})
assert result == {'b':'potatoes', 'c': {'e': 'pepsi', 'g': 'tea', 'h': 'cake'}}
result = merge_full({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'delete': {'c': None}, 'update_add': {'a': 'hotdog'}})
assert result == {'a':'hotdog', 'b':'fries'}
result = merge_full({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'update_add': {'c': {'e': 'pepsi'}}})
assert result == {'a':'hamburger', 'b':'fries', 'c': {'e': 'pepsi', 'f': 'coffee'}}
result = merge_full({'a':'hamburger', 'b':'fries', 'c': {'e': 'coke', 'f': 'coffee'}}, {'update_add': {'c': {'e': 'pepsi', 'h': 'cake'}}})
assert result == {'a':'hamburger', 'b':'fries', 'c': {'e': 'pepsi', 'f': 'coffee', 'h': 'cake'}}
result = merge_full({}, {'add': {'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke', 'f': 'coffee'}}})
assert result == {'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke', 'f': 'coffee'}}
result = merge_full({}, {'add': {'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke', 'f': {'coffee': {'h': 'black'}}}}})
assert result == {'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke', 'f': {'coffee': {'h': 'black'}}}}
result = merge_full({'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke', 'f': {'coffee': {'h': 'black'}}}}, {'add': {'c': {'f': {'coffee': {'j': 'sugar'}}}}})
assert result == {'a': 'hamburger', 'b': 'fries', 'c': {'e': 'coke', 'f': {'coffee': {'h': 'black', 'j': 'sugar'}}}}
old = {'charging': {
'batteryStatus': {'value': {'carCapturedTimestamp': '2024-10-19T05:34:34Z', 'cruisingRangeElectric_km': 192}},
'chargingStatus': {'value': {'carCapturedTimestamp': '2024-10-19T05:34:34Z'}},
'chargingSettings': {'value': {'carCapturedTimestamp': '2024-10-19T05:34:30Z'}},
'plugStatus': {'value': {'carCapturedTimestamp': '2024-10-19T05:34:29Z'}}}}
update = {'charging': {
'batteryStatus': {'value': {'carCapturedTimestamp': '2024-10-19T05:35:46Z', 'cruisingRangeElectric_km': 191}},
'chargingStatus': {'value': {'carCapturedTimestamp': '2024-10-19T05:35:46Z'}}}}
add = {'charging': {
'batteryStatus': {'value': {'currentSOC_pct': 55}}},
'fuelStatus': {'rangeStatus': {'value': {'primaryEngine': {'currentSOC_pct': 55}}}},
'measurements': {'fuelLevelStatus': {'value': {'currentSOC_pct': 55}}}}
delete = {'charging': {
'chargingSettings': {'value': {'carCapturedTimestamp': '2024-10-19T05:34:30Z'}},
'plugStatus': {'value': {'carCapturedTimestamp': '2024-10-19T05:34:29Z'}}}}
result = merge_full(old, {'update':update, 'add':add, 'delete':delete})
assert result == old
if __name__ == '__main__':
_legacy_assertions()