we_monitor/conversation_recap.md

Conversation Recap — we_monitor development

Overview

Starting from a single collect.py script that polled a VW WeConnect vehicle and wrote JSON to disk, the project was iteratively extended into a modular Python package with a TCP push server and a PyQt5 / pyqtgraph GUI client.

---

Feature development (in order)

1. Credentials file (-c / --credentials)

Added a JSON credentials file parameter alongside the existing -u / -p flags. File supports username, password, vin, and domains keys. Domains may be a JSON array or a comma-separated string. Explicit CLI flags and env vars (WC_USER / WC_PASS) take precedence over the file.

2. --list-domains

Added a CLI flag that prints all available domain names and exits.

3. Physical value format

Refactored all numeric measurements to {"value": x, "unit": "km"} objects, stripping unit suffixes from field names (e.g. odometer_km → odometer).

4. Default log path + auto-create directory

Output defaults to logs/YYYY_MM_DD_HH_MM_SS_<VIN>.json; the logs/ directory is created automatically if it does not exist.

5. Midnight UTC log rotation

When using the default logs/ directory the collector starts a new file after midnight UTC. The old file is kept intact; the new file is auto-named with the current date stamp.

6. TCP push server

Added --host / --port flags (default 0.0.0.0:9999). The server pushes every new snapshot to all connected clients as newline-delimited JSON immediately after each poll. The last snapshot is sent to newly connecting clients so they have something to display right away.

7. CLI test client (client.py)

Connects to the push server, pretty-prints every incoming JSON record, and prints a running record counter after each reception.

8. Refactor into modules

collect.py was split into focused server-side modules:

Module	Responsibility
server/network.py	TCP accept loop, broadcast
server/data_model.py	Domain extractors, collect_snapshot()
server/storage_helpers.py	Load / save JSON store, load_today_records()
server/we_connect.py	WeConnect login and vehicle selection
server/log_config.py	logging.basicConfig setup

collect.py remains in the project root as the sole entry point.

9. PyQt5 GUI client (gui_client.py)

Three-tab application:

• Connector — host / port, Connect / Disconnect, status label.
• Dashboard — QTreeWidget showing domain → object → field/value/unit; expand state preserved across live updates.
• Plot — time-series charts of all physical fields using pyqtgraph.

10. Full-day history on connect

On client connect the server now sends all records from the current day instead of only the most recent snapshot. Records are loaded from disk at startup into push_store_ref["preloaded"] and merged with live in-memory records (dedup by ts, sorted).

11. Plot tab — user-selectable traces (pyqtgraph)

Replaced static matplotlib plots with four pyqtgraph PlotWidgets. Users select up to 4 traces per plot from a dropdown; zoom/pan via mouse; crosshair cursor with nearest-value readout.

12. 8 plots with scrolling

Extended to 8 plots in a 4 × 2 scrollable grid (QScrollArea). Per-plot control bars (add-combo + + button + colored chip buttons) sit directly above each plot.

13. Abbreviated trace names + label mapping

_TRACE_LABELS dict maps full dotted keys to short human-readable labels (e.g. charging.batteryStatus.currentSOC → "SOC"). Combo boxes, chip buttons, legends, and cursor readout all use the short label; the full key is stored as item data.

14. Persist plot settings

Trace selections saved to ~/.config/we_monitor/plot_settings.json on every add/remove. Restored silently on the next launch once matching signals arrive from the server.

15. Status bar record counter

The main window status bar shows Last snapshot: <ts> | Records: <N>. Counter resets on each new connection.

16. Dark theme (nmea_client palette)

Color palette copied from nmea_client/gui/plot_widget.py:

Role	Hex
Main background	#1a1a2e
Plot / input background	#12121e
Borders	#2a2a4a / #3a3a5e
Normal text	#aaaacc
Bright text	#ddddee
Connected	#4CAF50
Error	#F44336

Trace colors use the Material palette from nmea_client (#2196F3, #F44336, #4CAF50, #FF9800, …).

---

Bugs fixed

Socket read timeout (gui_client spurious disconnect)

socket.create_connection(timeout=5) sets a permanent 5-second read timeout. After the first message, if the collection interval > 5 s, recv() raised socket.timeout and the GUI reported "Connection lost". Fix: self._sock.settimeout(None) immediately after connect.

load_today() never called after first poll

The lazy-load condition if not records was never true once the first WeConnect poll had written a record into memory, so disk history was never loaded for late-connecting clients. Fix: Eager loading — load_today_records() called at startup, result stored in push_store_ref["preloaded"]; merged with live records on every connect.

Erratic axis scaling on mouse hover

pyqtgraph's InfiniteLine (cursor crosshair) was included in auto-range calculations. Moving the cursor triggered view rescaling on every mouse event. Fix: pw.addItem(vl, ignoreBounds=True) in both the initial setup and after each _redraw() call.

---

Key design decisions

• store_ref mutable dict — shared between the main thread and the accept loop thread under a single threading.Lock. Keys: "current" (live store) and "preloaded" (today's disk records).
• NDJSON protocol — one JSON object per line over TCP; trivial to consume with nc, python client.py, or the GUI.
• pyqtgraph over matplotlib — chosen for native Qt integration, live update performance, and built-in zoom/pan without extra toolbars.
• _TRACE_LABELS dict — single place to maintain display names; falls back to key.split(".")[-1] for unknown keys.
• QComboBox item data — short label displayed, full key stored via addItem(label, key) / currentData(); no reverse-lookup dict needed.