Add README.md and conversation_recap.md

README covers installation, collector CLI reference, push server protocol, GUI client usage, and output format. conversation_recap documents the full development history: features, bugs fixed, and design decisions. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-25 22:42:00 +02:00
parent 9d7561ba2f
commit a01ae714c8
2 changed files with 311 additions and 0 deletions
@@ -0,0 +1,160 @@
+# we_monitor
+
+Periodically polls a VW WeConnect vehicle via the `weconnect` library, stores
+timestamped records as JSON, and pushes live data to connected TCP clients.
+A PyQt5 GUI client visualises the data in real time.
+
+## Repository layout
+
+```
+collect.py          — collector daemon (entry point)
+gui_client.py       — PyQt5 GUI client
+client.py           — minimal CLI test client
+server/
+  data_model.py     — domain extractors, collect_snapshot()
+  network.py        — TCP push server (accept loop + broadcast)
+  storage_helpers.py— load / save JSON store, midnight rotation helpers
+  we_connect.py     — WeConnect login and vehicle selection
+  log_config.py     — logging setup
+requirements.txt
+```
+
+## Requirements
+
+```
+pip install weconnect PyQt5 pyqtgraph
+```
+
+## Collector (`collect.py`)
+
+Connects to WeConnect, polls selected domains at a configurable interval, and
+appends records to a JSON file.  Simultaneously runs a TCP push server so
+clients receive every new snapshot the moment it is collected.
+
+### Quick start
+
+```bash
+# Credentials via environment variables, all defaults
+export WC_USER=me@example.com
+export WC_PASS=secret
+python collect.py
+
+# Credentials from a JSON file, 10-minute interval
+python collect.py -c credentials/my_car.json -i 600
+```
+
+### Credentials file (`-c FILE`)
+
+```json
+{
+  "username": "me@example.com",
+  "password": "secret",
+  "vin":      "WVWZZZE1ZMP000000",
+  "domains":  ["charging", "measurements", "readiness"]
+}
+```
+
+All keys are optional; explicit CLI flags and `WC_USER` / `WC_PASS`
+environment variables take precedence.
+
+### CLI reference
+
+| Flag | Default | Description |
+|------|---------|-------------|
+| `-c FILE` | — | JSON credentials file |
+| `-u / -p` | `WC_USER` / `WC_PASS` | Username / password |
+| `--vin` | first vehicle | Target VIN |
+| `-d DOMAINS` | `charging,measurements,readiness` | Comma-separated domains or `all` |
+| `-i SECONDS` | `300` | Collection interval |
+| `-o FILE` | `logs/YYYY_MM_DD_HH_MM_SS_<VIN>.json` | Output file |
+| `--max-records N` | unlimited | Keep only last N records |
+| `--host` | `0.0.0.0` | Push server bind address |
+| `--port` | `9999` | Push server TCP port |
+| `-v` | — | Verbose / debug logging |
+| `--list-domains` | — | Print available domains and exit |
+
+**Available domains:** `charging`, `climatisation`, `measurements`,
+`readiness`, `parking`, `access`
+
+### Log rotation
+
+When using the default `logs/` directory the collector automatically starts a
+new file after midnight UTC, keeping all previous files intact.  On startup the
+full day's records are pre-loaded from disk so newly connected clients receive
+the complete history.
+
+### Output format
+
+Each file is a JSON object:
+
+```json
+{
+  "meta": { "vin": "...", "created_at": "...", "interval_seconds": 300 },
+  "records": [
+    {
+      "ts": "2025-05-25T10:00:00+00:00",
+      "charging": {
+        "batteryStatus": {
+          "currentSOC":          { "value": 80, "unit": "%" },
+          "cruisingRangeElectric": { "value": 310, "unit": "km" }
+        }
+      }
+    }
+  ]
+}
+```
+
+Physical values are always `{"value": …, "unit": …}` objects.
+
+## Push server protocol
+
+The server listens on `--host`:`--port` (default `0.0.0.0:9999`).
+
+* On connect: sends all records from the current day, sorted by timestamp.
+* On each new poll: sends the new snapshot immediately.
+
+All messages are **newline-delimited JSON** (one record per line, `\n`
+terminated).
+
+## GUI client (`gui_client.py`)
+
+```bash
+python gui_client.py
+python gui_client.py --host 192.168.1.10 --port 9999
+```
+
+Dark theme matching nmea_client.  Three tabs:
+
+### Connector tab
+
+Host / port fields, Connect / Disconnect button, and connection status.
+
+### Dashboard tab
+
+Live tree view of the latest snapshot — domain → status object → field,
+showing value and unit.  Expand state is preserved across updates.
+
+### Plot tab
+
+8 plots (4 × 2) in a scrollable grid.  Each plot has its own control bar:
+
+* **Dropdown + `+`** — select a signal and add it as a trace (max 4 per plot).
+* **Colored chips (`name ×`)** — click to remove a trace.
+* Zoom / pan with mouse; crosshair cursor tracks mouse position.
+
+Trace selections are persisted to
+`~/.config/we_monitor/plot_settings.json` and restored automatically on
+the next launch once matching signals arrive.
+
+Signal labels use short human-readable names (e.g. *SOC*, *range*,
+*outside temp*) defined in `_TRACE_LABELS` near the top of `gui_client.py`.
+
+## CLI test client (`client.py`)
+
+```bash
+python client.py
+python client.py --host 192.168.1.10 --port 9999
+```
+
+Connects to the push server and prints every incoming snapshot as
+pretty-printed JSON with a running record counter.
@@ -0,0 +1,151 @@
+# Conversation Recap — we_monitor development
+
+## Overview
+
+Starting from a single `collect.py` script that polled a VW WeConnect vehicle
+and wrote JSON to disk, the project was iteratively extended into a modular
+Python package with a TCP push server and a PyQt5 / pyqtgraph GUI client.
+
+---
+
+## Feature development (in order)
+
+### 1. Credentials file (`-c / --credentials`)
+Added a JSON credentials file parameter alongside the existing `-u / -p` flags.
+File supports `username`, `password`, `vin`, and `domains` keys.  Domains may
+be a JSON array or a comma-separated string.  Explicit CLI flags and env vars
+(`WC_USER` / `WC_PASS`) take precedence over the file.
+
+### 2. `--list-domains`
+Added a CLI flag that prints all available domain names and exits.
+
+### 3. Physical value format
+Refactored all numeric measurements to `{"value": x, "unit": "km"}` objects,
+stripping unit suffixes from field names (e.g. `odometer_km` → `odometer`).
+
+### 4. Default log path + auto-create directory
+Output defaults to `logs/YYYY_MM_DD_HH_MM_SS_<VIN>.json`; the `logs/`
+directory is created automatically if it does not exist.
+
+### 5. Midnight UTC log rotation
+When using the default `logs/` directory the collector starts a new file after
+midnight UTC.  The old file is kept intact; the new file is auto-named with the
+current date stamp.
+
+### 6. TCP push server
+Added `--host` / `--port` flags (default `0.0.0.0:9999`).  The server pushes
+every new snapshot to all connected clients as newline-delimited JSON
+immediately after each poll.  The last snapshot is sent to newly connecting
+clients so they have something to display right away.
+
+### 7. CLI test client (`client.py`)
+Connects to the push server, pretty-prints every incoming JSON record, and
+prints a running record counter after each reception.
+
+### 8. Refactor into modules
+`collect.py` was split into focused server-side modules:
+
+| Module | Responsibility |
+|--------|---------------|
+| `server/network.py` | TCP accept loop, broadcast |
+| `server/data_model.py` | Domain extractors, `collect_snapshot()` |
+| `server/storage_helpers.py` | Load / save JSON store, `load_today_records()` |
+| `server/we_connect.py` | WeConnect login and vehicle selection |
+| `server/log_config.py` | `logging.basicConfig` setup |
+
+`collect.py` remains in the project root as the sole entry point.
+
+### 9. PyQt5 GUI client (`gui_client.py`)
+Three-tab application:
+
+* **Connector** — host / port, Connect / Disconnect, status label.
+* **Dashboard** — `QTreeWidget` showing domain → object → field/value/unit;
+  expand state preserved across live updates.
+* **Plot** — time-series charts of all physical fields using pyqtgraph.
+
+### 10. Full-day history on connect
+On client connect the server now sends all records from the current day instead
+of only the most recent snapshot.  Records are loaded from disk at startup into
+`push_store_ref["preloaded"]` and merged with live in-memory records (dedup by
+`ts`, sorted).
+
+### 11. Plot tab — user-selectable traces (pyqtgraph)
+Replaced static matplotlib plots with four pyqtgraph `PlotWidget`s.  Users
+select up to 4 traces per plot from a dropdown; zoom/pan via mouse;
+crosshair cursor with nearest-value readout.
+
+### 12. 8 plots with scrolling
+Extended to 8 plots in a 4 × 2 scrollable grid (`QScrollArea`).  Per-plot
+control bars (add-combo + `+` button + colored chip buttons) sit directly
+above each plot.
+
+### 13. Abbreviated trace names + label mapping
+`_TRACE_LABELS` dict maps full dotted keys to short human-readable labels
+(e.g. `charging.batteryStatus.currentSOC` → `"SOC"`).  Combo boxes, chip
+buttons, legends, and cursor readout all use the short label; the full key is
+stored as item data.
+
+### 14. Persist plot settings
+Trace selections saved to `~/.config/we_monitor/plot_settings.json` on every
+add/remove.  Restored silently on the next launch once matching signals arrive
+from the server.
+
+### 15. Status bar record counter
+The main window status bar shows `Last snapshot: <ts>  |  Records: <N>`.
+Counter resets on each new connection.
+
+### 16. Dark theme (nmea_client palette)
+Color palette copied from `nmea_client/gui/plot_widget.py`:
+
+| Role | Hex |
+|------|-----|
+| Main background | `#1a1a2e` |
+| Plot / input background | `#12121e` |
+| Borders | `#2a2a4a` / `#3a3a5e` |
+| Normal text | `#aaaacc` |
+| Bright text | `#ddddee` |
+| Connected | `#4CAF50` |
+| Error | `#F44336` |
+
+Trace colors use the Material palette from nmea_client
+(`#2196F3`, `#F44336`, `#4CAF50`, `#FF9800`, …).
+
+---
+
+## Bugs fixed
+
+### Socket read timeout (gui_client spurious disconnect)
+`socket.create_connection(timeout=5)` sets a permanent 5-second read timeout.
+After the first message, if the collection interval > 5 s, `recv()` raised
+`socket.timeout` and the GUI reported "Connection lost".
+**Fix:** `self._sock.settimeout(None)` immediately after connect.
+
+### `load_today()` never called after first poll
+The lazy-load condition `if not records` was never true once the first
+WeConnect poll had written a record into memory, so disk history was never
+loaded for late-connecting clients.
+**Fix:** Eager loading — `load_today_records()` called at startup, result
+stored in `push_store_ref["preloaded"]`; merged with live records on every
+connect.
+
+### Erratic axis scaling on mouse hover
+pyqtgraph's `InfiniteLine` (cursor crosshair) was included in auto-range
+calculations.  Moving the cursor triggered view rescaling on every mouse event.
+**Fix:** `pw.addItem(vl, ignoreBounds=True)` in both the initial setup and
+after each `_redraw()` call.
+
+---
+
+## Key design decisions
+
+* **`store_ref` mutable dict** — shared between the main thread and the accept
+  loop thread under a single `threading.Lock`.  Keys: `"current"` (live store)
+  and `"preloaded"` (today's disk records).
+* **NDJSON protocol** — one JSON object per line over TCP; trivial to consume
+  with `nc`, `python client.py`, or the GUI.
+* **pyqtgraph over matplotlib** — chosen for native Qt integration, live
+  update performance, and built-in zoom/pan without extra toolbars.
+* **`_TRACE_LABELS` dict** — single place to maintain display names; falls
+  back to `key.split(".")[-1]` for unknown keys.
+* **`QComboBox` item data** — short label displayed, full key stored via
+  `addItem(label, key)` / `currentData()`; no reverse-lookup dict needed.