we_monitor/README.md

# we_monitor

Periodically polls a VW vehicle via the `carconnectivity` library, stores
timestamped records as JSON, and pushes live data to connected TCP clients.
A PyQt5 GUI client visualises the data in real time.

## Repository layout

```
collect.py              — collector daemon (entry point)
gui_client.py           — PyQt5 GUI client
client.py               — minimal CLI test client
server/
  data_model.py         — domain extractors, collect_snapshot()
  network.py            — TCP push server (accept loop + broadcast)
  storage_helpers.py    — load / save JSON store, midnight rotation helpers
  we_connect.py         — CarConnectivity login and vehicle selection
  log_config.py         — logging setup
utils/
  jay_diff.py           — JSON diff / merge library
credentials/            — JSON config files (gitignored)
requirements.txt
```

## Requirements

```
pip install carconnectivity carconnectivity-connector-volkswagen PyQt5 pyqtgraph
```

Or use the installer which also sets up a systemd user service:

```bash
./install.sh
```

---

## Collector (`collect.py`)

Connects via CarConnectivity, polls selected domains at a configurable interval, and
appends timestamped records to a rotating JSON log.  Simultaneously runs a TCP
push server so clients receive every new snapshot the moment it is collected.

### Quick start

```bash
python collect.py -c credentials/my_car.json
```

Username and password can also be supplied via environment variables
(override the credentials file):

```bash
export WC_USER=me@example.com
export WC_PASS=secret
python collect.py -c credentials/my_car.json
```

### Credentials file

All connection and collection settings live in a single JSON file:

```json
{
  "credentials": {
    "username": "me@example.com",
    "password": "secret"
  },
  "vin":        "WVWZZZE1ZMP000000",
  "domains":    ["charging", "electric_drive", "connectivity"],
  "interval_s": 300,
  "host":       "0.0.0.0",
  "port":       9999,
  "log_dir":    "./logs",
  "diff":       false
}
```

| Key | Default | Description |
|-----|---------|-------------|
| `credentials.username` | — | MyVolkswagen email |
| `credentials.password` | — | MyVolkswagen password |
| `vin` | first vehicle | Target VIN |
| `domains` | `["charging","electric_drive","connectivity"]` | Domains to collect (list or `"all"`) |
| `interval_s` | `300` | Collection interval in seconds |
| `host` | `"0.0.0.0"` | Push server bind address |
| `port` | `9999` | Push server TCP port |
| `log_dir` | `"./logs"` | Directory for rotating JSON log files |
| `diff` | `false` | Enable diff-mode compression on the push server |

### CLI reference

CLI flags override the corresponding credentials-file value when given.

| Flag | Overrides | Description |
|------|-----------|-------------|
| `-c FILE` | — | Credentials / settings file |
| `-u / -p` | `credentials.username/password` | Username / password (also `WC_USER` / `WC_PASS`) |
| `-d DOMAINS` | `domains` | Domains to collect (comma-separated or `all`) |
| `-i SECONDS` | `interval_s` | Collection interval in seconds |
| `--host HOST` | `host` | Push server bind address |
| `--port PORT` | `port` | Push server TCP port |
| `--log-dir DIR` | `log_dir` | Directory for rotating log files |
| `--max-records N` | — | Keep only the last N records per file (default: unlimited) |
| `--diff` | `diff` | Enable diff-mode on the push server |
| `--dry` | — | Skip CarConnectivity login; run push server only (useful for GUI testing) |
| `-v` | — | Verbose / debug logging |
| `--list-domains` | — | Print available domains and exit |

**Available domains:** `charging`, `climatisation`, `electric_drive`,
`connectivity`, `vehicle`, `position`, `doors`

### Log rotation

A new file is started automatically after midnight UTC; previous files are kept
intact.  On startup and on each new client connection the collector sends the
last 24 hours of records, so clients always receive a full rolling window
regardless of when midnight falls.

### Output format

Log files are named `YYYY_MM_DD_HH_MM_SS_<VIN>.json` and contain:

```json
{
  "meta": { "vin": "...", "created_at": "...", "interval_seconds": 300 },
  "records": [
    {
      "ts": "2025-05-25T10:00:00+00:00",
      "charging": {
        "state": "readyForCharging",
        "power": { "value": 11.0, "unit": "kW" }
      },
      "electric_drive": {
        "range":   { "value": 310, "unit": "km" },
        "battery": { "soc": { "value": 80, "unit": "%" } }
      }
    }
  ]
}
```

Physical values are always `{"value": …, "unit": …}` objects.

---

## Push server protocol

The server listens on the `host`:`port` from the credentials file
(default `0.0.0.0:9999`).

* **On connect:** sends all records from the last 24 hours, sorted by timestamp.
* **On each new poll:** sends the new snapshot immediately.

All messages are **newline-delimited JSON** (one record per line, `\n` terminated).

### Diff mode (`--diff`)

When diff mode is enabled the history burst and every live snapshot are
compressed using `jay_diff_full` before transmission.  The wire format is
self-describing — diffs carry `update_add` / `delete` keys, full snapshots do
not — so no separate framing is needed.

The first message sent to a new client is always `jay_diff_full({}, record[0])`,
i.e. an `update_add` containing every field of the first record.  Subsequent
messages contain only the fields that changed.

Enable `--diff` on the server **and** on the client (`client.py --diff` or the
**Diff mode** checkbox in the GUI Connector tab).  Both sides must agree — a
mismatch produces garbled output.

---

## GUI client (`gui_client.py`)

```bash
python gui_client.py
```

Dark-theme PyQt5 application.  Three tabs: **Connector**, **Dashboard**,
**Plot**.  Connection settings, trace selections, and time windows are persisted
to `~/.config/we_monitor/plot_settings.json` and restored on the next launch.

### Connector tab

| Control | Description |
|---------|-------------|
| **Host / Port** | Address of the `collect.py` push server |
| **Diff mode** checkbox | Must match the server's `--diff` setting |
| **Connect / Disconnect** | Opens or closes the TCP connection |
| **Status** | Shows `Connected`, `Disconnected`, or the error message |

Clicking **Connect** saves the current host, port, and diff-mode setting.  All
plot data is cleared on each connect so the fresh server history is never mixed
with stale local data.

### Dashboard tab

Live tree view of the most recent snapshot.  The hierarchy mirrors the JSON
structure: domain → status object → field, with value and unit shown inline.
Expand / collapse state is preserved across updates.

### Plot tab

Eight plots arranged in a 4 × 2 scrollable grid.  Each plot is independent and
has its own control bar.

#### Control bar (per plot)

| Control | Description |
|---------|-------------|
| **Signal dropdown** | Lists every numeric `{value, unit}` field received so far |
| **`+` button** | Adds the selected signal as a new trace (max 4 per plot) |
| **Colored chips** (`name ×`) | One chip per active trace; click `×` to remove |
| **Time window spinner** | Show the last 1–24 hours of data (per plot) |
| **Date label** (right-aligned) | Shows the calendar date(s) of the visible window — single date or `YYYY-MM-DD – YYYY-MM-DD` when the view spans midnight |

#### Plot area

* **X axis** — time of day in `HH:MM:SS` format; no locale-dependent date prefix.
* **Y axis** — auto-zooms to fit the visible data on the first draw after connect
  or after a trace is added / removed.  Manual zoom is preserved between data
  updates.
* **Crosshair** — dashed vertical line tracks the mouse cursor.
* **Floating tooltip** — appears next to the cursor and shows the nearest
  sampled value for every active trace, with a colored bullet and unit.
* **Legend** — trace names and colors, shown inside the plot area.

#### Mouse interaction

| Action | Effect |
|--------|--------|
| Scroll wheel | Zoom X axis in / out, centred on cursor |
| Shift + scroll wheel | Zoom Y axis in / out, centred on cursor |
| Click + drag | Pan the view |
| Right-click | pyqtgraph context menu (export, auto-range, …) |

---

## CLI test client (`client.py`)

```bash
python client.py --host 192.168.1.10 --port 9999
python client.py --diff   # reconstruct from diff stream
```

Connects to the push server and prints every incoming snapshot as
pretty-printed JSON with a running record counter.

| Flag | Description |
|------|-------------|
| `--host HOST` | Server host (default: `127.0.0.1`) |
| `--port PORT` | Server port (default: `9999`) |
| `--diff` | Reconstruct full snapshots from diff stream |

---

## Module reference

### `collect.py`

Entry point for the collector daemon.  Parses CLI / credentials-file arguments,
establishes the CarConnectivity session, starts the push server, and runs the main
poll loop.

### `server/data_model.py`

`collect_snapshot(vehicle, domains) → dict`  
Iterates over the requested domains on a CarConnectivity vehicle object and extracts
fields into a flat dict keyed by domain.
Physical values are normalised to `{"value": …, "unit": …}`.  Returns a record
with a top-level `"ts"` ISO-8601 timestamp.

`ALL_DOMAINS`  
Tuple of all domain names the collector knows about.

### `server/network.py`

`start_push_server(host, port) → (clients, lock, store_ref)`  
Binds a TCP socket, starts the accept-loop thread (daemon), and returns the
shared state needed by `broadcast`.

`broadcast(snapshot, clients, lock)`  
Serialises `snapshot` to a JSON line and sends it to every connected client.
Dead connections are removed silently.

`store_ref` dict keys:
| Key | Type | Description |
|-----|------|-------------|
| `"current"` | `dict` | The currently-open JSON store |
| `"preloaded"` | `list` | 24-h history records loaded at startup |
| `"diff_mode"` | `bool` | Whether to compress the history burst as diffs |

### `server/storage_helpers.py`

`load_store(path, vin, interval) → dict`  
Loads an existing store file or creates a fresh one.

`save_store(path, store, max_records)`  
Writes the store to disk, optionally trimming to the last `max_records` entries.

`auto_out_path(log_dir, vin) → Path`  
Returns a timestamped file path for a new store.

`load_last_24h_records(log_dir, vin) → list`  
Scans the log directory for files belonging to `vin`, loads any records from
the last 24 hours, and returns them sorted by timestamp.

### `server/we_connect.py`

`connect(username, password) → CarConnectivity`  
Authenticates and returns a connected `CarConnectivity` instance.

`select_vehicle(wc, vin) → (vehicle, error)`  
Returns the target vehicle (first available if `vin` is empty) or an error string.

### `utils/jay_diff.py`

Lightweight JSON diff / patch library.

`jay_diff_full(old, new, combine_upd_add=True) → dict`  
Computes a diff between two dicts.  With `combine_upd_add=True` (the default
used everywhere in this project) the result uses `update_add` for all additions
and updates, plus a separate `delete` key.  Returns `{}` when nothing changed.

`jay_merge_full(old, diff) → dict`  
Applies a `jay_diff_full` diff to `old`.  The diff structure is self-describing:
if `diff` contains none of the known diff keys (`update`, `update_add`, `add`,
`delete`) and `old` is empty, `diff` is returned as-is (full-snapshot
passthrough for bootstrap).

### `gui_client.py` — class overview

| Class | Role |
|-------|------|
| `TcpReader` | Background thread; reads newline-delimited JSON from the socket, reconstructs full snapshots in diff mode via `jay_merge_full`, and emits each snapshot via the `message` Qt signal |
| `ConnectorTab` | Host / port / diff-mode form; emits `connect_requested` and `disconnect_requested` signals; persists settings |
| `DashboardTab` | Live `QTreeWidget` showing the latest snapshot; preserves expand state across updates |
| `PlotTab` | Eight independent time-series plots in a scrollable grid |
| `TimeAxisItem` | `pg.AxisItem` subclass; formats x-axis tick labels as `HH:MM:SS` |
| `PlotViewBox` | `pg.ViewBox` subclass; routes plain scroll to X zoom and Shift+scroll to Y zoom |
| `PlotTooltip` | Frameless child `QWidget` per plot; shows nearest sampled values for all active traces at the cursor position |
| `MainWindow` | `QMainWindow`; wires the three tabs together and owns the `TcpReader` |