From ab29ad282080d89ed228541ee2e7284c299355d3 Mon Sep 17 00:00:00 2001 From: Jens Ahrensfeld Date: Mon, 27 Jul 2026 19:31:54 +0200 Subject: [PATCH] Add README and TODO tracking for the testhost tool - README.md: what the tool is and isn't (generic, not JaySynth-specific), build instructions, CLI quick-start, full JSON scenario schema/field reference, the patch/bank state-format caveat (this tool's load/save goes through JUCE's host-side AudioPluginInstance state calls, not JaySynth's own GUI-triggered loadPatchFromFile/patchImportXml - a different code path worth knowing about), and an architecture summary. - TODO.md: tracks what's deferred by design (LV2/VST3/LADSPA format registration, Windows build), test coverage gaps (stereo effects untested, no regression-baseline tooling, no CI wiring, no JSON-parser unit tests, no polyphony testing), two things worth investigating (a non-fatal DPF plugin assertion seen on load, an unconfirmed WavAudioFormat channel-count limit), and one nice-to-have. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_011dhtwRLARk4eiPngcQykLJ --- tools/testhost/README.md | 146 +++++++++++++++++++++++++++++++++++++++ tools/testhost/TODO.md | 72 +++++++++++++++++++ 2 files changed, 218 insertions(+) create mode 100644 tools/testhost/README.md create mode 100644 tools/testhost/TODO.md diff --git a/tools/testhost/README.md b/tools/testhost/README.md new file mode 100644 index 0000000..553ad06 --- /dev/null +++ b/tools/testhost/README.md @@ -0,0 +1,146 @@ +# testhost + +A generic, headless command-line host for testing audio plugins. It loads a +plugin out-of-process and drives it entirely offline — no audio device, no +GUI event loop, no real-time pacing — so a test run is bounded by CPU speed +rather than wall-clock playback time (60-1000x+ real-time observed on this +machine, depending on the plugin). + +**This tool is not JaySynth-specific.** It was built to test JaySynth, but it +talks to plugins purely through JUCE's own `AudioPluginFormatManager` / +`AudioPluginInstance` hosting API, and it never assumes a fixed channel +layout — it always asks the loaded plugin how many input/output channels it +actually has. That means it drives an instrument (0 inputs, MIDI-triggered, +e.g. JaySynth) and an audio effect (real input required, e.g. a reverb, +EQ, or compressor) through the exact same code path. See +`/home/jens/.claude/plans/sharded-finding-tower.md` for the original design +plan and rationale. + +## Building + +```sh +MAKE_HOME=$(realpath ../../submodule/make) make -C tools/testhost +``` + +This produces `tools/testhost/build/linux/release/testhost`. It has its own +`JuceLibraryCode/` package (compiling only the module amalgams a host needs - +not `juce_audio_plugin_client`, which is the plugin-*side* VST entry point +and would collide with this executable's own `main()`), but reuses the +already-vendored `sdk/juce/JUCE-3.1.1` rather than a second JUCE checkout. +The host and any plugin it loads are decoupled at the VST2 ABI, not the JUCE +source level, so this doesn't need to track JaySynth's own JUCE version. + +`make CONFIG=debug -C tools/testhost` builds a debug variant the same way +the main plugin build does. + +## Quick start + +```sh +# Render 2s of a held note through JaySynth and record it: +./build/linux/release/testhost \ + --plugin ../../build/linux/release/JaySynth.so \ + --note 60 100 --duration 2 --wav out.wav + +# Feed pink noise through an audio effect (no MIDI needed): +./build/linux/release/testhost \ + --plugin /usr/lib/vst/ZamDelay-vst.so \ + --input pink --duration 2 --wav out.wav +``` + +Run `./build/linux/release/testhost --help` for the full flag reference +(patch/bank load & save, parameters, sample rate/block size, performance CSV +export, etc). + +## JSON test scenarios + +For anything beyond a one-off check, `--scenario file.json` drives a full +timed sequence of actions instead of a handful of CLI flags - this is what +makes tests repeatable and batchable (e.g. one file per regression case, run +unattended in CI). + +```json +{ + "sampleRate": 44100, + "blockSize": 512, + "durationSeconds": 2.0, + "input": { "type": "noise", "kind": "pink", "amplitude": 0.5 }, + "loadBank": "some_bank.fxb", + "events": [ + { "sample": 0, "type": "noteOn", "channel": 1, "note": 60, "velocity": 100 }, + { "sample": 22050, "type": "param", "index": 5, "value": 0.75 }, + { "sample": 44100, "type": "noteOff", "channel": 1, "note": 60 }, + { "sample": 88199, "type": "savePatch", "file": "result_patch.bin" } + ], + "recordWav": "out.wav" +} +``` + +Relative file paths inside the JSON (`loadBank`, `loadPatch`, `recordWav`, +and any event's `file`) resolve against **the scenario file's own +directory**, not the process's working directory, so a scenario file stays +portable no matter where it's run from. + +### Top-level fields + +| Field | Type | Notes | +|---|---|---| +| `sampleRate` | number | default 44100 | +| `blockSize` | number | default 512 | +| `durationSamples` **or** `durationSeconds` | number | one of these is required | +| `input` | object | `{"type": "silence"}` (default) / `"sine"` (+ `frequencyHz`) / `"noise"` (+ `kind`: `"white"`/`"pink"`) / `"impulse"`, all with optional `amplitude` (default 0.5) | +| `loadPatch` / `loadBank` | string (file path) | loaded once, before rendering starts | +| `recordWav` | string (file path) | omit to skip recording (e.g. pure performance runs) | +| `events` | array | see below; need not be pre-sorted, `TestScenario` sorts by `sample` | + +### Event types + +All events have `sample` (absolute sample index) and `type`: + +| `type` | Extra fields | Effect | +|---|---|---| +| `noteOn` | `channel`, `note`, `velocity` | `MidiMessage::noteOn`, timed to the exact sample within its block | +| `noteOff` | `channel`, `note` | `MidiMessage::noteOff`, same timing precision | +| `controller` | `channel`, `controllerNumber`, `controllerValue` | `MidiMessage::controllerEvent` | +| `param` | `index`, `value` (0.0-1.0) | `AudioProcessor::setParameter`, applied at the start of the block containing `sample` (see caveat below) | +| `loadPatch` / `loadBank` | `file` | reads the file's raw bytes and calls `setCurrentProgramStateInformation`/`setStateInformation` | +| `savePatch` / `saveBank` | `file` | calls `getCurrentProgramStateInformation`/`getStateInformation` and writes the raw bytes out | + +**Caveat:** MIDI events keep full per-sample timing (JUCE's `MidiBuffer` +carries a sample offset natively). Non-MIDI events (`param`/`loadPatch`/ +`loadBank`/`savePatch`/`saveBank`) scheduled for a sample in the middle of a +block are applied at the *start* of that block, not their exact offset - +fine for test purposes, but not sample-accurate. Use a smaller `blockSize` +if you need tighter timing on those. + +### Patch/bank format note + +`loadPatch`/`savePatch`/`loadBank`/`saveBank` read/write exactly the bytes +that JUCE's host-side `AudioPluginInstance::get/setCurrentProgramState +Information` and `get/setStateInformation` produce/consume for a hosted +VST2 plugin - i.e. real `.fxp`/`.fxb` file bytes (JUCE's own VST2 host +wrapper handles the fxb/fxp chunk header itself; this tool never parses +that format directly). This is a different code path from JaySynth's own +GUI-triggered "Load Patch from disk" button (`JaySynthAudioProcessor:: +loadPatchFromFile`/`patchImportXml` in `src/plug/PluginProcessor.cpp`) - +both are legitimate ways to get a patch into the plugin, but they aren't +the same call chain, so a fix to one doesn't automatically test the other. + +## Architecture + +- `PluginHost` — owns the `AudioPluginFormatManager`, loads/prepares/drives + one plugin instance. The only class that talks to `AudioPluginInstance` + directly; everything else in this tool is format-agnostic by construction. +- `AudioInputSource` — fills the plugin's input channels each block + (silence/sine/noise/impulse). +- `WavRecorder` — thin wrapper over `WavAudioFormat`/`AudioFormatWriter`. +- `TestScenario` — parses the JSON format above into a sorted event list + plus render/I-O config. +- `PerformanceStats` — per-block timing, summary + optional CSV. +- `main.cpp` — CLI parsing, and a single render loop shared by both the + ad-hoc-flags path and the `--scenario` path (ad-hoc flags are converted + into an equivalent in-memory `TestScenario` before rendering starts, so + there's only one loop to maintain). + +## Known limitations + +See `TODO.md` in this directory. diff --git a/tools/testhost/TODO.md b/tools/testhost/TODO.md new file mode 100644 index 0000000..124d05f --- /dev/null +++ b/tools/testhost/TODO.md @@ -0,0 +1,72 @@ +# testhost TODO + +Tracks known gaps and future work for `tools/testhost/`, the generic headless +plugin test host (see `README.md` for what it does and how to use it, and +`/home/jens/.claude/plans/sharded-finding-tower.md` for the original design +plan). Nothing here blocks current use — Phases 1-4 of the plan are done and +verified (PR https://git.jayfield.org/jens/JaySynth/pulls/1). + +## Deferred by design (Phase 5 of the plan) + +- [ ] **LV2 plugin format support.** JUCE 3.1.1 (the version currently + vendored at `sdk/juce/JUCE-3.1.1`) has no LV2 hosting backend. Once + JaySynth's own JUCE dependency is upgraded (or a custom `AudioPluginFormat` + subclass is written against the current version), register it in + `PluginHost`'s constructor alongside `VSTPluginFormat` - no other host code + should need to change, since everything downstream talks to the + format-agnostic `AudioPluginInstance`/`AudioProcessor` API already. +- [ ] **VST3/LADSPA format support.** JUCE already ships + `VST3PluginFormat`/`LADSPAPluginFormat` in this same vendored version - + registering them is a one-line addition to `PluginHost`'s constructor, + just not done yet since nothing has needed it. +- [ ] **Windows build (`Makefile.win`).** The plan calls for mirroring + `src/plug`'s Linux/Windows Makefile pattern; only the Linux side has been + built and tested so far. + +## Test coverage gaps + +- [ ] **Only mono (1 in/1 out) and asymmetric (2 in/1 out) effect channel + configs have been smoke-tested** (`ZamDelay-vst.so`, `ZamComp-vst.so`, + `ZamEQ2-vst.so`). A genuinely stereo-in/stereo-out effect hasn't been run + through yet - worth doing before relying on this tool for a stereo effect + plugin specifically. +- [ ] **No automated regression baseline comparison.** The plan's stretch + goal: batch-render every `.fxp`/`.xmp` under `extras/sounds/` through a + fixed note pattern and diff against stored baseline WAVs/checksums, to + catch DSP or save-load regressions automatically instead of via one-off + manual runs. Not built. +- [ ] **No CI wiring.** Nothing currently runs `testhost` automatically on + push/PR; it's a manual, opt-in tool for now. +- [ ] **No unit tests for `TestScenario`'s JSON parsing itself** (malformed + JSON, missing required fields, wrong types, out-of-range values) - only + exercised indirectly via a couple of hand-written scenario files during + development. Edge-case handling (e.g. a negative `sample`, an unterminated + `noteOn` with no matching `noteOff`) is untested. +- [ ] **Polyphony/multiple simultaneous notes untested** - only ever a single + held note in all smoke tests so far. + +## Investigate + +- [ ] **DPF-based plugins print a non-fatal assertion on load.** All three + Zam*-vst.so plugins tested (built with the DISTRHO Plugin Framework) print + `assertion failure: "fIsActive" in .../DistrhoPluginInternal.hpp` during + `PluginHost::load()`, despite JUCE's `VSTPluginInstance::prepareToPlay` + already sending `effMainsChanged`/`effStartProcess`. Rendering still + completes correctly (confirmed via WAV output), so this looks like a + DPF/JUCE VST2 lifecycle-ordering quirk rather than a testhost bug, but it + hasn't been root-caused - worth a closer look if it turns out to affect + correctness (not just log noise) for some other plugin. +- [ ] **`WavAudioFormat`'s base-class doc comment says channel count "must + be either 1 or 2"** (`AudioFormat::createWriterFor`'s Doxygen comment). + `WavRecorder` passes through whatever `PluginHost::getNumOutputChannels()` + reports with no clamping, and this has worked fine for the 1- and 2-channel + plugins tested so far - but a plugin with more than 2 output channels + hasn't been tried, so it's unconfirmed whether JUCE's WAV writer actually + enforces that limit or the doc comment is just generic/stale boilerplate + shared across format subclasses. + +## Nice-to-have (not planned, just noted) + +- [ ] A `--list-params` or similar introspection mode (print every + parameter's index/name/current value) would help writing new scenario + JSON files without cross-referencing the plugin's own GUI or source.