Add README and TODO tracking for the testhost tool
- README.md: what the tool is and isn't (generic, not JaySynth-specific), build instructions, CLI quick-start, full JSON scenario schema/field reference, the patch/bank state-format caveat (this tool's load/save goes through JUCE's host-side AudioPluginInstance state calls, not JaySynth's own GUI-triggered loadPatchFromFile/patchImportXml - a different code path worth knowing about), and an architecture summary. - TODO.md: tracks what's deferred by design (LV2/VST3/LADSPA format registration, Windows build), test coverage gaps (stereo effects untested, no regression-baseline tooling, no CI wiring, no JSON-parser unit tests, no polyphony testing), two things worth investigating (a non-fatal DPF plugin assertion seen on load, an unconfirmed WavAudioFormat channel-count limit), and one nice-to-have. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011dhtwRLARk4eiPngcQykLJ
This commit is contained in:
Claude Sonnet 5
@@ -0,0 +1,146 @@
|
||||
# testhost
|
||||
|
||||
A generic, headless command-line host for testing audio plugins. It loads a
|
||||
plugin out-of-process and drives it entirely offline — no audio device, no
|
||||
GUI event loop, no real-time pacing — so a test run is bounded by CPU speed
|
||||
rather than wall-clock playback time (60-1000x+ real-time observed on this
|
||||
machine, depending on the plugin).
|
||||
|
||||
**This tool is not JaySynth-specific.** It was built to test JaySynth, but it
|
||||
talks to plugins purely through JUCE's own `AudioPluginFormatManager` /
|
||||
`AudioPluginInstance` hosting API, and it never assumes a fixed channel
|
||||
layout — it always asks the loaded plugin how many input/output channels it
|
||||
actually has. That means it drives an instrument (0 inputs, MIDI-triggered,
|
||||
e.g. JaySynth) and an audio effect (real input required, e.g. a reverb,
|
||||
EQ, or compressor) through the exact same code path. See
|
||||
`/home/jens/.claude/plans/sharded-finding-tower.md` for the original design
|
||||
plan and rationale.
|
||||
|
||||
## Building
|
||||
|
||||
```sh
|
||||
MAKE_HOME=$(realpath ../../submodule/make) make -C tools/testhost
|
||||
```
|
||||
|
||||
This produces `tools/testhost/build/linux/release/testhost`. It has its own
|
||||
`JuceLibraryCode/` package (compiling only the module amalgams a host needs -
|
||||
not `juce_audio_plugin_client`, which is the plugin-*side* VST entry point
|
||||
and would collide with this executable's own `main()`), but reuses the
|
||||
already-vendored `sdk/juce/JUCE-3.1.1` rather than a second JUCE checkout.
|
||||
The host and any plugin it loads are decoupled at the VST2 ABI, not the JUCE
|
||||
source level, so this doesn't need to track JaySynth's own JUCE version.
|
||||
|
||||
`make CONFIG=debug -C tools/testhost` builds a debug variant the same way
|
||||
the main plugin build does.
|
||||
|
||||
## Quick start
|
||||
|
||||
```sh
|
||||
# Render 2s of a held note through JaySynth and record it:
|
||||
./build/linux/release/testhost \
|
||||
--plugin ../../build/linux/release/JaySynth.so \
|
||||
--note 60 100 --duration 2 --wav out.wav
|
||||
|
||||
# Feed pink noise through an audio effect (no MIDI needed):
|
||||
./build/linux/release/testhost \
|
||||
--plugin /usr/lib/vst/ZamDelay-vst.so \
|
||||
--input pink --duration 2 --wav out.wav
|
||||
```
|
||||
|
||||
Run `./build/linux/release/testhost --help` for the full flag reference
|
||||
(patch/bank load & save, parameters, sample rate/block size, performance CSV
|
||||
export, etc).
|
||||
|
||||
## JSON test scenarios
|
||||
|
||||
For anything beyond a one-off check, `--scenario file.json` drives a full
|
||||
timed sequence of actions instead of a handful of CLI flags - this is what
|
||||
makes tests repeatable and batchable (e.g. one file per regression case, run
|
||||
unattended in CI).
|
||||
|
||||
```json
|
||||
{
|
||||
"sampleRate": 44100,
|
||||
"blockSize": 512,
|
||||
"durationSeconds": 2.0,
|
||||
"input": { "type": "noise", "kind": "pink", "amplitude": 0.5 },
|
||||
"loadBank": "some_bank.fxb",
|
||||
"events": [
|
||||
{ "sample": 0, "type": "noteOn", "channel": 1, "note": 60, "velocity": 100 },
|
||||
{ "sample": 22050, "type": "param", "index": 5, "value": 0.75 },
|
||||
{ "sample": 44100, "type": "noteOff", "channel": 1, "note": 60 },
|
||||
{ "sample": 88199, "type": "savePatch", "file": "result_patch.bin" }
|
||||
],
|
||||
"recordWav": "out.wav"
|
||||
}
|
||||
```
|
||||
|
||||
Relative file paths inside the JSON (`loadBank`, `loadPatch`, `recordWav`,
|
||||
and any event's `file`) resolve against **the scenario file's own
|
||||
directory**, not the process's working directory, so a scenario file stays
|
||||
portable no matter where it's run from.
|
||||
|
||||
### Top-level fields
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| `sampleRate` | number | default 44100 |
|
||||
| `blockSize` | number | default 512 |
|
||||
| `durationSamples` **or** `durationSeconds` | number | one of these is required |
|
||||
| `input` | object | `{"type": "silence"}` (default) / `"sine"` (+ `frequencyHz`) / `"noise"` (+ `kind`: `"white"`/`"pink"`) / `"impulse"`, all with optional `amplitude` (default 0.5) |
|
||||
| `loadPatch` / `loadBank` | string (file path) | loaded once, before rendering starts |
|
||||
| `recordWav` | string (file path) | omit to skip recording (e.g. pure performance runs) |
|
||||
| `events` | array | see below; need not be pre-sorted, `TestScenario` sorts by `sample` |
|
||||
|
||||
### Event types
|
||||
|
||||
All events have `sample` (absolute sample index) and `type`:
|
||||
|
||||
| `type` | Extra fields | Effect |
|
||||
|---|---|---|
|
||||
| `noteOn` | `channel`, `note`, `velocity` | `MidiMessage::noteOn`, timed to the exact sample within its block |
|
||||
| `noteOff` | `channel`, `note` | `MidiMessage::noteOff`, same timing precision |
|
||||
| `controller` | `channel`, `controllerNumber`, `controllerValue` | `MidiMessage::controllerEvent` |
|
||||
| `param` | `index`, `value` (0.0-1.0) | `AudioProcessor::setParameter`, applied at the start of the block containing `sample` (see caveat below) |
|
||||
| `loadPatch` / `loadBank` | `file` | reads the file's raw bytes and calls `setCurrentProgramStateInformation`/`setStateInformation` |
|
||||
| `savePatch` / `saveBank` | `file` | calls `getCurrentProgramStateInformation`/`getStateInformation` and writes the raw bytes out |
|
||||
|
||||
**Caveat:** MIDI events keep full per-sample timing (JUCE's `MidiBuffer`
|
||||
carries a sample offset natively). Non-MIDI events (`param`/`loadPatch`/
|
||||
`loadBank`/`savePatch`/`saveBank`) scheduled for a sample in the middle of a
|
||||
block are applied at the *start* of that block, not their exact offset -
|
||||
fine for test purposes, but not sample-accurate. Use a smaller `blockSize`
|
||||
if you need tighter timing on those.
|
||||
|
||||
### Patch/bank format note
|
||||
|
||||
`loadPatch`/`savePatch`/`loadBank`/`saveBank` read/write exactly the bytes
|
||||
that JUCE's host-side `AudioPluginInstance::get/setCurrentProgramState
|
||||
Information` and `get/setStateInformation` produce/consume for a hosted
|
||||
VST2 plugin - i.e. real `.fxp`/`.fxb` file bytes (JUCE's own VST2 host
|
||||
wrapper handles the fxb/fxp chunk header itself; this tool never parses
|
||||
that format directly). This is a different code path from JaySynth's own
|
||||
GUI-triggered "Load Patch from disk" button (`JaySynthAudioProcessor::
|
||||
loadPatchFromFile`/`patchImportXml` in `src/plug/PluginProcessor.cpp`) -
|
||||
both are legitimate ways to get a patch into the plugin, but they aren't
|
||||
the same call chain, so a fix to one doesn't automatically test the other.
|
||||
|
||||
## Architecture
|
||||
|
||||
- `PluginHost` — owns the `AudioPluginFormatManager`, loads/prepares/drives
|
||||
one plugin instance. The only class that talks to `AudioPluginInstance`
|
||||
directly; everything else in this tool is format-agnostic by construction.
|
||||
- `AudioInputSource` — fills the plugin's input channels each block
|
||||
(silence/sine/noise/impulse).
|
||||
- `WavRecorder` — thin wrapper over `WavAudioFormat`/`AudioFormatWriter`.
|
||||
- `TestScenario` — parses the JSON format above into a sorted event list
|
||||
plus render/I-O config.
|
||||
- `PerformanceStats` — per-block timing, summary + optional CSV.
|
||||
- `main.cpp` — CLI parsing, and a single render loop shared by both the
|
||||
ad-hoc-flags path and the `--scenario` path (ad-hoc flags are converted
|
||||
into an equivalent in-memory `TestScenario` before rendering starts, so
|
||||
there's only one loop to maintain).
|
||||
|
||||
## Known limitations
|
||||
|
||||
See `TODO.md` in this directory.
|
||||
@@ -0,0 +1,72 @@
|
||||
# testhost TODO
|
||||
|
||||
Tracks known gaps and future work for `tools/testhost/`, the generic headless
|
||||
plugin test host (see `README.md` for what it does and how to use it, and
|
||||
`/home/jens/.claude/plans/sharded-finding-tower.md` for the original design
|
||||
plan). Nothing here blocks current use — Phases 1-4 of the plan are done and
|
||||
verified (PR https://git.jayfield.org/jens/JaySynth/pulls/1).
|
||||
|
||||
## Deferred by design (Phase 5 of the plan)
|
||||
|
||||
- [ ] **LV2 plugin format support.** JUCE 3.1.1 (the version currently
|
||||
vendored at `sdk/juce/JUCE-3.1.1`) has no LV2 hosting backend. Once
|
||||
JaySynth's own JUCE dependency is upgraded (or a custom `AudioPluginFormat`
|
||||
subclass is written against the current version), register it in
|
||||
`PluginHost`'s constructor alongside `VSTPluginFormat` - no other host code
|
||||
should need to change, since everything downstream talks to the
|
||||
format-agnostic `AudioPluginInstance`/`AudioProcessor` API already.
|
||||
- [ ] **VST3/LADSPA format support.** JUCE already ships
|
||||
`VST3PluginFormat`/`LADSPAPluginFormat` in this same vendored version -
|
||||
registering them is a one-line addition to `PluginHost`'s constructor,
|
||||
just not done yet since nothing has needed it.
|
||||
- [ ] **Windows build (`Makefile.win`).** The plan calls for mirroring
|
||||
`src/plug`'s Linux/Windows Makefile pattern; only the Linux side has been
|
||||
built and tested so far.
|
||||
|
||||
## Test coverage gaps
|
||||
|
||||
- [ ] **Only mono (1 in/1 out) and asymmetric (2 in/1 out) effect channel
|
||||
configs have been smoke-tested** (`ZamDelay-vst.so`, `ZamComp-vst.so`,
|
||||
`ZamEQ2-vst.so`). A genuinely stereo-in/stereo-out effect hasn't been run
|
||||
through yet - worth doing before relying on this tool for a stereo effect
|
||||
plugin specifically.
|
||||
- [ ] **No automated regression baseline comparison.** The plan's stretch
|
||||
goal: batch-render every `.fxp`/`.xmp` under `extras/sounds/` through a
|
||||
fixed note pattern and diff against stored baseline WAVs/checksums, to
|
||||
catch DSP or save-load regressions automatically instead of via one-off
|
||||
manual runs. Not built.
|
||||
- [ ] **No CI wiring.** Nothing currently runs `testhost` automatically on
|
||||
push/PR; it's a manual, opt-in tool for now.
|
||||
- [ ] **No unit tests for `TestScenario`'s JSON parsing itself** (malformed
|
||||
JSON, missing required fields, wrong types, out-of-range values) - only
|
||||
exercised indirectly via a couple of hand-written scenario files during
|
||||
development. Edge-case handling (e.g. a negative `sample`, an unterminated
|
||||
`noteOn` with no matching `noteOff`) is untested.
|
||||
- [ ] **Polyphony/multiple simultaneous notes untested** - only ever a single
|
||||
held note in all smoke tests so far.
|
||||
|
||||
## Investigate
|
||||
|
||||
- [ ] **DPF-based plugins print a non-fatal assertion on load.** All three
|
||||
Zam*-vst.so plugins tested (built with the DISTRHO Plugin Framework) print
|
||||
`assertion failure: "fIsActive" in .../DistrhoPluginInternal.hpp` during
|
||||
`PluginHost::load()`, despite JUCE's `VSTPluginInstance::prepareToPlay`
|
||||
already sending `effMainsChanged`/`effStartProcess`. Rendering still
|
||||
completes correctly (confirmed via WAV output), so this looks like a
|
||||
DPF/JUCE VST2 lifecycle-ordering quirk rather than a testhost bug, but it
|
||||
hasn't been root-caused - worth a closer look if it turns out to affect
|
||||
correctness (not just log noise) for some other plugin.
|
||||
- [ ] **`WavAudioFormat`'s base-class doc comment says channel count "must
|
||||
be either 1 or 2"** (`AudioFormat::createWriterFor`'s Doxygen comment).
|
||||
`WavRecorder` passes through whatever `PluginHost::getNumOutputChannels()`
|
||||
reports with no clamping, and this has worked fine for the 1- and 2-channel
|
||||
plugins tested so far - but a plugin with more than 2 output channels
|
||||
hasn't been tried, so it's unconfirmed whether JUCE's WAV writer actually
|
||||
enforces that limit or the doc comment is just generic/stale boilerplate
|
||||
shared across format subclasses.
|
||||
|
||||
## Nice-to-have (not planned, just noted)
|
||||
|
||||
- [ ] A `--list-params` or similar introspection mode (print every
|
||||
parameter's index/name/current value) would help writing new scenario
|
||||
JSON files without cross-referencing the plugin's own GUI or source.
|
||||
Reference in New Issue
Block a user