Add generic headless plugin test host #1

2026-07-27T19:30:12+02:00

jens commented

2026-07-27 19:30:12 +02:00

Summary

tools/testhost/ is a standalone CLI executable that loads and drives any VST2 plugin offline (no audio device, no GUI event loop) via JUCE's own format-agnostic AudioPluginFormatManager/AudioPluginInstance hosting API. It is deliberately generic, not JaySynth-specific - adding another plugin format later (VST3, LADSPA, eventually LV2) is just another addFormat() call, and it correctly handles instruments (0 audio inputs, MIDI-driven) and audio effects (real audio input, e.g. reverbs/EQs/compressors) identically by always querying the loaded plugin's actual channel counts rather than assuming a fixed layout.

Features

Offline audio rendering with WAV recording (WavRecorder)
Synthetic audio input generation for effect plugins: silence/sine/white noise/pink noise/impulse (AudioInputSource)
MIDI event generation (note on/off, controller) and parameter changes
Patch/bank load and save, via AudioProcessor's standard state-information API (real .fxp/.fxb byte compatibility)
JSON-driven test scenarios (TestScenario) for repeatable, timed sequences of events - or simple ad-hoc CLI flags for quick checks
Performance measurement: per-block render timing, min/mean/max, real-time factor, optional CSV export (PerformanceStats)

Why

Manual GUI-based testing (build, install, launch Reaper, click through patches) is slow, not repeatable, and risky to automate blindly. This tool replaces that loop for regression/smoke testing: renders run far faster than real-time (60-1000x+ observed) and can be scripted/batched in CI.

Verification

Smoke-tested against three real, very different plugins (see commit messages for full detail):

JaySynth.so (0 in/2 out, MIDI-driven): note events, param changes, and a full patch save→load round-trip all render correctly.
ZamDelay-vst.so (1 in/1 out) and ZamComp-vst.so (2 in/1 out, asymmetric channel count): pink-noise input correctly sized/fed, valid non-silent output.
ZamEQ2-vst.so: the host's own NaN/Inf detection caught a real NaN in this plugin's default state on first render - unprompted, exactly the kind of regression this tool exists to catch without a GUI.

Test plan

MAKE_HOME=... make -C tools/testhost builds cleanly (no new warnings)
Ad-hoc CLI mode against JaySynth.so
JSON scenario mode against JaySynth.so (timed events + patch save/load round-trip)
JSON scenario mode against third-party VST2 effects with generated audio input

## Summary `tools/testhost/` is a standalone CLI executable that loads and drives any VST2 plugin offline (no audio device, no GUI event loop) via JUCE's own format-agnostic `AudioPluginFormatManager`/`AudioPluginInstance` hosting API. It is deliberately generic, not JaySynth-specific - adding another plugin format later (VST3, LADSPA, eventually LV2) is just another `addFormat()` call, and it correctly handles instruments (0 audio inputs, MIDI-driven) and audio effects (real audio input, e.g. reverbs/EQs/compressors) identically by always querying the loaded plugin's actual channel counts rather than assuming a fixed layout. ### Features - Offline audio rendering with WAV recording (`WavRecorder`) - Synthetic audio input generation for effect plugins: silence/sine/white noise/pink noise/impulse (`AudioInputSource`) - MIDI event generation (note on/off, controller) and parameter changes - Patch/bank load and save, via `AudioProcessor`'s standard state-information API (real `.fxp`/`.fxb` byte compatibility) - JSON-driven test scenarios (`TestScenario`) for repeatable, timed sequences of events - or simple ad-hoc CLI flags for quick checks - Performance measurement: per-block render timing, min/mean/max, real-time factor, optional CSV export (`PerformanceStats`) ### Why Manual GUI-based testing (build, install, launch Reaper, click through patches) is slow, not repeatable, and risky to automate blindly. This tool replaces that loop for regression/smoke testing: renders run far faster than real-time (60-1000x+ observed) and can be scripted/batched in CI. ## Verification Smoke-tested against three real, very different plugins (see commit messages for full detail): - `JaySynth.so` (0 in/2 out, MIDI-driven): note events, param changes, and a full patch save→load round-trip all render correctly. - `ZamDelay-vst.so` (1 in/1 out) and `ZamComp-vst.so` (2 in/1 out, asymmetric channel count): pink-noise input correctly sized/fed, valid non-silent output. - `ZamEQ2-vst.so`: the host's own NaN/Inf detection caught a real NaN in this plugin's default state on first render - unprompted, exactly the kind of regression this tool exists to catch without a GUI. ## Test plan - [x] `MAKE_HOME=... make -C tools/testhost` builds cleanly (no new warnings) - [x] Ad-hoc CLI mode against JaySynth.so - [x] JSON scenario mode against JaySynth.so (timed events + patch save/load round-trip) - [x] JSON scenario mode against third-party VST2 effects with generated audio input

jens added 2 commits 2026-07-27 19:30:13 +02:00

Add generic headless plugin test host (Phase 1: core + audio I/O) 97b1c34cee

tools/testhost/ is a standalone CLI executable, not a JaySynth-specific
tool: it drives any VST2 plugin offline (no audio device, no GUI event
loop) via JUCE's own format-agnostic AudioPluginFormatManager /
AudioPluginInstance hosting API, so adding another format later (VST3,
LADSPA, eventually LV2) is just another addFormat() call - none of the
host logic changes.

- PluginHost: format-agnostic load/prepare/process/release, parameter
  get/set, and patch/bank state I/O, all delegating straight to
  AudioProcessor. Always queries the plugin's real input/output channel
  counts rather than assuming "0 in" (instrument) or a fixed channel
  layout - the same code path drives JaySynth (0 in/2 out, MIDI-driven)
  and a stereo/mono effect identically.
- AudioInputSource: fills the plugin's input channels with silence
  (default, correct no-op for instruments), a sine tone, white/pink
  noise, or an impulse - what makes the host equally useful for
  effect plugins, which need a real input signal to do anything.
- WavRecorder: records the plugin's actual output channel count to a
  WAV file via WavAudioFormat/AudioFormatWriter.
- main.cpp: CLI (--plugin/--patch/--param/--note/--input/--duration/
  --wav/--sampleRate/--blockSize), with basic NaN/Inf output detection.

Has its own JuceLibraryCode-equivalent package that compiles only the
module amalgams needed for hosting (not juce_audio_plugin_client,
which is the plugin-side VST entry point and would collide with this
executable's own main()), reusing the already-vendored
sdk/juce/JUCE-3.1.1 rather than a second JUCE copy - the host and any
plugin it loads are decoupled at the VST2 ABI, not the JUCE source
level, so this doesn't need to track JaySynth's own future JUCE
version bump.

Verified: builds clean (no new warnings) via
`MAKE_HOME=... make -C tools/testhost`. Smoke-tested against two very
different real plugins:
- build/linux/release/JaySynth.so (0 in/2 out, MIDI-driven): --note
  60 100 --duration 2 --wav produces correct-length, non-silent,
  NaN-free stereo audio.
- /usr/lib/vst/ZamDelay-vst.so (1 in/1 out, real third-party VST2
  effect, no MIDI): --input pink --duration 2 --wav correctly sizes
  input/output to mono and produces non-silent, NaN-free audio -
  proof the host is generic, not JaySynth-shaped.
- /usr/lib/vst/ZamEQ2-vst.so: the host's own NaN/Inf detection caught
  a real NaN in this plugin's default state on first render - exactly
  the kind of regression this tool exists to catch without a GUI.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011dhtwRLARk4eiPngcQykLJ

Add JSON test scenarios, patch/bank scripting, and performance stats (Phases 2-4) 39353751e6

- TestScenario: parses a JSON file describing sample rate/block size/
  duration, an input signal config (silence/sine/noise/impulse), and a
  timed sequence of events (noteOn/noteOff/controller/param/loadPatch/
  loadBank/savePatch/saveBank), sorted by sample position. Relative
  file paths inside the JSON resolve against the scenario file's own
  directory, not the process cwd, so scenario files are portable.
  Nothing here is JaySynth-specific: an event type a given plugin
  doesn't care about (e.g. MIDI sent to a plugin with no MIDI input)
  is simply inert, since AudioProcessor/MidiBuffer already tolerate
  that generically.
- main.cpp: unified the ad-hoc CLI flags (--note/--param/--input/etc.)
  and --scenario JSON files through one render loop, by synthesizing
  a TestScenario from the CLI flags when no JSON file is given. Added
  --bank/--savePatch/--saveBank for the ad-hoc path (loadPatch/wav
  already existed). Non-MIDI events (param/patch/bank) scheduled
  inside a block apply at the start of that block; MIDI events keep
  full per-sample timing via MidiBuffer's own offset parameter.
- PerformanceStats: wraps each processBlock call with a high-resolution
  timer, reports min/mean/max render time and a real-time factor
  (audio seconds rendered / wall-clock seconds), plus optional
  per-block CSV export for tracking regressions over time. Never
  paces the loop to real-time - the point is to run faster than that.

Verified end-to-end:
- Ad-hoc CLI path against JaySynth.so still renders correctly and now
  reports performance (e.g. ~74x real-time on this machine).
- A JSON scenario driving JaySynth (timed noteOn/param/noteOff/
  savePatch) produces correct WAV output and a valid saved patch file.
- That saved patch round-trips back in via --patch and renders
  cleanly (proves the save/load path is consistent, not just
  one-directional).
- A JSON scenario driving a real third-party effect (ZamComp-vst.so,
  2 in/1 out - a genuinely asymmetric channel count, not just "0 in"
  or "stereo") with generated pink noise and a param event produces
  correct mono output at >1000x real-time.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011dhtwRLARk4eiPngcQykLJ

jens added 1 commit 2026-07-27 19:32:03 +02:00

Add README and TODO tracking for the testhost tool ab29ad2820

- README.md: what the tool is and isn't (generic, not JaySynth-specific),
  build instructions, CLI quick-start, full JSON scenario schema/field
  reference, the patch/bank state-format caveat (this tool's load/save
  goes through JUCE's host-side AudioPluginInstance state calls, not
  JaySynth's own GUI-triggered loadPatchFromFile/patchImportXml - a
  different code path worth knowing about), and an architecture summary.
- TODO.md: tracks what's deferred by design (LV2/VST3/LADSPA format
  registration, Windows build), test coverage gaps (stereo effects
  untested, no regression-baseline tooling, no CI wiring, no JSON-parser
  unit tests, no polyphony testing), two things worth investigating
  (a non-fatal DPF plugin assertion seen on load, an unconfirmed
  WavAudioFormat channel-count limit), and one nice-to-have.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011dhtwRLARk4eiPngcQykLJ

jens referenced this issue from a commit

2026-07-27 20:07:03 +02:00

Add regression suite testing JaySynth's TODO.md findings via testhost

jens added 1 commit 2026-07-27 20:07:03 +02:00

Add regression suite testing JaySynth's TODO.md findings via testhost f54689f071

tools/testhost/tests/run_tests.sh dynamically exercises the tracked
hardening findings using testhost itself - no GUI, sub-second wall
time:

- oversized_blocksize (Critical #5): --blockSize 16384 under a
  timeout, catching either the overflow or the deadlock the original
  bug could cause.
- nrpn_out_of_range/nrpn_in_range (Critical #1): JSON scenarios
  sending the 5-message MIDI NRPN CC sequence, once targeting the
  max 14-bit ID (16383) and once a legitimate in-range ID (500), so
  a regression that made the bounds check too aggressive would also
  show up.
- patch_save_load_roundtrip / bank_save_load_roundtrip: save a
  distinctive continuous parameter value, reload in a fresh process,
  check it round-trips within tolerance - covers the
  setCurrentProgramStateInformation no-op bug and the
  patchDecodeXml/patchImportXml fixes.
- stability_sweep: loads every bundled .fxp under extras/sounds/ and
  renders a held note through each, as a broad crash/NaN net.

Added --printParams to main.cpp (prints every parameter's index and
value) to make the round-trip tests possible - this was also already
a noted nice-to-have in testhost/TODO.md.

While building the oversized_blocksize test, found and fixed (on the
hardening branch, commit d713ca5) a second, previously-unknown bug:
VCF_CalcCoeff_LPF/_HPF/_BPF advance the coefficient buffer pointer
once per (sample, filter-section) pair, but the buffer was only ever
allocated for bufsize samples, not bufsize*sections - a 4th-order
filter overflowed it for any block between 4097 and 8192 samples.
Found via bisecting the crashing block size then confirming with an
ASan build; see tests/README.md's debugging-notes section for exactly
how, including the ASan symbolizer hang encountered along the way and
the workaround.

Also discovered (documented in tools/testhost/TODO.md's Investigate
section, not fixed - out of scope): JaySynth's compiled VST2 wrapper
never advertises chunk support to the host, so real .fxp/.fxb sample
files have no effect when loaded through the generic AudioProcessor
state API. The round-trip tests work around this by using testhost's
own save output as the fixture rather than an external sample file.

All 10 checks pass against the current hardening-branch build.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011dhtwRLARk4eiPngcQykLJ