JaySynth/tools/testhost/TODO.md

testhost TODO

Tracks known gaps and future work for tools/testhost/, the generic headless plugin test host (see README.md for what it does and how to use it, and /home/jens/.claude/plans/sharded-finding-tower.md for the original design plan). Nothing here blocks current use — Phases 1-4 of the plan are done and verified (PR #1).

Deferred by design (Phase 5 of the plan)

• LV2 plugin format support. JUCE 3.1.1 (the version currently vendored at sdk/juce/JUCE-3.1.1) has no LV2 hosting backend. Once JaySynth's own JUCE dependency is upgraded (or a custom AudioPluginFormat subclass is written against the current version), register it in PluginHost's constructor alongside VSTPluginFormat - no other host code should need to change, since everything downstream talks to the format-agnostic AudioPluginInstance/AudioProcessor API already.
• VST3/LADSPA format support. JUCE already ships VST3PluginFormat/LADSPAPluginFormat in this same vendored version - registering them is a one-line addition to PluginHost's constructor, just not done yet since nothing has needed it.
• Windows build (Makefile.win). The plan calls for mirroring src/plug's Linux/Windows Makefile pattern; only the Linux side has been built and tested so far.

Test coverage gaps

• Only mono (1 in/1 out) and asymmetric (2 in/1 out) effect channel configs have been smoke-tested (ZamDelay-vst.so, ZamComp-vst.so, ZamEQ2-vst.so). A genuinely stereo-in/stereo-out effect hasn't been run through yet - worth doing before relying on this tool for a stereo effect plugin specifically.
• No automated regression baseline comparison. The plan's stretch goal: batch-render every .fxp/.xmp under extras/sounds/ through a fixed note pattern and diff against stored baseline WAVs/checksums, to catch DSP or save-load regressions automatically instead of via one-off manual runs. Not built.
• No CI wiring. Nothing currently runs testhost automatically on push/PR; it's a manual, opt-in tool for now.
• No unit tests for TestScenario's JSON parsing itself (malformed JSON, missing required fields, wrong types, out-of-range values) - only exercised indirectly via a couple of hand-written scenario files during development. Edge-case handling (e.g. a negative sample, an unterminated noteOn with no matching noteOff) is untested.
• Polyphony/multiple simultaneous notes untested - only ever a single held note in all smoke tests so far.
• tests/run_tests.sh never asserts specific expected values for PluginHost::load()/getNumInputChannels()/getNumOutputChannels()/ getNumParameters() - every test exercises them (they're printed in the load banner), but nothing pins e.g. "JaySynth must report exactly 0 in / 2 out / 140 parameters," so a regression there would only be caught if it also broke something else. Same underlying gap as "no automated regression baseline comparison" above, just called out separately since it's about the load-time metadata rather than the rendered audio. See tests/README.md's PluginHost coverage table.

Investigate

• Bundled sample .fxp files under extras/sounds/ silently have no effect when loaded via --patch (confirmed by diffing --printParams output with and without --patch - zero parameters differ), even though loading doesn't crash. Resolved: this is not a host/plugin capability mismatch - usesChunks() is genuinely true and test_chunk_roundtrip in tests/run_tests.sh proves the real bankEncodeXml/bankDecodeXml chunk path round-trips correctly for files this build produces (see tests/README.md's "corrected finding" note for the full disproof). What remains open is narrower: why these specific (10+ year old) external sample files fail to apply - most likely a compression or XML-schema shape that JaySynthAudioProcessor::setCurrentProgramStateInformation/ getXmlFromBinary no longer parses. Not one of the tracked repo-root TODO.md findings; worth a closer look at some point (hex-diff one of these files' decompressed XML against what this build produces), but not chased further here.
• DPF-based plugins print a non-fatal assertion on load. All three Zam*-vst.so plugins tested (built with the DISTRHO Plugin Framework) print assertion failure: "fIsActive" in .../DistrhoPluginInternal.hpp during PluginHost::load(), despite JUCE's VSTPluginInstance::prepareToPlay already sending effMainsChanged/effStartProcess. Rendering still completes correctly (confirmed via WAV output), so this looks like a DPF/JUCE VST2 lifecycle-ordering quirk rather than a testhost bug, but it hasn't been root-caused - worth a closer look if it turns out to affect correctness (not just log noise) for some other plugin.
• WavAudioFormat's base-class doc comment says channel count "must be either 1 or 2" (AudioFormat::createWriterFor's Doxygen comment). WavRecorder passes through whatever PluginHost::getNumOutputChannels() reports with no clamping. Resolved: Pianoteq 8 (Pianoteq 8.so, V8.3.2) loads as 0 in / 5 out and rendered a held-note test cleanly through --wav - the output is a valid, non-silent 5-channel/24-bit/44.1kHz WAV (verified with Python's wave module: correct nchannels/nframes, non-zero sample data). No clamping or writer error occurred, so the doc comment is stale/generic boilerplate, not an enforced limit in this JUCE version.

Nice-to-have (not planned, just noted)

• A --list-params or similar introspection mode (print every parameter's index/name/current value) would help writing new scenario JSON files without cross-referencing the plugin's own GUI or source. Done: --printParams (prints PARAM <index> <value> for every parameter, after any initial --patch/--bank load) - added to make the patch/bank round-trip regression tests in tests/ possible; only prints values, not names, so cross-referencing the plugin's own parameter-name list is still occasionally needed.