Add regression suite testing JaySynth's TODO.md findings via testhost
tools/testhost/tests/run_tests.sh dynamically exercises the tracked
hardening findings using testhost itself - no GUI, sub-second wall
time:
- oversized_blocksize (Critical #5): --blockSize 16384 under a
timeout, catching either the overflow or the deadlock the original
bug could cause.
- nrpn_out_of_range/nrpn_in_range (Critical #1): JSON scenarios
sending the 5-message MIDI NRPN CC sequence, once targeting the
max 14-bit ID (16383) and once a legitimate in-range ID (500), so
a regression that made the bounds check too aggressive would also
show up.
- patch_save_load_roundtrip / bank_save_load_roundtrip: save a
distinctive continuous parameter value, reload in a fresh process,
check it round-trips within tolerance - covers the
setCurrentProgramStateInformation no-op bug and the
patchDecodeXml/patchImportXml fixes.
- stability_sweep: loads every bundled .fxp under extras/sounds/ and
renders a held note through each, as a broad crash/NaN net.
Added --printParams to main.cpp (prints every parameter's index and
value) to make the round-trip tests possible - this was also already
a noted nice-to-have in testhost/TODO.md.
While building the oversized_blocksize test, found and fixed (on the
hardening branch, commit d713ca5) a second, previously-unknown bug:
VCF_CalcCoeff_LPF/_HPF/_BPF advance the coefficient buffer pointer
once per (sample, filter-section) pair, but the buffer was only ever
allocated for bufsize samples, not bufsize*sections - a 4th-order
filter overflowed it for any block between 4097 and 8192 samples.
Found via bisecting the crashing block size then confirming with an
ASan build; see tests/README.md's debugging-notes section for exactly
how, including the ASan symbolizer hang encountered along the way and
the workaround.
Also discovered (documented in tools/testhost/TODO.md's Investigate
section, not fixed - out of scope): JaySynth's compiled VST2 wrapper
never advertises chunk support to the host, so real .fxp/.fxb sample
files have no effect when loaded through the generic AudioProcessor
state API. The round-trip tests work around this by using testhost's
own save output as the fixture rather than an external sample file.
All 10 checks pass against the current hardening-branch build.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011dhtwRLARk4eiPngcQykLJ
This commit is contained in:
Claude Sonnet 5
@@ -141,6 +141,14 @@ the same call chain, so a fix to one doesn't automatically test the other.
|
||||
into an equivalent in-memory `TestScenario` before rendering starts, so
|
||||
there's only one loop to maintain).
|
||||
|
||||
## Regression suite
|
||||
|
||||
`tests/run_tests.sh` uses this tool to dynamically test the findings
|
||||
tracked in the repo-root `TODO.md` - see `tests/README.md` for what each
|
||||
test covers (and its limits). It's also how a real, previously-unknown VCF
|
||||
buffer-sizing bug got found and fixed (see that file and the repo-root
|
||||
`TODO.md`'s Critical section).
|
||||
|
||||
## Known limitations
|
||||
|
||||
See `TODO.md` in this directory.
|
||||
|
||||
+22
-1
@@ -47,6 +47,22 @@ verified (PR https://git.jayfield.org/jens/JaySynth/pulls/1).
|
||||
|
||||
## Investigate
|
||||
|
||||
- [ ] **JaySynth's compiled VST2 wrapper never advertises chunk support to
|
||||
the host, so real `.fxp`/`.fxb` sample files silently have no effect when
|
||||
loaded generically.** Found while building `tests/run_tests.sh`: loading
|
||||
a bundled sample under `extras/sounds/` via `--patch` produces byte-
|
||||
identical output to not loading anything (confirmed by diffing
|
||||
`--printParams`). The files are legitimately in VST2's opaque-chunk
|
||||
("FPCh") format, but JUCE 3.1.1's plugin-side VST wrapper
|
||||
(`juce_VST_Wrapper.cpp`, compiled into `JaySynth.so`) never sets
|
||||
`effFlagsProgramChunks` on the `AEffect` struct, so JUCE's own host-side
|
||||
`VSTPluginInstance::usesChunks()` returns false and silently no-ops both
|
||||
load and save. Not one of the tracked `TODO.md` (repo root) findings, and
|
||||
not attempted here - fixing it means patching JUCE's own VST wrapper,
|
||||
a bigger and riskier change than this pass's scope. See
|
||||
`tests/README.md` for the full writeup; `tests/run_tests.sh`'s round-trip
|
||||
tests work around it by using `testhost`'s own save output as the
|
||||
fixture instead of an external sample file.
|
||||
- [ ] **DPF-based plugins print a non-fatal assertion on load.** All three
|
||||
Zam*-vst.so plugins tested (built with the DISTRHO Plugin Framework) print
|
||||
`assertion failure: "fIsActive" in .../DistrhoPluginInternal.hpp` during
|
||||
@@ -67,6 +83,11 @@ verified (PR https://git.jayfield.org/jens/JaySynth/pulls/1).
|
||||
|
||||
## Nice-to-have (not planned, just noted)
|
||||
|
||||
- [ ] A `--list-params` or similar introspection mode (print every
|
||||
- [x] A `--list-params` or similar introspection mode (print every
|
||||
parameter's index/name/current value) would help writing new scenario
|
||||
JSON files without cross-referencing the plugin's own GUI or source.
|
||||
Done: `--printParams` (prints `PARAM <index> <value>` for every
|
||||
parameter, after any initial `--patch`/`--bank` load) - added to make
|
||||
the patch/bank round-trip regression tests in `tests/` possible; only
|
||||
prints values, not names, so cross-referencing the plugin's own
|
||||
parameter-name list is still occasionally needed.
|
||||
|
||||
@@ -39,6 +39,9 @@ namespace
|
||||
" --sampleRate <hz> default 44100\n"
|
||||
" --blockSize <n> default 512\n"
|
||||
" --wav <file> record the plugin's output to a WAV file\n"
|
||||
" --printParams print every parameter's index and current value\n"
|
||||
" (after any --patch/--bank/--param has been applied,\n"
|
||||
" before rendering) as \"PARAM <index> <value>\" lines\n"
|
||||
"\n"
|
||||
"Patch/bank state is exactly what JUCE's AudioPluginInstance::get/setCurrentProgram-\n"
|
||||
"StateInformation and get/setStateInformation read and write for a hosted VST2\n"
|
||||
@@ -66,6 +69,7 @@ namespace
|
||||
double inputFrequencyHz = 440.0;
|
||||
|
||||
String perfCsvFile;
|
||||
bool printParams = false;
|
||||
};
|
||||
|
||||
bool parseArgs (int argc, char *argv[], Args &args)
|
||||
@@ -96,6 +100,8 @@ namespace
|
||||
args.blockSize = String (argv[++i]).getIntValue();
|
||||
else if (arg == "--perfCsv" && i + 1 < argc)
|
||||
args.perfCsvFile = argv[++i];
|
||||
else if (arg == "--printParams")
|
||||
args.printParams = true;
|
||||
else if (arg == "--note" && i + 2 < argc)
|
||||
{
|
||||
args.hasNote = true;
|
||||
@@ -294,6 +300,12 @@ int main (int argc, char *argv[])
|
||||
if (scenario.initialLoadPatchFile.isNotEmpty() && ! loadStateFile (scenario.initialLoadPatchFile, false, host))
|
||||
return 1;
|
||||
|
||||
if (args.printParams)
|
||||
{
|
||||
for (int i = 0; i < host.getNumParameters(); ++i)
|
||||
printf ("PARAM %d %.6f\n", i, host.getParameter (i));
|
||||
}
|
||||
|
||||
AudioInputSource inputSource;
|
||||
inputSource.prepare (scenario.sampleRate);
|
||||
switch (scenario.inputType)
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
scenarios/*.wav
|
||||
@@ -0,0 +1,122 @@
|
||||
# Regression suite for JaySynth's hardening findings
|
||||
|
||||
`run_tests.sh` uses `testhost` itself to dynamically exercise the findings
|
||||
tracked in the repo-root `TODO.md` (the "hardening" review) — no GUI, no
|
||||
manual clicking, runs in well under a second of wall-clock time.
|
||||
|
||||
## Running
|
||||
|
||||
```sh
|
||||
MAKE_HOME=/path/to/submodule/make ./run_tests.sh [path/to/JaySynth.so]
|
||||
```
|
||||
|
||||
Builds `testhost` first if it isn't already built. Defaults to
|
||||
`build/linux/release/JaySynth.so` if no path is given. Exit code is 0 if
|
||||
everything passed, 1 otherwise (CI-friendly). On failure, per-test logs and
|
||||
WAVs are kept in a `mktemp -d` workdir printed at the end; on success
|
||||
that workdir is deleted.
|
||||
|
||||
## What each test actually covers
|
||||
|
||||
**`oversized_blocksize`** (Critical #5) — renders through `--blockSize
|
||||
16384` (double `SYNTH_MAX_BUFSIZE`). Before the fix this either overflowed
|
||||
`processBlock`'s fixed 8192-sample stack buffer or deadlocked the render
|
||||
worker threads; `timeout` is what actually catches the deadlock case, since
|
||||
a hung process would otherwise just sit here forever. **This test is also
|
||||
what caught a second, previously-unknown bug**: `VCF_CalcCoeff_LPF/_HPF/_BPF`
|
||||
advance their coefficient-buffer pointer once per *(sample, filter-section)*
|
||||
pair, but the buffer was only ever allocated for `bufsize` samples, not
|
||||
`bufsize × sections` — a 4th-order filter (2 sections) overflowed it for
|
||||
any block between 4097 and 8192 samples, a range this fix's own "chunk to
|
||||
≤8192" logic doesn't protect against on its own. Found via AddressSanitizer
|
||||
after this test kept crashing at blockSize values *below* 8192; fixed in
|
||||
`src/synth/vcf.c` (see TODO.md's Critical section). If this test ever
|
||||
regresses again, don't assume it's the same bug — bisect the block size
|
||||
first (see the debugging notes at the bottom of this file for how that
|
||||
investigation went).
|
||||
|
||||
**`nrpn_out_of_range`/`nrpn_in_range`** (Critical #1) — sends the standard
|
||||
5-message MIDI NRPN CC sequence (99/98/6/38/98) via a JSON scenario, once
|
||||
targeting the maximum possible 14-bit NRPN ID (16383) and once a legitimate
|
||||
in-range ID (500). Both must complete cleanly; the in-range case exists so
|
||||
a regression that made the bounds check *too* aggressive (rejecting valid
|
||||
IDs) would also be visible as a behavioural difference. **Caveat:** "doesn't
|
||||
crash" is a necessary but not fully sufficient check here. The original bug
|
||||
was an out-of-bounds *write* into `last_midiCC_info[]`, a plain array
|
||||
embedded inside the larger heap-allocated `JaySynth` object — not a
|
||||
separate heap allocation — so an overflow of a few hundred entries lands in
|
||||
*other member fields of the same object* rather than past its heap
|
||||
allocation. That means it can silently corrupt adjacent state without
|
||||
necessarily crashing in a short test run, and it's why the ID is pushed to
|
||||
the actual maximum (16383) rather than something more modest: to maximize
|
||||
the chance that, absent the fix, the write would land far enough away to
|
||||
hit unrelated memory and actually crash. The real guarantee is the
|
||||
bounds-check line itself (`src/plug/JaySynth.cpp`, `handleController`), not
|
||||
this test's silence.
|
||||
|
||||
**`patch_save_load_roundtrip`** / **`bank_save_load_roundtrip`** — covers
|
||||
two related fixes: (a) `setCurrentProgramStateInformation` used to pass
|
||||
`patchImportXml` an XML node one level too deep, making the VST2 "copy
|
||||
plugin state" path a complete no-op; (b) the `patchDecodeXml`/
|
||||
`patchDecodeXml_legacy` deduplication and `patchImportXml`'s legacy-branch
|
||||
restructuring. Sets a continuous parameter (index 0 = `SYNTH_PARAM_VOLUME`)
|
||||
to a distinctive value, saves, reloads in a fresh process, and checks the
|
||||
value reads back within a small tolerance (not exact equality — the
|
||||
slider/internal-value scaling curve, `toParam`/`toSlider`, isn't bit-exact
|
||||
across a round trip). Uses `testhost`'s own save→load cycle as the fixture,
|
||||
not an externally-provided `.fxp`/`.fxb` — see the chunk-format note below
|
||||
for why.
|
||||
|
||||
**`stability_sweep`** — loads every bundled `.fxp` under `extras/sounds/`
|
||||
and renders a held note through each, checking for crashes/hangs/NaN. A
|
||||
broad regression net, not a targeted test for any one finding.
|
||||
|
||||
## Two things discovered while building this suite that are worth knowing
|
||||
|
||||
**The VCF coefficient-buffer bug above** (already fixed, see TODO.md).
|
||||
|
||||
**JaySynth's compiled VST2 wrapper never advertises chunk support to the
|
||||
host**, so loading the bundled sample `.fxp` files through `testhost`'s
|
||||
generic `--patch` (i.e. through `AudioProcessor::setCurrentProgramState-
|
||||
Information`, exactly what any real VST2 host uses) has **zero effect** —
|
||||
confirmed by diffing `--printParams` output with and without `--patch`.
|
||||
Those sample files are legitimately in VST2's opaque-chunk ("FPCh") format
|
||||
(verified via hex dump: correct `CcnK`/`FPCh`/`Jsy1` header), but JUCE
|
||||
3.1.1's plugin-side VST wrapper (`sdk/juce/JUCE-3.1.1/modules/
|
||||
juce_audio_plugin_client/VST/juce_VST_Wrapper.cpp`, compiled into
|
||||
`JaySynth.so`) never sets the VST2 `effFlagsProgramChunks` flag on the
|
||||
`AEffect` struct, so JUCE's own host-side `VSTPluginInstance::usesChunks()`
|
||||
sees false and `setChunkData()` silently no-ops — for both loading *and*
|
||||
saving. This is **not** one of the tracked `TODO.md` findings and this
|
||||
suite doesn't attempt to fix it (it would mean patching JUCE's own VST
|
||||
wrapper, a materially bigger and riskier change). It's why
|
||||
`stability_sweep` only checks "doesn't crash," not "changes the patch," and
|
||||
why the round-trip tests use `testhost`'s own save output as the fixture
|
||||
rather than a real sample file — `testhost`'s save path has the same
|
||||
`usesChunks()`-is-false behaviour, so save and load are at least
|
||||
consistent with *each other*, even though neither talks to the external
|
||||
`.fxp` files' actual chunk data. Tracked as a discovered issue in
|
||||
`tools/testhost/TODO.md`'s "Investigate" section.
|
||||
|
||||
## Debugging notes: how the VCF bug was actually found
|
||||
|
||||
`oversized_blocksize` initially failed with `free(): invalid next size
|
||||
(normal)` - glibc's heap-corruption detector, tripped during process
|
||||
teardown, not at the point of the actual overflow. Bisecting `--blockSize`
|
||||
found the real threshold was 4096/4097, not 8192 as expected. Rather than
|
||||
guess further, an ASan build of just `src/synth` + `src/plug` (env-var
|
||||
override: `CFLAGS/CXXFLAGS="-fsanitize=address -fno-omit-frame-pointer -g
|
||||
-O1" LDFLAGS="-fsanitize=address"`, `CONFIG=asan` to keep it in its own
|
||||
build directory) run via `LD_PRELOAD=$(clang -print-file-name=libclang_rt.
|
||||
asan-x86_64.so)` pinpointed it exactly. Two gotchas worth remembering if
|
||||
this needs doing again: (1) `ASAN_OPTIONS=symbolize=0` is necessary in this
|
||||
environment — the default external `llvm-symbolizer` fork hangs
|
||||
indefinitely rather than resolving addresses, so use `symbolize=0` and
|
||||
resolve the reported `JaySynth.so+0x...` offsets with `addr2line -e
|
||||
JaySynth.so <offset>` (or just `nm`/`objdump` if `addr2line` needs exact
|
||||
offsets adjusted for the binary's load bias) instead of trusting ASan's own
|
||||
backtrace; (2) rebuilding the ASan variant needs a real `git worktree`
|
||||
avoided here — `sdk/juce/JUCE-3.1.1` is vendored as a zip+patch pair
|
||||
extracted on demand, not a plain tracked directory, so a fresh worktree
|
||||
doesn't have it. Simpler to `git stash` uncommitted work and build the
|
||||
ASan variant directly on the branch with the fix under investigation.
|
||||
Executable
+318
@@ -0,0 +1,318 @@
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# Regression suite for the findings tracked in JaySynth's TODO.md, run via
|
||||
# the testhost tool itself. See tests/README.md for what each test actually
|
||||
# covers, and - just as importantly - what it doesn't.
|
||||
#
|
||||
# Usage:
|
||||
# MAKE_HOME=/path/to/submodule/make ./run_tests.sh [path/to/JaySynth.so]
|
||||
#
|
||||
# Exit code is 0 if every test passed, 1 otherwise (so this is CI-friendly).
|
||||
|
||||
set -u
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
TESTHOST_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||
REPO_ROOT="$(cd "$TESTHOST_DIR/../.." && pwd)"
|
||||
|
||||
TESTHOST_BIN="$TESTHOST_DIR/build/linux/release/testhost"
|
||||
PLUGIN="${1:-$REPO_ROOT/build/linux/release/JaySynth.so}"
|
||||
|
||||
WORKDIR="$(mktemp -d /tmp/testhost-suite.XXXXXX)"
|
||||
PASS_COUNT=0
|
||||
FAIL_COUNT=0
|
||||
FAILED_NAMES=()
|
||||
|
||||
log() { printf '%s\n' "$*"; }
|
||||
pass() { log " PASS: $1"; PASS_COUNT=$((PASS_COUNT + 1)); }
|
||||
fail() { log " FAIL: $1"; FAIL_COUNT=$((FAIL_COUNT + 1)); FAILED_NAMES+=("$1"); }
|
||||
|
||||
# --- Preconditions -----------------------------------------------------
|
||||
|
||||
if [ ! -x "$TESTHOST_BIN" ]; then
|
||||
log "Building testhost..."
|
||||
if ! MAKE_HOME="${MAKE_HOME:-$(realpath "$REPO_ROOT/submodule/make")}" make -C "$TESTHOST_DIR" > "$WORKDIR/build.log" 2>&1; then
|
||||
log "FATAL: testhost build failed - see $WORKDIR/build.log"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
if [ ! -f "$PLUGIN" ]; then
|
||||
log "FATAL: plugin not found at '$PLUGIN' (build JaySynth first, or pass its path as \$1)"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
run_testhost() {
|
||||
# run_testhost <timeout_seconds> <args...>
|
||||
local timeout_s="$1"; shift
|
||||
timeout "${timeout_s}s" "$TESTHOST_BIN" --plugin "$PLUGIN" "$@"
|
||||
}
|
||||
|
||||
wav_is_valid_and_nonsilent() {
|
||||
# Very small, dependency-light sanity check: file exists, non-trivial
|
||||
# size (a 0/near-0 byte file means recording never really started).
|
||||
[ -s "$1" ] && [ "$(stat -c%s "$1" 2>/dev/null || echo 0)" -gt 1000 ]
|
||||
}
|
||||
|
||||
# --- Test 1: oversized block size (Critical #5) -------------------------
|
||||
# Before the fix, a host block size larger than SYNTH_MAX_BUFSIZE (8192)
|
||||
# either overflowed processBlock's fixed 8192-sample stack buffer or
|
||||
# deadlocked the render worker threads (JaySynthThread::go() silently
|
||||
# no-op'd, and renderNextBlock's wait(-1) blocked forever). The `timeout`
|
||||
# wrapper is what actually catches the deadlock case - a hang would
|
||||
# otherwise just sit here forever.
|
||||
test_oversized_blocksize() {
|
||||
local name="oversized_blocksize (Critical #5)"
|
||||
local out="$WORKDIR/oversized.wav"
|
||||
local log_file="$WORKDIR/oversized.log"
|
||||
|
||||
if run_testhost 20 --blockSize 16384 --note 60 100 --duration 1.5 --wav "$out" > "$log_file" 2>&1; then
|
||||
if grep -qi "WARNING - output contained NaN" "$log_file"; then
|
||||
fail "$name (rendered but produced NaN/Inf - see $log_file)"
|
||||
elif ! wav_is_valid_and_nonsilent "$out"; then
|
||||
fail "$name (no valid WAV produced - see $log_file)"
|
||||
else
|
||||
pass "$name"
|
||||
fi
|
||||
else
|
||||
local rc=$?
|
||||
if [ "$rc" -eq 124 ]; then
|
||||
fail "$name (HUNG - killed by timeout, matches the pre-fix deadlock)"
|
||||
else
|
||||
fail "$name (crashed, exit code $rc - see $log_file)"
|
||||
fi
|
||||
fi
|
||||
}
|
||||
|
||||
# --- Test 2/3: NRPN controller ID bounds (Critical #1) -------------------
|
||||
# handleController used to write last_midiCC_info[midiCC_info.ID] with no
|
||||
# bounds check; NRPN IDs assemble to up to 16383 (14-bit) but the array is
|
||||
# sized NUM_MIDI_CONTROLLERS=1024. This can't be observed from the outside
|
||||
# without a memory sanitizer (the write lands inside the same heap object,
|
||||
# not necessarily past its allocation), so "doesn't crash" is a necessary
|
||||
# but not fully sufficient check - the real guarantee is the bounds-check
|
||||
# line itself (see tests/README.md). Still worth running both an
|
||||
# out-of-range and an in-range NRPN sequence, so a regression that made the
|
||||
# check *too* aggressive (rejecting valid IDs) would also show up as a
|
||||
# behavioural difference between the two runs.
|
||||
test_nrpn_bounds() {
|
||||
local name_oor="nrpn_out_of_range, ID=16383 (Critical #1)"
|
||||
local name_ir="nrpn_in_range, ID=500 (Critical #1 regression guard)"
|
||||
|
||||
local out_oor="$WORKDIR/nrpn_oor.wav"
|
||||
local log_oor="$WORKDIR/nrpn_oor.log"
|
||||
if run_testhost 15 --scenario "$SCRIPT_DIR/scenarios/nrpn_out_of_range.json" > "$log_oor" 2>&1; then
|
||||
# recordWav in the scenario JSON resolves relative to the scenario
|
||||
# file's own directory, not $WORKDIR - move it out after the run.
|
||||
mv "$SCRIPT_DIR/scenarios/out_nrpn_out_of_range.wav" "$out_oor" 2>/dev/null
|
||||
if grep -qi "WARNING - output contained NaN" "$log_oor"; then
|
||||
fail "$name_oor (NaN/Inf in output - see $log_oor)"
|
||||
elif ! wav_is_valid_and_nonsilent "$out_oor"; then
|
||||
fail "$name_oor (no valid WAV - see $log_oor)"
|
||||
else
|
||||
pass "$name_oor"
|
||||
fi
|
||||
else
|
||||
local rc=$?
|
||||
fail "$name_oor (exit code $rc, expected clean completion - see $log_oor)"
|
||||
fi
|
||||
|
||||
local out_ir="$WORKDIR/nrpn_ir.wav"
|
||||
local log_ir="$WORKDIR/nrpn_ir.log"
|
||||
if run_testhost 15 --scenario "$SCRIPT_DIR/scenarios/nrpn_in_range.json" > "$log_ir" 2>&1; then
|
||||
mv "$SCRIPT_DIR/scenarios/out_nrpn_in_range.wav" "$out_ir" 2>/dev/null
|
||||
if grep -qi "WARNING - output contained NaN" "$log_ir"; then
|
||||
fail "$name_ir (NaN/Inf in output - see $log_ir)"
|
||||
elif ! wav_is_valid_and_nonsilent "$out_ir"; then
|
||||
fail "$name_ir (no valid WAV - see $log_ir)"
|
||||
else
|
||||
pass "$name_ir"
|
||||
fi
|
||||
else
|
||||
local rc=$?
|
||||
fail "$name_ir (exit code $rc - see $log_ir)"
|
||||
fi
|
||||
}
|
||||
|
||||
# --- Test 4: patch state save/load round-trip ---------------------------
|
||||
# Covers two related fixes together: (a) setCurrentProgramStateInformation
|
||||
# was passing patchImportXml an XML node one level too deep, making the
|
||||
# VST2 "copy plugin state" path a complete no-op (found while fixing the
|
||||
# Patch/Bank import-export findings); (b) patchDecodeXml/patchDecodeXml_legacy
|
||||
# deduplication and the patchImportXml legacy-branch restructuring. All of
|
||||
# these sit on the exact save->load call chain exercised here. Uses a
|
||||
# continuous parameter (index 0 = SYNTH_PARAM_VOLUME) rather than a
|
||||
# stepped/boolean one, and a tolerance rather than exact equality, because
|
||||
# the value/slider scaling curve (toParam/toSlider) is not bit-exact across
|
||||
# a round trip - see tests/README.md.
|
||||
test_patch_roundtrip() {
|
||||
local name="patch_save_load_roundtrip"
|
||||
local patch_file="$WORKDIR/roundtrip_patch.bin"
|
||||
local log1="$WORKDIR/roundtrip_save.log"
|
||||
local log2="$WORKDIR/roundtrip_load.log"
|
||||
local target_value="0.37"
|
||||
local tolerance="0.02"
|
||||
|
||||
if ! run_testhost 10 --param 0 "$target_value" --savePatch "$patch_file" --duration 0.05 > "$log1" 2>&1; then
|
||||
fail "$name (save step failed, exit $? - see $log1)"
|
||||
return
|
||||
fi
|
||||
|
||||
if [ ! -s "$patch_file" ]; then
|
||||
fail "$name (no patch file written)"
|
||||
return
|
||||
fi
|
||||
|
||||
if ! run_testhost 10 --patch "$patch_file" --printParams --duration 0.01 > "$log2" 2>&1; then
|
||||
fail "$name (load step failed, exit $? - see $log2)"
|
||||
return
|
||||
fi
|
||||
|
||||
local readback
|
||||
readback="$(grep '^PARAM 0 ' "$log2" | awk '{print $3}')"
|
||||
|
||||
if [ -z "$readback" ]; then
|
||||
fail "$name (PARAM 0 not found in output - see $log2)"
|
||||
return
|
||||
fi
|
||||
|
||||
local diff
|
||||
diff="$(awk -v a="$readback" -v b="$target_value" 'BEGIN { d = a - b; if (d < 0) d = -d; print d }')"
|
||||
local within_tolerance
|
||||
within_tolerance="$(awk -v d="$diff" -v t="$tolerance" 'BEGIN { print (d <= t) ? "1" : "0" }')"
|
||||
|
||||
if [ "$within_tolerance" = "1" ]; then
|
||||
pass "$name (wrote $target_value, read back $readback, within tolerance)"
|
||||
else
|
||||
fail "$name (wrote $target_value, read back $readback - outside +-$tolerance tolerance - see $log1 / $log2)"
|
||||
fi
|
||||
}
|
||||
|
||||
# --- Test 5: bank state save/load round-trip -----------------------------
|
||||
# Same idea as test 4 but through getStateInformation/setStateInformation
|
||||
# (the whole-bank path) and bankEncodeXml/bankDecodeXml/bankImportXml,
|
||||
# rather than the single-patch path.
|
||||
test_bank_roundtrip() {
|
||||
local name="bank_save_load_roundtrip"
|
||||
local bank_file="$WORKDIR/roundtrip_bank.bin"
|
||||
local log1="$WORKDIR/roundtrip_bank_save.log"
|
||||
local log2="$WORKDIR/roundtrip_bank_load.log"
|
||||
local target_value="0.62"
|
||||
local tolerance="0.02"
|
||||
|
||||
if ! run_testhost 10 --param 0 "$target_value" --saveBank "$bank_file" --duration 0.05 > "$log1" 2>&1; then
|
||||
fail "$name (save step failed, exit $? - see $log1)"
|
||||
return
|
||||
fi
|
||||
|
||||
if [ ! -s "$bank_file" ]; then
|
||||
fail "$name (no bank file written)"
|
||||
return
|
||||
fi
|
||||
|
||||
if ! run_testhost 10 --bank "$bank_file" --printParams --duration 0.01 > "$log2" 2>&1; then
|
||||
fail "$name (load step failed, exit $? - see $log2)"
|
||||
return
|
||||
fi
|
||||
|
||||
local readback
|
||||
readback="$(grep '^PARAM 0 ' "$log2" | awk '{print $3}')"
|
||||
|
||||
if [ -z "$readback" ]; then
|
||||
fail "$name (PARAM 0 not found in output - see $log2)"
|
||||
return
|
||||
fi
|
||||
|
||||
local diff
|
||||
diff="$(awk -v a="$readback" -v b="$target_value" 'BEGIN { d = a - b; if (d < 0) d = -d; print d }')"
|
||||
local within_tolerance
|
||||
within_tolerance="$(awk -v d="$diff" -v t="$tolerance" 'BEGIN { print (d <= t) ? "1" : "0" }')"
|
||||
|
||||
if [ "$within_tolerance" = "1" ]; then
|
||||
pass "$name (wrote $target_value, read back $readback, within tolerance)"
|
||||
else
|
||||
fail "$name (wrote $target_value, read back $readback - outside +-$tolerance tolerance - see $log1 / $log2)"
|
||||
fi
|
||||
}
|
||||
|
||||
# --- Test 6: stability sweep over bundled sample patches ------------------
|
||||
# Broad regression net: load every bundled .fxp under extras/sounds/ and
|
||||
# render a held note through it. NOTE: as discovered while building this
|
||||
# suite, these particular sample files are in VST2's opaque-chunk ("FPCh")
|
||||
# format, but JUCE 3.1.1's plugin-side VST wrapper (compiled into
|
||||
# JaySynth.so) never sets effFlagsProgramChunks, so JUCE's *host*-side
|
||||
# VSTPluginInstance::usesChunks() sees false and silently ignores the
|
||||
# chunk data entirely - confirmed by diffing --printParams output with and
|
||||
# without --patch: zero parameters differ. This is a real, separate
|
||||
# finding (see tests/README.md and testhost/TODO.md) - NOT one of the
|
||||
# tracked TODO.md findings, and NOT something this suite attempts to fix.
|
||||
# So this test only checks "loading this file doesn't crash the process",
|
||||
# not "loading this file changes the patch" - it still exercises the
|
||||
# malloc/delete[] paths touched by the Memory-management fixes as a broad
|
||||
# smoke test, just not the patch-content-application path.
|
||||
test_stability_sweep() {
|
||||
local sounds_dir="$REPO_ROOT/extras/sounds"
|
||||
local any=0
|
||||
|
||||
while IFS= read -r -d '' fxp; do
|
||||
any=1
|
||||
local base
|
||||
base="$(basename "$fxp")"
|
||||
local log_file="$WORKDIR/sweep_$(echo "$base" | tr -c 'A-Za-z0-9._-' '_').log"
|
||||
|
||||
if run_testhost 15 --patch "$fxp" --note 60 100 --duration 0.5 > "$log_file" 2>&1; then
|
||||
if grep -qi "WARNING - output contained NaN" "$log_file"; then
|
||||
fail "stability_sweep: '$base' (NaN/Inf - see $log_file)"
|
||||
else
|
||||
pass "stability_sweep: '$base'"
|
||||
fi
|
||||
else
|
||||
local rc=$?
|
||||
fail "stability_sweep: '$base' (exit code $rc, expected 0 - see $log_file)"
|
||||
fi
|
||||
done < <(find "$sounds_dir" -maxdepth 1 -iname '*.fxp' -print0 2>/dev/null)
|
||||
|
||||
if [ "$any" -eq 0 ]; then
|
||||
log " (no .fxp files found under $sounds_dir - skipping stability sweep)"
|
||||
fi
|
||||
}
|
||||
|
||||
# --- Run everything -------------------------------------------------------
|
||||
|
||||
log "testhost regression suite"
|
||||
log " testhost: $TESTHOST_BIN"
|
||||
log " plugin: $PLUGIN"
|
||||
log " workdir: $WORKDIR"
|
||||
log ""
|
||||
|
||||
log "Critical finding #5 - oversized block size:"
|
||||
test_oversized_blocksize
|
||||
log ""
|
||||
|
||||
log "Critical finding #1 - NRPN controller ID bounds:"
|
||||
test_nrpn_bounds
|
||||
log ""
|
||||
|
||||
log "Patch/Bank import-export fixes - state round-trip:"
|
||||
test_patch_roundtrip
|
||||
test_bank_roundtrip
|
||||
log ""
|
||||
|
||||
log "General stability sweep over bundled sample patches:"
|
||||
test_stability_sweep
|
||||
log ""
|
||||
|
||||
log "----------------------------------------"
|
||||
log "Results: $PASS_COUNT passed, $FAIL_COUNT failed"
|
||||
if [ "$FAIL_COUNT" -gt 0 ]; then
|
||||
log "Failed:"
|
||||
for n in "${FAILED_NAMES[@]}"; do
|
||||
log " - $n"
|
||||
done
|
||||
log "Artifacts (logs/WAVs) kept in: $WORKDIR"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
log "All tests passed. Cleaning up $WORKDIR."
|
||||
rm -rf "$WORKDIR"
|
||||
exit 0
|
||||
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"_comment": "Companion to nrpn_out_of_range.json: sends a LEGITIMATE, in-range NRPN ID (3<<7 | 116 = 500, well within NUM_MIDI_CONTROLLERS=1024) through the same CC sequence, to confirm the bounds-check fix in JaySynth::handleController did not break normal NRPN handling for valid IDs. Success = process completes cleanly, same as the out-of-range case (this tool can't directly observe internal state, so this is a regression guard against the fix being overly broad, not a positive functional check).",
|
||||
"sampleRate": 44100,
|
||||
"blockSize": 512,
|
||||
"durationSeconds": 1.0,
|
||||
"events": [
|
||||
{ "sample": 0, "type": "controller", "channel": 1, "controllerNumber": 99, "controllerValue": 3 },
|
||||
{ "sample": 1, "type": "controller", "channel": 1, "controllerNumber": 98, "controllerValue": 116 },
|
||||
{ "sample": 2, "type": "controller", "channel": 1, "controllerNumber": 6, "controllerValue": 64 },
|
||||
{ "sample": 3, "type": "controller", "channel": 1, "controllerNumber": 38, "controllerValue": 0 },
|
||||
{ "sample": 4, "type": "controller", "channel": 1, "controllerNumber": 98, "controllerValue": 116 },
|
||||
{ "sample": 10, "type": "noteOn", "channel": 1, "note": 60, "velocity": 100 },
|
||||
{ "sample": 44050, "type": "noteOff", "channel": 1, "note": 60 }
|
||||
],
|
||||
"recordWav": "out_nrpn_in_range.wav"
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"_comment": "Regression test for the fixed NRPN out-of-bounds write (TODO.md Critical finding #1: JaySynth::handleController wrote last_midiCC_info[midiCC_info.ID] with no bounds check; NRPN IDs assemble to up to 16383 but the array is sized NUM_MIDI_CONTROLLERS=1024). This sends the maximum possible 14-bit NRPN ID (127<<7 | 127 = 16383) via the standard MIDI NRPN CC sequence (99=NRPN MSB, 98=NRPN LSB, 6=Data Entry MSB, 38=Data Entry LSB, 98 again to complete/trigger the sequence - see MidiNrpn::process). Before the fix this reliably wrote far outside the array; after the fix, handleController returns immediately for any out-of-range ID. Success = process completes without crashing/hanging and produces valid, NaN-free audio.",
|
||||
"sampleRate": 44100,
|
||||
"blockSize": 512,
|
||||
"durationSeconds": 1.0,
|
||||
"events": [
|
||||
{ "sample": 0, "type": "controller", "channel": 1, "controllerNumber": 99, "controllerValue": 127 },
|
||||
{ "sample": 1, "type": "controller", "channel": 1, "controllerNumber": 98, "controllerValue": 127 },
|
||||
{ "sample": 2, "type": "controller", "channel": 1, "controllerNumber": 6, "controllerValue": 64 },
|
||||
{ "sample": 3, "type": "controller", "channel": 1, "controllerNumber": 38, "controllerValue": 0 },
|
||||
{ "sample": 4, "type": "controller", "channel": 1, "controllerNumber": 98, "controllerValue": 127 },
|
||||
{ "sample": 10, "type": "noteOn", "channel": 1, "note": 60, "velocity": 100 },
|
||||
{ "sample": 44050, "type": "noteOff", "channel": 1, "note": 60 }
|
||||
],
|
||||
"recordWav": "out_nrpn_out_of_range.wav"
|
||||
}
|
||||
Reference in New Issue
Block a user