As you can see in the screenshot, no bytes were dropped. All 99 messages have been received correctly.
Note that MT Pro splits Sys Ex messages in chunks of 1024 bytes, so each preset of 1178 bytes is split into one 1024 chunk and a following 154 byte chunk. I’ve saved the received file, which shows us that it has the same size. The Log Window was enabled in MT Pro and it showed all 99 messages in approx. 9 seconds – as expected.
I’m not sure why Steve C experienced data loss. Maybe you relied on the reported received bulk dump size? That’s not an accurate number, it may not have counted the first Sys Ex message or so. I’ve deactivated bulk dump detection entirely to minimize potential sources of error.
Saving the MIDI INPUT from the file menu will tell you the actual size.
Also, Steve C has used LoopMIDI for sending MIDI, which introduces yet another unknown to the equation.
I hope all this makes sense! I’m very keen on finding out where the data loss actually happens.
I debugged the vst , Windiows’ Long MIDI message call called 495 times (5 times per sysex message)
All of the 99 presets sent out from the vst to MT Pro
Unfortunately I must dig into it because of another reasons…
The vst editor plugin’s developer didn’t fixed the errors, and didn’t followed the new firmware of the synth.
So I implemented the most important things directly into the dll…
VST sends 99 messages in 495 chunks
(do you know the size of each chunk?)
when VST sends to Synth USB port directly, the synth receives all data
When VST sends at its normal speed (approx. 12KB/s), MT Pro does not receive the middle portion of the messages
VST takes less time to send all data when sending to MT Pro.
When VST is throttled manually (in debugger), MT Pro receives all messages
At this moment, my best guess is that the VST uses an unusual way to use and time the calls to midiOutLongMsg(). Windows MME is not trivial to use, you need to allocate, prepare, unprepare, and deallocate the buffers passed to midiOutLongMsg(). Sending is actually happening asynchronously, so if you don’t make sure that a buffer is entirely sent before reusing it, you may interrupt a pending buffer and prevent it from being sent. A different MIDI driver (USB) might behave differently then.
Although the Bome MIDI ports are extremely well tested and stable, you could try with a different loopback driver (like LoopMIDI) for passing MIDI data from VST to MT Pro. It might behave differently on unusual usage of midiOutLongMsg().
I analyzed, how the chunks are built up:
One sysex program size is 1178 bytes.
One chunk’s size is 0x100 (256) bytes
4*256bytes + 178bytes = 5 time / sysex . The math is ok, the dupm of 99 programs requires 495 calls.
Yes, exactly.
Yes, the presets between 13 and 66 are lost. (also not visible in the log)
Yes, it seems too fast to me.
Yes , exactly.
I don’t agree, because it works without problems with a different end point.
Hhmm, theoretically this could be the problem with MT Pro, because it can’t pump out the messages to the synth as fast how the new sysex messages are arriving from the VST. So the ‘waiting’ sysex messages could be overwritten with the new ones.
I like your product , it is the industrial standard for me, LoopMIDI and others don’t provide enough programming ability for customizing.
Thank you for taking the time to look into all this!
That was an attempt to explain the behavior we see. Exactly because it works with USB directly and not with the Bome virtual ports is the reason why the VST could be using some unusual way of sending MIDI – which happens to work fine with the USB driver and does not work well with the virtual driver.
As you see in the Send SX settings (screenshots above), sending MIDI on Windows can be done in numerous ways, and different drivers and devices behave very differently.
However, ‘normal’ MIDI software should work fine with virtual ports and also with USB MIDI devices.
Now it could also be that the USB driver is ‘misbehaving’ and the VST is optimized to still work with it. Is it the standard USB class driver, or a manufacturer-provided driver? Which synth is it?
MT Pro implements very very safe and proven MIDI buffer handling. It certainly never overwrites buffers that are still in use.
In case of performance problems, MT Pro either drops messages (and logs that) or sends them correctly. But as said, there cannot be performance problems with 10 messages per second, when it can process millions of messages per second.
The Send SX loopback test confirms the points above.
To rule out that the sending side of MT Pro could be part of the problem, what happens if you disable the MIDI route, but still keep the Bome Virtual MIDI Port 2 open (for receiving messages)?
I meant to only use LoopMIDI for sending MIDI from VST to MT Pro.
Route removed, sysex sending, nothing happened, this is the excpectation.
But, I routed the virtual port to another virtual port and turned on the logging. Voila , all of the sysex messages are visible in the log, nothing missing.
Just for fun I tried to route the routed port forward to the synth, but it does nothing. (hopefully by design)
The synth is a Dave Smith Instruments Pro 2 , so the MIDI implementation must be perfect
And it uses the standard MS USB driver.
I think, now we are a step closer to the localization of the error, because the MIDI endpoint’s receiving speed is affecting the number of forwarded sysex messages. Which should be independent of that.
Indeed, the virtual ports in MT Pro are not loopback ports.
So now we can safely assume that sending to the ‘Pro 2’ USB MIDI port is problematic. Because if we take that path out of the equation, everything works.
Could you please post the MT Pro log when you dump the presets without the Pro 2 port? I want to verify the time it takes to receive all messages.
(in MT Pro, you can check the Bome virtual port in the MIDI INPUT list to activate the INPUT port, so that it logs all incoming messages even without a MIDI Route)
dumps.zip (11.8 KB)
Both dumps made without the synth.
One of them with routing between virtual ports, the other made without routing.
The VST uses JUCE library’s standard sendMessageNow function, like many other applications.
Probably the event timers , around the call are too tight, but the MT Pro log must contains every incoming sysex messages in this case also.
Thank you very much! This is very interesting: it just takes ~2 seconds in both cases to receive all 99 messages.
Seems like that. It sends at 53KB/s (standard MIDI-DIN is 3KB/s). USB-MIDI has its own flow control. Few devices can receive MIDI over USB with such high speeds. Now if we assume from the previous discussions that the Pro 2 will receive with a max throughput of 12KB/s, MT Pro encounters the problem that it receives data at 53KB/s pipe and pushes it into a 12KB/s pipe. This is a classic back pressure problem. We cannot tell the sender to slow down, so there will be build up in MT Pro. And given that MT Pro is a real-time software, it is by design that it’ll drop messages eventually (somewhere) exactly to prevent build-up and to keep MIDI flowing. BUT it should tell the user in the Log Window about that. I’ve filed a bug report on that.
Oh that’s excellent, I can just look it up, and quickly build a test program to reproduce your issue.
So now the only idea for what is happening is that the USB MIDI OUT port is flooded so much that somehow the MIDI INPUT thread gets blocked. Normally, this shouldn’t happen, because input is using a dedicated thread. I’ll look into this.
With MT Pro, it is possible to force outgoing actions (and MIDI OUT) to be scheduled using the scheduler in the processing engine (which schedules up to 400 events): just set an arbitrary (small) delay on the outgoing action. But I don’t think that you can define such incoming/outgoing actions for 1178 byte Sys-Ex messages, because there are only ~710 variables available. We plan to allow infinite named variables, array variables and wild cards in version 2, but that does not help us now.
We will also overhaul the entire Bome MIDI engine to make it MIDI 2.0 compatible, and that might be a good opportunity to rework outgoing Sys-Ex.
Thanks for all your help!
From what I’ve seen in the log, it only works if it is OK to buffer up 10 seconds worth of MIDI data. This is OK for a software that only wants to safely get across Sys-Ex messages. But not for a real-time system. That’s why it does not work with MT Pro.
The underlying issue is that the sending VST silently assumes that the MIDI OUT path will buffer up everything. Better would be if it modeled the outgoing MIDI rate so that it matches the receiving end’s data rate to prevent build-up.
If you’re patching the VST anyway, maybe you can change the Sleep() amount, too? To play it really safe, I’d always choose MIDI-DIN speed, which means that you should sleep 377ms after sending each 1178-byte Sys-Ex message.
And a side note: we’re not a company which routinely blames “something else” whenever a problem with our products is reported. We always try to keep an open mind and consider all possibilities when it comes to finding the cause of a reported issue. This forum provides a good record of that.
But when it comes to core functionality, like receiving and sending MIDI, it is just very unlikely that there is a fundamental problem in a product of ours. MT Pro is used by hundreds of thousands of users, and pretty much all of them use it with MIDI, and they do so in many many different ways. That’s why we were hesitant in agreeing that MT Pro is the cause of this problem.
Now I think that we have found the underlying cause of this problem. MT Pro’s behavior is “by design”, but maybe not “as expected”. Plus a “missing log message” bug.
Unfortunately the sleeps what I referred are for error handling in the JUCE code, so I must
build a new one into the dll after the sending of the last chunk of every sysex message.
It works, mission accomplished.
A big thank you for your help and time also ! It really stands out from the attitudes of other companies!
Thanks, that’s great to hear. Thanks to you, too: we’ll consider your use case when overhauling the MIDI subsystem. Maybe make buffer size configurable, or generally use a larger buffer for Sys-Ex messages.
