MIDI IN buffer overflow

Bome Products
1

Hi Steve,
We’ve discussed this issue before, but I have another angle I’d you to weigh in on.
I’m still plagued by occurrence of MIDI IN buffer overflow (MIDI messages lost). It happens sporadically when I’m playing normally, but I can force it by rapidly sweeping a fader on Akai APC Mini through its full range as quickly as I can.

At that point, MIDI Translator becomes unresponsive to any messages from the APC Mini.

I can recover by unplugging the APC Mini USB cable and then reconnecting it. This gives me the message MIDI IN Buffer ok.This can be a bit complicated by the fact that I have so 8 APC Mini’s and need to determine which of the cables to pull, but it does allow me to continue playing once I’ve “fixed” the issue.

Would it be possible to make it so MT automatically ignores (or disconnects from) incoming MIDI messages upon sensing the overflow , and then quickly stops ignoring (reconnects)?

I remember Florian mentioning that he might consider configurable MIDI IN buffer. Not sure if this would solve it but it is a thought. I also saw your reply to another user who had the same problem. You mention not sending Fader messages back to the APC. I’m not sure I’m doing that. I think not, but I’ll check (I presume I’d see it in the MIDI log).

The problem is a nightmare for live performance - obviously not a problem during Covid, but hopefully it will be a problem again soon lol.

One more thing. Could you explain the difference between MIDI IN buffer overflow and ENGINE QUEUE overflow? I occasionally get ENGINE overflow and this error is not so easy to recover from. I have to shut down my computer to recover from this… and that’s really a bummer.
Thanks,
Gabriel

3

Hi Gabriel,

Your questions are in the guts of the code so I reached out to Florian for answers here. I’m hoping the new Beta version of MT Pro might help.

Steve

4

Hi Gabriel,
this is disturbing indeed!
I think there are 2 distinct issues:

1) Hardware
As you found in the forum, we have reason to believe the APC’s can freeze when getting too much MIDI INPUT. It’s likely to cause all sort of problems, including buffer overflows. Once one of the APC’s freezes, can you recover by closing/re-opening from within MT Pro? Simplest to try this is by closing the project file, then reopen it. If this works to recover the APC, the upcoming version of MT Pro will have an outgoing action for you, it’ll close then reopen a given MIDI device.

2) Software, i.e., MT Pro
The other issue is how MT Pro reacts on buffer overflows. Both types (MIDI and engine) are inside MT Pro and will not require a restart of MT Pro or the computer. It is essentially what you’re asking for: once a queue is full, new message to it are ignored until the queue gets drained.

But the reason why the overflow occurs might require a restart. For example, if a MIDI driver freezes when MT Pro sends a MIDI message to it, that thread is frozen in MT Pro and unless that driver releases the thread, MT Pro is a victim of that faulty MIDI driver and it cannot drain the queue because the driver’s freeze.

As said before, the buffers in MT Pro are of reasonable size, and an overflow is usually an indication of a hardware limitation of some sort. For example: MIDI connection to device is not fast enough, or the computer is not fast enough (unlikely), or a MIDI driver/device is blocking.

Are you on Windows or Mac?

5

Thanks Steve and hi Florian,
I’m doing some extensive testing under different conditions to see what happens and how and whether I can recover function without rebooting.
I’ll have more information for you by late tonight.

Is there some log file that might help? (I think you may have suggested that to me once before).
Thanks,
Gabriel

6

I’m on Windows. Do you think I might have better luck on a Mac?

7

I’ve done a lot of testing now, and have a pretty good idea of conditions which cause MIDI Input Buffer Overflow. I’ll write this up now and post it.
In the meantime, I’m interested to know if the following is a possible workaround:

I noticed that looping all APC Mini CC messages back to the APC can cause MIDI Input Buffer Overflow, whereas looping them back as Note on messages causes the Midi port to close, and then after a couple seconds, the port is automatically opened again and functionality returns.

This gave me the idea that the overflow could be avoided by monitoring the midi port to see when it is is close to overflowing. When it’s close, could the port be closed briefly and then reopened? This could be user configurable (as could the size of the buffer). I play real time, but the type of music I play can tolerate quite a long delay in the reaction to fader movements. The temporary dropout wouldn’t be a problem at all, and I presume that it would only happen very occasionally.

In any case, I’ll write up my findings now and post them.
Thanks,
Gabriel

8

Here is the hardware test setup:

2 APC Mini’s —> powered usb hub —> USB Port 1 of my laptop (I7 quad core 2.5GHz)
1 Behringer X-touch Mini —> USB Port 2 of my laptop (I7 quad core 2.5GHz)
(no particular reason for using the hub, or for connecting the X-touch Mini into a separate laptop port.)

MT Code: 4 cases: (I don’t know how to get the MT code to you… how would I attach it?).
CASE 1:

  • APC 1 faders 1, 4, and 7 (this choice is arbitrary) each send CC messages to MT.
  • MT loops the messages back to APC 1 but as note on rather than CC.
  • Fader 1 turns all LEDS on flashing red
  • Fader 4 turns all LEDS on flashing yellow
  • Fader 7 turns all LEDS on flashing green
    So… A LOT of messages are being sent.
    APC 2 fader 2 movement, turns off all LEDS in APC 1

CASE 2:
Same as CASE 1, but CC messages are looped back to APC’s as well as note ons and offs.
This is done to see if the extra CC feedback causes overflow more easily.

CASE 3:
Same as CASE 2, but ONLY CC messages are looped back to APC’s

CASE 4:
Same as CASE 1, but no messages are looped back to APC’s

METHOD USED TO CREATE MIDI Input Buffer Overflow:

  • In all cases, I “scrub” the faders as fast as I can through their entire range (0-127)
  • One scrub is both 0 to 127, and then down from 127 to 0.
  • I do this with one fader, then with two and then with all three faders.

RESULTS:
CASE 1: Loopback Note on messages only.

  • One fader: It takes at least 50 (sometimes 100 scrubs) to get the Overflow message.
  • Two faders: It takes far fewer scrubs to get Overflow (sometimes 10, sometimes 25)
    - I think the variance is caused by uneven scrubbing speed and range
  • Three Faders: It usually takes only 5 to 10 scrubs to get overflow.
    I can always recover by unplugging the APC’s (actually the hub they’re plugged into). When I do this, I get the message "MIDI IN buffer ok, and can continue using the APC.

CASE 2: Loopback Note on’s and CC’s

  • I’m not positive, but I think there’s no difference between this and CASE 1, so CC messages looped back to APC seem not to have an additional effect.

CASE 3: Loopback CC messages only.

  • Again… similar to CASE 1.

CASE 4: No loopback

  • Even moving all three faders rapidly, I can’t create an overflow.

Additional information:

  • After overflow, I can’t restart the project from within MT
  • I can close the project, but then, even though the MT is still shown in Task Manager as a running App,
    the GUI is frozen, I can’t do anything within MT regardless of the fact that I still see the project in the GUI.

I’ve also tested with all 9 faders in each APC enabled and causing loopback as above. In this case,

  • I’ve been able to get two Overflow messages - one immediately after the other.
  • Sometimes this happens after only a few scrubs.
  • Rarely, (And I think only in CASE 3 - only CC messages looped back- MT recovers without any input from me. I get the MIDI IN buffer ok message spontaneously and everything is working.

If there is a workaround for this, I’d prefer if it didn’t close the Project and then restart it. I find that sometimes to continue working, I have to go the the task manager and end the task in the App section, and then restart MT. Not good for playing publicly. There are even cases when after overflow, I can’t end the task in the App section, and I also see it in Background processes. I can’t end the task there either, and have to shut down the computer and reboot.

It’s also confounding that I get overflow sometimes when I have only moved 2 or maybe at most, 3 faders simultaneously. And in these cases, I certainly havent scrubbed them back and forth. I’ve just adjust some parameters pretty steadily and slowly. I wish there were some way to capture a log file of these occurrences because they seem unrelated to the density of the feedback.

I think it’s also worth noting that only the controller that has caused the overflow stops working. The other controllers continue to work after overflow.

MIDI Engine overflow has happened rarely, but I haven’t tried to figure out what causes it yet. I think I remember that I’d get it if I had already caused an overflow and then used a fader from a different controller.

10

I should mention that I never send CC messages to the APC. My tests included doing that just to see what would happen.

12

Hi Gabriel,
thank you very much for the tests and the detailed report. I know, for you it does not matter much what causes the problem as long as it doesn’t happen. But for us here, it is crucial to know what causes the problem.

In any case, as said, an overflow in MT Pro is usually the symptom of a more serious other problem:

  1. almost always a hardware or driver problem
  2. or possibly a feedback loop inside the MT Pro project

In case 2, an overflow will recover gracefully when the feedback loop is stopped.

Case 1, though, causes the associated MIDI thread to freeze. Only then the buffer will start filling up and eventually overflow. So the condition is the same when the buffer starts filling up and when it’s full. If you cannot get out of that situation at 100% buffer, you will not be able to fix it when the buffer is, say, 50% filled. Nor will resizing the buffer help. It will only make it faster or slower until the message appears. The error will always happen before that.

For completeness, it is also theoretically possible that MT Pro somehow has a deadlock internally. Especially when considering that MT Pro is heavily multi-threaded for maximum performance. However, in such an application-level deadlock, you can just kill MT Pro and restart it.

Your observed symptom that you cannot kill the MT Pro process is a strong indication that there is a freeze happening in the Windows kernel or in the APC (or standard Windows) MIDI driver. Searching for “APC Mini MIDI crash” will give you many other discussions about this problem. It seems to happen on macOS and Linux, too. I don’t know if AKAI has released a fix for this. At this point, I don’t think that MT Pro can overcome this hardware problem.

The way I read this, however, the freeze seems to be related to sending MIDI too fast to the APC. Can you create a copy of your main project and make sure to disable ALL MIDI to the APC Output ports?

  • remove all MIDI routes that have a destination of an APC
  • go through Project Default MIDI OUTPUT ports, Preset Default MIDI OUTPUT ports, and all MIDI Outgoing actions if they check an APC as MIDI OUTPUT port.
  • then all APC output ports should have a Port State “closed”

Not sure if the APC’s are still usable without visual feedback though…

I hope that helps.

13

In my ddj-1000 there is major problems if I send too many messages in the same millisecond

The easy workaround is to delay the messages with increasing delays. Use the delay option on midi out with a local variable set in the code for more convenience

14

Thanks Pedro!

Indeed, I have instructed him to use delays and even suppress messages if they come to fast. With his controller it is not just an inconvenience of bad messages. I have verified, that sending CC messages to fast to the APC-MINI will actually crash the controller requiring detaching and re-attaching the USB connection. He spoke with AkaiPro about this but since it doesn’t happen while running Ableton Live, they will not help him stating that the controller design is for Ableton Live. Too bad because other than that, they are pretty nice little Units. I actually recommend not sending CC message to it at all since there are not position indicators or motorized faders. It doesn’t make sense to send a CC, however it apparently also crashes if you flood it with note messages too fast. Gets really interesting when you have 8 of these controllers and you want to update 64 LEDs on all 8 of them (512 leds) that he wants refreshed rather quickly.

15

Hi Gabriel and Pedro, and everyone who stumbles on this topic with a similar problem.

If you can reproduce a way to make MT Pro unresponsive in some way (possibly after a buffer overflow), it would be great to send us a memory dump.

I’ve added a post with instructions how to get a memory dump for a frozen program here:

Thank you very much!