Can "timer zero" events be injected BEFORE any already buffered messages?

Bome Products
1

ISSUE:
I'm using "timer zero" events to a) merge multiple triggers; b) trigger multiple actions and c) modularize my code (with global variables). This works fine for messages that arrive slowly.

But this fails when messages arrive at the same time quantum. In that case, the timer zero events are processed AFTER the buffered messages (demo below).

QUESTION:

Is there a way to make timer-zero events be injected in the engine BEFORE any buffered messages, but AFTER any buffered timer-zero events that might be already there?

 

MANUAL page 43:

A Timer will, when its time has elapsed, inject an Incoming Event into the processing engine.

A trick is to use a one-shot timer with 0 delay: this will cause the current input event to be fully processed, and the timer event will be processed immediately, too – possibly already in parallel to the current event.

 

DEMO:

Please see the annexed project. In the log below, first 2x messages arrive slowly; in case 2, they arrive back to back, and "gb" becomes 108 instead of 107

 

 

Case 1: messages arrive slowly

742241 - MIDI IN [DDJ-SZ]: 91 1E 7F
742241 - IN 0.1 Note On on any channel=2 set 'qq' to ch. with any note and 'oo' to note=30 with any velocity and 'pp' to velocity=127
742241 - RULE 0.1:1 assignment: (gb=oo) = 30
742241 - OUT 0.1 One-shot timer "process_it": 0 ms delay
742241 - IN 0.4 On timer "process_it"
742241 - RULE 0.4:1 expression: (gb=gb+1) = 31 <<<<< OK!

 

747162 - MIDI IN [DDJ-SZ]: 91 1E 00

747162 - IN 0.2 Note Off on any channel=2 set 'qq' to ch. with any note and 'oo' to note=30 with any velocity and 'pp' to velocity=0
747162 - RULE 0.2:1 assignment: (gb=oo) = 30
747162 - OUT 0.2 One-shot timer "process_it": 0 ms delay
747162 - IN 0.4 On timer "process_it"
747162 - RULE 0.4:1 expression: (gb=gb+1) = 31 <<<<< OK!

 

CASE 2: messages arrive back to back

758921 - MIDI IN [DDJ-SZ]: 91 6A 7F
758921 - IN 0.1 Note On on any channel=2 set 'qq' to ch. with any note and 'oo' to note=106 with any velocity and 'pp' to velocity=127
758921 - RULE 0.1:1 assignment: (gb=oo) = 106

758921 - MIDI IN [DDJ-SZ]: 91 6A 00
758921 - IN 0.2 Note Off on any channel=2 set 'qq' to ch. with any note and 'oo' to note=106 with any velocity and 'pp' to velocity=0
758921 - OUT 0.1 One-shot timer "process_it": 0 ms delay
758921 - RULE 0.2:1 assignment: (gb=oo) = 106

758921 - IN 0.4 On timer "process_it"
758921 - RULE 0.4:1 expression: (gb=gb+1) = 107 <<<<<<<< OK!
758921 - OUT 0.2 One-shot timer "process_it": 0 ms delay
758921 - IN 0.4 On timer "process_it"
758921 - RULE 0.4:1 expression: (gb=gb+1) = 108 <<<<<<<< BAD: This should be 107!

 


Attachments:
1583918923930_example-back-to-back-prcessing.bmtp
2

Hi, in your example you would need 2 timers and a mechanism to "queue up" the next message for later processing.  When triggering the first timer, you would need to set a global variable to indicate "timer1 busy".  On the second incoming action, you would need to  immeediately store the incoming action for later processing but with no output.  When your first timer finishes, you would set the timer busy variable back to 0 and then trip a 2nd timer to release the action you wanted with the second event. Message "queuing" is not really pretty in MT Pro so if you describe in more detail an example of what you are trying to accomplish, maybe I can assist further. 

 

Steve Caldwell
Bome Q and A Moderator and
Independent Bome Consultant/Specialist
bome@sniz.biz

3

thanks for the answer.
Indeed I’m trying to avoid building my own internal message passing system with my own stack, my own event handler loop, etc :slight_smile:

Instead I’m requesting a specific change to the BOME event handler: that timer-zero events always get priority, instead of being put in the end of the queue.

(regular Timers and messages would be treated as of today.)

About my use case: I’m have a complex map for the DDJ-1000:
https://github.com/pestrela/music_scripts/blob/master/traktor/mapping_ddj_1000/DDJ-1000%20v6.7.0%20-%20BOME-side%20Mapping.bmtp

and I’ve now extended it to ADDITIONALLY translate other Pioneer devices, using the time-zero methos you suggested here

4

is there a way to keep basic formatting in the comments?

5

We are looking into other forum solution alternatives.

6

I have the same issue when receiving a midi dump from Cubase. Issue - a Timer can capture all input but the actioned translators don't process quick enough to keep up with messages arriving at the Timer.
Alternatives to writing your own queue manager are
1. Make your Timer more selective to reduce the number of times it 'fires'. Eg instead of monitoring for all note on messages, focus on the range of note on messages you actually want to process. If you still have the issue then 2.
2. Make the Timer store its input into global variables and have a timer related translator process those variables rather than the actual midi messages. A point to note is that although that translator may not always fire at the same time as the Timer (dropped messages), the last message in a fast stream arriving at the Timer will always be caught by the called translator so the last times the translator performs its actions it will be working on an up to date set of variables.
An example: In Cubase if a mixer solo button is actioned, Cubase will fire off a rapid stream of mixer mute button on messages as it automatically mutes all other mixer channels. If you have a Timer checking for note on messages with a called translator processing those messages, some will be dropped because the translator can't keep up. The solution is for the Timer to set a number of global variables, one for each mute button message received. Every time the translator is called it will examine all those mute variables and perform the required actions. It doesn't matter if it 'misses' a few Timer events as it's processing the variables not the input messages. When the Timer stops receiving the 'fast stream', the last message received will always trigger the underlying translator so no input has been lost.

Hope that helps.

7

Great comments @PAC.

One should also realize the translators themselves are extremely fast and can sometime work in parallel. Usually missing incoming MIDI messages is more of an issue of too many rules to process for the given incoming MIDI message. Minimize the number of rules. Maybe just set a global variable as you suggested. And do the heavy lifting later in timers.
I usually use a bitmap of note “states” to reduce the number of global variables needed. It is more complex set up but you can use only 1 global variable for every 32 on/off states that you want to track since Bome MIDI Translator Pro used 32 bit signed integers.

Steve Caldwell
Bome Q and A Moderator and
Independent Bome Consultant/Specialist
bome@sniz.biz

8

thanks @PAC. When ONLY the last state is relevant I’m already doing this. For an example see line 547 of https://github.com/pestrela/music/blob/master/traktor/mapping_ddj_1000/Support%20files/Technical%20Info%20-%20BOME%20DDJ%201000%20Screens.txt ; code is 12.43 to 12.57 of https://github.com/pestrela/music/blob/master/traktor/mapping_ddj_1000/DDJ-1000%20v6.8.0%20-%20BOME-side%20Mapping.bmtp

9

@Steve No doubt that with high number of rules may lead to missing messages; but the original example of this thread is something else (only 3 trivial rules)

10

Just to chime in again 6 weeks later. 

Right now I'm duplicating massive amounts of code in parallel translators to make each translator run as as a single action.
Removing timers was the only way to preserve atomicity; this damaged modularity a lot.

I'm still convinced my requested "cirurgical" OPTION would solve my probelms.

11

Hi Pedro,

What version of MT Pro are you running? I'm testing with 1.8.4 built 962 and cannot seem to duplication your issue.

See log below:

754450 - MIDI IN [Bome MIDI Translator 1 Virtual In]: 91 6A 7F
754450 - IN 0.1 Note On on any channel=2 set 'qq' to ch. with any note and 'oo' to note=106 with any velocity and 'pp' to velocity=127
754450 - RULE 0.1 assignment: (gb=oo) = 106
754451 - OUT 0.1 One-shot timer "process_it": 0 ms delay
754451 - MIDI IN [Bome MIDI Translator 1 Virtual In]: 91 6A 00
754451 - IN 0.4 On timer "process_it"
754451 - RULE 0.4 expression: (gb=gb+1) = 107
754451 - IN 0.2 Note Off on any channel=2 set 'qq' to ch. with any note and 'oo' to note=106 with any velocity and 'pp' to velocity=0
754451 - RULE 0.2 assignment: (gb=oo) = 106
754451 - OUT 0.2 One-shot timer "process_it": 0 ms delay
754451 - IN 0.4 On timer "process_it"
754451 - RULE 0.4 expression: (gb=gb+1) = 107

 

Your project file looks good, however be careful not to use the second delay value as if you do, the second delay will always fire

 

Steve Caldwell
Bome Q and A Moderator and
Independent Bome Consultant/Specialist
bome@sniz.biz

 


Attachments:
![](upload://xBsYI3mWrSkMsi7atwr6r2417aS.png)
12 
yes I'm using 1.8.4 built 962. I have no delayed messages, only timer=zero actions.

__QUESTION__: Did your messages came from the hardware simultaneoulsy? 
Because thats what my DDJ generates - two messages back to back.

I've reduced the log to the bare mininum.
It shows that the second message is processed BEFORE the timer triggered by the first message.

Pedro
  758921– MIDI IN [DDJ-SZ]: 91 6A 7F
  758921 – MIDI IN [DDJ-SZ]: 91 6A 00
  758921 – OUT 0.1 One-shot timer “process_it”: 0 ms delay

Steve:
  754450 - MIDI IN [Bome MIDI Translator 1 Virtual In]: 91 6A 7F
  754451 - OUT 0.1 One-shot timer "process_it": 0 ms delay
  754451 - MIDI IN [Bome MIDI Translator 1 Virtual In]: 91 6A 00

thanks,
13

pedro-estrela says:

yes I\'m using 1.8.4 built 962. I have no delayed messages, only timer=zero actions.

__QUESTION__: Did your messages came from the hardware simultaneoulsy?
Because thats what my DDJ generates - two messages back to back.

I\'ve reduced the log to the bare mininum.
It shows that the second message is processed BEFORE the timer triggered by the first message.

Pedro
758921– MIDI IN [DDJ-SZ]: 91 6A 7F
758921 – MIDI IN [DDJ-SZ]: 91 6A 00
758921 – OUT 0.1 One-shot timer “process_it”: 0 ms delay

Steve:
754450 - MIDI IN [Bome MIDI Translator 1 Virtual In]: 91 6A 7F
754451 - OUT 0.1 One-shot timer \"process_it\": 0 ms delay
754451 - MIDI IN [Bome MIDI Translator 1 Virtual In]: 91 6A 00

 

Interesting. Maybe the log messages themselves are delayed (not real time). Since MIDI data is serial, the only way to send two MIDI messages simultaneously would be if they came through two separate interfaces. I'm speculating that you have one thread processing the input messages and another thread processing the log messages and the incoming messages have higher priority than the log messages.

 

I used Bome SendSX to send the two messages I sent. (no delays).

In either case, I'm not sure what you are trying to solve.

According to the documentation, when a timer trips in succession before the timer has a chance to trigger, the second trigger will override the first. So I think a zero timer might have some chance of not completing it's processing before the second timer triggers. If I can reliably produce an error, I'm happy to report it as a bug. I think. At least I can have Florian look at it to see if there might be something wrong in the timing logic.

 

Steve Caldwell
Bome Q and A Moderator and
Independent Bome Consultant/Specialist
bome@sniz.biz

14

Hi,
I try to shed some light on these timers.

  • When you start a timer ("set" a timer), it will always retrigger that timer. So if a timer with that name is already pending, it is killed first.
  • One-Shot, 0ms timers are optimized, but are still enqueued asynchronously and executed concurrently to other processing
  • As Steve said, two MIDI simultaneous messages from one MIDI port are really coming one after another, even if they appear to arrive at once.
  • Now if two "simultaneous" MIDI messages trigger the same one-shot, 0ms timer, then the behavior is not defined.Either one of these two happens:
    1. The first timer gets enqueued and executes before the second MIDI message is processed. Then the second message will trigger the timer again. The timer gets executed twice.
    2. The first timer gets enqueued, then the second MIDI message is processed. Because the timer has not yet executed, it is first killed (dequeued), and then the timer for the second MIDI message is enqueued (which is the same). Eventually, the timer is only executed once.
  • In many other use cases, the retriggering behavior of timers is very useful. However, here, it leads to inconsistent results here (depending on processor speed, number of processor cores, overall system load, etc.).

That explains why in your example, the timer is sometimes executed twice, sometimes once.

Maybe there is a way to restructure your project file so that it does the per-message processing in the translator directly, and then trigger the timer?

I also think that a new feature in MT Pro would be useful: Call a Timer. It would be like a one-shot, 0ms timer, but without the retrigger behavior: every invocation would trigger exactly once.

I hope this helps some... at least to explain what's happening.

15

This trivial project counts 7-bit pending messages. Each message calls a zero-timer that resets the counter.

To see the race condition, just move a 14-bit high resolution fader/knob, and then searchin the log when "ga" becomes "2"

This is the crucial part of the log, see the text file for the whole log:

228461 - RULE 1.2:1 assignment: (ga=0) = 0
228465 - RULE 1.1:1 expression: (ga=ga+1) = 1
228465 - RULE 1.1:1 expression: (ga=ga+1) = 2 <<<<<<<<<<<< this is my problem
228465 - RULE 1.2:1 assignment: (ga=0) = 0
228465 - RULE 1.2:1 assignment: (ga=0) = 0

I would happily accept any lower-performance option that would remove this issue. Thank you.

 


Attachments:
1588679414357_zero-shot-timers-race-condition-14-bit-mesages-processed-as-7-bit.bmtp
1588679414418_zero-shot-timers-race-condition-14-bit-mesages-processed-as-7-bit.txt
16

Florian/Steve
I’ve now made a NEW trivial example for you to recreate the issue yourself.
Just inject 14-bit messages from a fader in this new project.

“Maybe there is a way to restructure your project file so that it does the per-message processing in the translator directly, and then trigger the timer?” >>>> I did that and it damaged the modularity a lot.

17

I’ve marked it as “best answer” so that you can finf it in this thread.

18

Thanks, Pedro, for the additional explanation and demo.

19

To emphasize again: while this behavior leads to a race condition in this project, and to unwanted results, it is not a bug in MIDI Translator Pro. It is by design that MT Pro is parallelizing processing where possible to achieve the best real time performance. Very tight timing, as in your example, inevitably leads to race conditions.

20

To analyze what’s happening: in most cases, the timer is executed directly after processing the incoming MIDI message. In the case you flagged as problem, one execution of the timer is omitted, because of the “retrigger” behavior, discussed in my answer below. Therefore, ga is increased one more time, before being reset in the timer.