Job priority

First goal

Being able to end workflows (multiple steps processes) before starting new ones.
This key function intends to limit the needs for storage but allow efficient parallel processing according to the user’s needs.

Technical description

Each step ID is unique and initially ordered and started by 0.
As the need it to increase the priority while the workflow progress, the first idea it to match this ID to the priority.
In RabbitMQ, the maximum priority is 255, but the maximum level recommended is 10 as more priority levels are defines, more the RabbitMQ server load increases.
For the use case, queues are defined with a maximum level of 100.


For Step Flow, the priority is taken directly from the step ID - clamped to 100. Each worker consumer creates the queue with the max priority header equal to 100. RabbitMQ orders jobs to consume first higher priority jobs.