Skip to main content

90. BullMQ Flows for Tournament Processing

Status: Accepted Date: 2025-07-06

Context​

A market tournament in the Dike module is not a single task, but a complex, multi-step workflow. A tournament consists of multiple rounds. Each round consists of multiple independent market comparison jobs. The next round can only begin after all the comparisons in the current round are complete. Orchestrating this logic manually (e.g., by having a parent job poll for the status of child jobs) is complex, inefficient, and error-prone.

Decision​

We will use BullMQ's "Flows" feature to orchestrate the entire tournament lifecycle. A Flow is a tree-like structure of jobs where jobs can have dependencies on other jobs.

The process for running a tournament will be:

  1. When a tournament is created, a "parent" flow producer will be initiated.
  2. It will create a set of "child" jobs, one for each pairwise market comparison required in the first round.
  3. It will then create a "next round" job that is configured to have all the comparison jobs of the current round as its dependencies.
  4. This "next round" job will only execute after all its children (the comparisons) have successfully completed. Its logic will then evaluate the results and create a new flow for the subsequent round.

This creates a declarative, robust, and scalable workflow directly within the queueing system.

Consequences​

Positive:

  • Declarative & Simple: The complex workflow logic is defined declaratively as a dependency tree of jobs. This is much simpler and easier to reason about than imperative orchestration code.
  • Robust & Reliable: BullMQ handles the state management of the workflow. If a worker crashes, the flow state is preserved in Redis, and processing will resume correctly when the worker restarts.
  • Scalability: The individual comparison jobs are independent and can be processed in parallel by multiple workers, allowing the tournament to scale horizontally.
  • Excellent Visibility: The bull-board UI provides visualization of these flows, making it easy to see the status of a tournament, which jobs are running, and where any bottlenecks or failures are occurring.

Negative:

  • Increased Conceptual Complexity: Flows are a more advanced feature of BullMQ than simple queues. Developers need to understand the concept of job dependencies and parent-child relationships.
  • Debugging Can Be Tricky: Debugging a complex, failed flow can sometimes be more difficult than debugging a linear piece of code, as the state is distributed across multiple jobs in Redis.

Mitigation:

  • Clear Documentation and Examples: We will create clear internal documentation and example implementations of tournament flows to ensure developers understand the pattern.
  • Idempotent Jobs: All jobs within the flow will be designed to be idempotent (safe to re-run). This means that if we need to manually re-run a failed part of a flow, it won't have unintended side effects.
  • Granular Logging: Each job in the flow will have detailed, structured logging with a shared tournamentId and flowId, allowing us to easily trace the execution of a single tournament across multiple jobs.