Skip to main content

49. Queue Consumer Pattern for Admin Tasks

Status: Accepted Date: 2025-07-06

Context​

Having decided to use a dedicated BullMQ queue for administrative tasks (adr://bullmq-admin-queue), we need to define the architectural pattern for how these tasks will be processed. We need a pattern that is reliable, scalable, and decouples the task-processing logic from the part of the application that enqueues the tasks.

Decision​

We will implement the Queue Consumer Pattern (also known as the Worker Pattern) for processing jobs from the admin-jobs queue.

This pattern consists of:

  1. Producers: Components (e.g., an AdminService) that create jobs and add them to the admin-jobs queue. The producer's responsibility ends once the job is successfully enqueued.
  2. Queue: The BullMQ admin-jobs queue, which acts as a buffer and communication channel.
  3. Consumers (Workers): A pool of one or more separate, long-running processes whose sole responsibility is to listen to the admin-jobs queue. When a new job appears, a consumer will pull it from the queue, execute the necessary business logic based on the job's type and payload, and then report the result (e.g., success or failure).

These consumer processes will be managed as separate, scalable units within our containerized infrastructure.

Consequences​

Positive:

  • Decoupling & Single Responsibility: The logic for producing a task is completely decoupled from the logic for consuming and executing it. Each component has a clear, single responsibility.
  • Scalability: We can easily scale the processing throughput by increasing the number of consumer processes (workers) listening to the queue, without changing any other part of the system.
  • Resilience: If a consumer process crashes while processing a job, BullMQ ensures that the job is not lost and will be picked up by another available consumer, making the system resilient to transient failures.
  • Asynchronous Processing: This pattern is inherently asynchronous, allowing the system to handle long-running administrative tasks without blocking web requests or other critical flows.

Negative:

  • Increased Operational Complexity: We need to deploy, monitor, and manage these consumer processes in addition to our main application servers.
  • Eventual Consistency: The result of an administrative task is not immediate. The producer enqueues the job and moves on. The system is eventually consistent, meaning there is a delay before the task is completed. The UI or client needs to handle this (e.g., by polling for a result or using WebSockets).

Mitigation:

  • Infrastructure as Code: The deployment and management of consumer processes will be defined in our Ansible and Docker configurations, making it automated and repeatable.
  • Monitoring: We will use BullMQ's built-in monitoring tools and our Grafana dashboard to monitor the health of the queues and consumers, including queue length and job failures.
  • Clear API Contracts for Asynchronicity: The APIs used to trigger admin tasks will be designed to reflect their asynchronous nature. For example, instead of returning the result directly, they will return a job ID that can be used to query the status of the task later.