Editorial illustration; the queue, worker and labels are representative, not a live Drupal administration screen.
A Drupal page request should not wait for every slow operation to finish. Sending a batch of remote notifications, resizing many files, importing records or calling a partner API inside the request can create timeouts and a poor visitor experience. It also makes failure harder to recover: one remote error may leave the whole page uncertain.
Drupal’s Queue API provides a way to record work for later processing. The important design work is not merely “put it in a queue.” You must define a small item, a worker that can retry safely, a trigger such as cron or a runner, and a way to tell whether the work completed.
Summary
- Put slow or retryable work into a queue instead of making the visitor wait.
- Make each item small, idempotent, and safe to retry after a partial failure.
- Monitor queue growth and log permanent failures; a queue without observability only hides the delay.
When is a queue appropriate?
Use a queue when the user does not need the remote operation’s final result before the page can respond. Good candidates include sending a notification, synchronising a record, generating a derivative, processing a large import or calling an unreliable external service.
Do not queue a decision that must be shown immediately. If a checkout cannot complete without a stock response, hiding that response behind a queue changes the business behaviour rather than improving performance.
The intended search question is “how do I use Drupal Queue API for slow tasks?” The useful answer is a decision and operating workflow, not a code fragment copied without a failure model.
The queue workflow
1. Define the work item
Store the smallest information a worker needs, such as an entity ID, operation name and a version or timestamp. Avoid storing a complete entity or a large payload if the worker can load current data safely later.
An item should be understandable when it appears in a log. sync_node:123 is more useful than an opaque blob that nobody can identify during an incident.
2. Add the item after the main action succeeds
Only queue work after the event that justifies it has been saved. If a form submission fails validation, it should not create a notification item. If an entity save and queue insertion must be treated as one business transaction, design that boundary explicitly and test what happens when one side fails.
Avoid adding the same item repeatedly when a hook runs more than once. Use a stable identifier or a deduplication strategy appropriate to the task.
3. Implement a queue worker with a clear contract
A worker should load the item, perform one bounded unit of work, and return only when the item is complete or the failure should be retried. Keep the operation small enough that a single execution does not exceed the time available to the runner.
If the worker calls a remote API, set a timeout and handle non-success responses deliberately. Do not assume a network exception means the remote system did nothing; an operation may have succeeded before the response was lost.
4. Make retries safe
Queue systems may retry work after a crash or temporary error. Design the worker to be idempotent: running it twice should not create two invoices, two notifications or two conflicting updates. Use a remote idempotency key, a local processed marker or a lookup before creating a side effect, depending on the service.
Separate transient failures from permanent ones. A timeout may be retried. Invalid data or a revoked remote account may need a visible failure state and human action.
5. Choose a reliable runner
Drupal queues are commonly processed by cron or another controlled runner. Verify that the runner actually executes in the environment you operate. A queue that grows because cron is disabled is a production failure, even though page requests remain fast.
Set a practical batch limit so one run does not monopolise the server. If work is high volume, use a dedicated process or worker strategy suited to the site’s hosting environment rather than silently increasing a web request timeout.
6. Observe completion and failure
Record the item ID, start time, outcome and useful error context without logging secrets. Monitor queue depth and age. A growing queue indicates that production is generating work faster than it is processing it, or that the worker is failing repeatedly.
Give support staff a way to identify failed items and retry them after fixing the cause. “The cron ran” is not the same as “the queued work completed.”
Example: importing product data
A product import receives 5,000 records from a partner. Processing all records during an admin page request risks a timeout. Instead, validate the upload, store the source reference, and queue one item per bounded group or record according to the required recovery model.
The worker loads a group, validates each record, writes a result, and records failures with enough context to correct them. If the partner API is unavailable, the group can retry. If one product is malformed, it should not erase the outcome of 499 valid products.
The page can show “import accepted; processing is in progress” only if the site also provides a trustworthy status path.
When not to use a queue
Do not use a queue to avoid fixing a slow database query that every visitor needs. Do not queue work merely because a function is inconvenient to debug. A queue adds state, retries, operational monitoring and possible duplicate effects. The delay must be asynchronous enough to justify that complexity.
For site-wide output that varies by context, compare this decision with Drupal cache contexts; caching and queuing solve different problems.
Mistakes to avoid
- Queueing a huge payload that becomes stale before processing.
- Treating every error as retryable forever.
- Calling an external API without a timeout or duplicate protection.
- Assuming cron is healthy because it is configured in Drupal.
- Logging tokens, personal data or full remote responses unnecessarily.
- Showing “complete” to users when only the queue item was created.
My Take
The strongest queue design is intentionally boring. A small item goes in, a bounded worker does one job, failures are visible, retries are safe, and somebody can tell whether the backlog is healthy. That operating model matters more than the queue call itself.
Bottom Line
Use Drupal Queue API when work can finish after the page responds. Define a small item, enqueue it after the triggering action succeeds, process it with a retry-safe worker, run it reliably, and monitor age, depth and failure. If the user needs the result immediately, improve the synchronous path instead.