Are Your Queue Workers ... Working?

I love dispatching jobs to queues. Jobs are simple, encapsulated chunks of code that do just one thing. This encourages me to write easily tested code, allowing me to forget about the breadth of the user experience for a moment and focus on simply testing a job's output based on given inputs. Jobs also help me keep my controller code concise, and my application's response times short.

In this post, I'm going to show you a simple and lightweight strategy to notify you when a queue isn't processing jobs. I'll give you the code, and talk about some ways I would consider extending it.

The Catch to Queues

There's just one catch to using jobs: if the queue worker isn't consuming jobs, my application is broken. A job sitting in a queue waiting to be processed doesn't make noise or throw errors, so it's very possible for me to not notice.

There are many reasons that a queue might stop, slow, or develop unbearable wait times. These can be part of the natural growing process of an application (e.g. when your application gets a bunch more traffic and you don't have enough threads on a given queue worker). It could be because something is broken with your application and it is dispatching too many jobs. Your server, Supervisor, or queue worker might be broken.

No matter what made your queue's processing pace unacceptable, it's critical that you know you'll be informed when a job sent to a queue can't be expected to be processed in a reasonable time.

Aren't Laravel Queues Reliable?

Yes! They are incredibly reliable!

After minimal configuration, queue workers fall into that amazing category of technology that just works. More so if they are configured and managed by a service such as Laravel Forge. So, for many programmers, queue workers can feel like mysterious and powerful elemental creatures whose inner workings and behaviors are only known to a select few.

But sometimes, as I mentioned above, your queues will stop functioning at the level you want them to. So that's what we're talking about here--not how to fix them, or discover why they broke, but simply how to know they're broken in the first place.

What Do I Want, Actually?

I want to know that my queue workers are processing any jobs they're given. If they slow significantly or stop, I want to receive an alert.

Queue Heartbeats

We'll be using some out-of-the-box Laravel tools and a "heartbeat" service; in this post, we'll use Envoyer's heartbeat, but many other services also offer them.

First, let's create a job with Artisan:

php artisan make:job QueueHeartbeat

This job exists for no reason other than to check that the queue works. That's it!

Here's the code we need, then, to check that:

// app/Jobs/QueueHeartbeat.php
 
namespace App\Jobs;
 
use GuzzleHttp\Client as HttpClient;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
 
class QueueHeartbeat implements ShouldQueue
{
    use Dispatchable, Queueable;
 
    protected $heartbeatUri;
 
    public function __construct($heartbeatUri)
    {
        $this->heartbeatUri = $heartbeatUri;
    }
 
    public function handle(HttpClient $http)
    {
        $http->request('GET', $this->heartbeatUri);
    }
}

In Envoyer, we're going to create a heartbeat for each of the queues you're running on your server.

Creating Your Heartbeats

Open your project on Envoyer, click the Heartbeats tab, and click Add Heartbeat. Choose an Alert After timeframe that makes sense for you; I recommend 10 minutes to make debugging less time-consuming.

Now let's update your application's scheduler. We want to dispatch a QueueHeartbeat job on the default queue every five minutes on production:

// app\Console\Kernel.php
 
use App\Jobs\QueueHeartbeat;
 
$schedule->call(function () {
    QueueHeartbeat::dispatch('http://beats.envoyer.io/heartbeat/[heartbeat-id]')
        ->onQueue('default');
})
    ->everyFiveMinutes()
    ->environments('production');

Note: With the above configurations, your heartbeats should be sent every five minutes, and your Envoyer Heartbeat is set to alert you after 10 minutes of silence. Thus, if you get a ping from Envoyer, you know at least two QueueHeartbeat jobs have not been processed.

Once this code deploys to production, you have a queue worker set up in Forge processing that queue, and you check to make sure your cron job is triggering the php artisan schedule:run command, it's time to check that the Envoyer heartbeat is being pinged correctly.

Open your Envoyer Heartbeats tab and find the row of the Queue Heartbeat you just created. Every heartbeat starts with a healthy status, so take note of the Last Check-In time for that heartbeat. Wait about five minutes, and refresh the page. If the time is updated, you're good to go!

If not, you'll begin to see the "Missing In Action" warning in 10-20 minutes:

This heartbeat will return to a green, healthy, status whenever those jobs start processing, so if it's Missing in Action, something needs to be remedied in your job, schedule.php, queue workers, or Supervisor.

Adding Notifications

Once your heartbeat is healthy, you can also configure Envoyer to send you notifications when your queue stops processing QueueHeartbeat jobs in a reasonable timeframe. To do this, open the Notifications tab, pick your preferred notification channel from the list, and fill out the connection information.

Multiple Queues?

If you're running more than just the default queue, create an Envoyer Heartbeat for each queue name, and add an additional line inside the same $schedule->call() closure.

// app\Console\Kernel.php
 
use App\Jobs\QueueHeartbeat;
 
$schedule->call(function () {
    QueueHeartbeat::dispatch('http://beats.envoyer.io/heartbeat/[heartbeat-id]')
        ->onQueue('default');
    QueueHeartbeat::dispatch('http://beats.envoyer.io/heartbeat/[other-heartbeat-id]')
        ->onQueue('otherQueueName');
})
    ->everyFiveMinutes()
    ->environments('production');

What about thenPing()?

You may be wondering why you'd do this when you could just use Laravel's thenPing functionality to ping your Heartbeat service after each job is dispatched.

The key is in that last phrase: "after each job is dispatched." thenPing doesn't actually ensure that the jobs were completed; it simply ensures that they were dispatched onto the queue without an exception being thrown.

So use thenPing for either synchronous actions or to ensure a job was dispatched, and use a tool like I'm describing here to ensure your queue is running.

Extending Functionality

If the behaviors of the above code don't suit your needs, you might consider extending its function.

You could update the code to use a health service other than Envoyer Heartbeats. Other services offer additional configurations that might match your notification preferences better (i.e. texting your phone when a queue is non-responsive for 30 minutes).

You could build a database table that persists data about each QueueHeartbeat job, such as when you dispatched it, and when it was processed. With this data, you can track the longterm health of your queues, as well as how long job waits are at different times of the day.

Sometimes server loads from other cron jobs, or user traffic can cause slowdowns on queues that you'd prefer to stay snappy. If you need very prompt processing times for a given queue, consider adjusting the sleep parameter in your worker configuration. Consider optimizing your job code to decrease processing time, adding processes for a queue, upgrading your server specs, or splitting out a separate server for running your queues!

Resist the Urge to Extend Too Much

With very simple blocks of code like the one above, I'll often opt out of the additional package. Constraining the number of packages my application uses reduces the mental overhead of the application, as well as the amount of work required for upgrades.

If you know you want a lot of data on your workers, there are very excellent packages that monitor and provide queue data. They may provide similar or different or better functionality with a fraction of the custom programming cost.

Laravel's Horizon is one excellent, and free, option. It has a similar notifications structure to what we've talked about here.

Set your Queue Anxiety Aside

For those of you who have ever been unpleasantly surprised by a silent but dead queue worker, I hope implementing something like this can help you relax.

Let me know via email if you come up with any other helpful concepts to extend this base code, and I'll consider adding the recommendation to this post!