[006.2] Sidekiq Enterprise: Unique Jobs

Using Sidekiq Enterprise to save you from duplicate jobs.

Subscribe now

Sidekiq Enterprise: Unique Jobs [08.19.2016]

You might have certain jobs that you want to ensure aren't duplicated, but perhaps various code paths could create them and verifying that you didn't duplicate them isn't something you wanted to add to your application code. Sidekiq Enterprise offers a solution, in the form of the Unique Jobs feature. Let's get started.

Project

We'll use the sidekiq_batches project, tagged with before_episode_006.2.

Let's provide some context for why this might be useful.

Example Context

We have a basic controller demonstrating our problem. It represents a controller action that takes a brief amount of time to complete, and enqueues a job:

class PutsController < ApplicationController
  def new
  end

  def create
    sleep 1
    PutsWorker.perform_async(params[:message])
    redirect_to root_path
  end
end

Our PutsWorker just takes the message, outputs it to the console, and waits a bit before considering itself complete.

class PutsWorker
  include Sidekiq::Worker

  def perform(message)
    puts message
    sleep 10
  end
end

If we open this up in the browser, without running sidekiq, we can see that a user that clicks on this button multiple times ends up enqueueing multiple copies of their job.

((( demonstrate this by clicking around and showing /sidekiq/queues/default )))

If this job instead charged the user, or sent an email, or something else that we didn't want to duplicate, this could range from minor problem to rage-inducing event.

You could solve this with javascript by disabling the button on click...but that won't actually save you because some people won't have javascript enabled.

Solving it with Unique Jobs

So that's the problem we have, how can we solve it with Sidekiq Enterprise's Unique Jobs?

Let's start off by enabling the feature in our initializer:

vim config/initializers/sidekiq.rb
Sidekiq::Enterprise.unique! unless Rails.env.test?
# You don't want this in tests, it will just cause you confusion.
# We also don't want the periodic jobs any more for this example.
# Sidekiq.configure_server do |config|
#   config.periodic do |mgr|
#     # see any crontab reference for the first argument
#     # e.g. http://www.adminschoice.com/crontab-quick-reference
#     mgr.register('* * * * *', PutsWorker, retry: 2, queue: 'default')
#     # Unsurprisingly, default is the default queue...but you can specify the
#     # queue in your options.
#     #
#     # In general, this is the following:
#     # mgr.register(cron_expression, worker_class, job_options={})
#   end
# end

Uniqueness is based on a combination of klass, args, and queue so you can create duplicates in different queues if that has value to you. Now let's make our PutsWorker unique:

vim app/workers/puts_worker.rb
class PutsWorker
  include Sidekiq::Worker
  sidekiq_options unique_for: 10.seconds # <--

  def perform(message)
    puts message
    sleep 10 # this bit is important because if a job is complete, it won't
             # block a duplicate being created.  We just make it take a little
             # bit of time.
  end
end

This tells sidekiq to not duplicate this job with the same arguments, in the same queue, for 10 seconds as long as an existing job hasn't been successfully processed. If your job is still working or yet to be worked when that time runs out, another job can be added. For this reason, you should write your unique workers under the understanding that they are provided best-effort uniqueness, but not 100% guarantees; they might have duplicates, but this way they won't have duplicates most of the time, if your worker completes these jobs in a reasonable amount of time and you've tuned the unique_for option appropriately.

10 seconds is probably a reallllly dumb uniqueness time but it makes sense since I'm going to be showing you the feature live, so let's see it.

((( run the same demonstration from the browser, show that it didn't make duplicate jobs )))

If we wait 10 seconds, we can create a duplicate per our configuration.

((( show that )))

Details

So in conjunction with scheduling jobs, the uniqueness period will be added to the scheduled difference. If you schedule a job to run in an hour, the uniqueness time for that job with the same arguments will now be one hour and 10 seconds in this case.

Bypassing uniqueness

If you know you want to run a job multiple times against the configured uniqueness, you can bypass it by disabling that sidekiq option before performing the job:

<%= form_tag "/puts" do |f| %>
  <div>
    <%= label_tag :message, "Message" %>
    <%= text_field_tag(:message) %>
  </div>
  <div>
    <%= label_tag :allow_dupes, "Allow Dupes" %>
    <%= check_box_tag(:allow_dupes) %>
  </div>
  <%= submit_tag "Schedule work" %>
<% end %>
class PutsController < ApplicationController
  def new
  end

  def create
    sleep 1
    worker =
      if params[:allow_dupes] == "1"
        PutsWorker.set(unique_for: false)
      else
        PutsWorker
      end

    worker.perform_async(params[:message])
    redirect_to root_path
  end
end

((( demonstrate that we can create dupes with the checkbox checked and we cannot with it unchecked - not that you want to do this from a UI, but it shows the feature easily )))

So that's it!

Summary

Today we learned how to use Sidekiq Enterprise's Unique Jobs feature to ensure jobs don't get enqueued more frequently than they need to be, as well as how to override the uniqueness requirement when we need to. I hope you enjoyed it. See you soon!

Resources