[005.4] Sidekiq Enterprise: Periodic Jobs

Scheduling jobs to run periodically; in other words, "cron for Sidekiq workers".

Subscribe now

Sidekiq Enterprise: Periodic Jobs [08.19.2016]

In today's episode we're going to begin looking at features provided by Sidekiq Enterprise. The first feature we're going to cover is Periodic Jobs.

You can think of periodic jobs like cron or recurring jobs. You can register jobs with a schedule on startup, and Sidekiq will guarantee that they get started at the appropriate time and are not duplicated. Let's get started.

Project

We'll use the sidekiq_batches project we've been using. It's tagged with before_episode_005.2. First, we'll enable Sidekiq Enterprise:

vim Gemfile
source ENV["SIDEKIQ_SOURCE_URL"] do
  gem 'sidekiq-ent'
end

And we'll install it with bundle.

Definition

Now we can define our first periodic job. First, we'll make a worker that just outputs with puts:

vim app/workers/puts_worker.rb
class PutsWorker
  include Sidekiq::Worker

  def perform
    puts "zomg"
  end
end

Then we'll configure the periodic jobs in sidekiq's initializer:

vim config/initializers/sidekiq.rb
Sidekiq.configure_server do |config|
  config.periodic do |mgr|
    # see any crontab reference for the first argument
    # e.g. http://www.adminschoice.com/crontab-quick-reference
    # We'll create a job that runs the putsworker every minute
    mgr.register('* * * * *', PutsWorker, retry: 2, queue: 'default')
    # Unsurprisingly, default is the default queue...but you can specify the
    # queue in your options.  If you didn't need options, you don't have to
    # provide them.
    # mgr.register('* * * * *', PutsWorker)
    #
    # In general, this is the following:
    # mgr.register(cron_expression, worker_class, job_options={})
  end
end

Now let's start sidekiq and watch it work:

sidekiq

Time Zones

OK, so while that's running let's talk about time zones. The times in your cron statements will be based on the system time zone. If you want to specify a different timezone, you can pass it as the TZ environment variable, for instance:

TZ=America/Los_Angeles bundle exec sidekiq

OK, so we'll wait to see that job get printed out. There we go, so you can see the periodic jobs working.

Dynamic Jobs

If you want to provide dynamic jobs to your users or via your database, the wiki lays out a pretty good starting point that uses periodic jobs as a primitive to build upon.

rails g migration create_dynamic_jobs
create_table :dynamic_jobs do |t|
  # these are totally optional, might want to track them for multi-tenancy or security purposes
  t.references :account_id, null: false, index: true
  t.references :user_id, null: false, index: true
  # We're not going to use them in our demo though...

  t.string :klass, null: false
  t.string :cron_expression, null: false
  t.timestamp :next_run_at, null: false, index: true
end

class DynamicJobWorker
  include Sidekiq::Worker

  def perform
    DynamicJob.find_each("next_run_at <= ?", Time.now) do |job|
      # Multi-tenant apps can use a server-side middleware to set DB connection based on account_id and/or user_id.
      Sidekiq::Client.push(:class => job.klass.constantize, :args => [],
                           :account_id => job.account_id, :user_id => job.user_id)
      x = Sidekiq::CronParser.new(job.cron_expression)
      job.update_attribute!(:next_run_at, x.next.to_time)
    end
  end
end

Sidekiq.configure_server do |config|
  config.periodic do |mgr|
    mgr.register("* * * * * *", DynamicJobWorker)
  end
end

So that's just a basic example, and there's more information in the Wiki.

Limitations

Granularity of 1 minute

You can only schedule jobs with 1 minute granularity with the Periodic Jobs feature. If you want to build something with different granularity, you can try to do it on top of the Leader Election feature with custom code or you could run a job that spawns a job 30 seconds later, to run on the minute.

Backfill

Also it's worth noting that Sidekiq doesn't backfill jobs if it wasn't running during a period that a job was supposed to run. You should solve this by writing your jobs to handle everything that should have been done since the last time it was run, rather than assume a guarantee that it runs on the hour every hour precisely, for instance.

API

You can list periodic jobs with the API:

rails c
loops = Sidekiq::Periodic::LoopSet.new
#=> #<Sidekiq::Periodic::LoopSet:0x007fc3d637eac8 @lids=["cecb90d0869fa89f840b7a250e909ce31a313bbb"]>
loops.each do |lop|
  p [lop.schedule, lop.klass, lop.lid, lop.options, lop.history]
end
# ["* * * * *", "PutsWorker", "cecb90d0869fa89f840b7a250e909ce31a313bbb", {"retry"=>"2", "queue"=>"default"}, [["54bbe0a5a9b6105f64201f4b", 1469565660.0], ["5bd1506c5dce881459930231", 1469565600.0], ["1908b4284ab80bd8f779cbbb", 1469565540.0], ["8254d90a10dfd14c2f008d46", 1469565480.0], ["5e6b871bec14a8ec8c23b88a", 1469565420.0], ["99fc1c84efd594981f331e53", 1469565360.0], ["fb56b0a79275e1332e924502", 1469565300.0], ["d14bf55cace4492bfe259828", 1469565240.0], ["60fa8bc5cc7eae3f57444401", 1469565180.0], ["67df4552320d986c1452bd8e", 1469565120.0], ["a4d966c9a7d5f008a02f5a6a", 1469565060.0], ["2c3c528a771093158f41e3ed", 1469565000.0], ["361242b8c9c00569f5236f19", 1469564940.0], ["1b14ad4956033fc5d71720db", 1469564880.0], ["06592ac5723e222c05176691", 1469564820.0], ["fb043ecc82d6a70ab3830626", 1469564760.0], ["60114566abdd962f15029d60", 1469564700.0], ["eb879f858073ca9f34c90471", 1469564640.0], ["0383ae27faef8675a78b77e1", 1469564580.0], ["4a5a0734b8ac112075ce02e3", 1469564520.0], ["8c505e530f97ea0667970b7d", 1469564460.0], ["495e697de1ef5cf36c5ad9cc", 1469564400.0], ["c83e3cd4b32d9a799ddf6818", 1469564340.0]]]
# => ["cecb90d0869fa89f840b7a250e909ce31a313bbb"]

Web UI

You can also see these schedules in the web UI. We'll make sure we've mounted the enterprise version of the web interface:

vim config/routes.rb
require 'sidekiq-ent/web'

Rails.application.routes.draw do
  mount Sidekiq::Web => '/sidekiq'
end

Then visit it in the browser at http://localhost:3000. Now you have a Cron tab...heheheh...that shows you the scheduled jobs. You can see the same info we just saw from the API by clicking into the job.

Summary

So that's Periodic Jobs in Sidekiq Enterprise. We saw how to schedule them and see the list of scheduled jobs and their history, and we learned their limitations. There's detail on the Wiki walking through how you can provide dynamic job scheduling for your users to create jobs as well. I hope you enjoyed it. See you soon!

Resources