[003.2] API

Diving into Sidekiq's Ruby API.

Subscribe now

API [05.13.2016]

Sidekiq provides a public API that allows you to access real-time information about your workers, queues, and jobs. It is used for everything that the Web UI can provide - if you can see it in the Web UI, you can script it with the public API. We saw a little bit of it in the last couple of episodes, but now we'll dive deep.

Project

We'll be using the basic Rails application from before as our project to explore the API, but it doesn't really matter what you use.

I'll kick off the Rails console.

rails c

To access the API, you need to require it:

require 'sidekiq/api

Queue

We'll start out looking at the Queues API. You can find all the Queues Sidekiq knows about:

Sidekiq::Queue.all
# => [#<Sidekiq::Queue:0x00000003773050 @name="default", @rname="queue:default">, #<Sidekiq::Queue:0x00000003773000 @name="mailers", @rname="queue:mailers">]

Here, we can see that sidekiq knows about the default queue and the mailers queue in our Rails app. Let's look at the mailers queue:

mailers = Sidekiq::Queue.new("mailers")
# => #<Sidekiq::Queue:0x00000003757cb0 @name="mailers", @rname="queue:mailers">

Let's see if it has any jobs in it:

mailers.size
# => 6

So we had 6 jobs in our mailers queue locally from before we told ActionMailer to just use the default queue.

We don't need this queue any more, since those are stale jobs from before a change we know about, so we could remove the queue by calling clear on it. We'll avoid that for now, as I'd like to look at the jobs in it.

We can enumerate all of the jobs:

mailers.each do |job|
  puts job.inspect
end

And here we can see each of the jobs was in an error state. We can delete a given job by enumerating and explicitly deleting it if the job_id, or jid, matches the id we want to remove. Let's remove the first one by job id.

(( We'll grab the job id from the inspected output )) ruby job_id = "688dd4463f192f737cc0acd9" mailers.each do |job| job.delete if job.jid == job_id end

You can find out a queue's latency in seconds, which is a measure of when the oldest job was enqueued:

mailers.latency

You can also explicitly find a job by its jid, though this will be inefficient if your queue is big:

mailers.find_job(job_id)

Finally, we can delete that queue since we don't need it to stick around:

mailers.clear

Named Queues

Sidekiq comes with some names queues as well, related to work that's being managed by the system. Let's have a look.

Scheduled

First up, is the Scheduled SortedSet. It holds all scheduled jobs in chronologically-sorted order. We can grab it and inspect it.

ss = Sidekiq::ScheduledSet.new
# => #<Sidekiq::ScheduledSet:0x0000000358a0b8 @name="schedule", @_size=0>
ss.size
# => 0

Now, let's enqueue an email to be sent at a specific time in the future, using ActionMailer's deliver_later:

VisitorMailer.contact_email("Josh", "josh@dailydrip.com", "foo").deliver_later(wait: 1.hour)
# Enqueued ActionMailer::DeliveryJob (Job ID: 3673c764-d7e8-476b-8898-f6c1bd358ac2) to Sidekiq(default) at 2016-03-15 21:20:43 UTC with arguments: "VisitorMailer", "contact_email", "deliver_now", "Josh", "josh@dailydrip.com", "foo"
# => #<ActionMailer::DeliveryJob:0x000000033d39b8 @arguments=["VisitorMailer", "contact_email", "deliver_now", "Josh", "josh@dailydrip.com", "foo"], @job_id="3673c764-d7e8-476b-8898-f6c1bd358ac2", @queue_name="default", @scheduled_at=1458076843.952317>
ss.size
# => 1

As expected, deliver_later with a wait option will place a job into the Sidekiq scheduled set.

You can enumerate over all of the scheduled jobs in your Sidekiq system using the Enumerable interface. Consequently, you could select into them to find all of the jobs of a particular type to remove them. Let's see the class of the job we just added:

ss.each {|job| puts job.klass }

All of the jobs that you initiate via ActiveJob will be of the same klass, so you might want to filter them down a bit more explicitly to remove them. For now, I'll just delete them all:

ss.each {|job| puts job.delete }
ss.size
# => 0

Retries

When a job raises an error, Sidekiq places it in the RetrySet for automatic retry later. Jobs are sorted based on when they will next retry.

rs = Sidekiq::RetrySet.new
rs.size
rs.clear

You can use this API to remove jobs that you do not wish to retry again.

Dead

Like RetrySet and ScheduledSet, the DeadSet holds all jobs considered dead by Sidekiq, ordered by when they died. It supports the same basic operations as the others.

ds = Sidekiq::DeadSet.new
ds.size
ds.clear

Process

Sidekiq::ProcessSet gets you access to near real-time info about the current set of Sidekiq processes running. You can remotely control the processes also:

ps = Sidekiq::ProcessSet.new
ps.size # => 2
ps.each do |process|
  p process['busy']     # => 3
  p process['hostname'] # => 'myhost.local'
  p process['pid']      # => 16131
end
ps.each(&:quiet!) # equivalent to the USR1 signal
ps.each(&:stop!) # equivalent to the TERM signal

Remote control can be useful in situations where signals are not supported: Windows, JRuby and the JVM or Heroku for instance.

Workers

The API also gives you programmatic access to the current active worker set for all Sidekiq processes. A 'worker' is defined as a thread currently processing a job.

This is live data that changes every millisecond. If size returns 5, it in no way suggests that enumerating the workers will result in a block being called 5 times. If you try to rely on that sort of thing, you will have errors in your code.

workers = Sidekiq::Workers.new
workers.size # => 0
# We need to add some work that will never complete, so that we have some
# workers to futz about with.
require_relative './app/workers/our_worker.rb'
20.times { OurWorker.perform_async("easy") }
workers.size
workers.each do |process_id, thread_id, work|
  puts "## process_id:"
  puts process_id.inspect
  puts "\n## thread_id:"
  puts thread_id.inspect
  puts "\n## work:"
  puts work.inspect
  puts "\n## work.payload:"
  puts work["payload"].inspect
  puts "-------\n"
end

Stats

You can look up various stats about your Sidekiq installation.

stats = Sidekiq::Stats.new
stats.processed # => 100
stats.failed # => 3
stats.queues # => { "default" => 1001, "email" => 50 }

Gets the number of jobs enqueued in all queues (does NOT include retries and scheduled jobs).

stats.enqueued # => 5

History

Sidekiq's API gives you detailed access to the history of your Sidekiq installation. All dates are UTC and history stats are cleared after 5 years.

Here we'll get the history of failed/processed stats, going 2 days back starting from today:

s = Sidekiq::Stats::History.new(2)
# => #<Sidekiq::Stats::History:0x00000003325d18 @days_previous=2, @start_date=Wed, 16 Mar 2016>
s.failed
# => {"2016-03-16"=>127, "2016-03-15"=>0}
s.processed
# => {"2016-03-16"=>127, "2016-03-15"=>0}

You can start from a different date:

s = Sidekiq::Stats::History.new( 3, Date.parse("2012-12-3") )
s.failed
# => { "2012-12-03" => 0, "2012-12-02" => 0, "2012-12-01" => 0 }
s.processed
# => { "2012-12-03" => 0, "2012-12-02" => 0, "2012-12-01" => 0 }

Summary

This was a brief overview of Sidekiq's Ruby API. In the resources section, I've linked to the Wiki page about the API as well as the RDocs for all of the classes in the API. If you want to dig in further, it's all there for the reading. See you soon!

Resources