[007.2] Sidekiq Enterprise: Encryption and Multi-Process

Encrypting secrets at rest, and managing multiple sidekiqs with a single binary

Subscribe now

Sidekiq Enterprise: Encryption and Multi-Process [11.30.2016]

In this episode we will take a look at two features from Sidekiq Enterprise: Encryption and Multi Process.

Let's get started!

Project

Imagine you have sensitive data and you would like to use it in your Workers. Without encryption, this sensitive data would be stored in Redis, available to anyone that can read from Redis.

Sidekiq Enterprise v1.3.0+ supports transparent encryption of the last job argument (aka the secret bag) so sensitive data at rest in Redis cannot be seen.

Let's set this up:

vim app/config/initializers/sidekiq.rb

We'll open up the Sidekiq configuration and enable crypto, specifying a key to encrypt our data with.

Sidekiq::Enterprise::Crypto.enable(active_version: 1) do |version|
  # this block should return the key for the version N
  #
  # every time you need to rotate the crypto key, you should bump
  # the active_version. this block should always return the key for
  # the given version.
  #
  # You can store the key for each version in a file, in an ENV variable or on a remote
  # keystore - it's up to you how to do key management.
  File.open("secret.1.key", "rb:ASCII-8BIT").read
end

For the moment, we don't have this key file. Let's just comment this line until we generate this file.

We're going to generate the file using rails c:

File.open("secret.1.key", "w:ASCII-8BIT") { |file| file.write(OpenSSL::Cipher.new("aes-256-gcm").random_key) }

And now let's uncomment that line.

vim app/config/initializers/sidekiq.rb

With this, the encryption is set up. Let's create a Worker using encryption, called PrivateWorker. It just sleeps for a long time.

class PrivateWorker
  include Sidekiq::Worker
  sidekiq_options encrypt: true

  def perform(x, y, secret_bag)
    sleep 900000
  end
end

Now using rails console, we can enqueue a job, pass some arguments. Let's do that two times:

PrivateWorker.perform_async(1, 2, 'secret')
PrivateWorker.perform_async(1, 2, 'secret')

Checking the SidekiqAPI, we should not be able to read the last argument, which contains 'secret':

workers = Sidekiq::Workers.new
workers.each do |process_id, thread_id, work, payload|  puts work.inspect end

As you can see, the content is encrypted. I still had some others queued jobs, if you were wondering why there are more than 2 jobs in there. Oops!

Also, we can check the Sidekiq WebUI, it will be encrypted.

Some important facts:

  • Only the last argument is encrypted.
  • The implementation uses OpenSSL's aes-256-gcm Cipher, which is the strongest widely available cipher as of 2016.
  • All the encrypted workers must take >= 2 arguments
  • If you need to execute a worker with only encrypted keys, you should pass only the second argument, something like: PrivateWorker.perform_async(nil, secret_bag)

Let's dive into the another cool feature from Sidekiq Enterprise: Multi Process.

Multi Process

Imagine you have a webserver with a container for your Sidekiq. If you don't think this is enough for you, you will probably want multiple processes. The problem with this is that they will be totally separated and must be managed individually.

Sidekiq Enterprise has a new binary, sidekiqswarm, which will spin up N child Sidekiq processes.

We'll create a service called TestService:

vim app/services/test_service.rb

This service will just do something, so we'll name our method appropriately. The method will just call a worker:

class TestService
  def self.do_something
    (1...999).each do
      TestWorker.perform_async
    end
  end
end

Let's create our Worker:

vim app/workers/test_worker.rb

Our worker looks like just a loop.

class TestWorker
  include Sidekiq::Worker

  def perform()
    999.times do
      OpenStruct.new(field1: '1', field2: '2')
      OpenStruct.new(field1: '1', field2: '2')
      OpenStruct.new(field1: '1', field2: '2')
      OpenStruct.new(field1: '1', field2: '2')
      OpenStruct.new(field1: '1', field2: '2')
    end
  end
end

We are just allocating OpenStruct.

First try will be just running sidekiq.

bundle exec sidekiq
rails console
TestService.do_something

Now let's try that using MultiProcess, by running the binary sidekiqswarm with COUNT=4:

COUNT=4 bundle exec sidekiqswarm

We can see that we have four sidekiq processes running right now!

Let's run our TestService again:

rails console
TestService.perform_async

If we use MultiProcess, execution time will be way better, because we will have 4 workers executing in parallel. If you look in top, you can see the difference.

We can check the processes using htop and here we can see we have four sidekiq processes. If we just use one and check htop, we just have the one process now.

Summary

So that's it. In today's episode, we saw how to use Sidekiq Enterprise's Encryption features, as well as the new sidekiqswarm binary for running multiple processes. I hope you enjoyed it!

Resources