[006.4] Sidekiq Enterprise: Rate Limiting

Using Sidekiq's Rate Limiters to get around pesky API SLAs.

Subscribe now

Sidekiq Enterprise: Rate Limiting [11.30.2016]

When dealing with third-party APIs you will commonly run into rate limits. It can be difficult to ensure that you are adhering to these rate limits in a typical Ruby application because it needs to be tracked across multiple threads and processes in any decently-sized application.

Sidekiq provides a Limiter that makes it easy to ensure rate limits are honored across processes. Let's check it out.

Project

We'll build a basic application that makes anonymous requests to the Twitter Trends API. It rate limits us to 75 requests per 15 minutes.

We'll start out introducing a gem to make it easy to interact with the API:

vim Gemfile
gem 'twitter'

As we are using the Twitter API, we can create a service in our application, this service will make all the requests and it will handle the requests for us. Let's call this Service TwitterService.

vim app/services/twitter_service.rb

Let's create our client that will connect with the TwitterAPI:

  def self.client
    # We'll need to set our consumer key and secret, and our access token and
    # secret, which will be fetched from our environment variables.
    client = Twitter::REST::Client.new do |config|
      config.consumer_key        = ENV['TWITTER_CONSUMER_KEY']
      config.consumer_secret     = ENV['TWITTER_CONSUMER_SECRET']
      config.access_token        = ENV['TWITTER_ACCESS_TOKEN']
      config.access_token_secret = ENV['TWITTER_ACCESS_TOKEN_SECRET']
    end
  end

Let's see which regions are available:

  def self.available_places_for_trends
     TwitterService.client.trends_available.map(&:name)
  end
rails console
TwitterService.available_places_for_trends

Here we can see a list of the available places for trends.

Let's create a method to get the trends for a specific region:

  # Let's call that `available_trends_in`, and it requires a place.
  def self.available_trends_in(place)
    # We're going to use the TwitterService API, call our client, and use the trends_available method from our client.
    id_places = TwitterService.client.trends_available.map { |place| { id: place.id, place: place.name } }

    id_places.each do |elem|
      return TwitterService.client.trends(elem[:id]) if place == elem[:place]
    end

    []
  end

Let's test that:

rails console
TwitterService.available_trends_in('New York')

Here we got the list of trends from New York.

So there's one request. If we make 74 more requests in the next 15min, we'll hit the limit and future requests will fail with the following error:

Twitter::Error::TooManyRequests: Rate limit exceeded

As we know all the available trending places, if we check all the trending topics for all the available places, we will have this error. So, let's get this error!

We're going to create a method called all_trends_from_all_places.

  # Here we are calling the Twitter API from all the available places
  # It exceeds the rate limit API
  def self.all_trends_from_all_places
    [].tap do |all|
      TwitterService.available_places_for_trends.each do |place|
        TwitterService.available_trends_in(place)
      end
    end
  end

Let's see this working:

rails console
TwitterService.all_trends_from_all_places

Here we go! We just got the rate limit error.

We're iterating over all the available places for trends and we are getting all the trends for a specific region, but we are doing that for all the World, there are about 450 places and consequently we get a TooManyRequests Error for the API. How do we solve that?

We'll build out a quick worker for fetching this request from the Twitter API and logging it out.

Sidekiq offers various rate limiting strategies. For the Twitter API, we want to use a window, as there is a 15-minutes sliding window for their rate limiting.

We are using Sidekiq::Limiter.window('trend_topic_places', 75, 900) and it will make 75 requests in a period of 15min. It schedules the jobs and it executes in the correct period of time.

We can set the Error reported by Twitter(Twitter::Error::TooManyRequests) as a custom error for our Limiter, so it is treated as a rate limit exception.

Let's add that in our Sidekiq initializer:

vim app/config/initializers/sidekiq.rb
# allow 75 operations within a 900 second(15min) window
TWITTER_API_LIMIT = Sidekiq::Limiter.window('trend_topic_places', 75, 900)
Sidekiq::Limiter.conf igure do |config|
  config.errors << Twitter::Error::TooManyRequests
end

Let's create our worker.

vim app/workers/trend_topics_worker.rb
class TrendTopicsWorker
  include Sidekiq::Worker

  def perform(place)
    TWITTER_API_LIMIT.within_limit do
      TwitterService.available_trends_in(place)
    end
  end
end

Our method will call the worker:

  # Here we are calling the Twitter API from all the available places
  # but we are passing the place name for a worker
  # This worker has a limit for the API. With this our we will not exceed
  # the API SLA
  def self.all_trends_from_all_places_using_worker
    [].tap do |all|
      TwitterService.available_places_for_trends.each do |place|
        TrendTopicsWorker.perform_async(place)
      end
    end
  end

We keep iterating over all the available places, but now with one difference. We are using the worker to handle the requests and the worker is being limited by the rate limit of the Twitter API.

If we check sidekiq's logs:

2016-10-25T16:39:00.883Z 62607 TID-ow57x2wiw TrendTopicsWorker JID-7c89ee4b8db7b42548f97be5 INFO: start
2016-10-25T16:39:00.886Z 62607 TID-ow57x2ryg TrendTopicsWorker JID-66ba2657d4a01f85cbb01d82 INFO: Limit 'trend_1477413540' over rate limit, rescheduling for later

It basically says the job will be rescheduled for a good period. Also if we check the WebUi, we can see all the scheduled jobs.

Summary

So, that's awesome. Using Sidekiq we were able to generate a Rate Limiting tool, so that we can ask Twitter for this information as quickly as we're allowed to, but no quicker. It's important to keep in mind that, since this is using Sidekiq, it works across processes - that's really hard to do without a tool like this. See you soon!

Resources