Blog

How to make everything background-processable through Sidekiq

I recently switched from Resque to Sidekiq for background processing in a project I'm working on. Sidekiq has a very slick feel to it and is basically a supercharged version of Resque, using threads instead of processed for its workers.

One thing that has irritated me when using Resque was that one was supposed to create separate worker classes for each job that should be processed in the background. While that makes sense for "real" jobs, e.g. syncing the Mailchimp database with your own or exporting some data into a XML file and mailing it out, often I just wanted to make an existing (class or instance) method processable in the background, e.g. user.perform_async :set_timezone_from_ip, request.remote_ip or Account.perform_async :disable_overdue.

Sidekiq offers the delay and delay_for methods to make class and instance method calls asynchronous, as detailed in their Wiki. It also says the following on that Wiki page, though:

I strongly recommend avoid delaying methods on instances. This stores object state in Redis and which can get out of date, causing stale data problems."

Redis likes IDs

Looking up the source code for these methods I saw what the problem is: when calling delay or delay_for on a object, the entire object is serialized to Redis, which is never a good idea. The common best practice is to only store simple data types (stings, Integers, etc.) in Redis.

I wondered why it wasn't possible to just save the object's ID, which could then be looked up by the worker. To verify that that was possible I created a simple module that did so and suggested it to Mike Perman, the creator of Sidekiq. His objection is that "instances can have transient state that the method relies on", which is a very valid concern, but, as long as the developer is aware of that, not a showstopper as far as I'm concerned.

So, without further ado, here's the module I have been using successfully for a while now. Mix it into any class (include Asyncable) to make instances of that class asyncable (instance.perform_async :foo or instance.perform_in 1.day, :foo).

module Asyncable
  extend ActiveSupport::Concern

  # The name of the parameter that is added to the parameter list when calling a method to be processed in the background.
  TargetParamName = :async_target_id

  included do
    include Sidekiq::Worker

    # The name of the Sidekiq queue can be set here.
    # By default, all jobs go to the "default" queue.
    # If a different queue name is set here, workers have to be specifically instructed to process those queues.
    # sidekiq_options queue: self.to_s.underscore
  end

  %w(perform_async perform_in).each do |method_name|
    define_method method_name do |*args|
      self.class.send method_name, *args, TargetParamName => self.id
    end
  end
  alias_method :perform_at, :perform_in

  def perform(*args)
    target = if args.last.is_a?(Hash) && args.last.keys.first.to_sym == TargetParamName
      self.class.find args.pop.values.first
    else
      self.class
    end

    target.send *args
  end
end

What it does

The module uses the ActiveSupport::Concern pattern which makes it very clean to write a module that is meant to be included in other classes.

This is what happens when you include Asyncable in another class:

  • the Sidekiq::Worker module is included, which makes your class a Sidekiq worker.
  • the perform_async and perform_in instance methods are defined, which basically take any parameters you pass in, add a hash { async_target_id: 123 } to the end (123 being the ID of the object you call the method on) and call the perform_async or perform_in class method with those parameters.
  • the perform method is defined which checks the passed parameters for the existance of the { async_target_id: 123 } hash at the end of the parameter list. If it's found, it is removed and the 123 is used to look up the correct instance of that class, which will be the target of the following method call. If the hash is not found, the class itself will be the target of the method call.

Discuss this post on Hacker News

Ideas? Constructive criticism? Think I'm stupid? Let me know in the comments!