The Evolution of Background Processing from Rails Beta to Rails 8

Background processing in Ruby on Rails has come a long way since the early days of the framework.

In the initial versions—Rails 0.x and 1.x—there was no native concept of background jobs.

Developers often hacked around long-running tasks using cron jobs, shell scripts, or external daemons.

There were no conventions or standardized interfaces, and each project implemented its own system for deferred execution.

This made background work brittle and hard to maintain. If a task failed, there was no built-in retry logic, visibility, or consistency across projects.

As Rails matured, the first real attempt at standardized background job processing came with Delayed Job, extracted from Shopify’s codebase around 2008.

Delayed Job was simple and clever—it used the database itself as a queue.

Jobs were just rows in a delayed_jobs table, serialized as YAML. A background worker process queried this table and executed the jobs.

While Delayed Job was easy to set up and Rails-native, it didn’t scale well.

Using the primary database as a job queue introduced contention, increased query latency, and suffered under high job volume.

Nonetheless, it became the de facto solution for many years because of its simplicity and ease of integration.

Here’s a typical Delayed Job usage:

#Enqueue a background job
UserMailer.welcome_email(@user).deliver_later

And in config/application.rb, you’d set the backend:

config.active_job.queue_adapter = :delayed_job

Rails 4 introduced Active Job, an abstraction layer over background job systems. It allowed Rails developers to write jobs in a standard way, while letting them choose the underlying engine—Delayed Job, Resque, Sidekiq, etc. This brought much-needed flexibility.

However, Active Job was limited by design choices like serializing arguments to global IDs and lack of advanced features like job prioritization, rate limiting, or concurrent execution out-of-the-box.

Still, it helped standardize how jobs were defined and dispatched in Rails:

class NotifyUserJob < ApplicationJob
  queue_as :default

  def perform(user)
    UserNotifier.send_notification(user).deliver_now
  end
end

As applications grew more demanding, Delayed Job and Resque started showing limits. Enter Sidekiq, which became the dominant background processing system in the Rails world.

Built on Redis, Sidekiq provided real multi-threaded job processing, high throughput, retries, monitoring, job prioritization, scheduled jobs, and plugins.

Sidekiq could handle thousands of jobs per second with minimal configuration. It also came with a slick web UI to inspect and manage jobs.

Rails 5, 6, and 7 continued to rely on Sidekiq for real-world background work. Although Active Job offered a generic interface, Sidekiq was the default for performance-conscious developers.

However, Sidekiq was never a part of Rails—it was a third-party gem with its own ecosystem, licensing model, and operational complexity due to Redis dependency.

Here's how Sidekiq usage looked:

#app/jobs/hard_worker.rb
class HardWorker
  include Sidekiq::Worker

  def perform(name, count)
    puts "Doing hard work for #{name} - #{count} times"
  end
end

#Enqueue the job
HardWorker.perform_async('Alice', 5)

The community began to feel the tension: Sidekiq was powerful but external. It didn’t align 100% with Rails’ ethos of convention over configuration.

You had to learn Redis, manage a separate server, and deal with connection limits. It worked—but it wasn’t native. Rails apps, even small ones, were now expected to bring in Sidekiq and Redis just to send a welcome email asynchronously.

That’s where Rails 8 stepped in with SolidQueue—a first-party, production-ready, database-backed queuing system built specifically as a backend for Active Job.

SolidQueue addresses the long-standing demand for a native, simple, and performant job processing solution that integrates tightly with Rails, without requiring external infrastructure like Redis.

SolidQueue works with standard SQL databases like PostgreSQL, MySQL, and even SQLite. It leverages SQL features like FOR UPDATE SKIP LOCKED (when available) to enable efficient, concurrent polling of jobs without introducing locking bottlenecks. It supports essential features out of the box: delayed execution, recurring jobs, concurrency controls, per-job numeric priorities, queue-based priority ordering, queue pausing, and bulk job enqueuing with perform_all_later.

Importantly, SolidQueue delegates job logic—such as retries, discards, serialization, and error handling—to Active Job, which means it plays well with all existing Rails conventions. It’s also compatible with Ruby on Rails’ multithreading capabilities, offering scalable background job processing without needing Redis or other external queues.

For developers and teams who value simplicity, transparency, and deep Rails integration, SolidQueue finally brings background processing fully into the Rails fold—powered entirely by the database and Active Record.

This evolution—from cron hacks to Delayed Job, from Active Job to Sidekiq, and finally to SolidQueue—shows how critical background processing has become to the Rails platform. And now, with SolidQueue, the journey has come full circle: Rails once again owns its job processing story, natively and powerfully.

The Evolution of Background Processing from Rails Beta to Rails 8 #

The Evolution of Background Processing from Rails Beta to Rails 8