How SolidQueue Works Internally? #

SolidQueue is engineered to bring native, database-backed job processing to Rails. Its internal workings are elegant yet powerful — composed of well-defined tables, efficient SQL operations, and deliberate use of ActiveJob’s abstractions.

Let us explore how SolidQueue operates under the hood, how it handles job lifecycle events (enqueueing, locking, execution, retrying, discarding), and how it ensures concurrency and resilience using only SQL primitives.

1. Job Lifecycle in SolidQueue #

At the core of SolidQueue lies the solid_queue_jobs database table.

Every background task (job) you enqueue creates a row in this table.

Here’s a simplified schema: #

create_table :solid_queue_jobs do |t|
  t.string :job_class, null: false
  t.text :serialized_params, null: false
  t.string :queue_name, default: "default"
  t.datetime :scheduled_at
  t.datetime :finished_at
  t.integer :priority, default: 0
  t.timestamps
end

When you invoke MyJob.perform_later(arg1, arg2), ActiveJob serializes the job and inserts it into this table. SolidQueue stores metadata like job class, parameters, queue name, and scheduling time. The scheduled_at column determines when the job should be eligible for execution — if it's in the future, job workers will ignore the job until that time.

What Is a Job Worker? #

A SolidQueue Job Worker is a long-running Ruby process responsible for polling the database, locking a job record, and executing the job logic.

You typically start it using the following command:

bin/rails solid_queue:work

Under the hood, this command spins up one or more threads (depending on your config) that continuously poll for jobs in a loop. Each thread uses an efficient SQL query to locate the next available job, safely lock it using FOR UPDATE SKIP LOCKED, and then instantiate and perform the job.

For example in production our this app, the one you are using right now to read this book, at https://allbooks.railsforgedev.com has a deploy.yml file that starts SolidQueue worker inside our docker container image:

servers:
  web:
    - allbooks.railsforgedev.com
  job:
    hosts:
      - allbooks.railsforgedev.com
    cmd: bin/jobs

Here’s a conceptual flow: #

1. Query the solid_queue_jobs table for jobs that are ready to run.
2. Use SKIP LOCKED to avoid interfering with other workers.
3. Lock the job row.
4. Instantiate the job class using ActiveJob.
5. Run the job’s perform method.
6. Update the finished_at timestamp on success (or leave it null for retries).

This polling loop ensures that job workers steadily consume jobs in a safe, concurrent fashion using just the database. There's no Redis, no third-party service — just Rails, your DB, and native threading.

2. Locking, Concurrency, and SKIP LOCKED #

Job workers need to pick jobs safely without stepping on each other’s toes.

SolidQueue uses SELECT … FOR UPDATE SKIP LOCKED to claim jobs in a concurrent-safe way.

This means that if one worker has already locked a row, others skip it, preventing race conditions or duplicate execution.

Here’s an approximation of the logic:

SELECT * FROM solid_queue_jobs
WHERE finished_at IS NULL
  AND scheduled_at <= NOW()
ORDER BY priority DESC, scheduled_at ASC
LIMIT 1
FOR UPDATE SKIP LOCKED

This gives us the next available job without waiting on locked rows. It’s fast, safe, and leverages SQL features available in PostgreSQL, MySQL, and SQLite.

3. Job Execution and Error Handling #

Once a worker has successfully claimed a job from the queue (via SQL locking), it proceeds to instantiate the job class using ActiveJob and then executes it.

What does this mean? #

When a job is enqueued using something like MyJob.perform_later(arg1, arg2), ActiveJob serializes the job class name, method arguments, and additional metadata into a JSON blob. This gets stored in the serialized_params column of the solid_queue_jobs table.

When the worker later processes this job, ActiveJob reads the serialized data, loads the correct job class (e.g., MyJob), and creates a Ruby object from it:

job_instance = MyJob.new
job_instance.perform(arg1, arg2)

This deserialization and instantiation step ensures that jobs behave exactly as written — running the correct logic with the intended parameters. This process also enables ActiveJob to stay backend-agnostic; the same job code can run under Sidekiq, SolidQueue, or any other compatible adapter.

If the job runs successfully, SolidQueue marks it as finished: #

job_instance.update!(finished_at: Time.current)

If an error occurs during execution, SolidQueue doesn't handle retries directly. Instead, it relies on ActiveJob’s retry semantics, which may re-enqueue the job for later execution depending on how you’ve defined your retry rules:

class MyJob < ApplicationJob
  retry_on SomeError, wait: 10.seconds, attempts: 3
end

However, if the job continues to fail beyond the configured retry attempts, ActiveJob may discard it, or call discard_on hooks if defined.

4. Delayed and Recurring Jobs #

SolidQueue supports delayed jobs using the scheduled_at timestamp column. If you call perform_later with a delay, such as:

MyJob.set(wait: 10.minutes).perform_later

ActiveJob sets the scheduled_at time accordingly, and SolidQueue workers will skip this job until that timestamp is reached. Workers continuously poll for due jobs (scheduled_at <= NOW()), so scheduling works smoothly out of the box — no extra configuration needed.

Recurring Jobs with SolidQueue::RecurringJob #

Beyond just delayed execution, SolidQueue also supports recurring jobs — a powerful feature traditionally handled using CRON or external schedulers like Whenever or Sidekiq-Cron.

With SolidQueue, recurring jobs are first-class citizens using the SolidQueue::RecurringJob model. You define a job, schedule, and optional conditions right in your Rails app:

SolidQueue::RecurringJob.create!(
  job: MyJob,
  arguments: [some_arg],
  interval: 10.minutes
)

This will enqueue MyJob every 10 minutes, internally managed by the database. You don’t need to write CRON expressions or rely on a system-level CRON daemon. Rails takes care of it — consistently and transparently.

CRON vs SolidQueue::RecurringJob #

By baking recurring scheduling directly into SolidQueue, Rails developers get a unified, database-driven job system with no external schedulers or YAML configs to maintain. It's readable, testable, and fully integrated into your ActiveJob lifecycle.

5. Job Cleanup and Pruning #

In production environments, the number of completed or failed jobs in your database can grow rapidly. Without cleanup, this can lead to:

• Database bloat
• Slower queries
• I/O overhead during polling
• Unnecessary storage costs

SolidQueue does not automatically prune jobs, but it offers rake tasks and Active Record models to help you manage this explicitly.

🧹 How to Prune Jobs #

SolidQueue provides a built-in rake task to clean up old jobs:

bin/rails solid_queue:prune

This will remove jobs that are considered safe to discard — typically those that are finished, discarded, or expired.

You can configure pruning options as needed. For example, if you only want to keep the last 7 days' worth of job history:

bin/rails solid_queue:prune DAYS=7

You can also call pruning directly via code in scheduled tasks or rake tasks:

SolidQueue::Job.finished.where("finished_at < ?", 7.days.ago).delete_all

💡 Best Practice in Production #

• Schedule pruning to run daily using a recurring job or a system-level cron (if needed).
• Do not prune immediately after job success, as job records can be valuable for auditing and debugging.
• Use database partitioning or archiving strategies if your app deals with high job volume.

❗️Important Note #

If you use SolidQueue extensively, pruning is not optional. Over time, millions of job records can accumulate. Cleaning them up is essential for performance and data hygiene.

Internally, SolidQueue is designed around a few powerful concepts: SQL tables for persistence, SKIP LOCKED for concurrency, and ActiveJob for orchestration.

There’s no Redis, no external process queue manager — just Rails, your DB, and a tight loop of polling and execution.

Understanding this machinery helps you build more reliable, observable, and Rails-native background processing pipelines.