rails/solid_queue

Solid Queue

Solid Queue is a database-based queuing backend for Active Job, designed with simplicity and performance in mind.

In addition to regular job enqueuing and processing, Solid Queue supports delayed jobs, concurrency controls, recurring jobs, pausing queues, numeric priorities per job, priorities by queue order, and bulk enqueuing (enqueue_all for Active Job's perform_all_later).

Solid Queue can be used with SQL databases such as MySQL, PostgreSQL, or SQLite, and it leverages the FOR UPDATE SKIP LOCKED clause, if available, to avoid blocking and waiting on locks when polling jobs. It relies on Active Job for retries, discarding, error handling, serialization, and delays, and it's compatible with Ruby on Rails's multi-threading.

Table of Contents

• Installation
  • Usage in development and other non-production environments
  • Single database configuration
  • Dashboard UI Setup
  • Incremental adoption
  • High performance requirements
• Workers, dispatchers, and scheduler
  • Fork vs. async mode
• Configuration
  • Optional scheduler configuration
  • Queue order and priorities
  • Queues specification and performance
  • Threads, processes, and signals
  • Database configuration
  • Other configuration settings
  • Validating the configuration
• Lifecycle hooks
• Errors when enqueuing
• Concurrency controls
  • Performance considerations
• Failed jobs and retries
  • Error reporting on jobs
• Puma plugin
• Jobs and transactional integrity
• Recurring tasks
  • Scheduling and unscheduling recurring tasks dynamically
• Inspiration
• License

Installation

Solid Queue is configured by default in new Rails 8 applications. If you're running an earlier version, you can add it manually following these steps:

1. bundle add solid_queue
2. bin/rails solid_queue:install

(Note: The minimum supported version of Rails is 7.1 and Ruby is 3.2.)

This will configure Solid Queue as the production Active Job backend, create the configuration files config/queue.yml and config/recurring.yml, and create the db/queue_schema.rb. It'll also create a bin/jobs executable wrapper that you can use to start Solid Queue.

Once you've done that, you will have to add the configuration for the queue database in config/database.yml. If you're using SQLite, it'll look like this:

production:
  primary:
    <<: *default
    database: storage/production.sqlite3
  queue:
    <<: *default
    database: storage/production_queue.sqlite3
    migrations_paths: db/queue_migrate

...or if you're using MySQL/PostgreSQL/Trilogy:

production:
  primary: &primary_production
    <<: *default
    database: app_production
    username: app
    password: <%= ENV["APP_DATABASE_PASSWORD"] %>
  queue:
    <<: *primary_production
    database: app_production_queue
    migrations_paths: db/queue_migrate

Then run db:prepare in production to ensure the database is created and the schema is loaded.

Now you're ready to start processing jobs by running bin/jobs on the server that's doing the work. This will start processing jobs in all queues using the default configuration. See below to learn more about configuring Solid Queue.

For small projects, you can run Solid Queue on the same machine as your webserver. When you're ready to scale, Solid Queue supports horizontal scaling out-of-the-box. You can run Solid Queue on a separate server from your webserver, or even run bin/jobs on multiple machines at the same time. Depending on the configuration, you can designate some machines to run only dispatchers or only workers. See the configuration section for more details on this.

Note: Future changes to the schema will come in the form of regular migrations.

Usage in development and other non-production environments

Calling bin/rails solid_queue:install will automatically add config.solid_queue.connects_to = { database: { writing: :queue } } to config/environments/production.rb. In order to use Solid Queue in other environments (such as development or staging), you'll need to add a similar configuration(s).

For example, if you're using SQLite in development, update database.yml as follows:

development:
+ primary:
    <<: *default
    database: storage/development.sqlite3
+ queue:
+   <<: *default
+   database: storage/development_queue.sqlite3
+   migrations_paths: db/queue_migrate

Next, add the following to development.rb

  # Use Solid Queue in Development.
  config.active_job.queue_adapter = :solid_queue
  config.solid_queue.connects_to = { database: { writing: :queue } }

Once you've added this, run db:prepare to create the Solid Queue database and load the schema.

Finally, in order for jobs to be processed, you'll need to have Solid Queue running. In Development, this can be done via the Puma plugin as well. In puma.rb update the following line:

# You can either set the env var, or check for development
plugin :solid_queue if ENV["SOLID_QUEUE_IN_PUMA"] || Rails.env.development?

You can also just use bin/jobs, but in this case you might want to set a different logger for Solid Queue because the default logger will log to log/development.log and you won't see anything when you run bin/jobs. For example:

config.solid_queue.logger = ActiveSupport::Logger.new(STDOUT)

Note about Action Cable: If you use Action Cable (or anything dependent on Action Cable, such as Turbo Streams), you will also need to update it to use a database.

In config/cable.yml

development:
-  adapter: async
+  adapter: solid_cable
+  connects_to:
+    database:
+      writing: cable
+  polling_interval: 0.1.seconds
+  message_retention: 1.day

In config/database.yml

development:
  primary:
    <<: *default
    database: storage/development.sqlite3
+  cable:
+    <<: *default
+    database: storage/development_cable.sqlite3
+    migrations_paths: db/cable_migrate

Single database configuration

Running Solid Queue in a separate database is recommended, but it's also possible to use one single database for both the app and the queue. Follow these steps:

1. Copy the contents of db/queue_schema.rb into a normal migration and delete db/queue_schema.rb
2. Remove config.solid_queue.connects_to from production.rb
3. Migrate your database. You are ready to run bin/jobs

You won't have multiple databases, so database.yml doesn't need to have primary and queue database.

Dashboard UI Setup

For viewing information about your jobs via a UI, we recommend taking a look at mission_control-jobs, a dashboard where, among other things, you can examine and retry/discard failed jobs.

Incremental adoption

If you're planning to adopt Solid Queue incrementally by switching one job at the time, you can do so by leaving the config.active_job.queue_adapter set to your old backend, and then set the queue_adapter directly in the jobs you're moving:

# app/jobs/my_job.rb

class MyJob < ApplicationJob
  self.queue_adapter = :solid_queue
  # ...
end

High performance requirements

Solid Queue was designed for the highest throughput when used with MySQL 8+, MariaDB 10.6+, or PostgreSQL 9.5+, as they support FOR UPDATE SKIP LOCKED. You can use it with older versions, but in that case, you might run into lock waits if you run multiple workers for the same queue. You can also use it with SQLite on smaller applications.

Workers, dispatchers, and scheduler

We have several types of actors in Solid Queue:

• Workers are in charge of picking jobs ready to run from queues and processing them. They work off the solid_queue_ready_executions table.
• Dispatchers are in charge of selecting jobs scheduled to run in the future that are due and dispatching them, which is simply moving them from the solid_queue_scheduled_executions table over to the solid_queue_ready_executions table so that workers can pick them up. On top of that, they do some maintenance work related to concurrency controls.
• The scheduler manages recurring tasks, enqueuing jobs for them when they're due.
• The supervisor runs workers and dispatchers according to the configuration, controls their heartbeats, and stops and starts them when needed.

Fork vs. async mode

By default, Solid Queue runs in fork mode. This means the supervisor will fork a separate process for each supervised worker/dispatcher/scheduler. This provides the best isolation and performance, but can have additional memory usage and might not work with some Ruby implementations. As an alternative, you can run all workers, dispatchers and schedulers in the same process as the supervisor, in different threads, with an async mode. You can choose this mode by running bin/jobs as:

bin/jobs --mode async

Or you can also set the environment variable SOLID_QUEUE_SUPERVISOR_MODE to async. If you use the async mode, the processes option in the configuration described below will be ignored.

The recommended and default mode is fork. Only use async if you know what you're doing and have strong reasons to

This supervisor mode is separate from a worker's concurrency model. Supervisor mode decides whether supervised processes live in forks or threads. Worker configuration decides whether claimed jobs run in a thread pool (threads: N) or as fibers on a single fiber reactor thread (fibers: N).

Because these are separate concerns, you can combine the default fork supervisor mode with fiber workers. In that setup, each worker process gets its own fiber reactor and bounded fiber count.

Configuration

By default, Solid Queue will try to find your configuration under config/queue.yml, but you can set a different path using the environment variable SOLID_QUEUE_CONFIG or by using the -c/--config_file option with bin/jobs, like this:

bin/jobs -c config/calendar.yml

You can also skip the scheduler process by setting the environment variable SOLID_QUEUE_SKIP_RECURRING=true. This is useful for environments like staging, review apps, or development where you don't want any recurring jobs to run. This is equivalent to using the --skip-recurring option with bin/jobs.

To run only the scheduler (no workers or dispatchers)—for example to isolate recurring tasks on a dedicated process—set SOLID_QUEUE_ONLY_RECURRING=true or use the --only-recurring option with bin/jobs.

This is what this configuration looks like:

production:
  dispatchers:
    - polling_interval: 1
      batch_size: 500
      concurrency_maintenance_interval: 300
  workers:
    - queues: "*"
      threads: 3
      polling_interval: 2
    - queues: [ real_time, background ]
      threads: 5
      polling_interval: 0.1
      processes: 3
    - queues: "api*"
      fibers: 100
      polling_interval: 0.05
  scheduler:
    dynamic_tasks_enabled: true
    polling_interval: 5

Everything is optional. If no configuration at all is provided, Solid Queue will run with one dispatcher and one worker with default settings. If you want to run only dispatchers or workers, you just need to include that section alone in the configuration. For example, with the following configuration:

production:
  dispatchers:
    - polling_interval: 1
      batch_size: 500
      concurrency_maintenance_interval: 300

the supervisor will run 1 dispatcher and no workers.

Here's an overview of the different options:

• polling_interval: the time interval in seconds that workers and dispatchers will wait before checking for more jobs. This time defaults to 1 second for dispatchers and 0.1 seconds for workers.

• batch_size: the dispatcher will dispatch jobs in batches of this size. The default is 500.

• concurrency_maintenance_interval: the time interval in seconds that the dispatcher will wait before checking for blocked jobs that can be unblocked. Read more about concurrency controls to learn more about this setting. It defaults to 600 seconds.

• queues: the list of queues that workers will pick jobs from. You can use * to indicate all queues (which is also the default and the behaviour you'll get if you omit this). You can provide a single queue, or a list of queues as an array. Jobs will be polled from those queues in order, so for example, with [ real_time, background ], no jobs will be taken from background unless there aren't any more jobs waiting in real_time. You can also provide a prefix with a wildcard to match queues starting with a prefix. For example:

  staging:
    workers:
      - queues: staging*
        threads: 3
        polling_interval: 5
  
  

  This will create a worker fetching jobs from all queues starting with staging. The wildcard * is only allowed on its own or at the end of a queue name; you can't specify queue names such as *_some_queue. These will be ignored.

  Also, if a wildcard (*) is included alongside explicit queue names, for example: queues: [default, backend, *], then it would behave like queues: *

  Finally, you can combine prefixes with exact names, like [ staging*, background ], and the behaviour with respect to order will be the same as with only exact names.

  Check the sections below on how queue order behaves combined with priorities, and how the way you specify the queues per worker might affect performance.

• threads: configures a worker to execute jobs in a thread pool of this size. By default, workers use threads: 3. Only workers have this setting, and it can't be combined with fibers. It is recommended to set this value less than or equal to the queue database's connection pool size minus 2, as each worker uses connections for polling and heartbeat and thread mode may use additional connections for job execution.

• fibers: configures a worker to execute jobs as fibers on a single fiber reactor thread, with this value as the maximum number of in-flight jobs. It can't be combined with threads. Fiber workers require fiber-scoped isolated execution state. In Rails apps, set config.active_support.isolation_level = :fiber before using fibers. Solid Queue refuses to boot fiber workers when isolation remains thread-scoped. On Rails 7.2 and later, a practical starting point is usually 3-5 queue database connections per worker process rather than matching the fibers value, because ordinary Active Record query paths can release connections between non-blocking waits. On Rails 7.1, size the queue database pool more conservatively, as in-flight fiber jobs may still retain connections roughly in proportion to fibers.

• processes: this is the number of worker processes that will be forked by the supervisor with the settings given. By default, this is 1, just a single process. This setting is useful if you want to dedicate more than one CPU core to a queue or queues with the same configuration. Only workers have this setting. This works with both threads and fibers workers as long as the supervisor is running in the default fork mode. Note: this option is ignored only when the supervisor itself is running in async mode.

• concurrency_maintenance: whether the dispatcher will perform the concurrency maintenance work. This is true by default, and it's useful if you don't use any concurrency controls and want to disable it or if you run multiple dispatchers and want some of them to just dispatch jobs without doing anything else.

Optional scheduler configuration

Optionally, you can configure the scheduler process under the scheduler section in your config/queue.yml if you'd like to schedule recurring tasks dynamically.

scheduler:
  dynamic_tasks_enabled: true
  polling_interval: 5

• dynamic_tasks_enabled: whether the scheduler should poll for dynamically scheduled recurring tasks. This is false by default. When enabled, the scheduler will poll the database at the given polling_interval to pick up tasks scheduled via SolidQueue.schedule_recurring_task.
• polling_interval: how frequently (in seconds) the scheduler checks for dynamic task changes. Defaults to 5.

Queue order and priorities

As mentioned above, if you specify a list of queues for a worker, these will be polled in the order given, such as for the list real_time,background, no jobs will be taken from background unless there aren't any more jobs waiting in real_time.

Active Job also supports positive integer priorities when enqueuing jobs. In Solid Queue, the smaller the value, the higher the priority. The default is 0.

This is useful when you run jobs with different importance or urgency in the same queue. Within the same queue, jobs will be picked in order of priority, but in a list of queues, the queue order takes precedence, so in the previous example with real_time,background, jobs in the real_time queue will be picked before jobs in the background queue, even if those in the background queue have a higher priority (smaller value) set.

We recommend not mixing queue order with priorities but either choosing one or the other, as that will make job execution order more straightforward for you.

Queues specification and performance

To keep polling performant and ensure a covering index is always used, Solid Queue only does two types of polling queries:

-- No filtering by queue
SELECT job_id
FROM solid_queue_ready_executions
ORDER BY priority ASC, job_id ASC
LIMIT ?
FOR UPDATE SKIP LOCKED;

-- Filtering by a single queue
SELECT job_id
FROM solid_queue_ready_executions
WHERE queue_name = ?
ORDER BY priority ASC, job_id ASC
LIMIT ?
FOR UPDATE SKIP LOCKED;

The first one (no filtering by queue) is used when you specify

queues: *

and there aren't any queues paused, as we want to target all queues.

In other cases, we need to have a list of queues to filter by, in order, because we can only filter by a single queue at a time to ensure we use an index to sort. This means that if you specify your queues as:

queues: beta*

we'll need to get a list of all existing queues matching that prefix first, with a query that would look like this:

SELECT DISTINCT(queue_name)
FROM solid_queue_ready_executions
WHERE queue_name LIKE 'beta%';

This type of DISTINCT query on a column that's the leftmost column in an index can be performed very fast in MySQL thanks to a technique called Loose Index Scan. PostgreSQL doesn't implement this technique natively, so Solid Queue uses a recursive CTE to emulate it, achieving similar performance by walking the B-tree index and jumping between distinct values. SQLite doesn't implement loose index scan either, but this is unlikely to be a problem since SQLite is typically used in development with small datasets.

Similarly to using prefixes, the same will happen if you have paused queues, because we need to get a list of all queues with a query like

SELECT DISTINCT(queue_name)
FROM solid_queue_ready_executions

and then remove the paused ones. Pausing in general should be something rare, used in special circumstances, and for a short period of time. If you don't want to process jobs from a queue anymore, the best way to do that is to remove it from your list of queues.

💡 To sum up, if you want to ensure optimal performance on polling, the best way to do that is to always specify exact names for them, and not have any queues paused.

Do this:

queues: [ background, backend ]

instead of this:

queues: back*

Threads, processes, and signals

By default, workers in Solid Queue use a thread pool to run work in multiple threads, configurable via the threads parameter above. Workers can also be configured with fibers, in which case claimed jobs are executed as fibers on a single reactor thread and bounded by the worker's fiber count. Besides this, parallelism can be achieved via multiple processes on one machine (configurable via different workers or the processes parameter above) or by horizontal scaling.

Fiber worker execution is best suited for cooperative, mostly I/O-bound jobs. Blocking or CPU-heavy work still blocks the single reactor thread, so it should not be expected to outperform thread mode for every workload.

Because fiber workers run multiple fibers on a single thread, Rails must also isolate execution state per fiber rather than per thread. If your app keeps the default thread-scoped isolation level, Solid Queue will raise a boot-time error instead of running fiber workers with shared Active Record state.

Keep in mind that config.active_support.isolation_level = :fiber applies to your whole application, not just to Solid Queue: if you run Solid Queue inside Puma via the plugin, or combine fiber workers with thread workers in the same process using the supervisor's async mode, everything in that process will use fiber-scoped execution state. This is fully supported by Rails, but it's a global setting worth being deliberate about.

On Rails 7.2 and later, fiber workers can often use a much smaller queue database pool than an equivalent thread pool. A practical starting point is 3-5 queue database connections per worker process: one for job execution, one for polling, one for heartbeats, plus some headroom. In the default fork supervisor mode, that guidance applies per worker process. In supervisor async mode, all workers share one process, so add together the requirements for the workers running there.

That lower-pool guidance depends on job code not holding connections open across non-blocking waits. APIs such as ActiveRecord::Base.connection, lease_connection, connection_pool.checkout, or long-lived with_connection / transaction blocks can pin connections and push fiber workers back toward thread-like pool usage. On Rails 7.1, plan conservatively and assume the configured fiber count can still grow queue database connection usage.

The supervisor is in charge of managing these processes, and it responds to the following signals when running in its own process via bin/jobs or with the Puma plugin with the default fork mode:

• TERM, INT: starts graceful termination. The supervisor will send a TERM signal to its supervised processes, and it'll wait up to SolidQueue.shutdown_timeout time until they're done. If any supervised processes are still around by then, it'll send a QUIT signal to them to indicate they must exit.
• QUIT: starts immediate termination. The supervisor will send a QUIT signal to its supervised processes, causing them to exit immediately.

When receiving a QUIT signal, if workers still have jobs in-flight, these will be returned to the queue when the processes are deregistered.

On Windows, the QUIT signal can't be trapped, so the supervisor only responds to TERM and INT there.

If processes have no chance of cleaning up before exiting (e.g. if someone pulls a cable somewhere), in-flight jobs might remain claimed by the processes executing them. Processes send heartbeats, and the supervisor checks and prunes processes with expired heartbeats. Jobs that were claimed by processes with an expired heartbeat will be marked as failed with a SolidQueue::Processes::ProcessPrunedError. You can configure both the frequency of heartbeats and the threshold to consider a process dead. See the section below for this.

Worker heartbeats are driven by a separate timer task, not by the worker execution backend itself. This means fiber workers do not rely on the reactor loop to prove liveness. However, liveness is still tracked at the worker-process level, not at the individual thread or fiber level.

This means finished and failed jobs still follow the normal Solid Queue lifecycle, but a single stuck job can remain claimed if the worker process itself is still alive. If you need stronger stuck-job detection, that requires an explicit timeout or watchdog mechanism on top of process heartbeats.

In a similar way, if a worker is terminated in any other way not initiated by the above signals (e.g. a worker is sent a KILL signal), jobs in progress will be marked as failed so that they can be inspected, with a SolidQueue::Processes::ProcessExitError. Sometimes a job in particular is responsible for this, for example, if it has a memory leak and you have a mechanism to kill processes over a certain memory threshold, so this will help identifying this kind of situation.

Database configuration

You can configure the database used by Solid Queue via the config.solid_queue.connects_to option in the config/application.rb or config/environments/production.rb config files. By default, a single database is used for both writing and reading called queue to match the database configuration you set up during the install.

All the options available to Active Record for multiple databases can be used here.

Other configuration settings

Note: The settings in this section should be set in your config/application.rb or your environment config like this: config.solid_queue.silence_polling = true

There are several settings that control how Solid Queue works that you can set as well:

• logger: the logger you want Solid Queue to use. Defaults to the app logger.

• app_executor: the Rails executor used to wrap background operations, defaults to the app executor

• on_thread_error: custom lambda/Proc to call when there's an error within a Solid Queue thread that takes the exception raised as argument. Defaults to

  -> (exception) { Rails.error.report(exception, handled: false) }
  

  This is not used for errors raised within a job execution. Errors happening in jobs are handled by Active Job's retry_on or discard_on, and ultimately will result in failed jobs. This is for errors happening within Solid Queue itself.

• use_skip_locked: whether to use FOR UPDATE SKIP LOCKED when performing locking reads. This will be automatically detected in the future, and for now, you only need to set this to false if your database doesn't support it. For MySQL, that'd be versions < 8; for MariaDB, versions < 10.6; and for PostgreSQL, versions < 9.5. If you use SQLite, this has no effect, as writes are sequential.

• process_heartbeat_interval: the heartbeat interval that all processes will follow—defaults to 60 seconds.

• process_alive_threshold: how long to wait until a process is considered dead after its last heartbeat—defaults to 5 minutes.

• fork_boot_timeout: how long a forked process can take to finish booting before the supervisor replaces it—defaults to 5 minutes. It only applies in the default fork mode.

• shutdown_timeout: time the supervisor will wait since it sent the TERM signal to its supervised processes before sending a QUIT version to them requesting immediate termination—defaults to 5 seconds.

• silence_polling: whether to silence Active Record logs emitted when polling for both workers and dispatchers—defaults to true.

• supervisor_pidfile: path to a pidfile that the supervisor will create when booting to prevent running more than one supervisor in the same host, or in case you want to use it for a health check. It's nil by default.

• preserve_finished_jobs: whether to keep finished jobs in the solid_queue_jobs table—defaults to true.

• clear_finished_jobs_after: period to keep finished jobs around, in case preserve_finished_jobs is true — defaults to 1 day. When installing Solid Queue, a recurring job is automatically configured to clear finished jobs every hour on the 12th minute in batches. You can edit the recurring.yml configuration to change this as you see fit.

• default_concurrency_control_period: the value to be used as the default for the duration parameter in concurrency controls. It defaults to 3 minutes.

Validating the configuration

You can validate the Solid Queue configuration ahead of time, without starting any process. This is handy in deploy scripts or CI to catch mistakes—a typo in recurring.yml, no processes configured, and so on—before they cause a supervisor to boot into a broken state:

# Using the bin/jobs binstub
bin/jobs check

# Or via rake
bin/rails solid_queue:check

Both commands validate the configuration for the current Rails environment. On success they print Solid Queue configuration is valid. and exit 0; otherwise they print the errors and exit non-zero. When the number of threads is larger than the database connection pool, they also print an advisory warning about it—the same one the supervisor logs on boot. They're tolerant of a missing database connection, so they can run on CI or deploy hosts without database credentials.

bin/jobs check accepts the same options as bin/jobs start (e.g. --config_file, --recurring_schedule_file, --skip-recurring). The rake task honors the same environment variables Solid Queue already uses: SOLID_QUEUE_CONFIG, SOLID_QUEUE_RECURRING_SCHEDULE, and SOLID_QUEUE_SKIP_RECURRING. To validate a specific environment's configuration, set RAILS_ENV, for example RAILS_ENV=production bin/jobs check.

Lifecycle hooks

In Solid queue, you can hook into two different points in the supervisor's life:

• start: after the supervisor has finished booting and right before it forks workers and dispatchers.
• stop: after receiving a signal (TERM, INT or QUIT) and right before starting graceful or immediate shutdown.

And into two different points in the worker's, dispatcher's and scheduler's life:

• (worker|dispatcher|scheduler)_start: after the worker/dispatcher/scheduler has finished booting and right before it starts the polling loop or loading the recurring schedule.
• (worker|dispatcher|scheduler)_stop: after receiving a signal (TERM, INT or QUIT) and right before starting graceful or immediate shutdown (which is just exit!).

Each of these hooks has an instance of the supervisor/worker/dispatcher/scheduler yielded to the block so that you may read its configuration for logging or metrics reporting purposes.

You can use the following methods with a block to do this:

SolidQueue.on_start
SolidQueue.on_stop

SolidQueue.on_worker_start
SolidQueue.on_worker_stop

SolidQueue.on_dispatcher_start
SolidQueue.on_dispatcher_stop

SolidQueue.on_scheduler_start
SolidQueue.on_scheduler_stop

For example:

SolidQueue.on_start do |supervisor|
  MyMetricsReporter.process_name = supervisor.name

  start_metrics_server
end

SolidQueue.on_stop do |_supervisor|
  stop_metrics_server
end

SolidQueue.on_worker_start do |worker|
  MyMetricsReporter.process_name = worker.name
  MyMetricsReporter.queues = worker.queues.join(',')
end

These can be called several times to add multiple hooks, but it needs to happen before Solid Queue is started. An initializer would be a good place to do this.

Errors when enqueuing

Solid Queue will raise a SolidQueue::Job::EnqueueError for any Active Record errors that happen when enqueuing a job. The reason for not raising ActiveJob::EnqueueError is that this one gets handled by Active Job, causing perform_later to return false and set job.enqueue_error, yielding the job to a block that you need to pass to perform_later. This works very well for your own jobs, but makes failure very hard to handle for jobs enqueued by Rails or other gems, such as Turbo::Streams::BroadcastJob or ActiveStorage::AnalyzeJob, because you don't control the call to perform_later in that cases.

In the case of recurring tasks, if such error is raised when enqueuing the job corresponding to the task, it'll be handled and logged but it won't bubble up.

Concurrency controls

Solid Queue extends Active Job with concurrency controls, that allows you to limit how many jobs of a certain type or with certain arguments can run at the same time. When limited in this way, by default, jobs will be blocked from running, and they'll stay blocked until another job finishes and unblocks them, or after the set expiry time (concurrency limit's duration) elapses.

Alternatively, jobs can be configured to be discarded instead of blocked. This means that if a job with certain arguments has already been enqueued, other jobs with the same characteristics (in the same concurrency class) won't be enqueued.

class MyJob < ApplicationJob
  limits_concurrency to: max_concurrent_executions, key: ->(arg1, arg2, *) { ... }, duration: max_interval_to_guarantee_concurrency_limit, group: concurrency_group, on_conflict: on_conflict_behaviour

  # ...

• key is the only required parameter, and it can be a symbol, a string or a proc that receives the job arguments as parameters and will be used to identify the jobs that need to be limited together. If the proc returns an Active Record record, the key will be built from its class name and id.
• to is 1 by default.
• duration is set to SolidQueue.default_concurrency_control_period by default, which itself defaults to 3 minutes, but that you can configure as well.
• group is used to control the concurrency of different job classes together. It defaults to the job class name.
• on_conflict controls behaviour when enqueuing a job that conflicts with the concurrency limits configured. It can be set to one of the following:
  • (default) :block: the job is blocked and is dispatched when another job completes and unblocks it, or when the duration expires.
  • :discard: the job is discarded. When you choose this option, bear in mind that if a job runs and fails to remove the concurrency lock (or semaphore, read below to know more about this), all jobs conflicting with it will be discarded until the interval defined by duration has elapsed.

When a job includes these controls, we'll ensure that, at most, the number of jobs (indicated as to) that yield the same key will be performed concurrently, and this guarantee will last for duration for each job enqueued. Note that there's no guarantee about the order of execution, only about jobs being performed at the same time (overlapping).

The concurrency limits use the concept of semaphores when enqueuing, and work as follows: when a job is enqueued, we check if it specifies concurrency controls. If it does, we check the semaphore for the computed concurrency key. If the semaphore is open, we claim it and we set the job as ready. Ready means it can be picked up by workers for execution. When the job finishes executing (be it successfully or unsuccessfully, resulting in a failed execution), we signal the semaphore and try to unblock the next job with the same key, if any. Unblocking the next job doesn't mean running that job right away, but moving it from blocked to ready. If you're using the discard behaviour for on_conflict, jobs enqueued while the semaphore is closed will be discarded.

Since something can happen that prevents the first job from releasing the semaphore and unblocking the next job (for example, someone pulling a plug in the machine where the worker is running), we have the duration as a failsafe. Jobs that have been blocked for more than duration are candidates to be released, but only as many of them as the concurrency rules allow, as each one would need to go through the semaphore dance check. This means that the duration is not really about the job that's enqueued or being run, it's about the jobs that are blocked waiting, or about the jobs that would get discarded while the semaphore is closed.

It's important to note that after one or more candidate jobs are unblocked (either because a job finishes or because duration expires and a semaphore is released), the duration timer for the still blocked jobs is reset. This happens indirectly via the expiration time of the semaphore, which is updated.

When using discard as the behaviour to handle conflicts, you might have jobs discarded for until the duration interval if something happens and a running job fails to release the semaphore.

For example:

class DeliverAnnouncementToContactJob < ApplicationJob
  limits_concurrency to: 2, key: ->(contact) { contact.account }, duration: 5.minutes

  def perform(contact)
    # ...

Where contact and account are ActiveRecord records. In this case, we'll ensure that at most two jobs of the kind DeliverAnnouncementToContact for the same account will run concurrently. If, for any reason, one of those jobs takes longer than 5 minutes or doesn't release its concurrency lock (signals the semaphore) within 5 minutes of acquiring it, a new job with the same key might gain the lock.

Let's see another example using group:

class Box::MovePostingsByContactToDesignatedBoxJob < ApplicationJob
  limits_concurrency key: ->(contact) { contact }, duration: 15.minutes, group: "ContactActions"

  def perform(contact)
    # ...

class Bundle::RebundlePostingsJob < ApplicationJob
  limits_concurrency key: ->(bundle) { bundle.contact }, duration: 15.minutes, group: "ContactActions"

  def perform(bundle)
    # ...

In this case, if we have a Box::MovePostingsByContactToDesignatedBoxJob job enqueued for a contact record with id 123 and another Bundle::RebundlePostingsJob job enqueued simultaneously for a bundle record that references contact 123, only one of them will be allowed to proceed. The other one will stay blocked until the first one finishes (or 15 minutes pass, whatever happens first).

Note that the duration setting depends indirectly on the value for concurrency_maintenance_interval that you set for your dispatcher(s), as that'd be the frequency with which blocked jobs are checked and unblocked (at which point, only one job per concurrency key, at most, is unblocked). In general, you should set duration in a way that all your jobs would finish well under that duration and think of the concurrency maintenance task as a failsafe in case something goes wrong.

Jobs are unblocked in order of priority but queue order is not taken into account for unblocking jobs. That means that if you have a group of jobs that share a concurrency group but are in different queues, or jobs of the same class that you enqueue in different queues, the queue order you set for a worker is not taken into account when unblocking blocked ones. The reason is that a job that runs unblocks the next one, and the job itself doesn't know about a particular worker's queue order (you could even have different workers with different queue orders), it can only know about priority. Once blocked jobs are unblocked and available for polling, they'll be picked up by a worker following its queue order.

Finally, failed jobs that are automatically or manually retried work in the same way as new jobs that get enqueued: they get in the queue for getting an open semaphore, and whenever they get it, they'll be run. It doesn't matter if they had already gotten an open semaphore in the past.

Scheduled jobs

Jobs set to run in the future (via Active Job's wait or wait_until options) have concurrency limits enforced when they're due, not when they're scheduled. For example, consider this job:

class DeliverAnnouncementToContactJob < ApplicationJob
  limits_concurrency to: 1, key: ->(contact) { contact.account }, duration: 5.minutes

  def perform(contact)
    # ...

If several jobs are enqueued like this:

DeliverAnnouncementToContactJob.set(wait: 10.minutes).perform_later(contact)
DeliverAnnouncementToContactJob.set(wait: 10.minutes).perform_later(contact)
DeliverAnnouncementToContactJob.set(wait: 30.minutes).perform_later(contact)

The 3 jobs will go into the scheduled queue and will wait there until they're due. Then, 10 minutes after, the first two jobs will be enqueued and the second one most likely will be blocked because the first one will be running first. Then, assuming the jobs are fast and finish in a few seconds, when the third job is due, it'll be enqueued normally.

Normally scheduled jobs are enqueued in batches, but with concurrency controls, jobs need to be enqueued one by one. This has an impact on performance, similarly to the impact of concurrency controls in bulk enqueuing. Read below for more details. We generally advise against mixing concurrency controls with waiting/scheduling in the future.

Performance considerations

Concurrency controls introduce significant overhead (blocked executions need to be created and promoted to ready, semaphores need to be created and updated) so you should consider carefully whether you need them. For throttling purposes, where you plan to have limit significantly larger than 1, we encourage relying on a limited number of workers per queue instead. For example:

class ThrottledJob < ApplicationJob
  queue_as :throttled

production:
  workers:
    - queues: throttled
      threads: 1
      polling_interval: 1
    - queues: default
      threads: 5
      polling_interval: 0.1
      processes: 3

Or something similar to that depending on your setup. You can also assign a different queue to a job on the moment of enqueuing so you can decide whether to enqueue a job in the throttled queue or another queue depending on the arguments, or pass a block to queue_as as explained here.

In addition, mixing concurrency controls with bulk enqueuing (Active Job's perform_all_later) is not a good idea because concurrency controlled job needs to be enqueued one by one to ensure concurrency limits are respected, so you lose all the benefits of bulk enqueuing.

When jobs that have concurrency controls and on_conflict: :discard are enqueued in bulk, the ones that fail to be enqueued and are discarded would have successfully_enqueued set to false. The total count of jobs enqueued returned by perform_all_later will exclude these jobs as expected.

Failed jobs and retries

Solid Queue doesn't include any automatic retry mechanism, it relies on Active Job for this. Jobs that fail will be kept in the system, and a failed execution (a record in the solid_queue_failed_executions table) will be created for these. The job will stay there until manually discarded or re-enqueued. You can do this in a console as:

failed_execution = SolidQueue::FailedExecution.find(...) # Find the failed execution related to your job
failed_execution.error # inspect the error

failed_execution.retry # This will re-enqueue the job as if it was enqueued for the first time
failed_execution.discard # This will delete the job from the system

Error reporting on jobs

Some error tracking services that integrate with Rails, such as Sentry or Rollbar, hook into Active Job and automatically report not handled errors that happen during job execution. However, if your error tracking system doesn't, or if you need some custom reporting, you can hook into Active Job yourself. A possible way of doing this would be:

# application_job.rb
class ApplicationJob < ActiveJob::Base
  rescue_from(Exception) do |exception|
    Rails.error.report(exception)
    raise exception
  end
end

Note that, you will have to duplicate the above logic on ActionMailer::MailDeliveryJob too. That is because ActionMailer doesn't inherit from ApplicationJob but instead uses ActionMailer::MailDeliveryJob which inherits from ActiveJob::Base.

# application_mailer.rb

class ApplicationMailer < ActionMailer::Base
  ActionMailer::MailDeliveryJob.rescue_from(Exception) do |exception|
    Rails.error.report(exception)
    raise exception
  end
end

Puma plugin

We provide a Puma plugin if you want to run the Solid Queue's supervisor together with Puma and have Puma monitor and manage it. You just need to add

plugin :solid_queue

to your puma.rb configuration.

If you're using Puma in development but you don't want to use Solid Queue in development, make sure you avoid the plugin being used, for example using an environment variable like this:

plugin :solid_queue if ENV["SOLID_QUEUE_IN_PUMA"]

that you set in production only. This is what Rails 8's default Puma config looks like. Otherwise, if you're using Puma in development but not Solid Queue, starting Puma would start also Solid Queue supervisor and it'll most likely fail because it won't be properly configured.

Note: phased restarts are not supported currently because the plugin requires app preloading to work.

Running as a fork or asynchronously

By default, the Puma plugin will fork additional processes for each worker and dispatcher so that they run in different processes. This provides the best isolation and performance, but can have additional memory usage.

Alternatively, workers and dispatchers can be run within the same Puma process(s). To do so just configure the plugin as:

plugin :solid_queue
solid_queue_mode :async

Note that in this case, the processes configuration option will be ignored. See also Fork vs. async mode.

The recommended and default mode is fork. Only use async if you know what you're doing and have strong reasons to

Jobs and transactional integrity

:warning: Having your jobs in the same ACID-compliant database as your application data enables a powerful yet sharp tool: taking advantage of transactional integrity to ensure some action in your app is not committed unless your job is also committed and vice versa, and ensuring that your job won't be enqueued until the transaction within which you're enqueuing it is committed. This can be very powerful and useful, but it can also backfire if you base some of your logic on this behaviour, and in the future, you move to another active job backend, or if you simply move Solid Queue to its own database, and suddenly the behaviour changes under you. Because this can be quite tricky and many people shouldn't need to worry about it, by default Solid Queue is configured in a different database as the main app.

Starting from Rails 8, an option which doesn't rely on this transactional integrity and which Active Job provides is to defer the enqueueing of a job inside an Active Record transaction until that transaction successfully commits. This option can be set via the enqueue_after_transaction_commit class method on the job level and is by default disabled. Either it can be enabled for individual jobs or for all jobs through ApplicationJob:

class ApplicationJob < ActiveJob::Base
  self.enqueue_after_transaction_commit = true
end

Using this option, you can also use Solid Queue in the same database as your app but not rely on transactional integrity.

If you don't set this option but still want to make sure you're not inadvertently relying on transactional integrity, you can make sure that:

• Your jobs relying on specific data are always enqueued on after_commit callbacks or otherwise from a place where you're certain that whatever data the job will use has been committed to the database before the job is enqueued.

• Or, you configure a different database for Solid Queue, even if it's the same as your app, ensuring that a different connection on the thread handling requests or running jobs for your app will be used to enqueue jobs. For example:

  class ApplicationRecord < ActiveRecord::Base
    self.abstract_class = true
  
    connects_to database: { writing: :primary, reading: :replica }
  

  config.solid_queue.connects_to = { database: { writing: :primary, reading: :replica } }
  

Recurring tasks

Solid Queue supports defining recurring tasks that run at specific times in the future, on a regular basis like cron jobs. These are managed by the scheduler process and are defined in their own configuration file. By default, the file is located in config/recurring.yml, but you can set a different path using the environment variable SOLID_QUEUE_RECURRING_SCHEDULE or by using the --recurring_schedule_file option with bin/jobs, like this:

bin/jobs --recurring_schedule_file=config/schedule.yml

You can completely disable recurring tasks by setting the environment variable SOLID_QUEUE_SKIP_RECURRING=true or by using the --skip-recurring option with bin/jobs.

To run only the scheduler (no workers or dispatchers), set SOLID_QUEUE_ONLY_RECURRING=true or use --only-recurring with bin/jobs.

The configuration itself looks like this:

production:
  a_periodic_job:
    class: MyJob
    args: [ 42, { status: "custom_status" } ]
    schedule: every second
  a_cleanup_task:
    command: "DeletedStuff.clear_all"
    schedule: every day at 9am

Tasks are specified as a hash/dictionary, where the key will be the task's key internally. Each task needs to either have a class, which will be the job class to enqueue, or a command, which will be eval'ed in the context of a job (SolidQueue::RecurringJob) that will be enqueued according to its schedule, in the solid_queue_recurring queue.

Each task needs to have also a schedule, which is parsed using Fugit, so it accepts anything that Fugit accepts as a cron. Schedules can include a time zone (e.g. 0 9 * * * America/New_York or every day at 9am America/New_York). When a schedule doesn't specify one, it's interpreted in the application's configured time zone (config.time_zone) by default. You can change or disable this default with config.solid_queue.time_zone; setting it to nil falls back to the system's local time.

You can optionally supply the following for each task:

• args: the arguments to be passed to the job, as a single argument, a hash, or an array of arguments that can also include kwargs as the last element in the array.

The job in the example configuration above will be enqueued every second as:

MyJob.perform_later(42, status: "custom_status")

• queue: a different queue to be used when enqueuing the job. If none, the queue set up for the job class.

• priority: a numeric priority value to be used when enqueuing the job.

Tasks are enqueued at their corresponding times by the scheduler, and each task schedules the next one. This is pretty much inspired by what GoodJob does.

For recurring tasks defined as a command, you can also change the job class that runs them as follows:

Rails.application.config.after_initialize do # or to_prepare
  SolidQueue::RecurringTask.default_job_class = MyRecurringCommandJob
end

It's possible to run multiple schedulers with the same recurring_tasks configuration, for example, if you have multiple servers for redundancy, and you run the scheduler in more than one of them. To avoid enqueuing duplicate tasks at the same time, an entry in a new solid_queue_recurring_executions table is created in the same transaction as the job is enqueued. This table has a unique index on task_key and run_at, ensuring only one entry per task per time will be created. This only works if you have preserve_finished_jobs set to true (the default), and the guarantee applies as long as you keep the jobs around.

Note: a single recurring schedule is supported, so you can have multiple schedulers using the same schedule, but not multiple schedulers using different configurations.

Finally, it's possible to configure jobs that aren't handled by Solid Queue. That is, you can have a job like this in your app:

class MyResqueJob < ApplicationJob
  self.queue_adapter = :resque

  def perform(arg)
    # ..
  end
end

You can still configure this in Solid Queue:

my_periodic_resque_job:
  class: MyResqueJob
  args: 22
  schedule: "*/5 * * * *"

and the job will be enqueued via perform_later so it'll run in Resque. However, in this case we won't track any solid_queue_recurring_execution record for it and there won't be any guarantees that the job is enqueued only once each time.

Scheduling and unscheduling recurring tasks dynamically

You can schedule and unschedule recurring tasks at runtime, without editing the configuration file. To enable this, you need to set dynamic_tasks_enabled: true in the scheduler section of your config/queue.yml, as explained earlier.

scheduler:
  dynamic_tasks_enabled: true

Then you can use the following methods to add recurring tasks dynamically:

SolidQueue.schedule_recurring_task(
  "my_dynamic_task",
  class: "MyJob",
  args: [1, 2],
  schedule: "every 10 minutes"
)

This accepts the same options as the YAML configuration: class, args, command, schedule, queue, priority, and description.

To remove a dynamically scheduled task:

SolidQueue.unschedule_recurring_task("my_dynamic_task")

Only dynamic tasks can be unscheduled at runtime. Attempting to unschedule a static task (defined in config/recurring.yml) will raise an ActiveRecord::RecordNotFound error.

To update an existing dynamic task, unschedule it and then schedule it again with the new options. A running scheduler only detects dynamic tasks being created and deleted, so updating a SolidQueue::RecurringTask record in place (for example, changing its schedule with update!) won't be picked up until the scheduler restarts.

Tasks scheduled like this persist between Solid Queue's restarts and won't stop running until you manually unschedule them.

Inspiration

Solid Queue has been inspired by resque and GoodJob. We recommend checking out these projects as they're great examples from which we've learnt a lot.

License

The gem is available as open source under the terms of the MIT License.