Purpose
The Sophora server runs several background tasks that need to be dealt with while requests are still coming in from clients such as authors using the Sophora Deskclient or automatic importer processes. If the server is under high load, some of those background tasks can be throttled in order to delay client requests as less as possible. This throttle mode is explained in this page.
Indication of high load
The decision on whether to throttle background tasks or not is based on whether the system considers itself under high load. This is the case if the system's average load throughout the last minute exceeds the high load threshold. This threshold is calculated as:
Number of available processors on all other Sophora Servers
Configuration
This threshold can be adjusted through the property highLoad of the JMX-bean com.subshell.sophora/ContentManager and through the property sophora.cpuload.throttleLoad. Changing this setting via JMX however just affects the running JVM and will not be persisted. In no case will the setting be synchronized to Sophora Sophora Replica Servers.
Monitoring
The throttling can be monitored via the JMX-bean com.subshell.sophora:Metrics/timer.throttledThreads. It exposes statistics about how often the throttling occurs and how long tasks are throttled. Additionally, the number of throttled tasks is logged by the Sophora server every minute while throttling is active.
Throttling tasks
Background tasks commonly perform iterative tasks. When throttled, after a certain amount of iterations they pause there work for an interval between 200ms and 5s. The exact value depends on the current load - the higher the load, the longer the delay.
Solr indexer queue
Working of the solr indexer queue may switch to throttle mode after each handled document
Tasks for filling the Solr indexer queue
Changing select values or structure nodes might require re-indexing several documents. Background tasks determine which documents exactly are involved before adding them to the queue. These tasks check for whether switching to throttle mode after 200 handled documents.
Thumbnail pregeneration
The thumbnail pregeneration task running on servers of type Sophora Primary Server and Sophora Replica Server is queueing steps on a per document basis. The thread reading from this queue is going to throttle after every document handled.
The throttle mode however does not affect the generation of thumbnails on demand, which is performed on a per-document basis anyway.
DeletedDocumentWorker
The task for removing documents marked as deleted is throttled unter high load after each document removed from the repository.
Other tasks checking for high load
There are some background tasks that use the above indication of high load but come with their own configurable delay.
ArchiveWorker
The archive worker has a configurable delay (at least 100ms), which is taken into account after each document added to or removed from the archive.
Limitations in a cloud setup
The underlying mechanics of determining the load will lead to undesired results in a cloud setup e.g. if your Sophora Servers are running in Kubernetes.
The used method OperatingSystemMXBean::getSystemLoadAverage will measure the load of the host machine on which the container is running. This will almost always make the server assume that the system is under high load, triggering the throttle mode constantly.
To prevent this, we propose to set the property sophora.cpuload.throttleLoad to some large value (e.g. 1000) whenever your Sophora Server is running in a container.
