Importer: Parallel Imports

Overview and use cases

By default, the Sophora Importer processes incoming documents sequentially through a single global queue. While sequential processing guarantees strict execution order across all incoming files, high-volume workloads or large bulk imports can delay time-sensitive updates.

Parallel imports let you run independent imports concurrently. Enable this feature for operational scenarios where strict sequential queuing across unrelated files is unnecessary or counterproductive:

• Process high-priority content immediately (such as sports ticker events or breaking news) without waiting behind routine background or batch operations.
• Isolate high-volume imports so they do not delay independent imports on other instances.
• Segregate independent imports on demand (such as sports leagues or news categories) without requiring configuration changes or operator intervention.

How parallel imports work

You can enable parallel imports individually for each importer instance. When enabled, the Sophora Importer routes incoming imports into separate queues and executes imports from different queues concurrently.

Parallel Imports are supported by Sophora Importer version 6.6.0 and newer.

Instance isolation

• Standard instances (default): Importer instances with parallel imports disabled route imports into a single global queue. All standard instances execute tasks in this shared queue sequentially.
• Parallel-enabled instances: Enabling parallel imports detaches that instance from the shared global queue. The Sophora Importer assigns dedicated queues to the instance.

Multiple parallel queues within an instance

After you enable parallel imports on an instance, you can split incoming imports into separate parallel queues that run concurrently.

• Instance default queue: Serves as the primary queue for a parallel-enabled instance. Any import that does not explicitly specify a sub-queue name automatically routes to this default queue.
• Custom sub-queues: Supplying systems can dynamically target named sub-queues (such as league1 or breaking-news). The importer creates sub-queues automatically on demand and executes them concurrently alongside other sub-queues and the instance default queue.

Sub-queue names have no character restrictions and are case-sensitive. The importer automatically cleans up the executor thread for an idle sub-queue after two hours of inactivity.

Queue selection methods

Supplying systems can route imports to specific sub-queues depending on the import channel:

• REST web service: Include the optional query parameter parallelImportQueueName=<queue_name> in the POST request URL. See the page on importing via webservice (REST).
• Watchfolder import: Prefix the filename with the queue name followed by the configured separator string. For example, league1__match42.xml targets the league1 queue when __ is configured as the separator. See the page on importing via watchfolder.

Note that the sub-queues are specific to the respective instance. Using the same queue name in two instances will result in two concurrent import queues.

Processing rules and ordering

In-queue guarantees

Within a single queue, processing remains single-threaded and behaves identically to standard Sophora Importer deployments:

• REST requests are imported sequentially in arrival order (FIFO).
• Watchfolder files are imported in ascending alphabetical order, with subfolder contents generally processing after files in the root folder.

Cross-queue guarantees and data dependencies

Execution across separate queues or parallel instances occurs simultaneously, with no ordering guarantees between items in different queues.