Importer: Import Via Watchfolder

The import process starts when one or more files are copied to the configured watchfolder of the importer instance.

How filenames control watchfolder imports

File selection

By default, the Sophora Importer handles files ending with '.xml' and ignores files ending with '.config.xml' or '.bin.xml'. To customize file filtering, set the watchFilesRegex configuration option.

If you set the watchRecursive configuration option to true, the importer also processes files in subfolders. Files in subfolders will generally be processed after files in the main folder.

Parallel queue routing

When parallel imports are enabled for an instance and a file-name-queue-separator is configured (for example, __), the Sophora Importer determines the target sub-queue from the filename prefix up to the first occurrence of the separator. For example, a file named league1__match42.xml routes to the league1 queue.

Files that omit or start with the separator route to the instance's default queue.

For detailed information on parallel queues, see Importer: Parallel Imports.

Processing order

By default, the importer processes files in strict ascending alphabetical order, with subfolder contents processing after files in the root directory. This ordering applies to all standard instances as well as within any individual queue on parallel-enabled instances.

Across different queues on parallel-enabled instances, files process concurrently, and the importer does not maintain global alphabetical ordering across different queue prefixes.

Importing a file

Importing a file comprises the following steps:

• The Importer determines the target queue for the input file based on its filename prefix and enqueues it for processing (see Parallel Imports).
• The file is given to the preprocessor script, if configured for this instance.
• The result ist given to the XSL transformation, if configured for this instance.
• The importer then checks whether the result is valid Sophora XML.
• Based on the Sophora XML, documents in the Sophora server will be created or updated.
• When the import is finished, the input file, along with intermediate files created by the preprocessor or XSLT, and additional files referenced from the import file, are moved to the success or failure folder.

Check for incomplete files

The importer uses some heuristics to determine if a file in the watchfolder is incomplete and still being written to by another process. Before importing a file, and if the instance does not use a preprocessor script, the importer checks whether the file is non-empty and contains valid XML. If you are using a preprocessor-script, that script can indicate a parse error by calling errorTracker.setParseError(). If the content is not valid and the modification date of the file is within the last ten seconds, the importer assumes that the file is still being written, and will retry the import after ten seconds.

Every ten seconds, a new attempt of importing the file will be done until one of the following two conditions are met:

• The modification date of the file is older than the date of the last attempt to import the file.
• The file is non-empty and contains syntactically valid XML.

If the file hasn't changed within the last ten seconds and ist still syntactically incorrect, it is no longer regarded as still in the process of being written and the import will ultimately fail. On the other hand, if the content of the file is valid XML or the preprocessor script does not indicate a parse error, the import will proceed.

Success or failure

If the import process fails, the source file will be moved to the failure directory. Additionally, this folder will contain an error protocol file named in the same way as the source file plus a timestamp.

If the import process finishes successfully, the source file is moved to the success directory. During a successful import there may be minor problems which did not prevent the import. Such problems will be logged to an error protocol file named in the same way as the source file plus a timestamp.

Automatically deleting old files

Old files from the success and failure folders can be deleted automatically using a configuration such as this:

importer:  
  cleanupFoldersCron: "0 0 0 * * ? *"
  cleanupFoldersSuccessfulMaxAge: 20
  cleanupFoldersFailureMaxAge: 20

With this example, the importer deletes all files from the success and failure folders of all importer instances that are older than 20 days. Empty subfolders will also be deleted. This process will happen each day at midnight, as specified by the cron expression.

The max-age properties can be set globally or in each instance. The configuration in the instance overrides the global one. Set the max-age to 0 in an instance configuration to disable deleting old files for this instance.

When this feature is combined with patterns in the configuration of the success and failure folders, there are some considerations. Considering the following importer instance configuration:

folders:
  watch: /foo/incoming
  temp: /foo/temp
  success: /foo/${date;yyyy.MM}/success/
  failure: /foo/${date;yyyy.MM}/failure

This configuration creates a new directory each month, containing the successful and failed imports for each month. When patterns are used, the deletion process will delete old files in folders matching a glob-style path expression like /foo/*/success and /foo/*/failure.
Note that any pattern is expected being resolved to a single-level folder or file which is why a pattern like /foo/${xslDeepPath} might lead to /foo/bar/batz/file.xml but won't be deleted by /foo/* lookup. If someone were to create the file /foo/bar/Readme.txt, it would be deleted after 20 days.
The deletion process explicitly ignores incoming, temp and xsl folders.