Importer: Installation & Configuration

Installation

To install the Importer you simply have to unpack the archive file "com.subshell.sophora.importer-sophoraversion.tar.gz". The resulting folder contains the starting scripts and the following subdirectories:

• "lib" which comprises the required Java libraries,
• "config" for the configuration files and
• "xsl" with exemplary XSL files. These files are needed, if the Importer should also process XML files that do not match the Sophora XML format (see section XSL Transformation Before Importing).

Configuration

The configuration files have to be located in the "config" directory. This folder needs to be on the same level as the Importer's starting script.

In the following sections the individual configuration files are described in detail. A ZIP file containing typical examples is attached here and can be downloaded: importer_conf.zip

sophora-importer.properties / sophora-importer_instance-NNN.properties (mandatory)

Within the importer process one or more importer instances run; each of them in an individual thread. Every importer instance can have its own watch folder, its own XSL transformation and so forth. So you can, for example, run an importer process with one instance responsible for video imports, one instance responsible for image imports from an image database and another instance responsible for live ticker imports.

There are some properties which are shared between all instances and therefore are declared in the mandatory base configuration file sophora-importer.properties. See subsection Properties in the base configuration file 'sophora-importer.properties' for further explanations about the settings in this file.

Other properties can (or must) be configured individually for any importer instance and therefore are declared in the instance property configuration files sophora-importer_instance-NNN.properties - see subsection Properties in the instance configuration file(s) 'sophora-importer_instance-NNN.properties'. NNN is a positive integer number. The numbering must begin with 1 and must be serial. If you want to have 3 importer instances, for example, you must provide the following configuration files:

• sophora-importer.properties
• sophora-importer_instance-1.properties
• sophora-importer_instance-2.properties
• sophora-importer_instance-3.properties

Properties in the Base Configuration File 'sophora-importer.properties'

Exemplary file: sophora-importer.properties

Properties which are marked as "Also configurable in instance configurations" in the following two tables, illustrated with a symbol, may alternatively (or exclusively) be defined in instance configuration files (see next section). In this case they override the value of the base configuration property (if the property is set in the instance configuration at all). Properties marked with a symbol cannot be overwritten in instance configuration files.

The subsequent table displays the required parameters in the base configuration file sophora-importer.properties.

Property	Description	Also configurable in instance configurations
sophora.importer.name	The Importer's name to be used for JMX.
sophora.contentmanager. serviceUrl	The address (RMI or HTTP) to connect with the content manager (e.g. rmi://demo.de:1199/ContentManager). Note: There is one connection which is shared among all importer instances of the importer process.
sophora.contentmanager.username	Username to access Sophora's content manager.
sophora.contentmanager. password	Password to access Sophora's content manager.
sophora.rmi.servicePort	Internal RMI port for the JMX communication.
sophora.rmi.registryPort	RMI port for external MBean requests.
sophora.importer. maximumImportsToKeep	Number of import operations to memorise. (This number has nothing to do with the amount of import log files that may be located in the success and failure directories.)
sophora.importer. minimumFailedImportsToKeep	The minimum number of failed imports that should be stored. (This number has nothing to do with the amount of import log files that may be located in the success and failure directories.)
sophora.importer.keepTempfiles	Determines whether to keep the temporary files after the Importer finishes. If the value is true, these files are moved to the success or failure directory together with the XML files.

The subsequent table shows optional parameters in the base configuration file sophora-importer.properties.

Property	Description	Also configurable in instance configurations
sophora.contentmanager.proxyHost	URL of the used proxy host, e.g. http://www.proxy.org.
sophora.contentmanager.proxyPort	Port of the used proxy host, e.g. 8181.
sophora.contentmanager.proxyUsername	Username to access the used proxy.
sophora.contentmanager.proxyPassword	Password to access the used proxy.
sophora.contentmanager.connectRetries	If a connection to the Sophora server is not possible try again a few times.
sophora.contentmanager.connectRetryInterval	The time in seconds to wait between connection attempts.
sophora.contentmanager.documentCacheSize	The size of the document cache. Default is 50. If you apply a transformation or a preprocessor that frequently accesses different existing documents from the Sophora server, you may want to increase this cache. Consider the increased memory footprint and assign more memory to the importer if necessary.
sophora.contentmanager.publishedDocumentCacheSize	The size of the published document cache. Default is 50. Similar to sophora.contentmanager.documentCacheSize, except that this value only considers the published versions of documents. If you retrieve the published version of documents in a transformation or a preprocessor, this is the value you may want to adjust.
sophora.contentmanager.migrationMode	Enables the migration mode when accessing the repository. When migration mode is switched on it is possible to set explicitly the last modification date, the last modifier and the sophora id of the document, which normally is not possible. Additionally the last modification date, the last modifier, the version date and the publisher can be controlled when a new version of the document is made (e.g. the document is published via a "publish" instruction); default value is false.
sophora.jmx.username	Username for the JMX connection.
sophora.jmx.password	Password for the JMX connection.
sophora.importer.jolokia.port	The Port for the jolokia JMX adapter service.
sophora.importer.webservice.active	Enables or disables the SOAP webservice interface. Possible values are true and false; default value is false.
sophora.importer.webservice.authentication.active	Enables basic authentication for the SOAP webservice interface. Possible values are true and false; default value is false. The users for the authentication can be configured in the file webservice_users.json.
sophora.importer.webservice.baseAddress	Base address including protocol, hostname and port where the webservice should be published if set to active. The WSDL description of the service can then be retrieved with the URL "/importService.wsdl", for example: http://localhost:8081/importService.wsdl
sophora.importer.webservice.defaultInstance	Sets default instance for the webservice interface. The default value is '1'. This property is used by all webservice methods, which do not have the string 'ToInstance' in its method name: importXml(...) importXmlWithBinaries(...) importXmlByReference(...) importXmlByReferenceWithBinaries(...)
sophora.importer.proxy.host	A proxy configuration is needed if the importer operates behind a proxy and the Import XML is passed to the webservice as a remote URL, the Import XML refers to binary files via http or https or feeds are importet. Example: http://www.proxy.org.
sophora.importer.proxy.port	Port of the used proxy host, e.g. 8181.
sophora.importer.proxy.user	Optional username to access the used proxy.
sophora.importer.proxy.password	Password to access the used proxy. If a username is set a password has to be set as well.
sophora.importer.filenames.addTimestamp	Determines whether a timestamp is attached to the names of the files that are imported and to the names of the temporary files. Default value is true.
sophora.importer.disableImport	This is for test purposes only (e.g. if you want to check the XSL transformation): The importer won't try to send XML files to the content manager if the value of this property is set to true. Values: true or false; default value is false.
sophora.importer.feedpolling.active	Set to true for polling configured feeds. Default is false.
sophora.importer.directory.feedpolling.data	Directory for saving data regarding the polling of feeds (e.g. last processed feed item per feed). If feed polling is active and no directory is given, a folder named feedpolling is created next to the properties-file of the importer.
sophora.importer.httpSoTimeout	Determines the timeout when accessing feeds or making other http requests (for example downloading images). The default is 30000 milli seconds.
sophora.importer.additionalClasspath	Additional classpath directory for the importer process. By using this property and putting client specific jars and classpath resources in the correspondent directory, it's not necessary to put these files in the common library folder of the importer. This has the advantage that you do not have to copy your client files to the common library folder of the importer again when upgrading to a new version of the importer.
sophora.importer.spring.additionalBasePackages	Additional packages which spring should scan when starting the importer. By using this property and putting client specific jars in the classpath of the importer you can use spring functionality in your importer specific code. If you want to specify more than one package you can do this by separating the different packages with commas, e.g.: sophora.spring.additionalBasePackages=com.my.package1,com.my.package2
sophora.client.dataDir	Defines a directory which may be used by the Sophora Client Api for persisting information like the available nodes in a cluster. The directory must be specified over an absolute path.
sophora.importer.cleanupFolders.cron	A cron expression that specifies when the "successful" and "failure" folders will be cleaned up. The expression uses the format of the Quartz CronTrigger. See also Automatically deleting old files.

Properties in the Instance Configuration File(s) 'sophora-importer_instance-NNN.properties'

Exemplary file: sophora-importer_instance-1.properties

Properties which are marked as "Also configurable in base configuration" in the following two tables, depicted with a symbol, may additionally (or exclusively) be set in the base configuration file sophora-importer.properties (see preceding section). In this case there is a default value for this property in the base configuraion which can be overwritten individually in each instance configuration file. Properties marked with a symbol are not part of the base configuration and thus can only be defined in here.

The subsequent table displays the required parameters in the instance configuration files sophora-importer_instance-NNN.properties.

Property	Description	Also configurable in base configuration
sophora.importer.directory.watchfolder	The import directory that the importer instance monitors. Only files with the ending .xml will be processed.
sophora.importer.watchfolder.checkInterval	Interval (in milliseconds) to check the import directory (watch folder); e.g. 30000.
sophora.importer.directory.successful	Target directory to move the XML files to, if the import process finished successfully. This property allows to use patterns within the given path in the form of ${pattern}. Supported patterns: ${date;<DateFormat>} - the date of the import in the given format. For supported date formats see https://docs.oracle.com/javase/8/docs/api/java/text/SimpleDateFormat.html. ${<xslParameter>} - XSL parameter keys given in feed imports or imports by web service.
sophora.importer.directory.failure	Target directory to move the XML files to, if the import process failed. This property allows to use patterns within the given path. For supported patterns see above.
sophora.importer.directory.temp	Directory to save temporary files in, which the importer instance produces.

The subsequent table shows optional parameters in the instance configuration files sophora-importer_instance-NNN.properties.

Property	Description	Also configurable in base configuration
sophora.importer.instance.name	The name of the particular importer instance to be used for JMX and logging purposes.
sophora.importer.instance. webservice.enabled	Enables or disables webservice for according instance. true or false; default value is true. Note that this property only has an effect if the webservice is activated in general (see sophora.importer.webservice.active).
sophora.importer.fileaccess.basedir	This optional property determines a directory which can be additionally accessed (recursively) during the import process. That means on one hand that you can use references to binary files in the sophora xml document which point to files within this folder (or its subfolders etc.). On the other hand it allows the webservice to read files in the specified directory (or its subfolders etc.). When the property is not configured, the webservice is not allowed to access any local files. It affects the possible URIs in the following webservice methods: importXmlByReference(URI, String) importXmlByReferenceToInstance(URI, String, int) importXmlByReferenceWithBinaries(URI, String, List) importXmlByReferenceWithBinariesToInstance(URI, String, List, int)
sophora.importer.watchfolder. includeSubfolder	If set to true all subfolders (and their subfolders etc.) of sophora.importer.directory.watchfolder are included when watching for incoming Sophora-XML files. Make sure that no system folders (like sophora.importer.directory.successful or sophora.importer.directory.failure) are configured as subfolders of sophora.importer.directory.watchfolder if this paramter is set to true. Default value is false. The importer instance executes the individual document imports in lexicographical order based on the relative paths of the documents; i.e. an incoming file subfolder-A/import.xml is handled before a file subfolder-B/import.xml.
sophora.importer.watchfolder. regex.filesToImport	This regular expression determines which files in the watch folder are processed by the importer. The default value is a regular expression matching all file names which end with '.xml', but do not end with '.config.xml' or '.bin.xml'. If you change this regular expression be careful that files ending with '.config.xml' and '.bin.xml' are still ignored because these file endings are produced when sophora documents including xml binary data or node type configurations are exported. (Hint: The regular expression, which is used for a watch folder instance, is printed out in the log file when the importer is started.)
sophora.importer.directory.xsl	The directory where the XSL files are located for the importer instance (see section XSL Transformation Before Importing). This property may only be omitted, if the sophora.importer.transformationMode property has the value skipTransform (see below).
sophora.importer.transformationMode	Defines the XSL transformation mode for the importer instance. The following values are valid: transformIfNotSophoraXml (Default): A XSL transformation will be performed, if the source XML file does not contain valid Sophora-XML. forceTransform: Always accomplish a XSL transformation before importing (independent from the validity of the source  XML file). skipTransform: Never execute a XSL transformation. If a XML file does not contain valid Sophora-XML, its import will fail.
sophora.importer. xslTransformerFactory	The classname of the XSL transformer factory, which is used for XSL transformations (see section Using a Custom XSL Transformer). The default value is org.apache.xalan.xsltc.trax.TransformerFactoryImpl.
sophora.importer.transformation. repairXml	Replace XML entities before performing a XSL transformation. Values: true or false; default value is false. Attention: This parameter should only be used, if it can be assured that the result is well-formed XML.
sophora.importer.defaultSite	The site to import the documents to. This parameter is only considered, if the XML neither contains an empty <site> nor empty <structureNode> tag (see Composition of the Import XML), the import operation is not an update of an existing document and the site mapping couldn't be retrieved from the file's name (see site mappings).
sophora.importer. defaultStructureNode	The structure node to import the documents to. This parameter is only considered, if the <structureNode> element in the XML is empty (see Composition of the Import XML), the import operation is not an update of an existing document and no default structure node from the site mapping is used (see site mappings).
sophora.importer.validate.documents	Defines whether documents are validated or not. By default documents are validated (value true). The validation should at most be disabled in very special situations (and should be activated afterwards!) - this might, for example, be a migration scenario from sophora to sophora where you have to migrate documents which lack a quite recently added mandatory property. When the validation is disabled (value false) invalid documents can be saved: You can save documents with missing mandatory properties and with property values that don't match the according validation expression - furthermore, results of validation scripts are ignored.
sophora.importer.cleanupFolders.successful.maxAge	When cleaning up the "successful" folder of the instance, files in the folder must be at least this many days old to be deleted. Set to 0 to disable deletion for this instance / folder.
sophora.importer.cleanupFolders.failure.maxAge	When cleaning up the "failure" folder of the instance, files in the folder must be at least this many days old to be deleted. Set to 0 to disable deletion for this instance / folder.

binary-property-names.xml (optional)

Exemplary file: binary-property-names.xml

Binary content within Sophora is modeled as a special property which needs to be treated separately: First, you have to declare properties as binary explicitly. Next, there must be a mapping of the properties' names and the according mimetypes. This assignment is done in the binary-property-names.xml file as shown in the following example:

<?xml version="1.0" encoding="ISO-8859-1"?>
<binaryPropertyNames>
  <binaryProperty name="sophora:binarydata" mimetypeName="sophora:mimetype" />
  <binaryProperty name="sophora-extension:binarydata" mimetypeName="sophora:mimetype" />
  <binaryProperty name="sophora-content:binarydata" mimetypeName="sophora:mimetype" />
</binaryPropertyNames>

Each <binaryProperty> element assigns a binary property to a mimetype.

While importing, those properties that match one of the "name" attributes of the <binaryProperty> elements are interpreted as binary. At the same time there must be another property on the same level of the XML that matches the "mimetypeName" attribute of the corresponding <binaryProperty> element (the reason for this is Sophora's need for the correct mimetype when creating binary data).

An example:
The import XML for an image document with an associated image file may look like the following:

<?xml version="1.0" encoding="ISO-8859-1"?>
<document nodeType="sophora-content-nt:imageobject"
          xmlns="http://www.sophoracms.com/import/2.8"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <properties>
    [...]
  </properties>
  <childNodes>
    <childNode nodeType="sophora-extension-nt:imagedata" name="sophora-extension:imagedata">
      <properties>
        [...]
        <property name="sophora-extension:binarydata">
          <value>olympiapeking_crowd.jpg</value>
        </property>
        <property name="sophora:mimetype">
          <value>image/jpeg</value>
        </property>
      </properties>
      <childNodes/>
      <resourceList/>
    </childNode>
  </childNodes>
  <resourceList/>
  <fields>
    [...]
  </fields>
  <instructions>
    [...]
  </instructions>
</document>

The referenced image file olympiapeking_crowd.jpg has to lie in the same directory as the import XML file. Alternatively, you can specify a relative path like images/import/olympiapeking_crowd.jpg. Due to security reasons it is not possible to reference files in a higher folder hierarchy, or the reference to file via an absolute path.

Default Behaviour

If the binary-property-names.xml is not present, a standard configuration will be applied. This standard configuration is also used, if the Sophora XML version is older than 2.0.

If a binary property is not covered by the standard configuration, it will be imported nonetheless. Its mimetype is then identified with the help of the file ending. In that case, the Importer will emit warnings saying that a mimetype property may not have been imported correctly.

site-mappings.xml (optional)

Exemplary file: site-mappings.xml

It is sometimes reasonable to retrieve the site, in which a document shall be imported, from the file name. Such configuration is done in the site-mappings.xml file. The Importer needs to be restarted for changes in the file to take effect.

<?xml version="1.0" encoding="ISO-8859-1"?>
<siteMappings>
  <siteMapping id="2" displayName="Demo" systemName="demo" structureNodeDefault="/media" ignore="false" />
  <siteMapping id="4" displayName="DemoIntern" systemName="demointern" structureNodeDefault="/media" ignore="false" />
  <siteMapping id="4711" displayName="System" systemName="system" ignore="true" />
</siteMappings>

The element <siteMapping> may have the following attributes (mandatory ones in bold font):

logback.xml (optional)

This optional configuration file defines the Importer's logging behaviour. The Importer does not need to be restarted for changes in the file to take effect, if the root attribute "scanPeriod" is set in the logging configuration file.

Introductory information about logback and its configuration can be found here: http://logback.qos.ch/

If you like to enable separate logging for each importer instance, you can use this exemplary configuration file and remove the comments at the two marked locations. If you do so, the importer will create the following log files:

• sophora-importer.log: the default log file with all information. This can be disabled by removing the "FILE" appender.
• sophora-importer_instance-main.log: the log file which contains all information, that can't be assigned to one specific instance.
• sophora-importer_instance-<number>.log: One log file for each configured instance, only showing instance specific information.

Exemplary logback.xml file:

<?xml version="1.0" encoding="UTF-8"?>

<!-- For more information on logback logging see: http://logback.qos.ch/manual/index.html -->
<configuration scan="true" scanPeriod="30 seconds">
    <jmxConfigurator />
    
    <!-- Name to be shown in the subject of email notifications.-->
    <property name="IMPORTER_NAME" value="Test-Importer" />
    <!-- Logging pattern: 'importerInstanceName' and 'sourceFileName' are references to MDC properties in the importer code. -->
    <property name="APPENDER_PATTERN" value="[%d{ISO8601}, %-5p] [%t] [%X{importerInstanceName}] [%X{sourceFileName}] %.40c:%L: %m%n" />
    <!-- Logging-Event-Class ("ERROR", "INFO" etc.) -->
    <property name="LOGGING_EVENT_CLASS" value="%-5p" />

    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <filter class="ch.qos.logback.core.filter.EvaluatorFilter">      
            <evaluator>
                <!-- No log messages marked as 'SPECIAL_EMAIL_NOTIFICATION' should be shown. -->
                <expression>marker != null &amp;&amp; marker.getName().equals("SPECIAL_EMAIL_NOTIFICATION")</expression>
            </evaluator>
            <OnMismatch>NEUTRAL</OnMismatch>
            <OnMatch>DENY</OnMatch>
        </filter>
        <encoder>
            <pattern>${APPENDER_PATTERN}</pattern>
        </encoder>
    </appender>
    
    <!-- Remove comment if you want to have separate logging for each importer instance.
    <appender name="SIFT" class="ch.qos.logback.classic.sift.SiftingAppender">
        <discriminator>
            <key>importerInstanceIndex</key>
            <defaultValue>main</defaultValue>
        </discriminator>
        <sift>
            <appender name="FILE-${importerInstanceIndex}" class="ch.qos.logback.core.FileAppender">
                <filter class="ch.qos.logback.core.filter.EvaluatorFilter">      
                    <evaluator>
                        <expression>marker != null &amp;&amp; marker.getName().equals("SPECIAL_EMAIL_NOTIFICATION")</expression>
                    </evaluator>
                    <OnMismatch>NEUTRAL</OnMismatch>
                    <OnMatch>DENY</OnMatch>
                </filter>
                
                <File>logs/sophora-importer_instance-${importerInstanceIndex}.log</File>
        
                <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
                    <fileNamePattern>logs/sophora-importer_instance-${importerInstanceIndex}.%d{yyyy-MM-dd}.log</fileNamePattern>
                    <maxHistory>7</maxHistory>
                </rollingPolicy>
                <encoder>
                    <pattern>${APPENDER_PATTERN}</pattern>
                </encoder>
            </appender>
        </sift>
    </appender>
    -->

    <appender name="LOGFILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <filter class="ch.qos.logback.core.filter.EvaluatorFilter">      
            <evaluator>
                <!-- No log messages marked as 'SPECIAL_EMAIL_NOTIFICATION' should be shown. -->
                <expression>marker != null &amp;&amp; marker.getName().equals("SPECIAL_EMAIL_NOTIFICATION")</expression>
            </evaluator>
            <OnMismatch>NEUTRAL</OnMismatch>
            <OnMatch>DENY</OnMatch>
        </filter>
        
        <File>logs/sophora-importer.log</File>

        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>logs/sophora-importer.%d{yyyy-MM-dd}.log</fileNamePattern>
            <maxHistory>7</maxHistory>
        </rollingPolicy>
        <encoder>
            <pattern>${APPENDER_PATTERN}</pattern>
        </encoder>
    </appender>
    
    <!-- For more information on logback email logging see: http://logback.qos.ch/manual/appenders.html#SMTPAppender -->
    <appender name="EMAIL" class="ch.qos.logback.classic.net.SMTPAppender">
        <evaluator class="ch.qos.logback.classic.boolex.OnMarkerEvaluator">
            <marker>EMAIL_NOTIFICATION</marker>
            <marker>SPECIAL_EMAIL_NOTIFICATION</marker>
        </evaluator>
        <SMTPHost>smtp.host.de</SMTPHost>
        <Username>xxx@yourmail.com</Username>
        <Password>your_password</Password>
        <To>importererror@yourcompany.com</To>
        <From>importer@yourcompany.com</From>
        <Subject>${LOGGING_EVENT_CLASS}: Importer '${IMPORTER_NAME}', Instanz '%X{importerInstanceName}'</Subject>
        <layout class="ch.qos.logback.classic.PatternLayout">
            <Pattern>${APPENDER_PATTERN}</Pattern>
        </layout>
        <CyclicBufferTracker class="ch.qos.logback.core.spi.CyclicBufferTracker">
            <!-- Send just one log entry per email. -->
            <BufferSize>1</BufferSize>
        </CyclicBufferTracker>
        <!-- Encoding of the email. -->
        <CharsetEncoding>ISO-8859-1</CharsetEncoding>
    </appender>

    <logger name="com.subshell.sophora" level="INFO" />
    <logger name="de.daserste" level="INFO" />

    <root level="WARN">
        <appender-ref ref="LOGFILE" />
        <!-- Remove comment if you want to have standard out logging.
        <appender-ref ref="STDOUT" />
        -->
        <!-- Remove comment if you want to have separate logging for each importer instance.
        <appender-ref ref="SIFT" />
        -->
        <!-- Remove comment if you want to have email notifications on particular importer errors.
        <appender-ref ref="EMAIL" />
        -->
    </root>
    
</configuration>

webservice_users.json (optional)

This optional configuration file, which has to be located next to sophora-importer.properties file, defines the users that have access to the webservice interface. All users are defined in an JSON file as an array of user objects. Users with blank password or blank user name will be ignored.

[
  {
    "name" : "user1",
    "password" : "password1"
  },
  {
    "name" : "user2",
    "password" : "password2"
  }
]

You have to set the property sophora.importer.webservice.authentication.active to true in the sophora-importer.properties, if you want the user configuration to apply.

Starting and Stopping the Importer

The Importer is started by executing the script sophora-importer.sh with the parameter start.

sophora@cmsserver:~/demo/sophora-importer/audiovideo <514>./sophora-importer.sh start

To stop the Importer prompt the following:

sophora@cmsserver:~/demo/sophora-importer/audiovideo <514>./sophora-importer.sh stop

If required configuration files cannot be detected or if they lack some mandatory parameters (see sophora-importer.properties / sophora-importer_instance-NNN.properties), the Importer cannot be started. The log file then contains information why the startup has been aborted automatically.

Health Check Endpoint

If the Importer's web service is activated, there is an additional REST endpoint /health to check if the Importer is up and running. The response will be the following JSON:

{"status":"UP"}