Importer: Installation & Configuration

Configuration Files

The Importer uses the following configuration files:

• application.yml: This is the main configuration file.
• sophora-importer-<version>.conf (optional): Used to set the JVM properties (JAVA_OPTS) such as heap size.
• loader.properties (optional): Used for adding the contents of a folder to the classpath of the Importer.
• logback-spring.xml: Logging configuration.

The Importer needs to be restarted for changes in any of these configuration files to take effect.

Installation

The Importer is a Spring Boot app and expects to find the configuration files next to the jar file, so we recommend to put the Importer application jar and the configuration files into the same directory. The name of the sophora-importer.conf file must exactly match the name of the jar file, i.e., if the name of the jar file includes the version number, the name of the conf file must include the version number as well.

Recommended directory structure:

/cms
    /sophora-importer
        /additionalLibs
        /groovy
        /logs
        application.yml
        sophora-importer-4.0.0.conf
        sophora-importer-4.0.0.jar
        loader.properties
        logback-spring.xml

Our Maven repository contains two files suitable for deploying the Importer:

• com.subshell.sophora.importer-<VERSION>-executable.jar
• com.subshell.sophora.importer-4.0.0-SNAPSHOT-bin.tar.gz

The executable jar is basically the Importer application without any configuration files. It is suitable when deploying the Importer using Ansible, Puppet or similar configuration management tools. The bin.tar.gz contains the executable jar as well as sample configuration files. Use this for manual deployments to get started quickly.

The "xsl" folder from the bin.tar.gz contains exemplary XSL files. These are examples for transforming XML files into Sophora XML using XSL (see section XSL Transformation Before Importing).

Starting and Stopping

The Importer jar file is a Spring Boot executable jar. On Linux and MacOS, the Importer can be started by running the executable jar as follows:

./sophora-importer-<version>.jar run

For running the Importer as a background daemon, use the options start and stop.

More options are documented in the Spring Boot documentation.

If required configuration files lack some required configuration options, the Importer cannot be started. The log file then contains information on why the startup has been aborted.

Adding jar files to the classpath

Jar files with additional classes for preprocessing etc. can be added to the classpath with the following entry in the file loader.properties:

# Loads resources (.class files etc.) from nested jar files in directories.
# Should contain comma-separated list of directories, archives, or directories within archives
# (e.g. lib,${HOME}/app/lib, earlier entries take precedence).
loader.path=additionalLibs

JAVA_OPTS

Options for the Java VM, such as heap size, can be set using the environment variable JAVA_OPTS or using an entry in the sophora-importer.conf file. For example:

JAVA_OPTS="-Xmx1G"

Management-Endpoints / Actuators

The Importer exposes a few HTTP endpoints for management and metrics. These are available at the same HTTP port as the SOAP web service. Access to the management endpoints is also using the authentication settings for the web service. Notable endpoints are:

• /actuator/health
• /actuator/jolokia
• /actuator/prometheus
• /actuator/sophora-server

Configuration using the application.yml

Within the importer process one or more importer instances run. Some configuration options are specific for each instance. For example, each instance has its own watch folder. So you can, for example, run an importer process with one instance responsible for video imports, one instance responsible for image imports from an image database and another instance responsible for live ticker imports.

Options for the connection to the Sophora Server

Options for the embedded HTTP server

Global options

Some of these options can be overridden for each instance.

Options for each instance

Each importer instance has a set of configuration options. For some options, it is possible to set defaults in the global options, which can then be overridden for a particular instance.

Instances are configured like this in the application.yml:

importer:
  instances:
    - name: Common Imports
      key: common
      folders:
        watch: /cms/data/import/incoming
      ...
    - name: Image importer
      key: images
      ...

Example application.yml:

sophora:
  client:
    server-connection:
      urls:
        - http://localhost:1196
      username: alice
      password: secret
      retries: 100
      retry-interval: 10

importer:
  name: Demo-Importer

  # JMX
  rmiServicePort: 5000
  rmiRegistryPort: 5001
  jmxLogin: importerjmx
  jmxPassword: password

  # Defaults for all instances.
  folders:
    watchCheckInterval: 1000
    watchRecursive: true

  webService:
    enabled: true
    authenticationRequired: true
    defaultInstance: common
    logins:
      admin: xxx

  filenamesAddTimestamp: false

  cleanupFoldersCron: "0 0 9 ? * * *"
  cleanupFoldersSuccessfulMaxAge: 90
  cleanupFoldersFailureMaxAge: 90

  # Configuration of the importer instances
  instances:
    - name: Common Imports
      key: common
      transform: skipTransform
      folders:
        watch: /cms/data/import/incoming
        temp: /cms/data/import/temp
        success: /cms/data/import/success
        failure: /cms/data/import/failure
        xsl: /cms/sophora-importer/xsl/
      defaultStructureNode: /import

server:
  port: 8081
  address: 0.0.0.0

logback-spring.xml (optional)

This optional configuration file defines the Importer's logging behaviour. The Importer does not need to be restarted for changes in the file to take effect, if the root attribute scanPeriod is set in the logging configuration file.

Introductory information about logback and its configuration can be found here: http://logback.qos.ch/

If you like to enable separate logging for each importer instance, you can use this exemplary configuration file and remove the comments at the two marked locations. If you do so, the importer will create the following log files:

• sophora-importer.log: the default log file with all information. This can be disabled by removing the "FILE" appender.
• sophora-importer_instance-main.log: the log file which contains all information, that can't be assigned to one specific instance.
• sophora-importer_instance-<number>.log: One log file for each configured instance, only showing instance specific information.

Exemple logback-spring.xml file:

<?xml version="1.0" encoding="UTF-8"?>

<!-- For more information on logback logging see: http://logback.qos.ch/manual/index.html -->
<configuration scan="true" scanPeriod="10 seconds">
	<jmxConfigurator/>

	<!-- Name to be shown in the subject of email notifications. -->
	<property name="IMPORTER_NAME" value="Test-Importer" />
	<!-- Logging-Event-Class ("ERROR", "INFO" etc.) for email subjects. -->
	<property name="LOGGING_EVENT_CLASS" value="%-5p" />
	<!-- Logging pattern: 'importerInstanceName' and 'sourceFileName' are references to MDC properties in the importer code. -->
	<property name="APPENDER_PATTERN"
			  value="%d{dd.MM.yyyy HH:mm:ss} %5level [%12.12thread] [%X{importerInstanceName}: %X{feedName} %X{sourceFileName}] %.40(%logger{0}:%L) --- %msg%n%ex"/>

	<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
		<filter class="ch.qos.logback.core.filter.EvaluatorFilter">
			<evaluator>
				<!-- No log messages marked as 'SPECIAL_EMAIL_NOTIFICATION' should be shown. -->
				<expression>marker != null &amp;&amp; marker.getName().equals("SPECIAL_EMAIL_NOTIFICATION")</expression>
			</evaluator>
			<OnMismatch>NEUTRAL</OnMismatch>
			<OnMatch>DENY</OnMatch>
		</filter>
		<encoder>
			<pattern>${APPENDER_PATTERN}</pattern>
		</encoder>
	</appender>

	<!-- Separate log files for each instance -->
	<appender name="INSTANCES" class="ch.qos.logback.classic.sift.SiftingAppender">
		<discriminator>
			<key>importerInstanceKey</key>
			<defaultValue>main</defaultValue>
		</discriminator>
		<sift>
			<appender name="INSTANCE-${importerInstanceKey}" class="ch.qos.logback.core.FileAppender">
				<filter class="ch.qos.logback.core.filter.EvaluatorFilter">
					<evaluator>
						<expression>marker != null &amp;&amp; marker.getName().equals("SPECIAL_EMAIL_NOTIFICATION")</expression>
					</evaluator>
					<OnMismatch>NEUTRAL</OnMismatch>
					<OnMatch>DENY</OnMatch>
				</filter>

				<File>logs/sophora-importer-instance-${importerInstanceKey}.log</File>

				<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
					<fileNamePattern>logs/sophora-importer-instance-${importerInstanceKey}.%d{yyyy-MM-dd}.log</fileNamePattern>
					<maxHistory>7</maxHistory>
				</rollingPolicy>
				<encoder>
					<pattern>${APPENDER_PATTERN}</pattern>
				</encoder>
			</appender>
		</sift>
	</appender>

	<appender name="LOGFILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
		<filter class="ch.qos.logback.core.filter.EvaluatorFilter">
			<evaluator>
				<!-- No log messages marked as 'SPECIAL_EMAIL_NOTIFICATION' should be shown. -->
				<expression>marker != null &amp;&amp; marker.getName().equals("SPECIAL_EMAIL_NOTIFICATION")</expression>
			</evaluator>
			<OnMismatch>NEUTRAL</OnMismatch>
			<OnMatch>DENY</OnMatch>
		</filter>

		<File>logs/sophora-importer.log</File>
		<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
			<fileNamePattern>logs/sophora-importer.%d{yyyy-MM-dd}.log</fileNamePattern>
			<maxHistory>7</maxHistory>
		</rollingPolicy>
		<encoder>
			<pattern>${APPENDER_PATTERN}</pattern>
		</encoder>
	</appender>

	<!-- For more information on logback email logging see: http://logback.qos.ch/manual/appenders.html#SMTPAppender -->
	<appender name="EMAIL" class="ch.qos.logback.classic.net.SMTPAppender">
		<evaluator class="ch.qos.logback.classic.boolex.OnMarkerEvaluator">
			<marker>EMAIL_NOTIFICATION</marker>
			<marker>SPECIAL_EMAIL_NOTIFICATION</marker>
		</evaluator>
		<SMTPHost>smtp.host.de</SMTPHost>
		<Username>xxx@yourmail.com</Username>
		<Password>your_password</Password>
		<To>importererror@yourcompany.com</To>
		<From>importer@yourcompany.com</From>
		<Subject>${LOGGING_EVENT_CLASS}: Importer '${IMPORTER_NAME}', Instanz '%X{importerInstanceName}'</Subject>
		<layout class="ch.qos.logback.classic.PatternLayout">
			<Pattern>${APPENDER_PATTERN}</Pattern>
		</layout>
		<CyclicBufferTracker class="ch.qos.logback.core.spi.CyclicBufferTracker">
			<!-- Send just one log entry per email. -->
			<BufferSize>1</BufferSize>
		</CyclicBufferTracker>
		<!-- Encoding of the email. -->
		<CharsetEncoding>ISO-8859-1</CharsetEncoding>
	</appender>

	<logger name="com.subshell.sophora" level="INFO"/>
	<logger name="org.springframework.boot" level="INFO"/>

	<root level="WARN">
		<appender-ref ref="LOGFILE" />
		<!-- Remove comment if you want log to the console.
		<appender-ref ref="STDOUT" />
		-->
		<!-- Remove comment if you want to have separate log files for each importer instance.
		<appender-ref ref="INSTANCES" />
		-->
		<!-- Remove comment if you want to have email notifications on particular importer errors.
		<appender-ref ref="EMAIL" />
		-->
	</root>
</configuration>

Binary property names for old versions of Sophora XML (optional)

Binary content within Sophora is modeled as a special property which needs to be treated separately in Sophora XML: First, you have to declare properties as binary explicitly. Next, there must be a mapping of the properties' names and the according mimetypes. This assignment is done in the application.yml file as shown in the following example:

# Provides a default mapping between binary property name and mimetype name.
# This is needed by the importer to successfully import documents with Sophora-XML <= 1.6 if custom binary properties are used.
importer:
  binaryProperties:
    'sophora:binarydata': 'sophora:mimetype'
    'sophora-extension:binarydata': 'sophora:mimetype'
    'core:binarydata': 'core:mimetype'

Each element in the map assigns a binary property to a mimetype property.

Properties in the Sophora XML that match one of the keys in this map are interpreted as binary. There must be another property on the same level of the Sophora XML that matches the value of the binary properties map. For example:

<childNodes>
 <childNode nodeType="sophora-extension-nt:imagedata" name="sophora-extension:imagedata">
 <properties>
 <property name="sophora-extension:binarydata">
 <value>olympiapeking_crowd.jpg</value>
 </property>
 <property name="sophora:mimetype">
 <value>image/jpeg</value>
 </property>
 </properties>
 <childNodes/>
 <resourceList/>
 </childNode>
 </childNodes>

If the binary properties options are not configured, a default configuration will be applied. If a binary property is not covered by the standard configuration, it will be imported nonetheless. Its mimetype is then identified with the help of the file ending. In that case, the Importer will write a warning to the log saying that a binary property may not have been imported correctly.