Sophora Metadata Supplier for ARD Mediathek/Audiothek Developing Guide

The Metadata Supplier as a link between the source system and the ARD Mediathek/Audiothek does not specify the contents of the resources that will be transferred. For this purpose, custom mappers are used as plugins in the Metadata Supplier application. They define the mapping from source objects to ARD Core API resources by creating Metadata Supplier model objects and filling in all the required (and optional) fields. The finished model objects are then returned to the Metadata Supplier application which will in turn continue with the preparation and uploading process to the ARD Mediathek/Audiothek.

Create a custom mapper plugin

To create a custom mapper plugin, a Java project with the following Maven dependency to the artifact metadata-supplier-mapper has to be created. Note that we provide a metadata-supplier-plugin-example project, which can be used as a starting point for your own custom mapper (GroupId: com.subshell.sophora.metadatasupplier, ArtifactId: metadata-supplier-plugin-example).

We provide helpers for handling Sophora documents and Spring-Data-Sophora entities in the projects metadata-supplier-sophora-commons and metadata-supplier-spring-data-sophora-commons. The example project demonstrates how to check for Sophora external IDs.

<dependencyManagement>
    <dependencies>
        <!-- Metadata Supplier -->
        <dependency>
            <groupId>com.subshell.sophora.metadatasupplier</groupId>
            <artifactId>metadatasupplier-parent</artifactId>
            <version>${sophora.metadatasupplier.version}</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

<dependencies>
    <!-- Sophora Metadata Supplier -->
    <dependency>
        <groupId>com.subshell.sophora.metadatasupplier</groupId>
        <artifactId>metadata-supplier-mapper</artifactId>
        <scope>provided</scope>
    </dependency>
    <!-- Optional Sophora Metadata Supplier Priority API -->
    <dependency>
        <groupId>com.subshell.sophora.metadatasupplier</groupId>
        <artifactId>metadata-supplier-priority-api</artifactId>
        <scope>provided</scope>
    </dependency>
    <!-- Optional helper for Sophora as source system -->
    <dependency>
	    <groupId>com.subshell.sophora.metadatasupplier</groupId>
	    <artifactId>metadata-supplier-sophora-commons</artifactId>
        <scope>provided</scope>
    </dependency>
    <!-- Optional helper for using Spring Data Sophora and Sophora as source system (includes metadata-supplier-sophora-commons) -->
    <dependency>
	    <groupId>com.subshell.sophora.metadatasupplier</groupId>
	    <artifactId>metadata-supplier-spring-data-sophora-commons</artifactId>
        <scope>provided</scope>
    </dependency>
 </dependencies>

Create a launcher project

To start the Metadata Supplier with your custom mapper plugin for development reasons from your IDE you have to use a launcher project. Note that we provide a metadata-supplier-plugin-example-launcher project, which can be used as a starting point (GroupId: com.subshell.sophora.metadatasupplier, ArtifactId: metadata-supplier-plugin-example-launcher).

1. Create a new Maven project that is used as launcher project (e.g. metadata-supplier-with-custom-plugin-launcher) with a dependency to metadata-supplier-application and your custom plugin project created above:

...
<artifactId>metadata-supplier-with-custom-plugin-launcher</artifactId>
...
<dependencies>
	<!-- Metadata Supplier Application -->
	<dependency>
 		<groupId>com.subshell.sophora.metadatasupplier</groupId>
 		<artifactId>metadata-supplier-application</artifactId>
		<!-- The released version of the Metadata Supplier -->
		<version>4.5.0</version>
	</dependency>
	<!-- Your custom Metadata Supplier plugin -->
	<dependency>
 		<groupId>com.mycompany.metadatasupplier</groupId>
		<artifactId>my-custom-metadata-supplier-plugin</artifactId>
		<!-- The development version of your plugin -->
		<version>4.x.y-SNAPSHOT</version>
	</dependency>
</dependencies>
...

2. Create (or copy) an application.yml to the root of your launcher project (next to the pom.xml) and configure it with your settings. See the administration documentation for examples.

3. In Eclipse IDE (IntelliJ IDEA should be similar) create a launch configuration for the launcher project with the following settings:

• Name: Sophora Metadata Supplier with custom plugin (or similar)
• Project: metadata-supplier-with-custom-plugin-launcher (your launcher project)
• Main class: com.subshell.sophora.metadatasupplier.application.Application
• Working Directory: ${workspace_loc:metadata-supplier-with-custom-plugin-launcher} (the launcher project root)

4. Finally run the launch configuration. The Metadata Supplier should start with your settings and your plugin.

Create custom mappers

The Metadata Supplier recognizes a mapper class by looking for two criteria. It needs to be a Spring component (with the @Component annotation) and it has to extend the IResourceMapper interface. This generic interface can be used to create any kind of resource. There are also ways to map specific kinds of resources with the interfaces that already extend from the base interface. An example would be the IShowMapper to map source objects to resources of type IShow. The following sections go into further detail as to how mappers are implemented. We also provide an ExampleShowMapper in the sources JAR file of the metadata-supplier-plugin-example project. It also determines if an unpublish or republish must be done at the ARD Core API based on the isPublished() on mapped objects that support it.

@Component
@Slf4j
public class ExampleShowMapper implements IShowMapper {

  @Override public boolean accept(SourceDescriptor sourceDescriptor, Parameters parameters) {
    // To be implemented
  }
  @Override public List<IShow> mapToResources(SourceDescriptor sourceDescriptor, IMappingContext mappingContext) {
    // To be implemented
  }
}

Implement IResourceMapper

The interface methods of the IResourceMapper will be triggered by the IMetadataSupplier (which is called by events from the source system for example). Before any mapping takes place, the accept(SourceDescriptor, Parameters) method will be called to select the correct mapper for the source object. Only if your custom mapper identifies with the given source descriptor, the mapToResources(SourceDescriptor, IMappingContext) method will be called. In the Sophora environment for example, the SourceDescriptor consists of the UUID and the node type of a Sophora document. So with the accept method you could define if the mapper is suitable to map documents of certain node types.

@Override
public boolean accept(SourceDescriptor sourceDescriptor, Parameters parameters) {
  boolean accepted;
  // For example you can check the type of the source.
  accepted = sourceDescriptor.hasSourceType(ACCEPTED_TYPE_NAME);
  // Sophora users may use the SophoraSourceDescriptor to use Sophora wordings.
  accepted = SophoraSourceDescriptor.isValidForNodeType(sourceDescriptor, ACCEPTED_TYPE_NAME);
  return accepted;
}

In the mapToResources(SourceDescriptor, IMappingContext) method, the mapping from your source objects to the Metadata Supplier model objects takes place. It provides you with the descriptor of the source object that needs to be mapped and an additional parameter of type IMappingContext which holds the Parameters and can be used for further things explained in the following sections. Since the mapper is responsible to retrieve the source object from the source system in order to map it, you may use a mechanism like Spring Data Sophora in the Sophora context. After the mapping process, the finished resource(s) have to be returned as a list, which may be empty if the mapping has failed. The Metadata Supplier automatically sorts the returned resource objects into the correct order for delivery to the ARD Mediathek/Audiothek. It also determines if an unpublish or republish must be done at the ARD Core API based on the isPublished() on mapped objects that support it.

@Override
public List<IShow> mapToResources(SourceDescriptor sourceDescriptor, IMappingContext mappingContext) {
  // The id of the source object is needed to get the source object from the source system.
  String sourceId = sourceDescriptor.getSourceId();
  // Sophora users may use the Sophora wordings of the SophoraSourceDescriptor.
  sourceId = SophoraSourceDescriptor.getExternalId(sourceDescriptor);
  IShowSourceObject myShowSourceObject = mySourceSystem.getShowSourceObject(sourceId);

  ShowBuilder<?, ?> showBuilder = Show.builder()
    .withTitle(myShowSourceObject.getTitle())
    .withExternalId(new ExternalId(sourceId))
    .withPublished(myShowSourceObject.isOnline())
    (...)
  // An image collection containing the added images will be created by the Metadata Supplier.
  for (IImage image : images) {
    showBuilder.withImage(image);
  }

  IShow show = showBuilder.build();
  return Collections.singletonList(show);
}

Unpublish/Republish without fully mapped resources

If the source object is no longer available in the source system to be fully mapped (e.g. it has been deleted) and the associated entity is to be unpublished in the ARD Mediathek/Audiothek, then a resource of type Publishable can be created instead of the full IResource object. This contains only the type, the external id and the instruction to unpublish.

@Override
public List<IResource> mapToResources(SourceDescriptor sourceDescriptor, IMappingContext mappingContext) {
	String sourceId = sourceDescriptor.getSourceId();
	ISeasonSourceObject seasonSourceObject = mySourceSystem.getSeasonSourceObject(sourceId);
	IResource season;
	if (seasonSourceObject == null) {
		season = Publishable.builder()
				.withType(Type.SEASON)
				.withExternalId(ExternalId.of(sourceId))
				.withPublished(false)
				.build();
	} else {
		// Otherwise map the full resource
		season = Season.builder()
				.withExternalId(ExternalId.of(sourceId))
				//.with...
				.build();
	}
	return List.of(season);
}

Unpublish resources that can no longer be mapped from the source object

A source object can be mapped multiple times and therefore create, update and publish several published resources in the ARD Mediathek/Audiothek, e.g. a varying number of publications. However, if the data from which the previous published resources were created is no longer available in the source object, in most cases the resources should be unpublished in the ARD Mediathek/Audiothek, too. To do this, the method IMappingContext.setUnpublishPreviouslyMappedResources(true/false) can be used. If set to true a Publishable.withPublished(false) is added for every publishable resource that meets the following conditions:

• The resource has not been mapped in this mapping operation anymore.
• The resource has been mapped as published in the previous mapping operation of the source.
• The resource has not been mapped as published for any other source objects.

@Override
public List<IResource> mapToResources(SourceDescriptor sourceDescriptor, IMappingContext mappingContext) {
    ...
    List<IResource> resources = new ArrayList<>();
    resources.add(...)

    // Set that all other resources that have been mapped before from source object should be unpublished (if possible)
    mappingContext.setUnpublishPreviouslyMappedResources(true);
    return resources;
}

In addition, IMappingContext.getResourceProvider() gives you access to the data stored by the Metadata Supplier of the last successful mapping of the current source.

Update resources only if they already exist in the ARD Mediathek/Audiothek

To prevent entities from being created when they should not (and do not yet) exist in the ARD Mediathek/Audiothek set the createIfNotExisting flag to false (default is true). In other words: Set createIfNotExisting of a resource to false to update the resource only if it already exists in the ARD Mediathek/Audiothek. In combination with withPublished(false) resources can be unpublished in the ARD Mediathek/Audiothek only if they already exist there (without creating them, only to be able to unpublish them).

// Determine whether the mapped source should exist in the ARD Mediathek/Audiothek (maybe it depends on a flag in the source object that has been switched)
boolean shouldExistInMediathek = ...
// Determine whether the mapped source should be visible in the ARD Mediathek/Audiothek (maybe it depends on the state of the source object that has been switched)
boolean shouldBePublishedInMediathek = ...
Show show = Show.builder()
		.withExternalId(...)
		.withTitle(...)
		//.with...
		.withCreateIfNotExisting(shouldExistInMediathek)
		.withPublished(shouldBePublishedInMediathek)
		.build();

Give (editorial) feedback

The Metadata Supplier provides feedback message codes and messages for standard situations and for technical information to the editors:

• No resource mapper accepts the SourceDescriptor => No feedback message
• Multiple resource mapper accept the SourceDescriptor => No feedback message
• Exactly one resource mapper accepts the SourceDescriptor but provides no IResource => No feedback message
• Exactly one resource mapper accepts the SourceDescriptor and returns exactly one IResource => See the following feedback messages table
• Exactly one resource mapper accepts the SourceDescriptor and returns multiple IResource objects => Combinations of the following messages table

Here is the default German messages.properties for all feedback messages codes:

# Formats
sms.format.datetime=yyyy-MM-dd'T'HH:mm:ssZ
sms.format.date=dd.MM.yyyy
sms.format.time=HH:mm:ss

# Messages
sms.mapping.error=Fehler beim Mapping: ${errorText}

sms.save.error=Fehler beim Versenden von '${externalId}' an die ARD Mediathek/Audiothek: ${errorText} (${resourceDescriptors})
sms.save.success=Folgende Inhalte wurden am ${date} um ${time} Uhr erfolgreich an die ARD Mediathek/Audiothek gesendet: ${resourceDescriptors}

sms.unpublish.error=Fehler beim Depublizieren in der ARD Mediathek/Audiothek (${resourceDescriptors})
sms.unpublish.success=Erfolgreich in der ARD Mediathek/Audiothek depubliziert (${resourceDescriptors})
sms.unpublish.unable=Ein Depublizieren in der ARD Mediathek/Audiothek war nicht m\u00F6glich (${resourceDescriptors})
sms.unpublish.unnecessary=Ein (erneutes) Depublizieren in der ARD Mediathek/Audiothek war nicht notwendig (${resourceDescriptors})

sms.republish.error=Fehler beim Ver\u00F6ffentlichen in der ARD Mediathek/Audiothek (${resourceDescriptors})
sms.republish.success=Erfolgreich in der ARD Mediathek/Audiothek ver\u00F6ffentlicht (${resourceDescriptors})
sms.republish.unable=Ein Ver\u00F6ffentlichen in der ARD Mediathek/Audiothek war nicht m\u00F6glich (${resourceDescriptors})
sms.republish.unnecessary=Ein (erneutes) Ver\u00F6ffentlichen in der ARD Mediathek/Audiothek war nicht notwendig (${resourceDescriptors})

In addition, custom feedback messages can be added to provide assistance to the editors. By calling the given addFeedback(...) methods on the IMappingContext visitor object of IResourceMapper.mapToResources(...), it can be used to collect information, warnings and errors (enum FeedbackType) that may occur during the mapping process. It is also possible to use I18N in custom feedback messages, see "I18N in custom mapper plugins" for further information.

// A show cannot be mapped without images (no I18N)
if (myImageSourceObjects.isEmpty()) {
  mappingContext.addFeedback(FeedbackType.ERROR, "The show has no images.");
  return Collections.emptyList();
}

After the new or updated resource has been sent to the ARD Mediathek/Audiothek (which may succeed or fail), the feedback messages from the mapper as well as additional feedback from the Metadata Supplier itself are sent to all registered IFeedbackHandlers. In case of the Sophora FeedbackHandler that feedback messages are attached as sticky notes to the source document. The messages are not taken over directly, but summarised and reformulated. The following table shows which feedback messages are converted to which sticky note messages.

Here is the default german messages.properties for all sticky note messages codes. The used sources sticky note message can be disabled by setting metadata-supplier-sophora.attach-used-sources-to-sticky-notes to false in the application.yml (default is true).

sms.stickynote.success=Die zugeh\u00F6rigen Inhalte wurden am ${date} um ${time} Uhr erfolgreich in der ARD Mediathek/Audiothek aktualisiert (${resourceDescriptors})
sms.stickynote.unpublish.success=Die zugeh\u00F6rigen Inhalte wurden am ${date} um ${time} Uhr erfolgreich in der ARD Mediathek/Audiothek depubliziert (${resourceDescriptors})
sms.stickynote.mapping.error=Die zugeh\u00F6rigen Inhalte konnten am ${date} um ${time} Uhr nicht in der ARD Mediathek/Audiothek aktualisiert werden: ${errorText}
sms.stickynote.save.error=Der zugeh\u00F6rige Inhalt '${externalId}' konnte am ${date} um ${time} Uhr nicht in der ARD Mediathek/Audiothek aktualisiert werden: ${errorText} (${resourceDescriptors})
sms.stickynote.publish.error=Die zugeh\u00F6rigen Inhalte konnten am ${date} um ${time} Uhr nicht in der ARD Mediathek/Audiothek aktualisiert werden (${resourceDescriptors})
sms.stickynote.usedSources=Zus\u00E4tzlich verwendet wurde(n): ${usedSources}

You are also able to create your own IFeedbackHandler. It needs to be a Spring component (with the @Component annotation) so that the Metadata Supplier can find it. The Metadata Supplier calls the handleFeedback(Feedback) method for each collected feedback during the mapping, saving, unpublishing and republishing process of a source. You can handle this feedback here in your own way, e.g. by logging, sending e-mails, writing back to the source system, etc.

@Component
@Slf4j
class ExampleFeedbackHandler implements IFeedbackHandler {

	private final MessageSource messageSource;

	@Autowired
	public ExampleFeedbackHandler(MessageSource messageSource) {
		this.messageSource = messageSource;
	}

	@Override
	public void handleFeedback(Feedback feedback) {
		// In this example we just log the info and error messages
		SourceDescriptor sourceDescriptor = feedback.getSourceDescriptor();
		log.info("Feedback for source {} and parameters {}", vSourceDescriptor(sourceDescriptor), vParameters(feedback.getParameters()));
		// You can get all feedback entries or filtered by type.
		for (FeedbackEntry entry : feedback.getEntries(FeedbackType.INFO)) {
			log.info("- {}", getMessage(entry));
		}
		for (FeedbackEntry entry : feedback.getEntries(FeedbackType.ERROR)) {
			log.error("- {}", getMessage(entry));
		}
	}

	/**
	 * Returns the (localized) message of the given feedback entry.
	 */
	private String getMessage(FeedbackEntry entry) {
		String defaultMessage = entry.getMessage();
		String messageCode = entry.getMessageCode();
		// Fallback for messages, that have no message code
		if (StringUtils.isBlank(messageCode)) {
			return defaultMessage;
		}
		IArgument[] arguments = ArgumentCreator.of(entry.getMessageArguments());
		return messageSource.getMessage(messageCode, arguments, defaultMessage, Locale.getDefault());
	}

}

Register and get previously used sources

The addUsedSource(...) method on the IMappingContext visitor object should be called for each additional source object that is used during the mapping of a source. The Metadata Supplier will save the connection between those objects in a database so that next time when one of the "used sources" is mapped, it will automatically trigger the mapping of all sources that have been "users" of this source object. This makes sure a source object is always up-to-date if one of its "used sources" has changed.

SourceDescriptor sourceDescriptorOfUsedSourceObject = new SourceDescriptor(mySourceObject.getId(), mySourceObject.getType());
mappingContext.addUsedSource(sourceDescriptorOfUsedSourceObject);

Sophora users may use the Sophora wordings of the SophoraSourceDescriptor.

SourceDescriptor sourceDescriptorOfUsedSourceObject = SophoraSourceDescriptor.create(mySourceObject.getId(), mySourceObject.getType());
mappingContext.addUsedSource(sourceDescriptorOfUsedSourceObject);

Exactly the used sources of the respective mapping process are saved. Usage relationships already registered from a previous mapping process of the source are removed, e.g. if less or other sources are used in this mapping process.

Sophora users must not forget to configure the document types and state changes for the used sources in the application.yml in order to trigger the Metadata Supplier when one of the used sources has been changed in Sophora.

It is also possible to access the sources used in the previous mapping of the source being mapped via IMappingContext.getPreviousUsedSources().

Prioritisation of incoming Sophora events

By default all Sophora document changes are handled on the same priority (NORMAL). This can be changed by implementing an IPrioritySetter component (from metadata-supplier-priority-api artifact) and provide it with your mapper plugin. The following priorities are supported: NORMAL and HIGH.

HIGH-priority events are always processed before the NORMAL-priority events. Within the same priority level, the events are processed one after the other (FIFO). This can lead to NORMAL-priority events never being processed if HIGH-priority events are constantly being added.

Here is an example for an IPrioritySetter component that prioritises all documents changed by the importer user as NORMAL and documents changed by other users (editors) as HIGH.

@Component
public class ExamplePrioritySetter implements IPrioritySetter {

	private final ISophoraClient client;

	private static final String IMPORTER_USER_NAME = "importer";

	@Autowired
	public ExamplePrioritySetter(ISophoraClient client) {
		this.client = client;
	}

	@Override
	public Priority getPriority(SourceDescriptor sourceDescriptor) {
		String externalId = SophoraSourceDescriptor.getExternalId(sourceDescriptor);
		return client.getDocumentByExternalIdIfExists(externalId)
				.map(document -> document.getString(SophoraConstants.SOPHORA_MODIFIED_BY))
				.map(lastModifier -> StringUtils.equals(lastModifier, IMPORTER_USER_NAME) ? Priority.NORMAL : Priority.HIGH)
				.orElse(Priority.NORMAL);
	}

}

Note the following restrictions:

• At most one IPrioritySetter component is allowed
• The IPrioritySetter component is not called if the Metadata Supplier is triggered by its REST API
• The IPrioritySetter component is not called for remapping sources due to the "used sources" feature

Install a custom mapper plugin

When the Java project with the custom mapper(s) is built, the result has to be a JAR that includes all its dependencies (jar-with-dependencies). Then place your built plugin JAR in the plugins folder which is located next to the Metadata Supplier Application JAR file.

By default the Metadata Supplier scans for new components in the package com.subshell.sophora.metadatasupplier (and all sub packages). If you created your mapper implementation in another package, you have to configure that in your application.yml.

---
# Application settings
spring:
  application:
    spring-additional-base-packages: 'my.custom.package.name'

Create a docker image

The Metadata Supplier Docker Image can be used to create your own custom docker image. The working directory of the base image is the /app folder, where the application.yml file must be copied to. The subdirectory plugins must be filled with your custom mapper plugin from the previous step. That's all and your Dockerfile should look like this:

ARG SMS_TAG

FROM docker.subshell.com/metadata-supplier/metadata-supplier:$SMS_TAG

COPY application.yml . 
COPY your-custom-mapper-plugin-jar-with-dependencies.jar ./plugins/

Now, run your own Metadata Supplier Docker Container with this image and enjoy the working Metadata Supplier Application wherever you want.