AVTool Media Configuration

The media configuration holds the information about the following parts:

• document types and formats
• media tagging
• server descriptions and paths
• YouTube global settings

The mediaconfig.xml is a Spring bean XML file.

Document type configuration

The document type of the video must be known to the AVTool. Set it in a MediaDocumentDescription for the property "nodeTypeName".

The "formats" define different resolutions or codecs for the same file. It is a free form text that is used as a list of all currently active file formats. This list may be used, for instance, to filter out the inactive formats. Any media file in a format which is not on this list will not be uploaded. Therefore, all active media file formats that appear in the media documents should be listed here.

Make sure your mediaconfig.xml contains something like the following block:

	<!-- List of document types -->
	<util:list id="documentDescriptionList">
		<!-- Video -->
		<bean class="com.subshell.sophora.avtool.api.MediaDocumentDescription">
			<property name="formats">
				<set>
					<value>h264-1080p</value> 
					<value>h264-720p</value>
				</set>
			</property>
			<!-- Only events for this document type are handled. -->
			<property name="nodeTypeName" value="example-nt:video"/>
			<!-- Optional: Name of a playout channel. The presence of a document on this channel corresponds to the visibility of
				the media files on YouTube. -->
			<property name="playoutChannelName" value="YouTube"/>
			<!-- Optional: Name of property that has the duration of the video. Format must be [[hours:]minutes:]seconds[.SSS]. Only
				used for captions if an intro/opener of this type is prepended to the video to be uploaded. -->
			<property name="durationPropertyName" value="example:duration"/>
		</bean>
	</util:list>

Media tagging

It is possible to add/extend MP3 tags in audio files and MP4 tags in videos. The MP3 tags are fully handled by the AVTool. For MP4 tagging, an external application AtomicParsley is used.

The values for the tags can be set in the configuration. You can use fixed values, property names, or Groovy scripts.

It is possible to control whether media files of a given document will be tagged by setting the property avtool:enableTagging.
Only when this is explicitly set to false no media file of the given document will be tagged. For true and null values media files are tagged as usual.

Configure MP3 tagging

For setting MP3 tags, you have to define an mp3TagsMapping property within the mediaTaggingConfig bean. The key of this mapping is the name of a tag. The value can be either a fixed value like "Example", a property name like "example:headline" (see tagging.propertyPrefix.regexp in configuration), or a script like "scriptClass:TagTitle" (see tagging.scriptClassPrefix.regexp in configuration).

For podcasts, different mappings can be specified. The mapping for podcast can use both the values from the podcast document itself and the values from the media documents.

To add chapter to the MP3 tags, you need to provide a mp3TagsChapterConfiguration. That in turn needs a mp3ChapterEmbeddedTagsMapping. The latter corresponds to a table in the document.

Configure MP4 tagging

In order to use MP4 Tagging, AtomicParsley must be installed and made executable for the AVTool.

You need to define the atomicParsleyExe, an mp4TagsMapping, and an mp4TagsAPMapping that will be used to create the command line to execute.

Media Tagging Example Configuration

	<bean id="mediaTaggingConfig" class="com.subshell.sophora.avtool.mediatagging.api.MediaTaggingConfiguration">
		<property name="mp3TagsMapping" ref="mp3TagsMapping" />
		<property name="mp3PodcastTagsMapping" ref="mp3PodcastTagsMapping" />
		<property name="mp3PodcastFromAudioTagsMapping" ref="mp3PodcastFromAudioTagsMapping" />
		<property name="mp3TagsChapterConfiguration" ref="mp3TagsChapterConfiguration" />
		<property name="mp4TagsMapping" ref="mp4TagsMapping" />
		<property name="mp4TagsAPMapping" ref="mp4TagsAPMapping" />
		<property name="formatSuffixMapping" ref="formatSuffixMapping" />
		<property name="atomicParsleyExe" value="/opt/local/bin/atomicparsley" />
		<property name="podcastImageVariant" value="podcast" />
		<property name="imageChildNodeName" value="ndr:image" />
		<property name="tagPodcastsOnly" value="true" />
	</bean>

	<!-- Optional tagging of chapters in audios. The child node must contain
		start offsets and may have addition properties for tags. -->
	<bean id="mp3TagsChapterConfiguration"
		class="com.subshell.sophora.avtool.mediatagging.api.ChapterTaggingConfiguration">
		<property name="childNodeName" value="ndr:chapterdata" />
		<property name="startTimePropertyName" value="ndr:chapterTime" />
		<property name="totalDurationPropertyName" value="ndr:duration" />
		<property name="embeddedTagsMapping" ref="mp3ChapterEmbeddedTagsMapping" />
	</bean>

	<!-- Optional tags for each chapter in audios. Map: Tag name -> expression
        Tagnamen: com.subshell.sophora.avtool.mediatagging.Mp3TagConverter -->
	<util:map id="mp3ChapterEmbeddedTagsMapping">
		<entry key="TITLE" value="ndr:chapterTitle" />
	</util:map>

	<!-- For tag names see com.subshell.sophora.avtool.mediatagging.Mp3TagConverter. -->
	<util:map id="mp3TagsMapping">
		<entry key="ARTIST" value="ndr:author" />
		<entry key="TYER" value="ndr:broadcastdate" />
		<entry key="GENRE" value="28" />
		<entry key="TRACK" value="0" />
		<entry key="TITLE" value="ndr:headline" />
		<entry key="COMMENT" value="ndr:url" />
		<entry key="ATTACHED_PICTURE" value="" />
		<entry key="COMPOSER" value="NDR" />
		<entry key="RADIO_OWNER" value="NDR" />
		<entry key="RADIO_NAME" value="ndr:broadcast" />
		<entry key="CONTENT_GROUP_DESC" value="ndr:shorttext" />
	</util:map>

	<util:map id="mp4TagsMapping">
		<entry key="TITLE" value="ndr:headline" />
		<entry key="COMMENT" value="ndr:shorttext" />
		<entry key="DESCRIPTION" value="ndr:shorttext" />
		<entry key="ARTIST" value="ndr:author" />
		<entry key="YEAR" value="ndr:broadcastdate" />
		<entry key="SHOW" value="ndr:broadcast" />
		<entry key="KEYWORD" value="ndr:keywords" />
		<entry key="COMPOSER" value="NDR" />
		<entry key="ALBUM_ARTIST" value="NDR" />
		<entry key="TV_NETWORK" value="NDR" />
		<entry key="PODCAST_FLAG" value="true" />
		<entry key="ARTWORK" value="" />
	</util:map>

	<util:map id="mp4TagsAPMapping">
		<entry key="TITLE" value="--title" />
		<entry key="COMMENT" value="--comment" />
		<entry key="DESCRIPTION" value="--description" />
		<entry key="ARTIST" value="--artist" />
		<entry key="YEAR" value="--year" />
		<entry key="SHOW" value="--TVShowName" />
		<entry key="KEYWORD" value="--keyword" />
		<entry key="COMPOSER" value="--composer" />
		<entry key="ALBUM_ARTIST" value="--albumArtist" />
		<entry key="TV_NETWORK" value="--TVNetwork" />
		<entry key="PODCAST_FLAG" value="--podcastFlag" />
		<entry key="ARTWORK" value="--artwork" />
	</util:map>

Servers

The AVTool must download the files from a server to be able to upload them to internet servers and YouTube. The source server is named "media server" because it holds all media files. The internet servers for the online files are called "streaming server" because clients stream videos from there.

The AVTool offfers integrations for the following storage solutions:

• SFTP (default)
• FTP(S)
• local file system
• Akamai NetStorage Usage API
• Amazon S3

You need to configure the credentials for accessing the server. Usually, a username and a password will be set.

The media server is defined as a ServerDescription. Multiple servers can be provided. A file will be looked up in order of the media servers until found.

The transporterFactory determines the protocol for transferring files. For locating the files, a pattern can be used to form the path on the server. The full path consists of two parts:

1. The "basedir" set in the ServerDescription. The interpretation depends on the used protocol especially for relative paths.
2. The "formatDirectoryMapping" provides further parts to the directory.

If the formatDirectoryMapping for a Path does not contain the {filename} placeholder it is automatically appended.

The folder on the server which contains the files must be set as "basedir". The media files can be organised in sub folders with a pattern that contains placeholders (e.g. a date). Such patterns can be configured per format in the map "formatDirectoryMapping".

After putting the parts together the following placeholders will be resolved in a path like /data/flashmedia/streams/ndr/%1$tY/%1$tm%1$td/{sophora_id}/TV-20130603-1637-5342.mp4:

The following example of a media server list describes some options:

  <util:list id="mediaServerDescriptionList">
 	<bean class="com.subshell.sophora.avtool.api.ServerDescription">
 		<!-- pre-defined: defaultTransporterFactory (SFTP), ftpTransporterFactory, ftpsTransporterFactory, localTransporterFactory, akamaiNetStorageTransporterFactory (HTTP), akamaiNetStorageSymlinkTransporterFactory, akamaiNetStorageSymlinkParentTransporterFactory, s3TransporterFactory, s3MoveTransporterFactory -->
 		<property name="transporterFactory" ref="defaultTransporterFactory"/>
 		<!-- Server name from which the video files should be retrieved. When local file transfer is used, this field is only 			used for logging. -->
 		<property name="host" value="mediaserver.example.com"/>
 		<property name="port" value="22"/>
 		<property name="user" value="sophora-avtool"/>
 		<property name="password" value="changeMe"/>
 		<property name="keyfile" value=""/>
 		<property name="basedir" value="/data/mediasource"/>
 		<property name="formatDirectoryMapping">
 			<map>
 				<entry key="h264-1080p" value="."/>
 				<entry key="h264-720p" value="h264/%1$tY/%1$tm%1$td"/>
 			</map>
 		</property>
 		<!-- Optional: A regex to parse the date from a file's name, which will then be applied to all entries in the formatDirecoryMapping 			that contain format specifiers, e.g. "%1$tY/%1$tm%1$td". The date in the file name must be written in the format "yyyyMMdd". -->
 		<property name="dateRegex" value="#?.{3}(\\d{8}).+"/>
 	</bean>
 	<bean class="com.subshell.sophora.avtool.s3.S3ServerDescription">
 		<property name="transporterFactory" ref="s3TransporterFactory" />
 		<property name="name" value="s3 streaming server" />
 		<property name="host" value="http://localhost" />
 		<property name="port" value="9090"/>
 		<property name="accessKeyId" value="s3mock" />
 		<property name="secretAccessKey" value="s3mock" />
 		<property name="region" value="us-east-1" />
 		<property name="bucketName" value="test-bucket" />
 		<property name="basedir" value="/data"/>
 		<property name="formatDirectoryMapping">
 			<map>
 				<entry key="h264-1080p" value="."/>
 				<entry key="h264-720p" value="h264/%1$tY/%1$tm%1$td"/>
 			</map>
 		</property>
 	</bean>
 </util:list>

Symlinks

The AVTool can create symlinks in the local file system, via SFTP or with the Akamai NetStorage API.
A usecase for this is for example:
The encoding system uploads files to Akamai or a SFTP Servern in a private folder. When the files are to be published, the AVTool creates symlinks from a public folder to the files in the private folder.

To use this functionality, a media server and a streaming server need to be configured. The media server must be configured so that the paths built with that media server point to the real files, i.e., the "private" files.
The streaming server must be configured to create the paths of the symlinks, i.e., the public files.
Also, both servers must use the same transporter factory.

It is advised to also set ignoreFileDigestCheckBeforeUpload to true and have any servers with these transporters first in the list of servers.

Media file digest check

By default the AVTool computes a hash for every file uploaded. If the same file would be uploaded again later the hash is checked against a list of known hashes. That way unecessary uploads can be avoided.
It is possible to skip that check with the ignoreFileDigestCheckBeforeUpload parameter.
That can be helpful if the AVTool does not need actually need to upload a file e.g. when only symlinks should be created or the s3MoveTransporterFactory is used.

VoD-Packager-API

The AVTool can control the publication state of media directories through the VoD-Packager-API. The VoD-Packager-API is based around media file storage in S3 buckets. Consequently the AVTool integration of the API expands on our S3 integration.
The AVTool uploads/moves files to/in S3 as usual and then calls the api to set the correct publication state for a given media directory. The upload/move step can be configured to be skipped if your workflow does not require the avtool to move/upload files. E.g. when your encoding platform already uploads the files to the final destination.

To use the VoD-Packager-API integration configure your media and streaming servers with the com.subshell.sophora.avtool.vodpackager.VodPackagerServerDescription server description and use either the vodPackagerApiTransporterFactory or the vodPackagerApiMoveTransporterFactory.

Authentication

The AVTool handles authentication with the api via client id and an initial client secret configured in the mediaconfig.xml.
The api requires that the client secret is regularly rotated. The avtool handles the secret rotation automatically and persists the current secret in a configurable file on disk. This file must be persisted since the previous secret, aswell as the initial secret stay valid only for a short time after rotating.

The secret will be rotated at a configurable time before the current secret expires. This time should be configured so that the secret is rotated slightly before the api sends out the expiration reminder mail. The default for this is 7 days, therefore the default time at which the avtool rotates the secret is set to 8 days before expiration. That way recieving the expiration reminder from the api also serves as an indicator that something is wrong with the avtool.
It is important that the secret is rotated before it expires as otherwise the rotation will fail and manual intervention is required. Therefore this time should be configured with enough of a grace period to retry in case of temporary problems such as network or api outages. In these cases the avtool will retry the rotation at a fixed and configurable interval until the current secret can be rotated or expires.
Since the expiration time of the initial secret is not known to the avtool, this secret will be rotated nearly immediatly after startup of the avtool.

When using the VoD-Packager-API it is not necessary to configure s3 credentials anymore. Bucket name, region and host are still required, but the s3 authentication is done with short lived session credentials supplied by the VoD-Packager-API. Aquisition and rotation of these credentials is handeled by the avtool automatically.

If the api secret needs to be managed externally this behaviour can be disabled with the externalApiSecretManagement parameter. If this parameter is set to true, the avtool just expects a valid secret file at the configured location, but will not attempt to rotate the secret.

Distribution configurations

The VoD-Packager-API allows the control of the publication state of media files through the so called "Distribution Configurations". These configurations represent for example a combination of geoblocking and age restrictions such as "de_fsk12" or "weltweit_fsk6". While there are default configurations available, arbitrary configurations are possible aswell. These have to be setup by the administration of the VoD-Packager-API first and cannot be configured in the AVTool alone.

The AVTool will never allow more than one configuration to be in the published state on a given media directory. That is, configurations it can "see". The VoD-Packager-API client (configured through the clientId parameter), can be configured on the api side to only see and edit certain configurations.
That way different AVTools are still able to publish/depublish the same dokument in different configurations. For example a "dev/test" AVTool could publish a file with a "dev_de_fsk12" configuration and never touch the regular "de_fsk12" configuration that might be used by a production AVTool. This can be used to avoid duplicating media data for different environments.
In use cases like that it is very important to properly configure s3 file permissions for every client as a safeguard to prevent e.g. a "dev/test" AVTool from deleting/moving files that are sill published on production.

Which configuration the AVTool will publish is controlled by providing the distribution configuration id in the media document. The AVTool expects said id in a string property. The name of that property can be configured through the "distributionConfigurationIdProperty" parameter. In most cases this property will not be part of the regular document model. Instead it should be computed and set in the AVTools preprocessor script. If done like that, explicit configuration of the property name is not required. Instead the default "script:distributionConfigurationId" can be used.

VoD-Packager-API Configuration

Example

<bean id="vodPackagerApi1" class="com.subshell.sophora.avtool.vodpackager.VodPackagerApiConfiguration">
	<property name="vodPackagerApiBaseUri" value="<api host>" /
	<property name="authenticationApiBaseUri" value="<auth api host>" /> 
	<property name="clientId" value="<api client id>" />
 	<property name="initialClientSecret" value="<api secret>" />
 </bean>

 <util:list id="mediaServerDescriptionList">
 	<bean class="com.subshell.sophora.avtool.vodpackager.VodPackagerServerDescription">
 		<property name="name" value="servername" />
 		<property name="host" value="<s3 host>" />
 		<property name="port" value="<s3 port>"/>
 		<property name="region" value="<s3 region>" />
 		<property name="bucketName" value="<s3 bucket name>" />
 		<property name="basedir" value="<cp-code/base directory>" />

 		<property name="transporterFactory" ref="vodPackagerApiTransporterFactory" />
 		<property name="vodPackagerApiConfiguration" ref="vodPackagerApi1" />

 		<property name="formatDirectoryMapping">
 			<map>
 				<entry key="avc270" value="." />
 				<entry key="avc360" value="." />
 				<entry key="avc540" value="." />
 				<entry key="avc720" value="." />
 				<entry key="avc1080" value="." />
 			</map>
 		</property>
 	</bean>
 </util:list> 

Full Example Configuration

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xmlns:util="http://www.springframework.org/schema/util"
	xsi:schemaLocation="http://www.springframework.org/schema/beans
	http://www.springframework.org/schema/beans/spring-beans.xsd
	http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util.xsd">

	<!--####################################################################
		Document type and media formats
		#################################################################### -->

	<!-- List of document types -->
	<util:list id="documentDescriptionList">
		<!-- Video -->
		<bean class="com.subshell.sophora.avtool.api.MediaDocumentDescription">
			<property name="formats">
				<set>
					<value>youtube</value>
				</set>
			</property>
			<!-- Only events for this document type are handled. -->
			<property name="nodeTypeName" value="example-nt:video"/>
			<!-- Optional: Name of a playout channel. The presence of a document on this channel corresponds to the visibility of
				the media files on YouTube. -->
			<property name="playoutChannelName" value="YouTube"/>
			<!-- Optional: Name of property that has the duration of the video. Format must be [[hours:]minutes:]seconds[.SSS]. Only
				used for captions if an intro/opener of this type is prepended to the video to be uploaded. -->
			<property name="durationPropertyName" value="example:duration"/>
		</bean>
	</util:list>

	<!-- Feedback to DeskClient. Map: error type to error string (you can use a select value for readable labels). -->
	<util:map id="deskclientFeedbackErrTypeConf">
		<entry key="TRANSPORT_MEDIASERVER" value="transport_mediaserver"/>
		<entry key="YOUTUBE" value="youtube"/>
		<entry key="OTHER" value="other"/>
		<entry key="MANY" value="many"/>
		<entry key="OK" value="ok"/>
	</util:map>

	<!--####################################################################
		Sources for video files
		#################################################################### -->

	<bean id="localTransporter" class="com.subshell.sophora.avtool.transporter.LocalFSTransporterFactory"/>

	<util:list id="mediaServerDescriptionList">
		<bean class="com.subshell.sophora.avtool.api.ServerDescription">
			<!-- pre-defined: defaultTransporterFactory (SFTP), ftpTransporterFactory, ftpsTransporterFactory, localTransporterFactory,
				akamaiNetStorageTransporterFactory (HTTP) -->
			<property name="transporterFactory" ref="defaultTransporterFactory"/>
			<!-- Server name from which the video files should be retrieved. When local file transfer is used, this field is only
				used for logging. -->
			<property name="host" value="mediaserver.example.com"/>
			<property name="port" value="22"/>
			<property name="user" value="sophora-avtool"/>
			<property name="password" value="changeMe"/>
			<property name="keyfile" value=""/>
			<property name="basedir" value="/data/mediasource"/>
			<property name="formatDirectoryMapping">
				<map>
					<entry key="youtube" value="."/>
				</map>
			</property>
			<!-- Optional: A regex to parse the date from a file's name, which will then be applied to all entries in the formatDirecoryMapping
				that contain format specifiers, e.g. "%1$tY/%1$tm%1$td". The date in the file name must be written in the format "yyyyMMdd". -->
			<property name="dateRegex" value="#?.{3}(\\d{8}).+"/>
		</bean>
	</util:list>

	<!--####################################################################
		YouTube
		#################################################################### -->

	<!-- YouTube settings for all channels -->
	<bean id="youtubeGlobalConfig" class="com.subshell.sophora.avtool.api.youtube.YoutubeConfiguration">
		<!-- Property of the document that indicates which channel to use. -->
		<property name="youtubeAccountPropertyName" value="example:youtubeChannel"/>
		<!-- The YouTube action to take when a video document is deleted or set offline. Options are DELETE, SET_PRIVATE and DO_NOTHING (default). -->
		<property name="deleteEventAction" value="DELETE"/>
		<property name="offlineEventAction" value="SET_PRIVATE"/>
		<!-- Optional: Application for adding opener and closer to the beginning/end of video. The specific videos are configured
			per channel. The local script file will be executed before each YouTube upload. It will receive the following parameters:
			- path to the opener file (if present)
			- path to the original video file (downloaded to a temp folder)
			- path to the closer file (if present)
			- path to the concatenated output file that the script must create
		-->
		<property name="concatVideosScript"
		          value="/cms-install-directory/ytc_opener_und_closer/addOpenerCloser_ffmpeg.sh"/>
	</bean>

	<util:list id="streamingServerDescriptionList">

		<!--####################################################################
		Optional list of servers to upload files to. Media files can be uploaded to Akamai or similar CDN servers using ftp, sftp
		and ftps. When a video document is set offline, the corresponding media files will be removed from these servers. Also
		purging of files from Akamai can be configured.
		#################################################################### -->

		<bean class="com.subshell.sophora.avtool.api.AkamaiServerDescription">

			<property name="name" value="video-cdn"/>
			<property name="transporterFactory" ref="sftpTransporterFactory"/>
			<property name="host" value="akamai.example.net"/>
			<property name="port" value="22"/>
			<property name="user" value="alice"/>
			<property name="password" value="secret"/>
			<property name="keyfile" value=""/>

			<!-- Base directory for all uploads -->
			<property name="basedir" value="/video"/>

			<!-- Optional: A regex to parse the date from a file's name. -->
			<property name="dateRegex" value="#?.{3}(\\d{8}).+"/>

			<!-- Akamai FastPurge -->
			<property name="akamaiUrlPatterns">
				<list>
					<value>http://download.mysite.com/video/%1$tY/%1$tm%1$td/{filename}</value>
					<value>http://media.mysite.com/video/%1$tY/%1$tm%1$td/{filename}</value>
				</list>
			</property>
			<property name="akamaiTagPatterns">
				<list>
					<value>{filename}</value>
				</list>
			</property>

			<!-- Akamai HLS/HDS Purge -->
			<property name="akamaiRestPurgeConfiguration">
				<list>
					<bean class="com.subshell.sophora.avtool.api.AkamaiRestPurgeRequestConfiguration">
						<property name="requestName" value="HLS"/>
						<property name="requestPath" value="/config-media-live/v1/vod/purge/HLS"/>
						<property name="manifestUrl" value="https://mysite.akamaihd.net/i/video/%1$tY/%1$tm%1$td/{sequencename}.,webs,webm,webl,webxl,.h264.mp4.csmil/master.m3u8"/>
						<property name="fileNamePattern" value="%1$tY/%1$tm%1$td/{filename}"/>
					</bean>
				</list>
			</property>

			<!-- Mapping of format to server path. Can use date fields parsed using the dateRegex. -->
			<property name="formatDirectoryMapping">
				<map>
					<entry key="youtube" value="%1$tY/%1$tm%1$td"/>
				</map>
			</property>
		</bean>
	</util:list>

</beans>