AVTool Overview

Features

The AVTool has many features for processing files for audio and video documents.

Transfer

• When a media document is published, the files are transferred from the media server to the streaming servers.
• If a media document is deleted or taken offline, the associated files are deleted from the streaming servers unless they are referenced by other documents.
• The AVTool can upload a file to multiple target paths on a server.
• Supported protocols for transfer: FTP, FTPS, SFTP, local file system, and Akamai NetStorage API
• Additional transport protocols can be implemented via plugins.
• After successful transfer, a pre-published media document is automatically published.
• The media formats to be transferred should be selected in the media document.
• There can be conditional formats that are only uploaded if a boolean property is set.
• Several files can be transferred at the same time. The AVTool ensures that parallel operations do not affect the same media files.
• The file paths on the streaming servers can be made dependent on several factors:
• - The structure path of the document
• - A property in the document (e.g. geoblock)
• Patterns can be used for the file paths in the configuration file. E.g. date from the sequence, Sophora ID, etc. See also placeholders for file names.
• Before an action (command) is executed, a pre-processor script is executed, which should alter the triggering document to adapt the document for the AVTool Document Model.

Monitoring

• The queue of jobs and the currently executed transfers are displayed via JMX and a web page.
• Jobs can be removed from or added to the queue via JMX.
• Emails can be sent in the event of errors during transmission.
• Any errors that occur are written to the video document as a sticky note and can thus be displayed in the DeskClient. The colors of the sticky notes are configurable.
• The job status and, if applicable, the exception can be attached to the AV document as YellowData.

YouTube

• Video files can be transferred to YouTube. Several formats can be specified; the first one available in the video will be used.
• If the video document is deleted or taken offline, the video will be deleted from YouTube or made private (see: AVTool YouTube Configuration/Global YouTube configuration).
• Videos can be uploaded as “private”. They must be then manually set to “public”.
• The YouTube ID is saved into the video document as YellowData (to retain the document status).
• The YouTube channel, to which the video is to be transferred, is selected in the video document itself. You can choose between several channels, but a video can only be assigned to one channel at a time.
• The title and the description of the YouTube videos can be customized using Groovy scripts.
• The preview image can be included (teaser image).
• Before uploading, an external program can be called up to pre-process the video (e.g. for opener and closer videos).
• In Sophora, a preview can be implemented that displays the corresponding YouTube page for a video document.
• After the file has been uploaded, a check is made every minute to see whether the video has been processed. If the upload status does not change to “processed”, an error message is logged.
• Subtitle files can be uploaded to YouTube.

Akamai

• After uploading or deleting files, the associated tags and URLs are flushed to Akamai.
• Several URLs per media file are possible.
• Only one flush request per document is sent to Akamai for all media files and formats it contains. The job is only split if it becomes too large.
• Symlinks for files and directories can be created.

Miscellaneous

• MP3s and MP4s can be tagged (incl. teaser image) with values of the media document.
• Support for podcasts. Podcasts are documents that contain video or audio documents as components. If the podcast is published, the files of all linked video and audio documents will be transferred. It is possible that different target servers or file paths are used on the streaming servers for the media files than if the video or audio documents were published independently of the podcast.
• When deleting or placing media documents offline, the associated files can be deleted from the media server (only via the local file system, not via FTP).

Workflow

Sophora offers the option to create media documents. Each media document references one or more media files via one or more sequences (see document model for details). Each media file can be available in different formats and qualities. For example, the video encoder offers videos in Flash, Real and Windows Media format, each in low and high quality. The exact formats and qualities to be offered are set in the media document.

The AVTool reacts when a media document is published, taken offline or deleted.

Publication

If an editor publishes a media document, it first enters the PRE_PUBLISHED state. The AVTool then carries out the following steps:

• For all sequences listed in the document, the media files are loaded from the media server in all formats selected in the document.
• MP3 and MP4 files are tagged.
• The media files are uploaded to the streaming servers.
• If selected in the document, the first sequence is transferred to YouTube. The YouTube ID is saved into the document as YellowData.
• A result code and any error text are attached to the document as a sticky note.
• If no error has occurred, the document is set to the PUBLISHED state.

Media files are only transferred to the streaming servers if they have changed or the content of the tags has changed. For this purpose, a hash value is generated for each media file after tagging and stored locally in the file system. The media file is only transferred to the streaming servers if its hash value has changed.

Podcasts are a special case. A podcast document does not contain any sequences itself, but contains a list of audio or video documents. If a podcast document is published, the media files of all media documents referenced in the podcast document are transferred. When tagging, information from the respective media document as well as from the podcast document itself can be used. In addition, only the formats selected in the podcast document are transferred. The target directories on the streaming servers can also be configured differently than if the media documents are published directly.

For transmission to YouTube a channel must be configured in the document. If neither a YouTube channel nor the required format is available for a video, the video cannot be transferred to YouTube.

If an error occurs during transmission or tagging, all sequences of the media document in all formats are deleted from all streaming servers (if they are no longer referenced by any other documents).

Taking offline and deleting

If a media document is taken offline or deleted, the following steps happen:

• All sequences are deleted from the streaming servers in all formats. A sequence is only deleted if it is not referenced by any other published document.
• If the document contains a YouTube ID, the corresponding video is deleted from YouTube or set to “private”. Which action should be performed can be configured differently for "offline" and for "delete" event.
• If configured, the sequences are deleted from the archive (media server).
• The result and any error text are written to the document.

Scripting

There are two kinds of Groovy scripts that can be used to customize the AVTool: Tagging scripts and preprocessor scripts. Scripts of both types must be put in a folder which is configured in a Java property named "avtool.groovy.dir" (with "-Davtool.groovy.dir=/path/to/groovy/scripts" in the .conf file).

Tagging Script

The content for fields in the "Video Data" section of the YouTube channel configuration document can be computed by Groovy scripts. If the content starts with the script class prefix as given by the property tagging.scriptClassPrefix.regexp in the application.properties, e.g. "scriptClass:", the remaining portion of the string is interpreted as the name of a Groovy class. For example, if the field "Title" contains the value scriptClass:YoutubeTitle, the AVTool looks for a script defining the class "YoutubeTitle", calls its processDocument() method and uses the result as the actual title of the YouTube video. Here is an example of a tagging script:

import com.subshell.sophora.api.content.INode;
import com.subshell.sophora.avtool.api.scripting.AbstractMediaTaggingScript;
import java.text.SimpleDateFormat;
 
class YoutubeTitle extends AbstractMediaTaggingScript {
	String processDocument(INode doc) {
		def parts = []
		parts << doc.getString('your-prefix:headline')
		parts << context.getSelectValueLabel(doc, 'your-prefix:broadcast')
		parts << "TV"
		parts.removeAll { it == null || it == '' }
		return parts.join(' | ')
	}
}

Pre-processor Script

Every time the AVTool processes a video, a pre-processor script is called with the ISophoraDocument node of the video document. The script can then change the node, i.e. change or add properties or child nodes. The changed document is not required to match the node type configuration of the video document, i.e. the script can add arbitrary properties, which can then be referenced by the YouTube channel configuration document. The following example shows a simple preprocessor script.

import org.slf4j.Logger

import com.subshell.sophora.api.content.INode
import com.subshell.sophora.avtool.api.scripting.IPreprocessorContext
import com.subshell.sophora.avtool.api.scripting.IPreprocessorScript

class YouTubePreprocessor implements IPreprocessorScript {
	private Logger log
	private IPreprocessorContext context

	@Override
	public void init(IPreprocessorContext context) {
		log = context.logger
		this.context = context
	}

	@Override
	public void preprocess(INode video) {
		// Set a fixed YouTube channel for all videos.
		video.setString("your-prefix:youtubeChannelId", "channel1")

		// Set a new property which can then be referenced by the channel configuration.
		if (video.hasProperty("your-prefix:title")) {
			video.setString("youtube:title", video.getString("your-prefix:title"))
		} else {
			video.setString("youtube:title", "My video")
		}

		// The script has access to the Sophora Client.
		String description = context.sophoraClient.getDocumentByExternalIdIfExists("your-special-document").getString("your-prefix:youtubeDescription")
		video.setString("youtube:description", description)
	}
}

Flowchart

The following flowchart depicts how the AVTool handles Document Changed Events to upload or remove mediafiles from streaming servers, YouTube, or Akamai Netstorage.