Architectural Viewpoint
- Sophora Primary Server
- Fall-back Sophora Replica Server – The fall-back Sophora Replica Server is used as substitute when the Sophora Primary Server is temporarily unavailable; for example when updating the Sophora Primary Server.
- Backup system – Is also a fall-back Sophora Replica Server but with a different role. It should be exclusively run as backup system.
- Sophora Staging Servers (existing n times)
From the Components' Point of View
For each Sophora Primary Server:
- Repository database
- Repository with the Lucene index within the file system
Lucene Index and Database
When a database backup of the repository is executed, the Lucene index has to be synchronised. This can be achieved by bringing in a Lucene index backup at the same time; e.g. a copy of the entire repository, including configuration files. Alternatively, the entire index can be deleted so that it is rebuilt from scratch when the system is restarted. This operation might take a while since it depends on the size of the repository. Thus, this is usually not viable.
Data Replication
To transport content from the Sophora Primary Server repository to the delivery servers the Java Message Service (JMS) is applied (refer to http://www.oracle.com/technetwork/java/index-jsp-142945.html). By that, the transportation mechanism is not located at database level but shifted to the application. Sophora therefore uses the JMS implementation ActiveMQ (see http://activemq.apache.org/).
Two use cases have to be distinguished:
- Replication of content for the fall-back or backup system
- Publishing content (i.e. Sophora Staging Server)
The replication operates in a Sophora Primary Server ""slave"" configuration. That means changes in the Sophora Primary Server repository are replicated to all connected replica systems.
Besides the concerned documents the replicated data also contains system properties such as user information and node types. This ensures that the fall-back system is a complete copy of the Sophora Primary Server.
In normal mode the main system acts as Sophora Primary Server and the fall-back systems are Sophora Replica Servers. If the Sophora Primary Server system is unavailable (for whatever reason), the fall-back system can be declared as the Sophora Primary Server temporarily. Testing the availability and switching the Sophora Primary Server slave roles is done manually by a Sophora administrator. She is also responsible to inform the editorial office/staff which system currently acts as Sophora Primary Server. Editorial journalists are supposed to work on the Sophora Primary Server system exclusively.
If the designated Sophora Primary Server system is available again, it first has the role of a Sophora Replica Server and needs to synchronise with the fall-back system. To guarantee a complete synchronisation both systems have to be locked for a short period. Afterwards the roles can be switched back. Do not forget to inform the editorial office about the switchover so that they continue working on the correct Sophora Primary Server.
The ActiveMQ database does not need to be backed up since it is only used for a short time when the Sophora Primary Server system is turned off. In the ideal case the stored queue in this database is empty which implies that the connected Sophora Replica Servers are synchronised. If that is not the case, inserting a filled queue after a break-down is rather destructive than helpful since the synchronisation of the Sophora Replica Servers will not be executed appropriately/correctly.
Creating a Backup
Creating a component's backup needs to be done in a consistent state. This is achieved, if the backup is created when the component is shut down. Thereby, no inconsistencies can occure while the backup process is running.
Backing Up a Sophora Primary Server
The following directories need to be saved:
<sophora.home>/repository<sophora.home>/config
If the database for JackRabbit has been configured as an embedded database, the database files are located in the directories repository. Thus, they are enclosed by the backup. Otherwise, the database files of the main repository have to be backed up additionally.
Backing Up a Sophora Staging Server
For the creation of a Sophora Replica Server's backup there are two possibilities:
- A Sophora Staging Server that is going to be backed up needs to be disconnected from the delivery and shut down. Afterwards, its data is copied to the backup location and the Sophora Replica Server can be restarted. Next, the Sophora Staging Server synchronises automatically with the Sophora Primary Server to catch up with changes that happened in the meantime. Such backup operation may be automated using a script in order to repeat/renew the backup at regular intervals; e.g. every 24 hours.
- A separated Sophora Staging Server, which never actually connects to the delivery, might be run as backup Sophora Staging Server concurrently. By that, the backup Sophora Staging Server synchronises as the other Sophora Staging Server but never forward the received data to a delivery. The actual backup procedure is the same as in the first alternative with the benefit that a potential blackout in the delivery is avoided entirely.
However, the following directories of the Sophora Staging Server to back up have to be secured:
<sophora.home>/repository<sophora.home>/config
For each configured/connected delivery (i.e. Tomcat context) one additionally has to save the following files:
- Docbase of the Tomcat context containing the templates
<localConfigDirectory>(as defined in the Tomcat context's configuration)<sophora.delivery.cache.directory>(as defined in thesophora.propertieswithin<localConfigDirectory>)
If the configuration of the Tomcat (and Apache) is not generated automatically by a configuration system, you should also secure the according files.
Backing Up the Dashboard
The Dashboard also has its own data management about the information collected over time. These information can be found in
<dashboard.home>/data
Create a backup of this directory after stopping the Dashboard process.
Recovery
A component's backup can only be restored when the concerning process has been stopped beforehand. When a Sophora Primary Server system is recovered the connected Sophora Replica Servers also need to be recovered.
Restoring a Sophora Primary Server's Backup
To restore the backup of a Sophora Primary Server the following directories need to be copied:
<sophora.home>/repository<sophora.home>/config
Following directories have to be removed:
<sophora.home>/data
If the backup is restored from a backup or fall-back system, you might need to adjust some ports and IP addresses within the sophora.properties file.
If an embedded database is used for the repositories, these information are already contained in the restored data. Otherwise the database file corresponding to the used database system have to be restored from the backup.
Such parameters can be found in the following files:
<sophora.home>/repository/repository.xml<sophora.home>/repository/workspaces/default/works
Restoring a Sophora Staging Server's Backup
The subsequent directories have to be restored:
<sophora.home>/repository<sophora.home>/config
Following directories have to be removed:
<sophora.home>/data
If the backup has been created on a different machine, you might need to adjust the configured ports and IP addresses with the sophora.properties file.
Additionally, you have to restore the following directories for each configured Tomcat context:
- Docbase of the Tomcat context containing the templates
<localConfigDirectory>(as defined in the Tomcat context's configuration)<sophora.delivery.cache.directory>(as defined in thesophora.propertieswithin<localConfigDirectory>)
Here again you might have to adjust ports and IP addresses within <localConfigDirectory>/sophora.properties.
Scenario 1: The Sophora Primary Server Drops Out
In such case, none of the editorial journalists can access the system so that no changes can be done to the repository. The delivery is not affected from a breakdown of the Sophora Primary Server.
- Usually the Sophora Primary Server can simply be restarted. The Sophora Staging Server and fall-back Sophora Replica Servers as well as the deskclient log in again on their own.
- If the Sophora Primary Server cannot be restarted, the fall-back Sophora Replica Server should be declared as the new Sophora Primary Server. Therefore, you need to customise its configuration and restart it. The Sophora Staging Servers, deskclients and all other Sophora modules (like the Sophora Indexer or the Sophora Importer) also need to be restarted manually in order to connect to the new Sophora Primary Server. As soon as the problems with the original Sophora Primary Server are solved it can be announced as new fall-back Sophora Replica Server and thereby synchronises with the new Sophora Primary Server automatically after restarting. If you want the former Sophora Primary Server become Sophora Primary Server again, you have to stop both the new Sophora Primary Server and fall-back Sophora Replica Server (old Sophora Primary Server), change their mode of operations and restart them afterwards.
- If the fall-back Sophora Replica Server is unavailable too, you have to use the backup system as temporary Sophora Primary Server (refer to the previous item).
Scenario 2: A Sophora Staging Server Drops Out
When a Sophora Staging Server drops out the delivery on this server fails. Thus, it should be removed from the according delivery (group) until the problem has been fixed.
Usually the concerned Sophora Staging Server can simply be restarted. It automatically synchronises with the Sophora Sophora Primary Server. Afterwards, this Sophora Staging Server can be reintegrated into the delivery.
If the Sophora Staging Server cannot be restarted: Restore this Sophora Staging Server with the backup of a different Sophora Replica Server (see section "Backing Up a Sophora Staging Server"). Restart the Sophora Staging Server again so that it synchronises with the Sophora Primary Server (done automatically beginning at the configured date). Finally, it can be reintegrated into the delivery.
Alternatively you can also restore the Sophora Staging Server using the backup system and process as in the previous step.
Scenario 3: Sophora Primary Server and fall-back Sophora Replica Server Drops Out And the Backup System Is Not Updated Or Unavailable as Well
In such scenario the Sophora system has to be restored from the backup system. All information that have not been integrated in the backup before the total crash are lost. That means the system can only be restored with an outdated version since no synchronisation can be done starting from the point in time of the breakdown.
The Sophora Staging Servers might contain more recent content than the Sophora Primary Server does. Nonetheless, they need, in order to get back to a consistent state, to synchronise with the outdated Sophora Primary Server. Otherwise, a correct application flow of the Sophora Staging Servers cannot be guaranteed.
A backup of the repository and the repository directories of the Sophora Primary Server need to exist for this case.
A backup of the ActiveMQ queue must not be restored. In fact, you should even clean it because it is most likely not synchronous to the backup.
When the backups have been restored the Sophora Primary Server can be restarted. Afterwards, the fall-back Sophora Replica Servers and backup systems can also be restarted by and by as well as the Sophora components connected to the Sophora Primary Server.
