3. TrilioVault Administration Guide¶
3.1. TrilioVault Documentation¶
The purpose of this document is to help the user execute various tasks of TrilioVault. It gives step-by-step instructions for creating workloads and taking snapshots. The other documents that are created as a part of the TrilioVault product line covers the information mentioned in the table below.
|Installation/Deployment Guide||Step-by-step installation procedure for the TrilioVault backup & recovery system.|
|Administration & User Guide||Detailed documentation for performing TrilioVault administrative and user tasks along with troubleshooting.|
|API Guide||Detailed documentation about CLI and API.|
|Release Notes||Late breaking information about the specific version of TrilioVault.|
Figure 1: TrilioVault System Architecture.
TrilioVault is a tenant-driven Data Protection-as-a-Service (DPaaS) solution for OpenStack clouds. The TrilioVault solution takes point-in-time backup of workloads and supports multiple recovery options to recover your workloads in case of any outages. This document provides detailed instructions on the administration of the system. The target audience of the TrilioVault are OpenStack tenants as well as OpenStack Administrators who are proficient with basic backup and recovery concepts.
The amount of data coupled with the need for computing power in large businesses has grown tremendously in recent times. One of the biggest challenges when deploying an OpenStack cloud in an organization is the ability to provide a policy based, automated, comprehensive backup and recovery solution. For IT leaders, the major concern today is the threat of data loss and their ability to recover applications from any outage and bringing the business back online. Whether it is data corruption, data loss or system failure, when it happens, this can have huge financial impact on businesses of all sizes. Unfortunately, OpenStack does not provide enough support to implement the data protection solution that enterprises require today. With Trilio Data’s one of a kind VAST (Virtual Snapshot Technology) solution, TrilioVault can easily meet the data protection requirements of any OpenStack customers today, enabling organizations to recover workloads and improve their Recovery Time Objective (RTO) and Recovery Point Objective (RPO). This multi-tenant, self-service, policy-based solution is designed to protect application workloads from data corruption or data loss, providing point-in-time snapshots, with configuration and change awareness to seamlessly recover a workload with one click.
Some of the key features of TrilioVault include:
- Provides backup and disaster recovery solution for OpenStack.
- Provides multi-tenant, self-service and policy based solution.
- Provides point-in-time backup of data and configuration.
- Provides one-click seamless recovery.
- Provides the flexibility of selectively restoring instances on to same or different networks, different availability zones, regions or different clouds.
- Provides the flexibility of recovering backups on to different clouds.
- Recovery either a single file or a directory either through a web portal or ssh
- In-place restoring a volume or a VM
- Search for files in list of snapshots
- Take regular backup of OpenStack configuration including individual service configuration files, data and mysql database.
- Results in reducing Total Cost of Operation (TCO) with excellent Return on Investment (ROI).
3.3. System Overview¶
TrilioVault provides a tenant driven point-in-time data backup and seamless recovery solution for Enterprises and Cloud Service Providers. The solution takes backup(s) of entire workload clusters which consist of compute resources, network configuration, and storage data as one backup unit. It also provides the flexibility of taking incremental backup(s) thereby reducing the time to capture only that portion of data where changes have been made rather than taking the backup of the entire system.
3.3.1. Understanding Concepts¶
Before starting to use TrilioVault, it is necessary to know a few commonly used concepts in TrilioVault. Below are some of the important terms in OpenStack and TrilioVault.
184.108.40.206. TrilioVault Dashboard¶
TrilioVault dashboard is a plug-in into horizon dashboard and allows a user to create and manage backup jobs. A dashboard is a self-managed portal to schedule and execute on-demand backups of workloads.
Instances are the running virtual machines within an OpenStack cloud. In the context of TrilioVault, instances are members of workloads (backup-jobs). TrilioVault takes the backup of those OpenStack Instances which are members of the workloads.
A workload is defined as a collection of instances, the interconnectivity between them, the virtual disks mapped to those machines and any metadata associated with each of these resources. The tenant can add or remove instances to/from a workload. An instance can only be a member of a single workload at any given time. If a tenant wants to move an instance from one workload to another, the instance first need to be removed from old workload and then add to the new workload.
220.127.116.11. Workload Policy¶
Workload policy defines various aspects of the backup process including number of backups to retain, frequency at which backups are taken and full backups between incrementals.
A snapshot is the actual copy of the data backup stored in some storage designated as backup store or respository. A snapshot can also be defined as a state of a system at a particular point-in-time. TrilioVault provides its users with the ability to take full and incremental snapshots. A Full type snapshot takes a complete backup of all instances included in the workload independent of previous snapshots. In an Incremental type snapshot, TrilioVault takes backups of only data modfied since the last snapshot.
18.104.22.168. Backup target¶
Backup target is the storage repository where TrilioVault keeps its backup data. TrilioVault supports both NFS and Swift as a backup target.
Restore operation recreates a selected snapshot. TrilioVault supports multiple restore options for user to choose from.
22.214.171.124. One-click restore¶
One-click restore restores the selected snapshot to the exact location including the same network/subnet, volume types, security groups, IP addresses and so on. One-click Restore only works when original instances are deleted.
126.96.36.199. Selective Restore¶
The selective restore method provides a significant amount of flexibility to recover instances. With the selective restore, user can choose different target networks, target volume types, include/exclude specific instances to restore, and target flavor for each instance, etc.
188.8.131.52. In-place restore¶
One click and selective restores creates brand new resources when restoring virtual resources from the backup media. In some cases it is not desirable to construct new resources. Instead user may want to restore an existing volume to a particular point in time. In-place restore functionality will overwrite existing volume with the data from the backup media.
3.4. Managing Workloads¶
TrilioVault Horizon dashboard plugin which can be accessed using the URL http://<horizonip> from either Mozilla Firefox and Google Chrome browsers. Please refer to the TrilioVault Deployment Guide for horizon plugin installation.
TrilioVault Horizon plugin adds a Backup Panel in the tenant space. A TrilioVault user interface is displayed in Figure 2: TrilioVault Dashboard is described in Table 3.
Figure 2: TrilioVault Dashboard.
Table 3: Panels in the TrilioVault Dialog Box.
|Top Panel||Displays the OpenStack logo, the project you are logged into, and the role.|
|Left Panel||Displays the various menu items, along with description.|
|Right Panel||Displays the information about the menu selected in the Left Panel.|
Table 4: Fields and Descriptions of Menus in TrilioVault.
lists the Manage Compute and Other sub-menu items. Manage Compute lists the following options.
Other menu lists the following submenu.
lists the System sub-menu items.
|Backups||This panel is provided by the TrilioVault plugin. It is listed under the Project menu. All the above tabs are inherited from the OpenStack dashboard. Here a user can create and manage his/her backup jobs.|
3.4.1. Creating a Workload¶
A workload can be of two types:
- Serial Workload: When the selected workload type is Serial, TrilioVault executes the snapshot process of instances in the workload successively in the order they were added to the workload.
- Parallel Workload: When the selected workload type is Parallel, TrilioVault executes the snapshot operation of instances in the workload in arbitrary order.
Workload policy is an admin defined policy template. Cloud administrator defines these policy templates and makes them available for tenants. Each policy defines various aspects of the backup job including snapshot retention, snapshot interval etc. When one or more policies are assigned to a tenant, tenant can only choose one of those policies for their workloads.
To create workload, follow the steps.
- Click Project from the left panel. Navigate to Backups then click Create Workload. The Create Workload screen is displayed as shown in Figure 3.
Figure 3: Create Workload.
- Select the desired values as explained in the following table:
|Details tab||Refer to Figure 3 for details.|
|Workload Name||Enter the name of the workload to be created.|
|Workload Description||Enter brief description.|
|Workload Type||Select either Serial or Parallel.|
|Workload Policy||Cloud Administrator defined workload policies to choose from.|
|Workload Members tab||Refer to Figure 7 for details.|
|Instance checkbox||Add the instances for your workload.|
|Policy tab||Refer to Figure 13 for details.|
|Enabled||This checkbox is selected by default. If it is deselected, Start Date, End Date, Start Time, Repeat Every cannot be entered.|
|Start Date||The date on which backup process will start. It is in mm/dd/yyyy format and the default value is always Today’s date.|
|End Date||The date until which the backup process will take place. It is in mm/dd/yyyy format and the default value is always No End Date.|
|Start Time||The time when the backup process will begin. It has to be in the format hh:mm AM/PM. The default value is 09:00 PM UTC.|
|Repeat Every||The time interval in which the backup process will be repeated, for example, repeat every hour/day/week. It has to be in format 24 hrs, or 1 Day, or 1 week. The default value is 24 hrs.|
|Snapshot Retention Type||Can be either a Number of Snapshots to keep or Number of days to retain Snapshots.|
|Snapshot Retention Value||Depending upon the Snapshot Retention Type, enter the value of Snapshot Retention Type. The default value is 30. For example, if the type is a Number of Snapshots to keep, the system will keep the last 30 snapshots and if the type is Number of days to retain Snapshots, then the Snapshots for the last 30 days will be retained. All other snapshots will be deleted.|
|Full Backup Interval||Frequency at which Full Backup should be taken. The options are: * Never (The default value selected is Never). * Always (Value 0 means Always). * No. of Days (Number of incremental snapshots to take full backup between 1 and 999). If this option is selected, enter the number provided in the text box.|
|Pause at Snapshot||TrilioVault will pause guest instances while snapshotting them, otherwise not. The default setting is not to pause the VM.|
- Click Create.
- The created workload will be seen listed the Workload Types in Backups under Project menu as shown in Figure 4.
Figure 4: Serial/Parallel Workload Created.
3.4.2. Edit Workload¶
Edit Workload allows user to change the workload name or workload details. The following steps show how an existing workload can be modified.
- Click Edit Workload corresponding to the workload to be edited as shown in Figure 5.
Figure 5: Edit Workload Menu
- Update Workload screen is displayed as shown in Figure 6.
Figure 6: Edit Workload.
You can change all the fields except:
- ID where workload ID is displayed.
- Type where workload type is displayed.
- Start Time of the workload policy.
- Also if a policy template is associated with the workload, any individual elements of the policy cannot be modified.
184.108.40.206. Assigning Instances to Workload¶
To assign an instance to a workload follow these steps:
- Click Edit Workload corresponding to the workload to be edited as shown in Figure 5.
- Select the tab Workload Members as seen in the screen diplayed in Figure 7.
Figure 7: Assigning Instance to Workload.
To assign an instance to a workload, click + (plus) sign corresponding to the instances list displayed under All Instances. The selected instances will be displayed under Workload Members. Click Update.
220.127.116.11. Removing Instances from a Workload¶
To remove an instance from a workload follow these steps.
- Click Edit Workload corresponding to the workload to be edited as shown in Figure 5.
- Select Workload Members. The screen is displayed as in Figure 7.
- To remove an instance from a workload click – (minus) sign corresponding to the instances list displayed under Workload Members.
- Click Update.
3.4.3. Workload Details¶
To get the information about the workload details follow these steps.
- Using TrilioVault login, navigate to Backups.
- Click <workload name>. The Workload Details screen is displayed as shown in Figure 8.
Figure 8: Workload Details.
Workload Details displays the details mentioned in Table 5.
Table 5: Workload Details Description.
|Details||Displays the details of workload such as Workload Name, Workload Description, Availability Zone, VM names attached to it and its ID.|
|Snapshots||Displays information of the snapshot attached to the workload. Refer to Figure 17 for details.|
|Policy||Displays information of the backup schedule and retention policy. Details such as Scheduler Enabled, Start Date, Start Time, End Date, Repeat every, Next snapshot runs at, Full Backup interval can be seen here. It also displays Number of snapshots to keep. Referto Figure 11 for details.|
|Misc.||Displays information about the workload like Created at (time of creation), Last updated at (time of last update), Workload ID and Workload type ID. Refer to Figure 9 for details.|
Figure 9: Workload Miscellaneous Information.
18.104.22.168. Instance Details¶
To get the information about the Instance details follow these steps.
- Using TrilioVault login, navigate to Backups.
- Click <workload name>. The Workload Details screen is displayed as shown in Figure 8.
- Click VM RecoveryManager on workload Details page, the Instance Overview details is displayed as shown in Figure 10.
- To easily navigate between workload and the instance members, TrilioVault adds metadata workload_id and workload_name keys to each instance that is part of a workload.
Figure 10: Instance Details.
3.4.4. Deleting Workload¶
A previously created workload can be deleted from TrilioVault using Delete Workloads. However, to delete a workload you must delete all the snapshots attached to it. To delete a workload follow these steps:
- Click Project from the left panel. Navigate to Backups under Other.
- Select the workload to be deleted.
- Click Delete Workloads. Refer to Figure 11 for details.
Figure 11: Delete Workload.
- A message will be displayed asking for the confirmation for deleting the workload as displayed in Figure 12.
Figure 12: Delete Confirmation Message.
- The workload will be deleted and will no longer appear under Workload list.
3.5. Policy Configurations¶
3.5.1. Configuring a Scheduling Policy¶
In order to utilize the available bandwidth properly and have a smooth passage of input and output data, TrilioVault allows for the configuring and scheduling of policies. To schedule a policy, follow these steps:
- Login to Horizon dashboard using Administrator login id and password.
- Click Edit Workload corresponding to the workload to be edited as shown in Figure 5.
- Select Policy tab. The screen as shown in Figure 13 is displayed.
3.5.2. Configuring a Retention Policy¶
To create a persistent and automatic backup, retention policies are configurable. Follow these steps to configure a retention policy:
- Login to the TrilioVault dashboard using the Administrator login ID and password.
- Click Edit Workload corresponding to the workload to be edited as shown in Figure 5.
- Select Schedule tab. The Scheduling and Retaining Policy screen as shown in Figure 13 is displayed.
The Schedule details are described in the section Creating a Workload.
Figure 13: Scheduling & Retention Policy.
To view the protection policy attached to the workload, navigate to the Policy tab as mentioned in section Workload Details. The policy details screen is displayed in Figure 14.
Figure 14: View Policy Details.
The policy details screen is described in Table 6.
Table 6: Policy Details.
3.5.3. Workload Policy Template¶
TrilioVault 2.6 has introduced a new concept called workload policy template. The template defines attributes of an SLA. For example, a business critical application may have stringent SLA with hourly backups and 100 snapshots to retain. Cloud administrator can define a policy template that captures this SLA and publish it to tenants. Policy template feature also gives cloud administrator more control over what backup policies that tenants can choose for their applications.
From workflow point of view policy template is very similar to nova flavors where nova flavors are defined by cloud administrator and tenants are only limited to those flavors when creating instances. With workload policy templates, cloud administrator creates various templates and publishes to tenants and tenants are limited to only those policy templates that they are assigned. However, workload policy template is not automatically assigned to each tenant. Cloud administrator should assign each template selectively to a tenant. If a tenant is not assigned any template, that tenant has the freedom to choose/define the backup policy based on Configuring a Scheduling Policy
3.6. Managing Snapshots¶
3.6.1. Taking a Snapshot¶
TrilioVault creates a snapshot of the workload automatically as per the policy settings created as described in Policy Configurations. A snapshot can also be executed manually (on demand). To create a snapshot for a workload, follow these steps.
- Navigate to Backups and click Workload.
- Click Create Snapshot corresponding to the workload for which snapshot is to be created as shown in Figure 15.
Figure 15: Creating snapshot for Workload.
A new Create Snapshot window is displayed as shown in Figure 16.
Figure 16: Create Full/Incremental Snapshot.
Create Snapshot in Table 7 gives the description of the Create Snapshot window.
Table 7: Create Snapshot.
|Snapshot Name||Name of the snapshot.|
|Snapshot Description||Description of the snapshot to be created.|
|Snapshot Type||Type of snapshot Full or Incremental. The default value is Incremental.|
22.214.171.124. Full Snapshot¶
To create a full snapshot follow these steps.
- Click Create Snapshot corresponding to the workload for which snapshot is to be created.
- Enter Snapshot details like Snapshot Name, Snapshot Description and Snapshot Type as Full.
- Click Create.
- Click <workload name>, click Snapshot tab. The created snapshot can be seen as displayed in Figure 17.
Figure 17: Full Snapshot Creation.
126.96.36.199. Incremental Snapshot¶
During the creation of an incremental snapshot, TrilioVault creates a temporary volume which is cleared after the snapshot has been completed.
An Incremental snapshot is the backup of the latest changes since the last backup. When a recovery is needed, the restoration process would need the last full backup plus all the incremental backups until the point-in-time of the restoration. Incremental backups require minimum storage space and are quick to perform. To create incremental snapshot follow these steps.
- Click Create Snapshot corresponding to the workload for which the snapshot is to be created.
- Enter the Snapshot details like Snapshot Name, Snapshot Description and Snapshot Type as Incremental.
- Click Create.
- Click <workload name>, click Snapshot tab. The created snapshot can be seen as displayed in Figure 18.
Figure 18: Incremental Snapshot Creation.
188.8.131.52. Snapshot Details¶
Snapshot Details displays the information regarding instances attached to the snapshot, Security Group, Flavor, Networks, Volumes, and so on. The details are described in Table 8 and displayed in Figure 19.
Table 8: Snapshot Details Info. & Instance details tab.
|Name||Displays name of the snapshot.|
|Description||Displays brief description of the snapshot.|
|Snapshot Type||Displays the type of snapshot. Snapshot type can be Full or Incremental.|
|Size||Displays the size of the snapshot.|
|Status||Displays the status of snapshot operation. The different statuses are as follows: Executing: Snapshot of workload instances is in progress. Uploading: Uploading data to snapshot storage. Available: Snapshot is successfully finished. Error: Snapshot failed.|
|Scheduled on||Displays the TrilioVault node on which the workload snapshot operation is scheduled.|
|Time taken||Displays the time taken for snapshot operation in seconds.|
|Restore Size||Displays size of the restore, if user wants to restore this snapshot.|
|Progress||Displays the progress of snapshot as a percentage of completed task.|
|VMs||Displays the list of VMs in the selected snapshot.|
|VMs – Instance Info.|
|Name||Displays the name of the Instance.|
|Status||Displays the status of the Instance.|
Figure 19: Snapshot Details – Instance Info tab.
Security tab lists the security groups attached to the Instance. The Security tab details are displayed in Figure 20 and described in Table 9.
Figure 20: Snapshot Details – Security Group tab.
Table 9: Security tab.
|Name||Displays the name of the security group, which the VM is using.|
|Type||Displays the type of the security group.|
Flavor tab describes the flavor attached to the Instance. The Flavor tab details are displayed in Figure 21 and described in Table 10.
Figure 21: Snapshot Details – Flavor tab.
Table 10: Flavor tab. Details Description Flavor VCPUs Displays the number of VCPUs that the VM is using. Disk Displays the amount of disk that the VM is using. RAM Displays the amount of memory that the VM is using.
Network tab details the network on which Instance is created. The Network tab details are displayed in Figure 22 and described in Table 11.
Figure 22: Snapshot Details – Networks tab.
Table 11: Network tab. Details Description Networks IP Address Displays the fixed IP Address of the VM. Network Displays the internal/tenant network name that the VM is connected to. Mac Address Displays the Mac Address of the VM.
Volume tab describes the volume attached to the instance. The Volume tab details are displayed in Figure 23 and described in Table 12.
Figure 23: Snapshot Details – Volumes tab.
Table 12: Volume tab. Details Description Volumes Name Displays the name of the attached volumes. Type Displays the name of the Cinder volume type. Size Displays the size of the volume. Mount Point Displays the directory name where the volume is mounted on the VM. Restore Size Displays the size of the volume for restore.
Misc. tab gives information about the UUID of the Instance. The Misc. tab details are displayed in Figure 24.
Figure 24: Snapshot Details – Misc. tab.
3.6.2. Mount a Snapshot¶
TrilioVault allows you to view or download a file from the snapshot. Any changes to the files or directories when snapshot is mounted are temporary and are discarded when the snapshot is unmounted. Mounting is a faster way to restore a single or multiple files. To mount a snapshot follow these steps.
TrilioVault ships with a ubuntu based file manager image. This image includes a web based file manager application that helps with browsing and download files when snapshot is mounted. Before you create a File Manager instance, File Manager (a.k.a Recovery Manager) image needs to be uploaded to the Glance. The cloud administrator can upload the image to Glance and mark the image public so that every tenant will have access to the image. File Manager image includes a qemu guest agent. TrilioVault data mover communicates with the qemu agent to map backup images and discover file systems in the backup images. Since the file manager image includes qemu guest agent, the image should be uploaded to glance with property hw_qemu_guest_agent=yes so Nova will create a qmp channel when creating an instance from this image. TrilioVault horizon plugin uses a special property tvault_recovery_manager, when set to yes it will filter out instances that don’t have this property. If a tenant has a huge number of instances, setting this property on the image will help TrilioVault horizon plug-in present only the instances whose image has this property. In this release we only support launching File Manager instance with virtio bus so please set hw_disk_bus to virtio. Use the following command to upload the image to Glance and mark it as public image and set the property.
glance image-create --name tvault-file-manager \ --file <File Manager Image Path> \ --container-format bare --disk-format qcow2 \ --progress --is-public=True \ --property hw_qemu_guest_agent=yes \ --property tvault_recovery_manager=yes \ --property hw_disk_bus=virtio
Once the image is successfully loaded, a tenant can instantiate file manager instance as described below.
- Launch the File Manager Instance. Refer to Figure 25 for details.
Figure 25: Launch File Manager Instance.
- Assign Security Group to open port 80
- Associate a floating IP to the launched instance.
- To mount a snapshot, login to TrilioVault and Navigate to Backups. Click <workload name>, then click Snapshots tab.
- Click More drop-down, then click Mount Snapshot. Refer to Figure 26 for details.
Figure 26: Mounting Snapshot.
- Assign the File Manager instance to the snapshot launched in Step 1. Refer to Figure 27 for details.
- Click Mount.
Figure 27: Assigning File Manager Instance to snapshot.
- Access File Manager using <floating IP> created in step 3. Refer to Figure 28 for details.
Figure 28: File Manager.
- Select the preferred action and double-click the selected action.
Figure 29: File Manager Options.
- The web-interface displays 5 options as shown in Figure 29.
Table 14: File Manager Options.
|Parent Folder||Allows end user to go back to parent folder of the selected file.|
|Download||Allows end user to download the file.|
|Rename||Allows end user to rename the file. (The change is local. It won’t change the backup copy.)|
|Delete||Allows end user to delete the file.(This change is local. It won’t change the backup copy.)|
|Replace file||Allows end user to replace the file with another file.(The change is local. It won’t change the backup copy.)|
- Choose appropriate action.
File manager instance contains the metadata that describes what snapshot is currently mounted.
184.108.40.206. In-place restore of a file/directory¶
The previous section explained how to recover a file or directory from a backup by using the snapshot mount command. However, the web portal based file/directory retrieval is cumbersome and is not very friendly for scripting. With this release, the TrilioVault Recovery Manager image is shipped with cloud-init functionality so you can instantiate the recovery manager instance with a key-pair so you can do a password-less ssh session to recovery manager instance. To do an in-place restore of files or directories, the user first performs a snapshot-mount operation on a backup image and uses the recovery manager instance. Once this operation is completed successfully, the user can mount mounted[CORRECT?] snapshot VM images using sshfs command as shown in the following figure.
sshfs email@example.com:/home/ubuntu/tvault-mounts/mounts mnt
In the above example, 172.24.4.4 is the floating IP address of recovery manager instance. mnt is the mount point. The mnt point can be browsed for VM images/file systems using standard Linux tools and copy required files.
sshfs-based mount operation is still subjected to standard Linux user permissions. For example a file that is owned by root and does not have read permissions to ssh user may not be copied
3.6.3. Unmount a Snapshot¶
Snapshot can be unmounted when access to individual files/directories are no longer needed. To unmount the snapshot, follow these steps.
- To unmount a snapshot, login to Horizon dashboard and navigate to Backups. Click <workload name>, then click Snapshots tab.
- Click Unmount Snapshot as shown in Figure 30.
Figure 30: Unmount Snapshot.
- On confirming the Unmount Snapshot action, TrilioVault will unmount the snapshot. Refer to Figure 31 for details.
Figure 31: Confirm Unmount.
- The snapshot gets unmounted successfully. Refer to Figure 32 for details.
Figure 32: Snapshot Unmounted
Sometimes it becomes daunting to find the snapshot that is mounted, in particular when there are many workloads and many snapshots. When a snapshot is mounted, the file manager instance metadata includes the snapshot ID that is currently mounted with the file manager. Please see the following screen grab from the file manager instance details.
Figure 33: File Manager instance metadata with mounted snapshot id
3.6.4. Restore a Workload Snapshot¶
To return the workload instances to their original state, a snapshot must be restored. When a snapshot is restored, it recreates the original workload instances in the exact same state as they were in, when the snapshot was taken. A single snapshot can be restored any number of times and the existing snapshots continue to be valid even after restoring any snapshots from the list. A snapshot can be restored in three ways.
- One-click Restore
- Selective Restore
- In-place Restore
220.127.116.11. One-click Restore¶
One-click Restore is a single click restore of the snapshot irrespective of the size of the snapshot. The snapshot may include multiple instances, networks or Cinder volumes. One-click is a convenient way of reverting an application to a predetermined point in time. For One-click Restore to succeed, the user needs to make sure that the original workload instances are deleted before triggering the restore operation. This method will restore the selected snapshot to the exact same location same tenant, network, volume types, and so on. To restore a snapshot with One-click, follow these steps.
- Delete all original workload instances from the OpenStack cloud as shown in Figure 34. The Instances display the deleted workload instances.
Figure 34: VM(s) to be Deleted.
- Login to Horizon dashboard and navigate to Backups. Click <workload name>, and then click Snapshots tab.
- Click One-click Restore menu option of the corresponding snapshot as shows in Figure 35.
Figure 35: Snapshot Selected for One-click Restore.
- One-click Restore screen as shown in Figure 36 is displayed.
Figure 36: One-click Restore.
- Enter the details OneClickRestore Name and OneClickRestore Description.
- Click Restore.
- It will restore all the workload instances at the same location with the same settings on OpenStack as shown in Figure 37.
Figure 37: One-click Restore in progress.
18.104.22.168. Selective Restore¶
Selective Restore allows for lot of flexibility to recover instances. With the Selective Restore method, end user can choose the target network, target volume types, include/exclude instances to restore, and change target flavor for each VM. instances will get restored as per the end user’s choice. For example, if a tenant would like to restore only selected instances and not all of the workload instances, then the tenant can do that using Selective Restore. Unlike the One-click Restore method, an end user does not need to delete the original workload instances to perform the selective restore operation. To proceed with Selective Restore, follow these steps.
- Login to Horizon and Navigate to Backups.
- Click <workload name>, then click Snapshots tab.
- Click Selective Restore as shown in Figure 38.
Figure 38: Snapshot Selected for Selective Restore.
- Selective Restore screen will be displayed as shown in Figure 39.
Figure 39: Selective Restore Details.
- Enter the details and navigate to the Network Mapping tab. Refer to Figure 40 for details. Select the target network you want to restore the snapshot to.
Figure 40: Selective Restore-Network Mapping.
- Navigate to Volume Types Mapping tab and select the target volume type you want to restore to, as shown in Figure 41.
Figure 41: Selective Restore-Volume Types Mapping.
- Each of the subsequent tabs correspond to workload instances created. There is a separate tab for each VM. Using these tabs you can edit restore settings of the respective VM. Select an instance’s tab. You will see options to choose to include/exclude that VM from restore, flavor type, and instance name.
- Deselect the checkbox if you do not want to include that particular VM in the restore process as shown in Figure 42.
Figure 42: Selective Restore-VM not included in Restore.
- Select Flavor, Enter Instance Name. Instance Name if changed will restore the instance with the new name.
- Click Restore.
- It will restore all the workload instances on OpenStack as shown in Figure 43. In the figure the restored VM in target IP can be seen.
Figure 43: Selective Restore instances.
22.214.171.124. In-place restore¶
Both OneClick Restore and Selective Restore create brand new resources as part of the restore process. In some cases, it is not desirable to create new resources. For example, an instance may have an application data residing on a Cinder volume and user only wanted to restore Cinder volume but leave the rest of the VM intact. In-Place Restore allows users to restore selected volume without the need to recreate the whole instance. In-Place restore was introduced in TrilioVault version 2.4.
Figure 44 In-place Resore Menu Item
The current release added a new menu item called In-place Restore which opens a new dialog.
Figure 45 In-place Restore Wizard
Figure 46 Choice of resources to restore
3.6.5. File Search¶
Another important introduced in 2.4 release is the ability to search for a file or list of files within a selected list of snapshots. TrilioVault takes image level backups, yet it provides file level/directory level recovery. However, if a user doesn’t know the latest snapshot where the desired file may reside, it is a painful process to mount numerous snapshots and look for the file. With file search feature, user can quickly find the list of snapshots that the file exists. File Search feature tab is added to workload details page as show below.
Your workload may have more than one VM. First select the VM and then choose the file path. File path may include wildcards which allows listing of files within a directory. For example if you want to seach all files with in/var/log/ directory, you can provide a search path /var/log/*.
By default, it searches all the snapshots available for the workload. TrilioVault file search feature directly searches the backup images. TrilioVault backup images are fully formed and are instantly accessible without the need to restore backup images. TrilioVault leverage this ability to search files and directories on the backup image itself. Though this process appears little slow compared to using some index based solutions, our file search is radically simple and easy to maintain. Also, if you were to transfer a workload from one cloud to another cloud, you can still search backup images in the new cloud. Though it supports multiple snapshots search facility, the search slows down if the number of snapshots in the search increases. Our file search facility provides various filters to narrow down the list of snapshots. From the list of snapshots in the table, user can deselect any snapshots from search. User can also narrow the search by latest number snapshots or by time range. Once user clicks ‘Search’, the results are displayed in a format that looks like ls –la.
126.96.36.199. Restore Details¶
To obtain Restore details information, follow these steps.
- Using Horizon dashaboard, navigate to Backups.
- Click <workload name> on the Workload Details screen.
- Navigate to the Snapshots tab and click the name of snapshot. The screen as shown in Figure 47 is displayed.
Figure 47: Snapshot Details.
- Navigate to the Restore tab and click <restore name>. The Restore Details screen as shown in Figure 48 is displayed.
Figure 48: Restore Details.
Table 15: Restore Details and Description.
|Name||Displays the name of Selective Restore.|
|Description||Displays the description of the Selective Restore.|
|Restore Type||Displays the type of restore. Restore can be One-click or Selective.|
|Status||Displays the status of the restore operation.|
|Time taken||Displays the time taken for restore operation in seconds.|
|Size||Displays the size of the restore.|
|Progress Message||Displays the progress message of the restore.|
|Progress||Displays the status of the progress.|
|Host||Displays the host name where the restore operation is taking place.|
|Name||Displays the name of the instance.|
|Status||Displays the status of the instance.|
|ID||Displays the ID of the instance.|
188.8.131.52. Deleting a Snapshot¶
Deleting the snapshot removes the snapshot from the workload. A parent or child snapshot can be deleted from the snapshot tree.
To delete a snapshot, follow these steps.
- Login to Horizon dashboard and navigate to Backups tab.
- Select the <workload name>. click Snapshots tab.
- Select the snapshot to be deleted and click Delete Snapshots.
Figure 49: Delete Snapshot.
3.7. Admin Console¶
This section details the Admin Console provided for the “Backup Administrator”. It is available in the TrilioVault dashboard as well as in the OpenStack dashboard. This console is typically provided for an Administrator to monitor workload snapshots, restores, snapshot storage, activity details on TrilioVault cluster and take necessary action. To use this feature user needs “admin role” on the respective tenant.
- Login to Horizon using admin user.
- Click on Admin Tab.
- Navigate to Backups-Admin Tab.
- Navigate to TrilioVault page.
Features of Admin Console
This tab displays a real time count of successful and failed snapshots/restores per workload. Also it has information regarding amount of storage used and time remaining for next snapshot for a given workload as shown in Figure 50.
Figure 50: Workloads
3.7.2. Configuration Backup¶
Since TrilioVault 2.5 it is possible to backup Openstack configurations. Admin can manage configuration backups under this tab. For additional details on how to enable configuration backup, please refer to KB article OpenStack Configuration Backup. The configuration tab looks as follows:
Figure 51: Configuration Backup
OpenStack clouds are typically managed by sophisticated devops processes that include either Anisible/Salt or similar devops tools. Usually configuration changes made with these tools are versioned. Some enterprises with compliance regulations or enterprises without the mentioned tools still need to take backups of the Openstack configuration on a regular basis. Triliovault supports configuration backup, by saving all configuration files and databases releated to all Openstack services.
Configuration backup operating parameters can be managed by Config Backup Settings. These settings can be adjusted in the Settings tab of the Configuration Backup tab. The Schedule tab allows the set up of a configuration backup on a regular basis.
The policy tab allows to define how many backups should be retained.
Services to Backup tab can be used to specify the services to backup. YAML based syntax is used to configure the services to backup. TrilioVault backups the core services in OpenStack by default. The default config dirs and log directories of each service is specified in the file. Ofcourse each service configuration can be customed to adjust each OpenStack cloud environment.
This tab displays information about TrilioVault cluster nodes like node IP address, hostname, version of TrilioVault installed etc. For more details refer to Figure 52.
Figure 52: Nodes
3.7.4. Data Movers (TrilioVault Data Mover Service)¶
This tab displays information about TrilioVault contego service such as service status, compute node, version etc. For more details refer to Figure 53.
Figure 53: Data Movers
This tab displays information about snapshot storage such as percentage storage utilized, storage usage by snapshots per workload. It also displays information about full vs incremental snapshot’s storage usage, count. For more details refer to Figure 54.
Figure 54: Storage panel with Swift as backup target
Figure 55: Storage panel with NFS as backup target
Audit logs provide the sequence of workload related activities done by users, like workload creation, snapshot creation, etc. A user can search the audit logs based on a date for which a filter is available. For more details refer to Figure 56.
Figure 56: Audit
This tab manages all global settings for the whole cloud. TrilioVault has two types of settings:
- Email settings
- Job scheduler settings.
184.108.40.206. Email Settings¶
These settings will be used by TrilioVault to send email reports of snapshots and restores to users. Users have to enable email alerts setting in “TrilioVault Settings” tab in the users tenant to activate the feature. For more details refer to Figure 57: Email Settings.
Figure 57: Email Settings
220.127.116.11. Disable/Enable Job Scheduler¶
Deactivating the Job Scheduler will result in deactivating all scheduled jobs for all workloads in the entire cloud. User defined schedules can’t override this setting. For more details see Figure 58
Figure 58: Global Job Scheduler
This tab displays the TrilioVault license information, as shown in Figure 59: License
Figure 59: License
Admin can upload new license or check the validity of existing license.
3.7.9. Workload Policy¶
TrilioVault’s tenant driven backup service gives tenants control over backup policies. However, sometimes it may be too much control to tenants and the cloud admins may want to limit what policies are allowed by tenants. For example, a tenant may become over zealous and only uses full backups every 1 hr interval. If every tenant were to pursue this back up policy, it puts severe stain on cloud infrastructure. Instead if cloud admin can define predefined backup policies and each tenant is only limited to those policies then cloud administrator can excert better control over backup service.
Workload policy is similar to nova flavor where a tenant cannot create arbitrary instances. Instead each tenant is only allowed to use the nova flavors published by the admin.
The policy tab in Figure 60: Policy lists the workload policies defined for the cloud. Administrator can create new workload policy as shown in following figure.
Figure 60: Policy create option
Figure 61: New Policy name and description
Figure 62: Attributes of new policy
Admin should assign one or more policies to a tenant to limit the tenant to only those policies. If a tenant is not assigned any policy, the tenant is allowed to choose any workload policy. Admin can assign the policy by choosing the following item from drop down menu.
Figure 63: Assign policy to tenant
Figure 64: Assign policy to tenant
Usage tab provides administrator to monitor backup resources usage for each tenant.
Figure 65: Tenant usage
3.7.11. TrilioVault Settings¶
This section details the “TrilioVault Settings” feature. This feature is provided to configure global settings that can be used across all tenants in TrilioVault. In current release user has only setting available i.e email alerts. For more details see following steps with Figure 66
User email adresses can be set up in identity->users section of Horizon.
Using this feature from Horizon Dashboard:
- Login to Horizon (Admin not required).
- Click on Project Tab.
- Navigate to Backups Tab.
- Navigate to TrilioVault Settings page.
- Check “Enable Email Alerts” checkbox to receive email alerts.
Figure 66 TrilioVault Settings
3.7.12. Disaster Recovery with TrilioVault¶
TrilioVault introduced two new APIs in this release that helps users to implement disaster recovery usecases for their workloads. To recovery workloads on a remote cloud, TrilioVault relies on the underlying backup target replication technology as shown in the following diagram.
In the above diagram, TrilioVault is configured to use NFS as backup target. The NFS backup repository is then replicated to a remote site. All the backup images including full and incremental backups (including the metadata) are replicated to the remote site. If the production site is down, any point in time copy of the backup image can be recovered to the remote cloud with a click of a button. This section describes how the backup images are recovered from the replicated backup repository to the remote cloud.
When a site failure happens in a production environment it is assumed that the NFS repository fails over to the DR site. The remote repository is then available in read/write mode at the DR site. It is also assumed that TrilioVault is deployed at the remote site as per the TrilioVault deployment guidelines.
18.104.22.168. Discover Workloads from a replicated NFS target at DR Cloud¶
TrilioVault provides a CLI command that discovers all workloads, which are not assigned to a tenant.
[centos@centos7 devstack]$ workloadmgr workload-get-orphaned-workloads-list +------------+--------------------------------------+----------------------------------+----------------------------------+ | Name | ID | Project ID | User ID | +------------+--------------------------------------+----------------------------------+----------------------------------+ | workload-1 | 7e27a3ed-10ac-44c0-94cc-f6df0297d9a6 | b67703419b0b47558d1ff349c78f94c7 | cd9b679980a049abacec183e4246461a | | workload-2 | 686c17fa-dc65-4f5b-91fd-0df0ee3dbee0 | b67703419b0b47558d1ff349c78f94c7 | cd9b679980a049abacec183e4246461a | | workload-3 | c06ec529-002f-4cca-b2c6-04c3cff6151b | b67703419b0b47558d1ff349c78f94c7 | cd9b679980a049abacec183e4246461a | | workload-4 | 18be9b1c-0da3-44df-b2fa-e6f4f9e45bc9 | b67703419b0b47558d1ff349c78f94c7 | cd9b679980a049abacec183e4246461a | | workload-5 | 5bbfce15-0b4f-43fa-afbc-19fb1cecd22d | b67703419b0b47558d1ff349c78f94c7 | cd9b679980a049abacec183e4246461a | +------------+--------------------------------------+----------------------------------+----------------------------------+
22.214.171.124. List Tenants on Disaster Recovery Cloud¶
[centos@centos7 devstack]$ keystone tenant-list +----------------------------------+--------------------+---------+ | id | name | enabled | +----------------------------------+--------------------+---------+ | f16b93dc2cfb486c91942ba559846c48 | admin | True | | 6ff8b527114e445fb2ed2e15cb357d39 | alt_demo | True | | f70ad4d8cd134eef9a8448a6720e37b4 | demo | True | | 078c8244166541abb17e0918cb57dffe | invisible_to_admin | True | | d70d345c58614770937d85dd991b01b2 | service | True | +----------------------------------+--------------------+---------+
126.96.36.199. List Users on Disaster Recovery Cloud¶
[centos@centos7 devstack]$ keystone user-list +----------------------------------+-------------+---------+----------------------------+ | id | name | enabled | email | +----------------------------------+-------------+---------+----------------------------+ | e5e74c85d2594139a0149665cbbc66ed | admin | True | | | 8db192578c174debbfceebf07cd66cd0 | alt_demo | True | firstname.lastname@example.org | | e9c207d9849a45b6b84746371bc3ca93 | cinder | True | | | 18d9e959ce354f3f92bd3437810ba5aa | demo | True | email@example.com | | a50d15c7a58e459e9f54eca3902fff86 | glance | True | | | 89cb90200f1e4c5aae0e1dbe228f72ef | neutron | True | | | d7657b20b4ce4874add512f9b49ebd49 | nova | True | | | 164b800bb070465dbb5b552d1fad075b | triliovault | True | firstname.lastname@example.org | +----------------------------------+-------------+---------+----------------------------+
188.8.131.52. Reassign a Workload to desired tenant at Disaster Recovery Cloud¶
[centos@centos7 devstack]$ workloadmgr workload-reassign-workloads --new_tenant_id f70ad4d8cd134eef9a8448a6720e37b4 --user_id 18d9e959ce354f3f92bd3437810ba5aa --workload_ids 686c17fa-dc65-4f5b-91fd-0df0ee3dbee0 --migrate_cloud True +------------+--------------------------------------+----------------------------------+----------------------------------+ | Name | ID | Project ID | User ID | +------------+--------------------------------------+----------------------------------+----------------------------------+ | workload-2 | 686c17fa-dc65-4f5b-91fd-0df0ee3dbee0 | f70ad4d8cd134eef9a8448a6720e37b4 | 18d9e959ce354f3f92bd3437810ba5aa | +------------+--------------------------------------+----------------------------------+----------------------------------+
workload-reassing-workloads command imports the specified workload into the OpenStack tenant. Once the workload is imported, the user can utilize the TrilioVault Selective Restore to restore any snapshot in the imported workload
One-Click restore won’t work
184.108.40.206. Bulk Workload Recovery¶
workload-reassign-workloads command to assign workloads provides an additional feature to enable cloud administrators to do bulk workload imports. To use this feature a YAML file needs to be created, that describes all desired workload to tenant/user mappings. An example is shown below:
reassign_mappings: - new_tenant_id: f70ad4d8cd134eef9a8448a6720e37b4 user_id: 18d9e959ce354f3f92bd3437810ba5aa workload_ids: - "7e27a3ed-10ac-44c0-94cc-f6df0297d9a6" - "c06ec529-002f-4cca-b2c6-04c3cff6151b" - "18be9b1c-0da3-44df-b2fa-e6f4f9e45bc9" - "5bbfce15-0b4f-43fa-afbc-19fb1cecd22d" migrate_cloud: True
workloadmgr workload-reassign-workloads --map_file /tmp/reassign.yml +------------+--------------------------------------+----------------------------------+----------------------------------+ | Name | ID | Project ID | User ID | +------------+--------------------------------------+----------------------------------+----------------------------------+ | workload-1 | 7e27a3ed-10ac-44c0-94cc-f6df0297d9a6 | f70ad4d8cd134eef9a8448a6720e37b4 | 18d9e959ce354f3f92bd3437810ba5aa | | workload-3 | c06ec529-002f-4cca-b2c6-04c3cff6151b | f70ad4d8cd134eef9a8448a6720e37b4 | 18d9e959ce354f3f92bd3437810ba5aa | | workload-4 | 18be9b1c-0da3-44df-b2fa-e6f4f9e45bc9 | f70ad4d8cd134eef9a8448a6720e37b4 | 18d9e959ce354f3f92bd3437810ba5aa | | workload-5 | 5bbfce15-0b4f-43fa-afbc-19fb1cecd22d | f70ad4d8cd134eef9a8448a6720e37b4 | 18d9e959ce354f3f92bd3437810ba5aa | +------------+--------------------------------------+----------------------------------+----------------------------------+
The Admin can also transfer workloads belonging to a tenant on the production site to a tenant ID at the disaster recovery cloud using the following YAML file.
reassign_mappings: - new_tenant_id: f70ad4d8cd134eef9a8448a6720e37b4 user_id: 18d9e959ce354f3f92bd3437810ba5aa old_tenant_ids: - "634bd395-c5ad-4f1d-9d2a-db5a6809c99c" - "82967045-f4f8-494c-b3bd-d569afe54433" migrate_cloud: True
In the above example all workloads that belong to old_tenant_ids are transferred to new_tenant_id.
3.8. Best Practices¶
The checklist given below can help in minimizing the risk associated with data management and ensure stability and growth as your OpenStack cloud grows
- The best way to deploy a TrilioVault cluster in production environment is as a VM(s) on a standalone KVM box.
- Best TrilioVault node configuration (TrilioVault appliance flavor) to get optimum performance in production environment is: RAM: 24GB, Root Disk: 40GB, VCPUs: 4
- To increase the backup performance, the bandwidth between compute nodes and snapshot storage (NFS/Swift) should be enough for TrilioVault copy snapshot data from compute node to snapshot storage.
- For snapshot storage mechanisms, devise a plan for a. Backup of snapshot storage. b. Security of the snapshot. c. Using deduplication for efficient use of snapshot storage.
- Frequency of backups depend upon the Recovery Time Objective (RTO) and Recovery Point Objective (RPO). TrilioVault uses forever forward incremental backup method by default. It means it creates a backup chain that consists of the first full backup and a set of forward incremental backups. But its behavior can be changed by scheduling full backups frequency. For example, a user can configure a workload on TrilioVault to take a full backup after every 10 incremental backups. However, incremental backups complete quickly and takes less storage on backup media, so compared to full backups incremental backups are efficient and saves space and time.
- Retention policy can be set up depending upon the environmental need. For example, if the retention policy is set to retain 100 snapshots, it will retain latest 100 snapshots and will delete older snapshots. While configuring this policy, decide how many old snapshots are required in future to recover/revert the environment. Retention policies can also be set by configuring a number of days to retain a snapshot. For example, if it is set to 10 days, TrilioVault will retain all the snapshots taken in last 10 days and will delete older snapshots. Frequency of taking snapshots also impacts on retention policy, so it is necessary to configure it accordingly.
- Decide on the best workload size. A workload is a logical grouping of related instances. For example, if there is a 50 node MongoDB cluster deployed on OpenStack cloud, all 50 nodes will be part of one backup job. As another example, if an application is deployed as 5 webserver instances, 10 DB server instances and 1 security gateway instances, workload definition will incude all 16 instances so they all backed up in single snapshot.
- Decide on throttling the number of simultaneous uploads per hypervisor. User can trigger any number of workload snapshots simultaneously. If the instances are all running on a single compute node all the backups are performed by that single compute node. In order not to overload a compute node with backup jobs, TrilioVault has a feature by which admin can configure a maximum number of allowed backups on a given compute at any given time. This setting need to be done on each compute node in /etc/tvault-contego/tvault-contego.conf with parameter key max_uploads_pending. By default, it is set to 3.
- Deciding which Restore mechanism to restore the lost or corrupted data. One-click Restore allows to restore all the data in one click to its original place, but to use this you need to clean all old instances or applications of the same workload. Selective Restore which allows to select a target network, volume type, and instances. To restore only selected instances that are corrupted, Selective restore mechanism can be used. If you would like to see a file content before restoring a snapshot, you can use File Recovery Manager under Mount a Snapshot
3.8.1. Frequently Asked Questions¶
|Instance||An instance is a virtual server in OpenStack for running applications/Operating system on the OpenStack infrastructure.|
|Workload||A workload is a collection of work defined in TrilioVault for data protection process.|
|Snapshot||A snapshot in TrilioVault is the backup of your Virtual machine which you can use during failure or data loss.|
|QCOW2||QCOW2 is the format of the TrilioVault image used for the deployment of TrilioVault.|
|TrilioVault Image||An image is a collection of files for a specific OS. TrilioVault Image is the image created with the help of the QCOW2 file provided to the user for deployment.|
|Flavor||Flavor is the set of resources that are made available to the user for performing backup and restore.|
|Controller Node||Controller node of the OpenStack.|
|Compute Node||Compute node of the OpenStack.|
|Floating IP||Floating IP is the IP created on the fly to launch an instance. This IP is associated with the project so that the user can work on the same instance each time it logs in.|
|Tvault-contego||Tvault-contego is TrilioVault Nova Extension which should be installed on all OpenStack Controller and Compute node.|
|Tenant/Projects||Tenants or Projects are units in OpenStack to which you can assign users. A project is a group of one or more user.|