Backing Up and Restoring Runtime Fabric

Use the following procedures to back up and restore Runtime Fabric services and the required state necessary for all deployed applications and their configurations.

Data about the state of Runtime Fabric is distributed across the Kubernetes cluster. To preserve applications in the event of system failure, confirm that Runtime Fabric is automatically backed up. When scheduling backups, follow best practices by configuring hourly backups. Store your backup on external storage outside the Runtime Fabric cluster.

For Runtime Fabric on VMs / Bare Metal, the backup and restore process includes backing up and restoring appliance-related custom resources such as logforwarder and smtp. The backup and restore process does not include backing up metrics data stored in InfluxDB that is generated by the applications.

You cannot use the back and restore process to migrate from Runtime Fabric for VMs / Bare Metal to Runtime Fabric on Self-Managed Kubernetes.

When to Use the Back Up and Restore Process

Common scenarios for using the backup and restore process include:

Performing a major infrastructure upgrade, such as migrating nodes in a Runtime Fabric cluster to a new version of an operating system.
Upgrading Runtime Fabric for VMs / Bare Metal. In some cases, you may want to back up your cluster, install a new cluster using a new Runtime Fabric version on that same cluster, and then restore your backup on that cluster.
Performing a failover to a standby cluster.

Do not use the backup and restore process to create concurrent running instances of Runtime Fabric. In other words, do not back up Runtime Fabric A, use that back up to create a new Runtime Fabric B while Runtime Fabric A is still active, and attempt to run them concurrently. This creates application deployment issues.

What Gets Backed Up

The Runtime Fabric backup process affects only the Runtime Fabric cluster. The backup process does not back up data from the control plane.

The following is backed up:

Runtime Fabric cluster details, such as controller and worker node configurations
Cluster-level secure properties
All data in the Runtime Fabric and application namespace, such as data about deployments, services and service accounts, secrets, ingress resources, role bindings, cluster role configurations, and ConfigMap settings
The current rtfctl version
The current Runtime Fabric agent version
Application details including:
- Application configurations for pods, replicasets, application metadata, and templates
- Application properties configured in Runtime Manager
Data from other Mulesoft configurations, including:
- Object Store
- Edge
- Anypoint Monitoring and external logforwarders

What Does Not Get Backed Up

Data about application state
Data from Ops Center
Data from the control plane

Changes Made After Performing a Backup Are Not Restored

Any changes you make after you perform a backup will not be present if you restore the cluster from that backup. For example, if you perform a backup, delete an application, and then restore from that backup, the application you deleted will be recreated in the cluster, but it will be marked as deleted in Runtime Manager. As a result the application’s status will show as applying in Runtime Manager.

Runtime Fabric does not restore the following control plane changes if you make them after you take a backup:

Created apps
Delete apps
Updated app configurations
Agent upgrades

You need to manually resolve such inconsistencies by reapplying the changes you made after performing the backup, as these changes can’t be reapplied from the control plane.

Create a Backup

Before creating a backup:

Verify that Runtime Fabric is installed and running successfully.
Verify that rtfctl command-line tool is upgraded to version 0.3.102 or later. Verify the version by running rtfctl version on any Runtime Fabric node.

Creating a backup per node, such as via a VM snapshot, is not supported.

To create a backup run the following command:

./rtfctl backup <path_to_backup_file>

This command creates a backup of the current system state in <path_to_backup_file>, which can be any path in the file system that you have write access to, such as: /opt/anypoint/runtimefabric/backup.tar.gz.

For Runtime Fabric on VMs / Bare Metal, the rtfctl binary is in directory /opt/anypoint/runtimefabric/. Confirm that you run the backup command on a controller node in the cluster as a privileged user. To use the backup for restoring later, copy the backup file to secure storage on a different machine that is not part of Runtime Fabric.

The following command, for example, uses scp to copy the backup to another drive:
```
scp your_username@remotehost:/opt/anypoint/runtimefabric/backup.tar.gz /backup-path/to-restore.tar.gz
```
For Runtime Fabric on Self-Managed Kubernetes, confirm that the rtfctl binary is present in the current directory and kubectl is present in the user $PATH. Additionally, confirm that you’ve set the Kubernetes (kubectl) context to the intended backup cluster.

Perform a Restore

Before you restore a cluster, review the following information:

You must perform the restore on the same Runtime Fabric cluster that is listed in Runtime Manager (the same cluster in the control plane).
Configuration changes you make to deployed applications and management services after a back up are not restored.
Application monitoring metrics are not restored.
When restoring on an existing Runtime Fabric cluster, use the same version of the rftctl command-line utility that you used to create the backup.
For Runtime Fabric on Self-Managed Kubernetes, a restore does not require the control plane. For Runtime Fabric on VMs / Bare Metal a connection with the control plane is required, as you must first install the applianace.

Runtime Fabric provides two target options when restoring a cluster from a backup:

Use an existing Runtime Fabric cluster.
Create a new Kubernetes cluster with the same configuration as the backed-up cluster. This includes the same number of servers, disks, etc.

To restore Runtime Fabric appliance cluster on VMs / Bare Metal, run the Runtime Fabric installer scripts without providing the Runtime Fabric activation data. After you run the install scripts, complete the restore process.

To restore a cluster on Runtime Fabric on Self-Managed Kubernetes, create a cluster without running the rtfctl install command. After you create the cluster, complete the restore process.

Choose a target option for restoring a cluster.
Ensure you have installed rftctl in the cluster.
Copy the backup file you previously created, and make sure it is available to rtfctl.
- For Runtime Fabric on VMs / Bare Metal, copy the compressed backup file to a directory in a controller node of the environment to be restored. For example, you can transfer this file securely via the following command:
  scp /backup-path/to-restore.tar.gz your_username@remotehost:/opt/anypoint/runtimefabric/
Confirm your Kubernetes (kubectl) context is set to the backed-up cluster, and scale down all Runtime Fabric components on the original backed-up cluster:
```
kubectl scale --replicas=0 -n rtf deployment.apps/agent
```
Restore the cluster from the backup:

./rtfctl restore <path_to_backup_file>

+ This process may require several minutes to complete.

For Runtime Fabric on Self-Managed Kubernetes, confirm the rtfctl binary is present in the current directory and the Kubernetes (kubectl) context is set to the cluster you are restoring to.