Backing Up and Restoring Runtime Fabric

Use the following procedures to back up and restore Runtime Fabric services and the required state necessary for all deployed applications and their configurations.

Data about the state of Runtime Fabric is distributed across the Kubernetes cluster. To preserve applications in the event of system failure, confirm that Runtime Fabric is automatically backed up. When scheduling backups, follow best practices by configuring hourly backups. Store your backup on external storage outside the Runtime Fabric cluster.

Starting with rtfctl version 1.0.70, if you use an rtf-agent version higher than 2.6.52, the rtfctl backup bundle does not contain the Mule app deployments. The deployments are synched from the Runtime Manager UI at the time of restoration.

When to Use the Back Up and Restore Process

Common scenarios for using the backup and restore process include:

Performing a major infrastructure upgrade, such as migrating nodes in a Runtime Fabric cluster to a new version of an operating system.
Migrating your Runtime Fabric cluster from on-premises to the cloud or migrating from one cloud provider to another.
Performing a failover to a standby cluster.

Do not use the backup and restore process to create concurrent running instances of Runtime Fabric. In other words, do not back up Runtime Fabric A, use that back up to create a new Runtime Fabric B while Runtime Fabric A is still active, and attempt to run them concurrently. This creates application deployment issues.

What Gets Backed Up

The Runtime Fabric backup process affects only the Runtime Fabric cluster. The backup process does not back up data from the control plane.

For regular Kubernetes clusters, the following is backed up:

Runtime Fabric cluster details, such as controller and worker node configurations
Cluster-level secure properties
All data in the Runtime Fabric and application namespace, such as data about deployments, services and service accounts, secrets, ingress resources, role bindings, cluster role configurations, and ConfigMap settings
The current rtfctl version
The current Runtime Fabric agent version
Application details including:
- Application configurations for pods, replicasets, application metadata, and templates
- Application properties configured in Runtime Manager
Data from other Mulesoft configurations, including:
- Object Store
- Edge
- Anypoint Monitoring and external logforwarders

For OpenShift clusters, the following is backed up:

Custom Resources
Application namespaces and for each one:
- Secrets
- Service Account
- ConfigMap
- Services
- CronJobs
- DaemonSets
- Deployments
- Ingress
- Role
- Role Bindings
Ingress from Runtime Fabric namespace

What Does Not Get Backed Up

Data about application state
Data from Ops Center
Data from the control plane

Changes Made After Performing a Backup Are Not Restored

Any changes you make after you perform a backup will not be present if you restore the cluster from that backup. For example, if you perform a backup, delete an application, and then restore from that backup, the application you deleted will be recreated in the cluster, but it will be marked as deleted in Runtime Manager. As a result the application’s status will show as applying in Runtime Manager.

Runtime Fabric does not restore the following control plane changes if you make them after you take a backup:

Created apps
Delete apps
Updated app configurations
Agent upgrades

You need to manually resolve such inconsistencies by reapplying the changes you made after performing the backup, as these changes can’t be reapplied from the control plane.

Create a Backup

Before creating a backup:

Verify that Runtime Fabric is installed and running successfully.
Verify that rtfctl command-line tool is upgraded to version 0.3.102 or later. Verify the version by running rtfctl version on any Runtime Fabric node.
Confirm that the rtfctl binary is present in the current directory and kubectl is present in the user $PATH. Additionally, confirm that you’ve set the Kubernetes (kubectl) context to the intended backup cluster.

Creating a backup per node, such as via a VM snapshot, is not supported.

To create a backup, run the following command:

./rtfctl backup <path_to_backup_file>

This command creates a backup of the current system state in <path_to_backup_file>, which can be any path in the file system that you have write access to, such as: /opt/anypoint/runtimefabric/backup.tar.gz.

Perform a Restore

Before you restore a cluster, review the following information:

You must perform the restore on the same Runtime Fabric cluster that is listed in Runtime Manager (the same cluster in the control plane).
Configuration changes you make to deployed applications and management services after a back up are not restored.
Application monitoring metrics are not restored.
When restoring on an existing Runtime Fabric cluster, use the same version of the rftctl command-line utility that you used to create the backup.

Runtime Fabric provides two target options when restoring a cluster from a backup:

Use an existing Runtime Fabric cluster.
Create a new Kubernetes cluster with the same configuration as the backed-up cluster. This includes the same number of servers, disks, etc.

To restore a cluster on Runtime Fabric, create a cluster without running the rtfctl install command. After you create the cluster, complete the restore process.

Choose a target option for restoring a cluster.
Ensure you have installed rftctl in the cluster.
Copy the backup file you previously created, and make sure it is available to rtfctl.
Confirm your Kubernetes (kubectl) context is set to the backed-up cluster, and scale down all Runtime Fabric components on the original backed-up cluster:
```
kubectl scale --replicas=0 -n rtf deployment.apps/agent
```
Restore the cluster from the backup:

./rtfctl restore <path_to_backup_file>

This process may require several minutes to complete.

For Runtime Fabric, confirm the rtfctl binary is present in the current directory and the Kubernetes (kubectl) context is set to the cluster you are restoring to.

Create a Backup and Restore for OpenShift Clusters

In the following example, you have a Runtime Fabric with a set of Mule applications and configurations deployed on cluster (named cluster-1) inside an OpenShift cluster. To create a backup and perform a restore, follow these steps:

Run the backup command:

./rtfctl backup <path_to_backup_file> -n <rtf_namespace>

MuleSoft performs the back up based on the detected cluster you have. Though, you can also manually specify the type of back up to perform using the flag --backup-type string. The flag options are auto, openshift, and full. With the default auto flag option, MuleSoft automatically detects the infrastructure cluster provider to perform the back up.

Shutdown cluster-1.
Delete all Mule application deployments by running the following command:
```
kubectl delete ns <app_namespace>
```
Do not remove applications from the Runtime Manager UI.
Remove the Runtime Fabric installation.
Remove the Runtime Fabric operator from the OpenShift cluster (optional).
Send a curl command to the management plane API endpoint to regenerate the activation ID.
The Runtime Fabric shows as ready for activation. You can install and activate Runtime Fabric again on a new OpenShift cluster.
```
curl --location --request PATCH 'https://anypoint.mulesoft.com/runtimefabric/api/organizations/{org-id}/fabrics/{fabric-id}/regenerateActivationData' \
--header 'Authorization: Bearer {bearer-token}
```
Install the Runtime Fabric operator on cluster-2 using the regenerated activation data ID.
The installation must be on the same Runtime Fabric namespace. All labels on the backup point to that installation.
Run the restore command to pick the set of resources and restore them in the cluster:
```
./rtfctl restore <path_to_backup_file> -n <rtf_namespace>
```
Check the status of your Runtime Fabric deployments.