Upgrading Runtime Fabric on VMs/Bare Metal

Where possible, we changed noninclusive terms to align with our company value of Equality. We maintained certain terms to avoid any effect on customer implementations.

To upgrade Anypoint Runtime Fabric on VMs/Bare Metal, upgrade the Runtime Fabric appliance, components, the rtfctl command-line utility, and node configurations.

About Appliance Upgrades

Appliance upgrades affect platform-layer components, which include Docker, Kubernetes, and the Ops Center. These platform-layer components are installed on virtual machines (VMs) or bare-metal servers that act as nodes on the cluster.

Appliance upgrades are applied using the rtfctl command-line utility.

Appliance upgrades are applied to each node, one at a time. This rolling upgrade process minimizes the impact to running applications by ensuring that the minimum three controller nodes are still active, ensuring cluster availability. This process also ensures that at least one worker node’s worth of unused resources are available to allow for applications to be redeployed as each worker node is taken offline to be upgraded.

About Component Upgrades

Upgrading Runtime Fabric components means upgrading services that:

Facilitate communication to the control plane
Provide load balancing
Log and forward metrics

When component upgrades are available, you apply the upgrades within Runtime Manager via a button that’s visible when an upgrade is available. Review the Anypoint Runtime Fabric Release Notes for information about the latest component versions.

Runtime Fabric component upgrades do not require adding a node.

Application monitoring metrics are not restored after an upgrade.

Determine the Component Versions You Are Using

On a Runtime Fabric controller node, run the following command:

/opt/anypoint/runtimefabric/rtfctl version

The output looks something like this:

For example:

[root@ip-10-0-245-74 /]# /opt/anypoint/runtimefabric/rtfctl version
COMPONENT    VERSION
rtfctl       0.3.122
agent        1.9.3
appliance    1.1.1719717580-73a133d
kubernetes   1.17

The rtfctl command-line utility version in the rtfctl field
The appliance component version in the appliance field
The Runtime Fabric component version in the agent field

The Runtime Fabric component and appliance versions that are in use are also displayed in the Runtime Fabrics tab in Runtime Manager. If an upgrade is available, an "Upgrade to vx.x" message is displayed.

About `rtfctl` Command-line Utility Upgrades

The rtfctl command-line utility is used to manage Runtime Fabric and perform appliance and node configuration upgrades.

You upgrade the rtfctl utility from the command line.

It is important to use the most current version of the rtfctl command-line utility.

About Node Configuration Upgrades

Node configuration upgrades involve scripts for provisioning AWS and Azure infrastructure, preparing each node, bootstrapping a new Runtime Fabric, adjusting operating system limits, and performing jobs to clean up system resources.

Node configuration upgrades are delivered as an updated install script version using the Runtime Fabric downloads page in Runtime Manager.

If you are upgrading the node configuration with updated install scripts, the latest version of the appliance is automatically downloaded and used.

How the Upgrade Process Works

To upgrade Anypoint Runtime Fabric on VMs/Bare Metal, perform the following tasks:

Upgrade the rtfctl command-line utility.
Verify that Runtime Fabric is healthy.
Upgrade Runtime Fabric components.
Upgrade the Runtime Fabric appliance.
Run the upgrade in the cluster.
Upgrade node configurations.
Update the Mule deployments running on Runtime Fabric to the latest patch version.

Before You Perform the Upgrade Process

Before upgrading your production environment, perform these steps on a nonproduction Runtime Fabric instance to verify compatibility with your environment. Plan to upgrade when your team is not deploying new applications or managing existing applications on Runtime Fabric. When your nonproduction upgrade succeeds, perform these steps in your production environment. Review the Anypoint Runtime Fabric Release Notes information.
Provision additional worker nodes as needed to maintain availability of applications when applying appliance upgrades.

If the amount of available resources for application deployments is less than the resources of one worker node, add another worker node with the same amount of CPU, memory, and disk resources to Runtime Fabric.

See How to Add a Node to Runtime Fabric.
Review What Items to Check Before an RTF Appliance Upgrade from the MuleSoft Knowledge Base.

Upgrade the `rtfctl` Command-Line Utility

The rtfctl command-line utility is used in several steps during the upgrade process.

If you are not using the latest version of the rtfctl command-line utility, download the latest version with the following command:
```
cd /opt/anypoint/runtimefabric
sudo curl -L https://anypoint.mulesoft.com/runtimefabric/api/download/rtfctl/latest -o rtfctl
```
Change file permissions for the rtfctl binary:
```
sudo chmod +x rtfctl
```
If you are a user who owns the rtfctl binary, you can update rtfctl without needing to download a new version. To do so, run the following command:
```
`rtfctl update`
```

Verify That Runtime Fabric is Healthy

Before running the upgrade, ensure the cluster is active and healthy.

Using your web browser, navigate to the Runtime Fabrics tab in Runtime Manager and verify that Runtime Fabric reports an Active status.
Open a terminal to a Runtime Fabric controller node.
Run /opt/anypoint/runtimefabric/rtfctl status and confirm that the cluster status is healthy.
Keep your browser session open to perform the remainder of the upgrade subtasks.

Upgrade Runtime Fabric Components

The first step in the Runtime Fabric process is upgrading the Runtime Fabric components.

Before You Upgrade Runtime Fabric Components

Verify that you have allowlisted the following endpoints on port 443, used by the runtime-registry to retrieve Docker containers for app deployments.
- US control plane: rtf-runtime-registry.kprod.msap.io
- EU control plane: rtf-runtime-registry.kprod-eu.msap.io
Verify that you have allowlisted the endpoints used for Mutual Transport Layer Security (mTLS) between the Anypoint Platform control plane and your Runtime Fabric.

After upgrading to Runtime Fabric, you must restart applications so they can update their deployment manifests for the runtime-registry endpoint.

Navigate to your Runtime Fabric by selecting it on the Runtime Fabrics tab in Runtime Manager.
Click Upgrade to vx.x.
Click Run Network Test to perform the network check to ensure you’ve allowlisted the correct endpoints for the runtime registry and mTLS with the Anypoint Control plane.

If you haven’t allowlisted the endpoints, you must do so before continuing.

The network test may take several minutes to complete. If you encounter any errors, refer to Network Test Errors.
When the network test completes, click Apply Upgrade.

The status of the Runtime Fabric is updated to show that the upgrade procedure is in progress.
Wait for the status to transition to Active.

In most cases, the upgrade takes fewer than 10 minutes. When the status transitions to Active and the version of the Runtime Fabric reflects the new version, the component upgrade is complete.

If the Runtime Fabric and Runtime Manager connection disconnects during the upgrade, connectivity is restored after the upgrade finishes.
Keep your browser session open to perform the remainder of the upgrade subtasks.

Upgrade the Runtime Fabric Appliance

After you upgrade the Runtime Fabric components, you can upgrade the Runtime Fabric appliance.

Before You Upgrade the Runtime Fabric Appliance

Verify that /opt/anypoint/runtimefabric/installer directory has a minimum of 4 GiB of disk space available.
Verify that you have root privileges.
Verify that your environment includes:
- An internal load balancer running on at least three replicas, to maintain availability. On newer versions of Runtime Fabric, the internal load balancer runs on all controller nodes by default.
- An external load balancer configured to load-balance incoming requests to the controller nodes, with a health check configured for TCP port 443.
- CPU and memory resources for at least one additional worker node available in the cluster.
  
  This enables safe removal of a worker node from the cluster to apply upgrades without impacting application availability.
- Applications serving inbound requests scaled to a minimum of two replicas.
If you’ve customized Runtime Fabric email alerts, the appliance upgrade overwrites the customizations. Refer to Modify Email Alert Contents for instructions on recreating the modifications.

To upgrade the Runtime Fabric appliance:

Using your web browser, navigate to Runtime Manager and select the Runtime Fabrics tab to view a list of Runtime Fabric instances.
Select the Runtime Fabric instance you want to upgrade.
Next to the appliance version, click Upgrade Available.
Select the Copy Link icon next to Installer package to copy the code snippet, as shown in the following example:
```
rtfctl appliance upgrade --url https://<upgrade-package-information>
```
Paste the code snippet in a text editor for reference.

Run the Upgrade in the Cluster

The upgrade can take more than an hour to complete.

The connection between Runtime Fabric and Runtime Manager can disconnect multiple times during the upgrade procedure. Connectivity is restored after the upgrade finishes.

The upgrade procedure is designed to keep deployed applications running and available to serve requests by applying changes to one node at a time.

A node is selected by the procedure to apply the upgrade.
Applications running on the selected node are redeployed to another node.
The selected node is removed from the cluster.
The upgrade is applied on the selected node.
The selected node is added to the cluster after upgrading.

To run the upgrade:

Using your terminal open on a controller node, switch to the root user.
Paste the command you copied in Step 5, "Upgrade the Runtime Fabric appliance".
```
sudo /opt/anypoint/runtimefabric/rtfctl appliance upgrade --url <cluster-installer-url>
```
A confirmation is displayed that indicates the upgrade is being performed in the background.
Run sudo gravity status on a node and verify that the cluster status is “updating”.
To monitor the upgrade procedure progress, using your terminal on a controller node, run the following command:
```
sudo /opt/anypoint/runtimefabric/installer/gravity plan
```
To confirm that the upgrade has completed successfully, run the following command:
```
sudo gravity status
```
Verify that the cluster status transitioned from Updating to Active.
If the Runtime Fabric cluster version was 1.0.x prior to upgrading and an HTTP proxy is in use, apply the HTTP proxy settings with:
```
sudo ./rtfctl apply http-proxy --confirm existing
```

Upgrade Node Configurations

Perform the following steps on every node to ensure that system configurations are current:

Open a terminal to a Runtime Fabric controller or worker node.

Run the apply system-configurations command in rtfctl:

sudo ./rtfctl apply system-configuration --force

Or if you are using the EU control plane, run:

sudo ./rtfctl apply system-configuration --force --control-plane eu

When you run the apply system-configuration command on worker nodes, you might see an error similar to the following:

Warning: Agent pod is not running.Warning: Failed to get agent version from chartWarning: Failed to get agent version from podDownloading latest scripts

You can ignore this warning.

Update the Mule Deployments Running on Runtime Fabric to the Latest Patch Version

After you upgrade the Runtime Fabric appliance and components, update your Mule applications deployed on Runtime Fabric to the latest date patch versions.

Using your web browser, navigate to the Applications tab in Runtime Manager.
Select an application and select Manage Application.
Ensure that the value in the Runtime version field indicates that an update is available and select it.
Select Deploy to redeploy the application with the updated Mule runtime engine version.
Repeat this for each application running on this Runtime Fabric instance.

Resume an Appliance Upgrade

If the appliance (installer package) upgrade encounters a failure, try to resume the upgrade.

Resumed upgrades are attached to your terminal session, so ensure that you have a stable connection before attempting to resume an upgrade.

On the terminal open to the controller node that was used to start the upgrade, navigate to the directory with the installer bundle files, as shown in the following example:
```
cd /opt/anypoint/runtimefabric/installer
```

To resume the upgrade, run:

sudo /opt/anypoint/runtimefabric/installer/gravity upgrade --resume

The upgrade procedure streams output to your terminal session. If the previous error reoccurs, perform the troubleshooting steps described in the following section.

Troubleshooting Runtime Fabric Upgrade Errors

If you encounter upgrade errors when attempting to upgrade Runtime Fabric, troubleshoot them as follows.

Network Test Errors

When upgrading Runtime Fabric components to v1.11 and later, you may encounter the following errors during the network test:

mTLS handshake error: check certs

You may encounter this error if your certificates are out of date. Update your certificates and re-attempt the upgrade.
ConnectionRefused:check port’s open

This error occurs if the required outbound ports are not available on the cluster. Review the Runtime Fabric component upgrade steps and ensure you’ve made the necessary ports available.
Connection timed out

This error occurs if the required endpoints are not allowlisted on the cluster. Review the Runtime Fabric component upgrade steps and ensure you’ve allowlisted the necessary endpoints.
Network test takes longer than five minutes

You may encounter a timeout if retrieval of the necessary Docker image takes too long due to a network issue or an issue with the localhost network.

Appliance Upgrade Errors

A specific sequence of steps is performed during an appliance (installer package) upgrade. If an error occurs, the upgrade pauses and displays an error. In most cases, the availability of running applications is not impacted when running multiple replicas of each application on a production Runtime Fabric configuration.

Most errors are due to insufficient disk performance on the etcd block device running on the controller nodes.

To resolve common errors, perform the following steps:

Use the gravity plan command to identify the phase where the upgrade paused:

sudo /opt/anypoint/runtimefabric/installer/gravity plan

The following partial list provides examples of upgrade phases:

* init
* checks
* bootstrap
  * node-1
* masters
  * node-1
    * drain
    * system-upgrade
    * taint
    ...
* runtime
  * rbac-app
  * site
  * kubernetes
* app
  * telekube

Resume the upgrade using the --debug flag with the phase in which the error occurred.

The following example resumes an upgrade by restarting the masters/node-1/drain phase:
```
sudo /opt/anypoint/runtimefabric/installer/gravity upgrade --phase=/masters/node-1/drain --force --debug
```
Wait for the command to run. If the command does not terminate with an error, resume the upgrade with:
```
sudo /opt/anypoint/runtimefabric/installer/gravity plan resume
```
If the upgrade again terminates with an error:
1. Read the logs to identify which node requires repair.
2. Submit a ticket to MuleSoft Support if you require additional assistance.
Open another terminal to the Runtime Fabric node identified in the error logs.

On the controller node running the upgrade, manually run the failed phase:

sudo /opt/anypoint/runtimefabric/installer/gravity plan execute --phase=< insert phase > --force --debug

If this command returns an error, wait several minutes and repeat the previous steps.

See How to Troubleshoot RTF Appliance Upgrade for additional troubleshooting information.

Roll Back an Appliance Upgrade

If you cannot resolve issues with an upgrade, you can roll back the upgrade using one of the following options:

Automatic rollback with a single command
Manual rollback

Automatic Rollback With a Single Command

Run:

rtfctl appliance rollback

This option is only supported for newer appliance versions. If the command output shows that the rtfctl appliance rollback command is not supported, then perform a manual rollback.

Manual Rollback

To perform a manual rollback of the upgrade steps that executed:

Use the gravity plan command to identify the phase in which the upgrade paused:

sudo /opt/anypoint/runtimefabric/installer/gravity plan

The following partial list provides upgrade phase examples:

* init
* checks
* bootstrap
  * node-1
* masters
  * node-1
    * drain
    * system-upgrade
    * taint
    ...
* runtime
  * rbac-app
  * site
  * kubernetes
* app
  * telekube

Roll back the steps that completed in reverse order, as shown in the following example:

$ sudo gravity plan rollback --phase=/masters/node-1/taint
$ sudo gravity plan rollback --phase=/masters/node-1/system-upgrade
$ sudo gravity plan rollback --phase=/masters/node-1/drain
...

To roll back a group of steps:

$ sudo /opt/anypoint/runtimefabric/installer/gravity plan rollback --phase=/masters

After all steps are rolled back, update the plan status to mark the upgrade as completely rolled-back:
```
$ sudo gravity plan complete
```