/opt/anypoint/runtimefabric/rtfctl version
Upgrade Runtime Fabric on VMs/Bare Metal
To upgrade Anypoint Runtime Fabric on VMs/Bare Metal, upgrade the Runtime Fabric appliance, components, the rtfctl
command-line utility, and node configurations.
About Appliance Upgrades
Appliance upgrades affect platform-layer components, which include Docker, Kubernetes, and the Ops Center. These platform-layer components are installed on virtual machines (VMs) or bare-metal servers that act as nodes on the cluster.
Appliance upgrades are applied using the rtfctl
command-line utility.
Appliance upgrades are applied to each node, one at a time. This rolling upgrade process minimizes the impact to running applications by ensuring that the minimum three controller nodes are still active, ensuring cluster availability. This process also ensures that at least one worker node’s worth of unused resources are available to allow for applications to be redeployed as each worker node is taken offline to be upgraded.
About Component Upgrades
Upgrading Runtime Fabric components means upgrading services that:
-
Facilitate communication to the control plane
-
Provide load balancing
-
Log and forward metrics
When component upgrades are available, you apply the upgrades within Runtime Manager via a button that’s visible when an upgrade is available. Review the Anypoint Runtime Fabric Release Notes for information about the latest component versions.
Runtime Fabric component upgrades do not require adding a node.
Application monitoring metrics are not restored after an upgrade. |
Determine the Component Versions You Are Using
On a Runtime Fabric controller node, run the following command:
The output looks something like this:
For example:
[root@ip-10-0-245-74 /]# /opt/anypoint/runtimefabric/rtfctl version
COMPONENT VERSION
rtfctl 0.3.122
agent 1.9.3
appliance 1.1.1719717580-73a133d
kubernetes 1.17
-
The
rtfctl
command-line utility version in thertfctl
field -
The appliance component version in the
appliance
field -
The Runtime Fabric component version in the
agent
field
The Runtime Fabric component and appliance versions that are in use are also displayed in the Runtime Fabrics tab in Runtime Manager. If an upgrade is available, an "Upgrade to vx.x" message is displayed. |
About rtfctl
Command-line Utility Upgrades
The rtfctl
command-line utility is used to manage Runtime Fabric and perform appliance and node configuration upgrades.
You upgrade the rtfctl
utility from the command line.
It is important to use the most current version of the rtfctl
command-line utility.
About Node Configuration Upgrades
Node configuration upgrades involve scripts for provisioning AWS and Azure infrastructure, preparing each node, bootstrapping a new Runtime Fabric, adjusting operating system limits, and performing jobs to clean up system resources.
Node configuration upgrades are delivered as an updated install script version using the Runtime Fabric downloads page in Runtime Manager.
If you are upgrading the node configuration with updated install scripts, the latest version of the appliance is automatically downloaded and used. |
How the Upgrade Process Works
To upgrade Anypoint Runtime Fabric on VMs/Bare Metal, perform the following tasks:
-
Upgrade the
rtfctl
command-line utility. -
Verify that Runtime Fabric is healthy.
-
Upgrade Runtime Fabric components.
-
Upgrade the Runtime Fabric appliance.
-
Run the upgrade in the cluster.
-
Upgrade node configurations.
-
Update the Mule deployments running on Runtime Fabric to the latest patch version.
Before You Perform the Upgrade Process
-
Before upgrading your production environment, perform these steps on a nonproduction Runtime Fabric instance to verify compatibility with your environment. Plan to upgrade when your team is not deploying new applications or managing existing applications on Runtime Fabric. When your nonproduction upgrade succeeds, perform these steps in your production environment. Review the Anypoint Runtime Fabric Release Notes information.
-
Provision additional worker nodes as needed to maintain availability of applications when applying appliance upgrades.
If the amount of available resources for application deployments is less than the resources of one worker node, add another worker node with the same amount of CPU, memory, and disk resources to Runtime Fabric.
-
Review What Items to Check Before an RTF Appliance Upgrade from the MuleSoft Knowledge Base.
Upgrade the rtfctl
Command-Line Utility
The rtfctl
command-line utility is used in several steps during the upgrade process.
-
If you are not using the latest version of the
rtfctl
command-line utility, download the latest version with the following command:cd /opt/anypoint/runtimefabric sudo curl -L https://anypoint.mulesoft.com/runtimefabric/api/download/rtfctl/latest -o rtfctl
-
Change file permissions for the
rtfctl
binary:sudo chmod +x rtfctl
-
If you are a user who owns the
rtfctl
binary, you can updatertfctl
without needing to download a new version. To do so, run the following command:`rtfctl update`
Verify That Runtime Fabric is Healthy
Before running the upgrade, ensure the cluster is active and healthy.
-
Using your web browser, navigate to the Runtime Fabrics tab in Runtime Manager and verify that Runtime Fabric reports an Active status.
-
Open a terminal to a Runtime Fabric controller node.
-
Run
/opt/anypoint/runtimefabric/rtfctl status
and confirm that the cluster status is healthy. -
Keep your browser session open to perform the remainder of the upgrade subtasks.
Upgrade Runtime Fabric Components
The first step in the Runtime Fabric process is upgrading the Runtime Fabric components.
Before You Upgrade Runtime Fabric Components
-
Verify that you have allowlisted the following endpoints on port 443, used by the runtime-registry to retrieve Docker containers for app deployments.
-
US control plane:
rtf-runtime-registry.kprod.msap.io
-
EU control plane:
rtf-runtime-registry.kprod-eu.msap.io
-
-
Verify that you have allowlisted the endpoints used for Mutual Transport Layer Security (mTLS) between the Anypoint Platform control plane and your Runtime Fabric.
After upgrading to Runtime Fabric, you must restart applications so they can update their deployment manifests for the runtime-registry endpoint. |
-
Navigate to your Runtime Fabric by selecting it on the Runtime Fabrics tab in Runtime Manager.
-
Click Upgrade to vx.x.
-
Click Run Network Test to perform the network check to ensure you’ve allowlisted the correct endpoints for the runtime registry and mTLS with the Anypoint Control plane.
If you haven’t allowlisted the endpoints, you must do so before continuing.
The network test may take several minutes to complete. If you encounter any errors, refer to Network Test Errors.
-
When the network test completes, click Apply Upgrade.
The status of the Runtime Fabric is updated to show that the upgrade procedure is in progress.
-
Wait for the status to transition to Active.
In most cases, the upgrade takes fewer than 10 minutes. When the status transitions to Active and the version of the Runtime Fabric reflects the new version, the component upgrade is complete.
If the Runtime Fabric and Runtime Manager connection disconnects during the upgrade, connectivity is restored after the upgrade finishes.
-
Keep your browser session open to perform the remainder of the upgrade subtasks.
Upgrade the Runtime Fabric Appliance
After you upgrade the Runtime Fabric components, you can upgrade the Runtime Fabric appliance.
Before You Upgrade the Runtime Fabric Appliance
-
Verify that
/opt/anypoint/runtimefabric/installer
directory has a minimum of 4 GiB of disk space available. -
Verify that you have root privileges.
-
Verify that your environment includes:
-
An internal load balancer running on at least three replicas, to maintain availability. On newer versions of Runtime Fabric, the internal load balancer runs on all controller nodes by default.
-
An external load balancer configured to load-balance incoming requests to the controller nodes, with a health check configured for TCP port 443.
-
CPU and memory resources for at least one additional worker node available in the cluster.
This enables safe removal of a worker node from the cluster to apply upgrades without impacting application availability.
-
Applications serving inbound requests scaled to a minimum of two replicas.
-
-
If you’ve customized Runtime Fabric email alerts, the appliance upgrade overwrites the customizations. Refer to Modify Email Alert Contents for instructions on recreating the modifications.
To upgrade the Runtime Fabric appliance:
-
Using your web browser, navigate to Runtime Manager and select the Runtime Fabrics tab to view a list of Runtime Fabric instances.
-
Select the Runtime Fabric instance you want to upgrade.
-
Next to the appliance version, click Upgrade Available.
-
Select the Copy Link icon next to Installer package to copy the code snippet, as shown in the following example:
rtfctl appliance upgrade --url https://<upgrade-package-information>
-
Paste the code snippet in a text editor for reference.
Run the Upgrade in the Cluster
The upgrade can take more than an hour to complete.
The connection between Runtime Fabric and Runtime Manager can disconnect multiple times during the upgrade procedure. Connectivity is restored after the upgrade finishes.
The upgrade procedure is designed to keep deployed applications running and available to serve requests by applying changes to one node at a time.
-
A node is selected by the procedure to apply the upgrade.
-
Applications running on the selected node are redeployed to another node.
-
The selected node is removed from the cluster.
-
The upgrade is applied on the selected node.
-
The selected node is added to the cluster after upgrading.
To run the upgrade:
-
Using your terminal open on a controller node, switch to the root user.
-
Paste the command you copied in Step 5, "Upgrade the Runtime Fabric appliance".
sudo /opt/anypoint/runtimefabric/rtfctl appliance upgrade --url <cluster-installer-url>
A confirmation is displayed that indicates the upgrade is being performed in the background.
-
Run
sudo gravity status
on a node and verify that the cluster status is “updating”. -
To monitor the upgrade procedure progress, using your terminal on a controller node, run the following command:
sudo /opt/anypoint/runtimefabric/installer/gravity plan
-
To confirm that the upgrade has completed successfully, run the following command:
sudo gravity status
-
Verify that the cluster status transitioned from Updating to Active.
-
If the Runtime Fabric cluster version was 1.0.x prior to upgrading and an HTTP proxy is in use, apply the HTTP proxy settings with:
sudo ./rtfctl apply http-proxy --confirm existing
Upgrade Node Configurations
Perform the following steps on every node to ensure that system configurations are current:
-
Open a terminal to a Runtime Fabric controller or worker node.
-
Run the
apply system-configurations
command inrtfctl
:sudo ./rtfctl apply system-configuration --force
Or if you are using the EU control plane, run:
sudo ./rtfctl apply system-configuration --force --control-plane eu
Update the Mule Deployments Running on Runtime Fabric to the Latest Patch Version
After you upgrade the Runtime Fabric appliance and components, update your Mule applications deployed on Runtime Fabric to the latest date patch versions.
-
Using your web browser, navigate to the Applications tab in Runtime Manager.
-
Select an application and select Manage Application.
-
Ensure that the value in the Runtime version field indicates that an update is available and select it.
-
Select Deploy to redeploy the application with the updated Mule runtime engine version.
-
Repeat this for each application running on this Runtime Fabric instance.
Resume an Appliance Upgrade
If the appliance (installer package) upgrade encounters a failure, try to resume the upgrade.
Resumed upgrades are attached to your terminal session, so ensure that you have a stable connection before attempting to resume an upgrade.
-
On the terminal open to the controller node that was used to start the upgrade, navigate to the directory with the installer bundle files, as shown in the following example:
cd /opt/anypoint/runtimefabric/installer
-
To resume the upgrade, run:
sudo /opt/anypoint/runtimefabric/installer/gravity upgrade --resume
-
The upgrade procedure streams output to your terminal session. If the previous error reoccurs, perform the troubleshooting steps described in the following section.
Troubleshooting Runtime Fabric Upgrade Errors
If you encounter upgrade errors when attempting to upgrade Runtime Fabric, troubleshoot them as follows.
Network Test Errors
When upgrading Runtime Fabric components to v1.11 and later, you may encounter the following errors during the network test:
-
mTLS handshake error: check certs
You may encounter this error if your certificates are out of date. Update your certificates and re-attempt the upgrade.
-
ConnectionRefused:check port’s open
This error occurs if the required outbound ports are not available on the cluster. Review the Runtime Fabric component upgrade steps and ensure you’ve made the necessary ports available.
-
Connection timed out
This error occurs if the required endpoints are not allowlisted on the cluster. Review the Runtime Fabric component upgrade steps and ensure you’ve allowlisted the necessary endpoints.
-
Network test takes longer than five minutes
You may encounter a timeout if retrieval of the necessary Docker image takes too long due to a network issue or an issue with the localhost network.
Appliance Upgrade Errors
A specific sequence of steps is performed during an appliance (installer package) upgrade. If an error occurs, the upgrade pauses and displays an error. In most cases, the availability of running applications is not impacted when running multiple replicas of each application on a production Runtime Fabric configuration.
Most errors are due to insufficient disk performance on the etcd
block device running on the controller nodes.
To resolve common errors, perform the following steps:
-
Use the
gravity plan
command to identify the phase where the upgrade paused:sudo /opt/anypoint/runtimefabric/installer/gravity plan
The following partial list provides examples of upgrade phases:
* init * checks * bootstrap * node-1 * masters * node-1 * drain * system-upgrade * taint ... * runtime * rbac-app * site * kubernetes * app * telekube
-
Resume the upgrade using the
--debug
flag with the phase in which the error occurred.The following example resumes an upgrade by restarting the
masters/node-1/drain
phase:sudo /opt/anypoint/runtimefabric/installer/gravity upgrade --phase=/masters/node-1/drain --force --debug
-
Wait for the command to run. If the command does not terminate with an error, resume the upgrade with:
sudo /opt/anypoint/runtimefabric/installer/gravity plan resume
-
If the upgrade again terminates with an error:
-
Read the logs to identify which node requires repair.
-
Submit a ticket to MuleSoft Support if you require additional assistance.
-
-
Open another terminal to the Runtime Fabric node identified in the error logs.
-
On the controller node running the upgrade, manually run the failed phase:
sudo /opt/anypoint/runtimefabric/installer/gravity plan execute --phase=< insert phase > --force --debug
If this command returns an error, wait several minutes and repeat the previous steps.
See How to Troubleshoot RTF Appliance Upgrade for additional troubleshooting information.
Roll Back an Appliance Upgrade
If you cannot resolve issues with an upgrade, you can roll back the upgrade using one of the following options:
-
Automatic rollback with a single command
-
Manual rollback
Automatic Rollback With a Single Command
Run:
rtfctl appliance rollback
This option is only supported for newer appliance versions. If the command output shows that the rtfctl appliance rollback command is not supported, then perform a manual rollback.
|
Manual Rollback
To perform a manual rollback of the upgrade steps that executed:
-
Use the
gravity plan
command to identify the phase in which the upgrade paused:sudo /opt/anypoint/runtimefabric/installer/gravity plan
The following partial list provides upgrade phase examples:
* init * checks * bootstrap * node-1 * masters * node-1 * drain * system-upgrade * taint ... * runtime * rbac-app * site * kubernetes * app * telekube
-
Roll back the steps that completed in reverse order, as shown in the following example:
$ sudo gravity plan rollback --phase=/masters/node-1/taint $ sudo gravity plan rollback --phase=/masters/node-1/system-upgrade $ sudo gravity plan rollback --phase=/masters/node-1/drain ...
To roll back a group of steps:
$ sudo /opt/anypoint/runtimefabric/installer/gravity plan rollback --phase=/masters
-
After all steps are rolled back, update the plan status to mark the upgrade as completely rolled-back:
$ sudo gravity plan complete