Contact Us 1-800-596-4880
  1. Homepage
  2. Runtime Fabric (1.8)
  3. On VMs / Bare Metal

  4. Upgrade Runtime Fabric on VMs / Bare Metal

Upgrade Runtime Fabric

Anypoint Runtime Fabric is a software appliance that combines packages that work together but are upgraded individually:

  • Appliance upgrades

    Appliance upgrades affect platform-layer components, which include Docker, Kubernetes, and the Ops Center. These platform-layer components are installed on virtual machines (VMs) or bare-metal servers that act as nodes on the cluster.

    Appliance upgrades are applied using the rtfctl command-line utility.

    Appliance upgrades are applied as a rolling upgrade to each node, one at a time. To minimize impact to running applications, a minimium of three controller nodes is required to ensure the cluster remains available. In addition, there should be at least one worker node’s worth of unused resources available to allow for applications to be redeployed as each worker node is taken offline to be upgraded.

  • Component upgrades

    Runtime Fabric component upgrades involve the internal services that run within the appliance used to facilitate communication to the Anypoint control plane, provide load balancing, and log and metrics forwarding.

    Runtime Fabric component upgrades are applied within Runtime Manager via a button that’s visible when an upgrade is made available.

    Runtime Fabric component upgrades do not require adding a node.

  • rtfctl command-line utility upgrades

    The rtfctl command-line utility is used to manage Runtime Fabric and perform appliance and node configuration upgrades.

    Upgrades for the rtfctl utility are delivered as a new binary via the Runtime Fabric downloads page in Runtime Manager.

  • Node configuration upgrades

    Node configuration upgrades involve scripts for provisioning AWS and Azure infrastructure, preparing each node, bootstrapping a new Runtime Fabric, adjusting operating system limits, and performing jobs to clean up system resources.

    Node configuration upgrades are delivered as an updated install script version using the Runtime Fabric downloads page in Runtime Manager.

    If you are upgrading the node configuration with updated install scripts, the latest version of the appliance is automatically downloaded and used.

It is important to use the most current versions of the appliance and rtfctl command-line utility.

Determine the Component Versions You Are Using

Run the following command on a Runtime Fabric controller node:

/opt/anypoint/runtimefabric/rtfctl version

Output includes:

  • The rtfctl command-line utility version in the rtfctl field

  • The appliance component version in the appliance field

  • The Runtime Fabric component version in the agent field

For example:

[root@ip-10-0-245-74 /]# /opt/anypoint/runtimefabric/rtfctl version
COMPONENT    VERSION
rtfctl       0.2.96+2868e14
agent        1.5.4
appliance    1.1.1581641679-5884bce
kubernetes   1.13.12
The Runtime Fabric component and appliance versions that are in use are also displayed in the Runtime Fabrics tab in Runtime Manager. If an upgrade is available, an "Upgrade to vx.x" message is displayed.

Determine the Most Recent Component Versions Available

The most recent component versions are displayed in the Downloads page, located in Runtime Manager in the Runtime Fabrics tab. A link to each package’s release notes is provided to show the changes.

You can also review the Anypoint Runtime Fabric Release Notes information.

Apply the Upgrades

Before upgrading your production environment, perform the following steps on a nonproduction Runtime Fabric to verify compatibility with your environment.

  1. Review the Anypoint Runtime Fabric Release Notes information.

  2. Provision additional worker nodes as needed to maintain availability of applications when applying appliance upgrades.

    If the amount of available resources for application deployments is less than the resources of one worker node, add another worker node with the same amount of CPU, memory, and disk resources to Runtime Fabric.

    See How to Add a Node to Runtime Fabric.

  3. To upgrade Runtime Fabric components:

    1. Plan to upgrade when your team is not deploying new applications or managing existing applications on Runtime Fabric.

    2. Navigate to your Runtime Fabric by selecting it on the Runtime Fabrics tab in Runtime Manager.

    3. Select "Upgrade to vx.x". The status of the Runtime Fabric is updated to show that the upgrade procedure is in progress.

    4. Wait for the status to transition to Active.

      In most cases, the upgrade takes fewer than 10 minutes. When the status transitions to Active and the version of the Runtime Fabric reflects the new version, the component upgrade is complete.

      If the Runtime Fabric and Runtime Manager connection disconnects during the upgrade, connectivity is restored after the upgrade finishes.

      Keep your browser session open to use in later steps.

      To upgrade to Runtime Fabric version 1.5 or later, you must be running version 1.4.54. If you are not running version 1.4.54, a button to upgrade to 1.4.54 is displayed. Upgrade to this version in order to enable upgrades to 1.5.

  4. If you are not using the latest version of the rtfctl command-line utility, download the latest version:

    1. Using your terminal on a controller node, run the following command:

      cd /opt/anypoint/runtimefabric
sudo curl -L https://anypoint.mulesoft.com/runtimefabric/api/download/rtfctl/latest -o rtfctl

    2. Change file permissions for the rtfctl binary:

      sudo chmod +x rtfctl

  5. If you are not using the latest version of the appliance, upgrade to the latest version:

    1. Make sure the /opt/anypoint/runtimefabric/installer directory has a minimum of 4 GiB of disk space available.

    2. Verify you have root privileges. Root privileges are required to perform an upgrade.

    3. Verify your environment meets the prerequisites:

      • The internal load balancer must be running on at least three replicas to maintain availability. On newer versions of Runtime Fabric, the internal load balancer runs on all controller nodes by default.

      • An external load balancer must be configured to load balance incoming requests to the controller nodes, with a health check configured for TCP port 443.

      • The CPU and memory resources for at least one additional worker node must be available in the cluster.

        This enables safe removal of a worker node from the cluster to apply upgrades without impacting application availability.

      • Applications serving inbound requests must be scaled to a minimum of two replicas.

        [WARNING] Runtime Fabrics running on cluster version 1.0.x encounter application downtime if configured to use an HTTP proxy. You must apply the HTTP proxy configuration on each node after the upgrade. Locate the Runtime Fabric version by running rtfctl version on a node and referencing the appliance version.

  6. Update the Mule deployments running on Runtime Fabric to the latest patch version.

    1. Using your web browser, navigate to the Applications tab in Runtime Manager.

    2. Select an application and select Manage Application.

    3. The value in the Runtime version field indicates if an update is available. Select the updated version of Mule runtime engine.

    4. Select Deploy to redeploy the application with the updated Mule runtime engine version.

    5. Repeat this for each application running on this Runtime Fabric.

  7. Verify Runtime Fabric is healthy:

    1. Using your web browser, navigate to the Runtime Fabrics tab in Runtime Manager and verify that Runtime Fabric reports an Active status.

    2. Open a terminal to a Runtime Fabric controller node.

    3. Run /opt/anypoint/runtimefabric/rtfctl status and confirm that the cluster status is healthy.

    4. Keep your terminal open to use in later steps.

  8. Get the latest version of the Runtime Fabric appliance package:

    1. Using your web browser, navigate to Runtime Manager and select the Runtime Fabrics tab to view a list of Runtime Fabrics.

    2. Select the Downloads link.

    3. Select the Copy Link icon next to Installer package to copy the code snippet, as shown in the following example:

      rtfctl appliance upgrade --url https://<upgrade-package-information>

    4. Paste the code snippet in a text editor for reference.

  9. Start the upgrade procedure:

    1. Using your terminal open on a controller node, run the following command, substituting the value of the <cluster-installer-url> parameter with the Installer package URL value:

      sudo /opt/anypoint/runtimefabric/rtfctl appliance upgrade --url <cluster-installer-url>

      This command also supports using the --file parameter to reference an installer package already downloaded on the node.

      A confirmation is displayed that indicates the upgrade is being performed in the background.

    2. Run sudo gravity status on a node and verify that the cluster status is “updating”.

    3. Monitor the upgrade procedure progress:

      1. Using your terminal on a controller node, run the following command:

        sudo /opt/anypoint/runtimefabric/installer/gravity plan

        The upgrade can take more than an hour to complete.

        The connection between Runtime Fabric and Runtime Manager can disconnect multiple times during the upgrade procedure. Connectivity is restored after the upgrade finishes.

        The upgrade procedure is designed to keep deployed applications running and available to serve requests. The upgrade is applied in a rolling method:

        1. A node is selected by the procedure to apply the upgrade.

        2. Applications running on the selected node are redeployed to another node.

        3. The selected node is removed from the cluster.

        4. The upgrade is applied on the selected node.

        5. The selected node is added to the cluster after upgrading.

    4. To confirm that the upgrade has completed successfully run the following command:

      sudo gravity status

      Then, verify that the cluster status transitioned from Updating to Active.

  10. If the Runtime Fabric cluster version was 1.0.x prior to upgrading and an HTTP proxy is in use, run the following command to apply the HTTP proxy settings:

    sudo ./rtfctl apply http-proxy --confirm existing

  11. Perform the following step on every node to ensure that system configurations are current:

    1. Open a terminal to a Runtime Fabric controller or worker node.

    2. If needed, download the latest rtfctl command-line utility:

      cd /opt/anypoint/runtimefabric
curl -L https://anypoint.mulesoft.com/runtimefabric/api/download/rtfctl/latest -o rtfctl

    3. Change file permissions for the rtfctl binary:

      chmod +x rtfctl

    4. Run the apply system-configurations command in rtfctl:

      sudo ./rtfctl apply system-configuration --force

      If you are using the EU control plane, use the following command:

      sudo ./rtfctl apply system-configuration --force --control-plane eu

Resume an Appliance Upgrade

If the appliance (installer package) upgrade encounters a failure, try to resume the upgrade.

Resumed upgrades are attached to your terminal session, so ensure that you have a stable connection before attempting to resume an upgrade.

  1. On the terminal open to the controller node that was used to start the upgrade, navigate to the directory with the installer bundle files, as shown in the following example:

    cd /opt/anypoint/runtimefabric/installer

  2. Run the following command to resume the upgrade:

    sudo /opt/anypoint/runtimefabric/installer/gravity upgrade --resume

  3. The upgrade procedure streams output to your terminal session. If the previous error reoccurs, perform the troubleshooting steps described in the following section.

Troubleshooting Appliance Upgrade Errors

A specific sequence of steps is performed during an appliance (installer package) upgrade. If an error occurs, the upgrade pauses and displays an error. In most cases, the availability of running applications is not impacted when running multiple replicas of each application on a production Runtime Fabric configuration.

Most errors are due to insufficient disk performance on the etcd block device running on the controller nodes.

Perform the following steps to resolve common errors:

  1. Use the gravity plan command to identify the phase where the upgrade paused:

    sudo /opt/anypoint/runtimefabric/installer/gravity plan

    The following partial list provides examples of upgrade phases:

    * init
* checks
* bootstrap
  * node-1
* masters
  * node-1
    * drain
    * system-upgrade
    * taint
    ...
* runtime
  * rbac-app
  * site
  * kubernetes
* app
  * telekube

  2. Resume the upgrade using the --debug flag with the phase in which the error occurred.

    The following example resumes an upgrade by restarting the masters/node-1/drain phase:

    sudo /opt/anypoint/runtimefabric/installer/gravity upgrade --phase=/masters/node-1/drain --force --debug

  3. Wait for the command to run. If the command does not terminate with an error, resume the upgrade by running the following command:

    sudo /opt/anypoint/runtimefabric/installer/gravity plan resume

  4. If the upgrade again terminates with an error:

    1. Read the logs to identify which node requires repair.

    2. Submit a ticket to MuleSoft Support if you require additional assistance.

  5. Open another terminal to the Runtime Fabric node identified in the error logs.

  6. On the controller node running the upgrade, manually run the failed phase:

    sudo /opt/anypoint/runtimefabric/installer/gravity plan execute --phase=< insert phase > --force --debug

If this command returns an error, wait several minutes and repeat the previous steps.

See How to Troubleshoot RTF Appliance Upgrade for additional troubleshooting information.

Roll Back an Appliance Upgrade

If you cannot resolve issues with an upgrade, you can roll back the upgrade using one of the following options:

  • Automatic rollback with a single command

  • Manual rollback

Automatic Rollback With a Single Command

Enter the following command:

rtfctl appliance rollback
This option is only supported for newer appliance versions. If the command output shows that the rtfctl appliance rollback command is not supported, then perform a manual rollback.

Manual Rollback

To perform a manual rollback of the upgrade steps that executed:

  1. Use the gravity plan command to identify the phase in which the upgrade paused:

    sudo /opt/anypoint/runtimefabric/installer/gravity plan

    The following partial list provides upgrade phase examples:

    * init
* checks
* bootstrap
  * node-1
* masters
  * node-1
    * drain
    * system-upgrade
    * taint
    ...
* runtime
  * rbac-app
  * site
  * kubernetes
* app
  * telekube

  2. Roll back the steps that completed in reverse order, as shown in the following example:

    $ sudo gravity plan rollback --phase=/masters/node-1/taint
$ sudo gravity plan rollback --phase=/masters/node-1/system-upgrade
$ sudo gravity plan rollback --phase=/masters/node-1/drain
...

    To roll back a group of steps:

    $ sudo /opt/anypoint/runtimefabric/installer/gravity plan rollback --phase=/masters

  3. After all steps are rolled back, update the plan status to mark the upgrade as completely rolled-back:

    $ sudo gravity plan complete