Contact Us 1-800-596-4880

Using the Runtime Fabric Command Line Tool

Install and use the rtfctl command line tool to locally manage Runtime Fabrics. The rtfctl tool supports the following actions:

  • Get the status of a Runtime Fabric.

  • Perform a heap dump or a thread dump of a running application.

  • Display memory usage of a running application.

  • Manage proxy settings.

  • Manage secure properties.

  • Manage a Mule license.

  • Manage the alert sender email address.

If you are using the default namespace (rtf), you do not need to specify the additional flag --namespace <rtf_namespace>. Note that this flag is introduced only as of rtfctl version 0.4.1.

Install rtfctl

The tasks in the section must be performed by an IT administrator.

If you’re using Runtime Fabric for AKS and have the AKS-managed Azure AD integration enabled, the following command returns the credentials needed to run rtfctl commands:

az aks get-credentials --name <NAME> --resource-group <RESOURCE_GROUP> --admin

  1. Download the rtfctl command-line tool:

    rtfctl is supported on Windows, MacOS (Darwin), and Linux. Choose the appropriate method:

    • Windows

      curl -L https://anypoint.mulesoft.com/runtimefabric/api/download/rtfctl-windows/latest -o rtfctl.exe
    • MacOS (Darwin)

      curl -L https://anypoint.mulesoft.com/runtimefabric/api/download/rtfctl-darwin/latest -o rtfctl
    • Linux

      curl -L https://anypoint.mulesoft.com/runtimefabric/api/download/rtfctl/latest -o rtfctl
  2. Change file permissions for the rtfctl command-line tool:

    chmod +x rtfctl

Running Commands

The rtfctl command line tool consumes kubeconfig files as described in Organizing Cluster Access Using kubeconfig Files. Use this method to configure which cluster you want to manage. Confirm that the rtfctl binary is present in the current working directory and in the user $PATH.

  • Run rtfctl commands as a privileged user.

  • To list all supported commands, run rtfctl -h.

  • For more information about a specific command, run rtfctl <command> -h.

Checking for Proxy Settings

When you run commands, rtfctl checks for any proxy settings in your environment. The tool looks for a NO_PROXY key (in upper case) and will accept a no_proxy key (in lower case) if NO_PROXY is not used. If both keys are present, rtfctl uses NO_PROXY. The rtfctl tool sets the key in your local Kubernetes context to honor your proxy settings.

Commands for Configuring Runtime Fabric

Use the following commands to configure Runtime Fabric.

Command Description

Applies changes to a Runtime Fabric configuration

Updates rtfctl

Waits for Runtime Fabric components to become ready

apply

Applies changes to a Runtime Fabric configuration.

Any value used with rtfcl apply updates does not work for OpenShift. You have to update the values in the OpenShift console.
  • Applies an HTTP proxy configuration

    rtfctl apply http-proxy http://<host>:<port> --namespace <rtf_namespace>
  • Applies an HTTP proxy configuration with hostnames bypassing the proxy

    rtfctl apply http-proxy http://<host>:<port> --no-proxy "DOMAIN1.com,DOMAIN2.com" --namespace <rtf_namespace>
  • Applies an HTTP proxy configuration with authentication

    rtfctl apply http-proxy http://<username>:<password>@<host>:<port> --namespace <rtf_namespace>
  • Clears an HTTP proxy configuration

    rtfctl apply http-proxy "" --no-proxy "" --namespace <rtf_namespace>
  • Applies a secure property

    rtfctl apply secure-property --key <key> --value <value> --app-namespace <app_namespace> --namespace <rtf_namespace>
  • Applies a system configuration on a host node

    rtfctl apply system-configuration
  • Applies a system configuration on a host node without downloading the latest scripts

    rtfctl apply system-configuration --skip-download
  • Applies a Mule license

    rtfctl apply mule-license <base64_encoded_license> --namespace <rtf_namespace>

    If Runtime Fabric is installed via Helm, you must update the mule-license property in the Helm values.

  • Applies a monitoring proxy

    rtfctl apply monitoring-proxy http://<host>:<port> --namespace <rtf_namespace>
  • Clears a monitoring proxy configuration

    rtfctl apply monitoring-proxy "" --namespace <rtf_namespace>

update

Updates rtfctl.

  • Updates from the US control plane

    rtfctl update
  • Updates rtfctl from the EU control plane

    rtfctl update --host eu1.anypoint.mulesoft.com

wait

Waits for Runtime Fabric components to become ready

  • Specifies the number of seconds to wait

rtfctl wait <value_in_seconds> --namespace <rtf_namespace>
  • Specifies the number of seconds to pass before timing out the wait (Default is 600 seconds)

rtfctl wait --timeout <value_in_seconds> --namespace <rtf_namespace>

Commands for Installing and Upgrading Runtime Fabric

Use the following commands when installing or upgrading Runtime Fabric, or when performing a back up or restore.

Command Description

Installs Runtime Fabric

Performs a back up of the local Runtime Fabric state

Restores local Runtime Fabric state from a backup

Validates the cluster for Runtime Fabric installation

Displays the Runtime Fabric version information

install

  • Installs Runtime Fabric

rtfctl install <activation_data>
  • Override the Docker image registry URL and secret

    rtfctl install <activation_data> --image-pull-registry <string>  --image-pull-secret <string>

backup

Performs a back up of the local Runtime Fabric state.

rtfctl backup <backup_file_name>.tar.gz --namespace <rtf_namespace>

restore

Restores the local Runtime Fabric state from a backup

rtfctl restore <backup_file_name>.tar.gz --namespace <rtf_namespace>

validate

Validates the cluster for a Runtime Fabric installation

  • Performs a validation check

    rtfctl validate <activation_data>
  • Performs a validation check and passes the fabric.tf configuration file that contains proxy settings

    rtfctl validate -f <path_to_settings_file>

    To validate installation with a proxy, you must specify proxy settings in your settings file:

MULE_LICENSE=<your-mule-license>
ACTIVATION_DATA=<activation_data>
HTTP_PROXY=http://<proxy_domain>.com:<port>
  • Override the Docker image registry URL and secret

    rtfctl validate <activation_data> --image-pull-registry <string> --image-pull-secret <string>

version

Displays the Runtime Fabric version information

rtfctl version --namespace <rtf_namespace>

Commands for Managing Applications on Runtime Fabric

Use the following commands to manage applications deployed to Runtime Fabric.

Command Description

Deletes a secure property

Displays details for an application

Displays disk usage for an application

Displays the JVM memory usage in a pod

Creates a debugging information package for an application

Restarts an application

Captures a JVM heap dump

Captures a JVM thread dump

delete

Deletes a secure property

rtfctl delete secure-property <sample_key> -n <app_namespace>

describe

Displays details for an application in a specific namespace

rtfctl describe apps <app_name> --namespace <rtf_namespace>

rtfctl describe apps <app_name> --app-namespace <app_namespace> --namespace <rtf_namespace>

disk

Displays disk usage for an application

  • Displays the disk usage for an application, using the first replica by default

    rtfctl disk <app_name> --app-namespace <app_namespace> --namespace <rtf_namespace>
  • Displays the disk usage for an application, using a specific replica

    rtfctl disk <app_name> --details --pod <replica_name> --app-namespace <app_namespace> --namespace <rtf_namespace>

memory

Displays the JVM memory usage in a pod

  • Gets the JVM memory for each replica

    rtfctl memory <app_name> --app-namespace <app_namespace> --namespace <rtf_namespace>
  • Gets the JVM memory for a particular replica

    rtfctl memory <app_name> --pod <replica_name> --app-namespace <app_namespace> --namespace <rtf_namespace>

package

Creates a debugging information package for an application

  • Creates a package for an application using the first pod by default

    rtfctl package <app_name> <application_package_filename> --app-namespace <app_namespace> --namespace <rtf_namespace>
  • Creates a package for an application containing the apps, policies, and .mule directories

    rtfctl package <app_name> <application_package_filename> --apps --policies --dotmule --app-namespace <app_namespace> --namespace <rtf_namespace>
  • Creates a package for an application containing a heap dump

    rtfctl package <app_name> <application_package_filename> --heap-dump --app-namespace <app_namespace> --namespace <rtf_namespace>

restart

Restarts an application

rtfctl restart <app_name> --app-namespace <app_namespace> --namespace <rtf_namespace>

heapdump

Triggers a JVM heap dump

  • Performs a JVM heap dump for an application, saving it as `/tmp/dump.hprof, using the first replica by default

    rtfctl heapdump <app_name> <destination_path> --app-namespace <app_namespace> --namespace <rtf_namespace>
  • Performs a JVM heap dump for an application, saving it as /tmp/dump.hprof, using a specific replica

    rtfctl heapdump <app_name> <destination_path> --pod <replica_name> --app-namespace <app_namespace> --namespace <rtf_namespace>

threaddump

Triggers a JVM threaddump

  • Performs a JVM thread dump for an application using the first replica by default

    rtfctl threaddump <app_name> --app-namespace <app_namespace> --namespace <rtf_namespace>
  • Performs a JVM thread dump for an application using a specific replica

    rtfctl threaddump <app_name> --pod <replica_name> --app-namespace <app_namespace> --namespace <rtf_namespace>

    == Commands for Troubleshooting Runtime Fabric

Use the following commands to troubleshoot Runtime Fabric.

Command Description

get

Displays one or more resources

Tests network connectivity

Generates a diagnostics report

Displays the status of Runtime Fabric

Verifies outbound network connectivity to the Runtime Fabric control plane

get

Displays one or more resources

  • Prints a list of all applications in all environments

    rtfctl get apps --namespace <rtf_namespace>
  • Prints a list of all applications in a specific environment

    rtfctl get apps --app-namespace <app_namespace> --namespace <rtf_namespace>
  • Displays an HTTP proxy configurations

    rtfctl get http-proxy --namespace <rtf_namespace>
  • Displays a monitoring proxy configuration

    rtfctl get monitoring-proxy --namespace <rtf_namespace>
  • Displays secure properties

    rtfctl get secure-properties --app-namespace <app_namespace> --namespace <rtf_namespace>
  • Displays the Mule license

    rtfctl get mule-license --namespace <rtf_namespace>
  • Displays Runtime Fabric registry credentials

    rtfctl get registry-credentials <activation_data>

ping

Tests network connectivity

  • Pings example.com from inside an application, using the first replica by default

    rtfctl ping <app_name> <host_or_ip_address> --app-namespace <app_namespace> --namespace <rtf_namespace>
  • Attempts to connect to example.com on port 12345 from inside an application, using the first replica by default

    rtfctl ping <app_name> <host_or_ip_address> --port <port> --app-namespace <app_namespace> --namespace <rtf_namespace>

report

Generates a diagnostics report at rtf-report.tar.gz

rtfctl report --namespace <rtf_namespace>

Run rtfctl report tasks in parallel by using the workers flag that controls the number of tasks to run in parallel. The default value is 10.

rtfctl report --workers <num_of_workers>

Faster compression of the tarball generated by rtfctl report with the --use-fast-compression flag. The default value is false.

rtfctl report --use-fast-compression=true

Select which application namespaces to include in the report besides the installation namespace by using the --namespaces flag. The flag controls what to include in the reports and can result in faster collection if only selected namespaces are reported.

  # Reports rtf installation default namespace (rtf)
  rtfctl report
  # Reports only <rtf_namespace>
  rtfctl report -n <rtf_namespace>
  # Reports for <app_namespace_1> and <app_namespace_2> application namespaces that belong to <rtf_namespace>
  rtfctl report -n <rtf_namespace> --app-namespaces <app_namespace_1>,<app_namespace_2>
  rtfctl report -n <rtf_namespace> --app-namespaces <app_namespace_1> --app-namespaces <app_namespace_2>
  # Report for all application namespaces that belong to <rtf_namespace>
  rtfctl report -n <rtf_namespace> --app-namespaces all

status

Displays the status of Runtime Fabric

  • Displays status output in JSON

    rtfctl status --output json --namespace <rtf_namespace>
  • Checks Kubernetes DNS health

    rtfctl status dns-check --node-label "<k>:<v>"

test

Verifies outbound network connectivity to the Runtime Fabric control plane.

  • Schedules a check on any nodes

    rtfctl test outbound-network --namespace <rtf_namespace>
  • Schedules a check the nodes satisfying the node-label parameter

    rtfctl test outbound-network --node-label "<k>:<v>" --namespace <rtf_namespace>
  • Schedules a check on all nodes

    rtfctl test outbound-network --all --namespace <rtf_namespace>

Usage

Install a Mule license

  1. Make sure to base64 encode the license.

  2. Run the following command:

    $ sudo ./rtfctl apply mule-license '<license-key-information>'  --namespace rtf1
    Updating rtf namespace... OK

Capture a Heap Dump

$ sudo /usr/local/bin/rtfctl heapdump hello-world /tmp/dump.hprof --app-namespace bed42b62-XXXX --namespace rtf1
Dumping heap for hello-world-f76484d8-l44qv...
Heap dump written to /tmp/dump.hprof

For additional information, run the rtfctl heapdump -h command.

Working with Multiple Replicas

For application management commands that accept a --pod flag, you can obtain the list of replicas using the describe command:

$ sudo ./rtfctl describe apps hello-world --app-namespace bed42b62-XXXX --namespace rtf1
Name:              hello-world
Namespace:         bed42b62-XXXX
Status:            PROGRESSING
Runtime:           2
ID:
Version:
Organization ID:   d30f74c0-XXXX
Replicas:          1 desired | 1 updated | 1 total | 0 available | 1 unavailable

  Name                             Status
  ----                             ------
  hello-world-77f5897949-z54gb   Pending

Auditing rtfctl Operations

The rtfctl command line tool (from v1.0.29 and above) audits the rtfctl operations on the cluster and it helps cluster administrators and you to track operations that impact Runtime Fabric core software.

Consider the following points:

  • Audit logs are stored in a Kubernetes secret in the rtfctl-audit namespace.

  • Audit logs are cluster scoped and not attached to Runtime Fabric core software lifecycle.

  • Command executions remain operational even if the auditing process fails to audit the operation.

To audit the operations, run the following command:

$ sudo ./rtfctl audit-logs

An output example:

[{"timestamp":"2023-10-26 00:52:24.446 +0000 UTC","kubeContext":"test-k8scontext","command":["./rtfctl","restart","test-application","--app-namespace","8456f9c7-2149-4983-b02b-6b18ddd28efe","--namespace","rtf"]}]
Currently audit logs rollover or rotation is not supported.

Understanding rtfctl Output

The rtfctl command line tool commonly calls other applications and system tools, and the output of those calls is sent to standard output and standard error output. This output is written in the context of each operation performed.

The rtfctl command line tool includes a last output line on the most verbose commands to make it clear whether the operation succeed or failed. Normal operation may include some intermediate failure messages, such as when a stopped system unit is attempted to be stopped again.

If you are in doubt about whether a message represents a problem or not, discuss it with your system administrator or contact MuleSoft support.

Download Previous rtfctl CLI Versions

Use the following rest API call to download different rtfctl tool CLI versions:

curl -L https://anypoint.mulesoft.com/runtimefabric/api/download/rtfctl/{version} -o rtfctl