Adding or Removing Nodes from Runtime Fabric on VMs / Bare Metal

When using Runtime Fabric on VMs / Bare Metal, you can add and remove nodes from a Runtime Fabric. Scenarios where you may need to add or remove a node include:

Add a node to scale-up a Runtime Fabric.
Delete or add nodes to ensure the health and stability of a Runtime Fabric.
Delete a node, perform an OS patch, then re-add the node to a Runtime Fabric.

Adding a node from the Ops Center UI is not supported. Use the following procedure to add a new node to Runtime Fabric.

Add a Node to Runtime Fabric

Before adding a node, provision the infrastructure for the new node. To be able to join the cluster, the new node must be on the same network as your Runtime Fabric installation.

Before You Begin

Before adding a new node, ensure you have:

Provisioned the necessary hardware as outlined in Anypoint Runtime Fabric Installation Prerequisites.
Listed the required disks as block devices for each VM. The disks should not be mounted or formatted.

Refer to Production Configuration Requirements or Development Configuration Requirements for full disk requirements.
Opened the necessary ports and added hostnames to allowlists as described in Network and Port Requirements.
Disabled any antivirus agents, such as McAfee, running in your environment.

To add a new node:

On the virtual machine of the new node create the /opt/anypoint/runtimefabric directory:
```
mkdir -p /opt/anypoint/runtimefabric
```
Copy the init.sh installation script to the new node under the /opt/anypoint/runtimefabric directory.

This script is included in the files you downloaded when installing Runtime Fabric.
Copy the environment file from the /opt/anypoint/runtimefabric directory of an existing node to the same directory on the new node.

Ensure that you copy the environment file from the correct node type (controller or worker).
Modify the environment file you just copied to contain only the parameters required for the correct node type.
- Worker node parameters
  
  The following example shows the environment parameters that are required for a worker node. Use the values in the file you copied for RTF_DOCKER_DEVICE_SIZE, RTF_TOKEN, and RTF_INSTALLER_IP. For RTF_DOCKER_DEVICE, replace device_name with the appropriate value.
  - RTF_NODE_ROLE=worker_node
  - RTF_INSTALL_ROLE=joiner
  - RTF_DOCKER_DEVICE=/dev/device_name
  - RTF_TOKEN="R6J4UR6J4UR6J4UO"
  - RTF_INSTALLER_IP="10.0.42.86"
  - Remove RTF_ACTIVATION_DATA if it is present.
- Controller node parameters
  
  The following example shows the environment parameters that are required for a controller node. Use the values in the file you copied for RTF_DOCKER_DEVICE_SIZE, RTF_TOKEN, and RTF_ETCD_DEVICE. For RTF_DOCKER_DEVICE, replace device_name with the appropriate value. For RTF_INSTALLER_IP, specify the IP address of the lead controller node.
  - RTF_NODE_ROLE=controller_node
  - RTF_INSTALL_ROLE=joiner
  - RTF_DOCKER_DEVICE=/dev/device_name
  - RTF_TOKEN="R6J4UR6J4UR6J4UO"
  - RTF_ETCD_DEVICE=/dev/etcd_device
  - RTF_INSTALLER_IP="10.1.255.93"
  - Remove RTF_ACTIVATION_DATA if it is present
- Dedicated node parameters
  
  The following example shows the environment parameters that are required for a load balancer node. If you copied the environment file from a controller node, delete all lines in the file except those shown in the example. Use the values in the file you copied for RTF_DOCKER_DEVICE_SIZE and RTF_TOKEN. For RTF_DOCKER_DEVICE, replace device_name with the appropriate value. For RTF_INSTALLER_IP, specify the IP address of the lead controller node.
  - RTF_NODE_ROLE=dedicated_ingress_node
  - RTF_INSTALL_ROLE=joiner
  - RTF_DOCKER_DEVICE=/dev/device_name
  - RTF_DOCKER_DEVICE_SIZE=50G
  - RTF_TOKEN="R6J4UR6J4UR6J4Uy"
  - RTF_INSTALLER_IP="10.1.255.93"
Run the init.sh script on the new node as a privileged user. You may need to add executable permissions when running this script:
```
sudo chmod +x ./init.sh
```
This script causes the new node to join the Runtime Fabric cluster.
Verify the new node was added to the Runtime Fabric:
```
sudo gravity status
```
When a dedicated internal load balancer node is added to the cluster, the option to specify a dedicated node is provided when you enable inbound traffic. For dedicated nodes, you cannot specify the amount of CPU cores and memory.

Remove a Node from Runtime Fabric

To remove a controller node from a Runtime Fabric cluster, run the following command:

sudo kubectl drain <node-name>
sudo gravity leave

To remove a worker node from a Runtime Fabric cluster, run the following command:

sudo gravity leave

This command shuts down all Runtime Fabric services running on the node. Additionally, all software and data is removed.

If successful, this command outputs the ID of the initiated removal operation. You can monitor the the progress of this command using the following command:

gravity status

If the node cannot be removed using sudo gravity leave you may need to run the command again using the --force option. You can also remove a node remotely by running the following command:

sudo kubectl drain <node-name>
sudo gravity remove <node-name>

<node> specifies the node to remove and can be either the node’s assigned hostname or its IP address.