Enable Inbound Traffic on Anypoint Runtime Fabric (On VMs / Bare Metal)

Runtime Fabric on VMs / Bare Metal provides load balancing to manage inbound traffic to Mule applications and API proxies. Network traffic is balanced across multiple replicas of an application, which, by default, are deployed across different worker VMs.

Runtime Fabric can be shared across multiple environments. After installation, configure one or more environments for each Runtime Fabric to enable application deployment. You must associate at least one environment with each Runtime Fabric to be able to deploy Mule applications or API gateways.

Applications deployed to a production environment must be deployed to an instance of Runtime Fabric on VMs / Bare Metal that is separate from nonproduction applications.

By default, inbound traffic is disabled to prevent consuming unnecessary resources. To allow Mule applications or API gateways to listen on inbound connections, you must enable inbound traffic.

TLS Context Configuration Prerequisites

Complete the prerequisite network-related tasks in Infrastructure Requirements before enabling inbound traffic on Runtime Fabric on VMs / Bare Metal.
Be able to create or configure an external TCP load balancer to manage traffic to applications running in Runtime Fabric on VMs / Bare Metal:
- Gather all IP addresses of controller or dedicated nodes (depending on the configuration you choose during installation) that can be reached by the external TCP load balancer. These IP addresses may not be known until you complete the Runtime Fabric on VMs / Bare Metal installation.
  
  For smaller traffic loads, controller nodes can be shared for cluster administration and inbound traffic TLS termination.
  
  For larger traffic loads, use dedicated nodes. The controller and dedicated nodes must have IP addresses that can be reached (are routable and allowed at applicable firewalls) by the external load balancer. Otherwise, traffic cannot be routed from the load balancer to Runtime Fabric.
- Know the domain or domains to use for routing traffic to your applications.
- Understand that you need to configure an external load balancer to proxy the TCP protocol. The load balancer does not terminate TLS but does allow Runtime Fabric to do so.
Be able to configure the DNS settings to associate domains that are signed in the TLS certificate to the IP address or addresses served by the external load balancer.
You can change only the TLS configuration and policies for Runtime Fabrics that you own. For inherited Runtime Fabric instances, the TLS configuration is read-only.

Step 1: Configure Inbound Traffic Using Anypoint Runtime Manager

Navigate to Runtime Manager and select Runtime Fabric.
Select the name of your Runtime Fabric to open the management page.
In the Associated Environments tab, ensure that your Runtime Fabric is associated with the environments in which applications are to be deployed.
Select Inbound Traffic and activate the Enable inbound traffic toggle.

If you disable the Inbound Traffic tab, changes can take several minutes to complete.

In the Resource Allocation section:

By default, the inbound load balancer runs on each controller node configured in Runtime Fabric on VMs / Bare Metal. Refer to Resource Allocation and Performance on Anypoint Runtime Fabric for more information.

Select a mode:

Shared Mode

Dedicated Mode

Dedicated nodes are used in environments with high traffic demands. Refer to Internal Load Balancer for additional information.

The number of replicas for inbound traffic is set to the number of controller nodes for Shared Mode or the number of dedicated internal load balancer nodes for Dedicated Mode. You must include all IP addresses of the controller nodes or the dedicated internal load balancer nodes in your external load balancer or DNS resolution.

Refer to Internal Load Balancer and Network Architecture for additional information.

When switching between shared and dedicated modes:

All inbound traffic is lost if the shared or dedicated IP addresses are not included in the load balancer or DNS resolution prior to making the change.
If you change from Shared Mode to Dedicated Mode, you must include the dedicated internal load balancer node IP addresses in place of the controller node IP addresses in your external load balancer. Temporarily include both controller and dedicated internal load balancer nodes until the switch is completed to properly route inbound requests.
If you change from Dedicated Mode to Shared Mode, configure the CPU Cores and Memory fields appropriately for your deployment. The amount of available resources can significantly change between Dedicated Mode, in which all node resources can be consumed, and Shared Mode.

Specify the minimum number of cores to allocate for each controller node for the internal load balancer.

Note that Runtime Fabric on VMs / Bare Metal can use up to the maximum number shown if the cores are available. Transport Layer Security (TLS) termination is computationally expensive, so allocate enough CPU to increase throughput and decrease latency. Refer to Internal Load Balancer Memory Allocation for more information on CPU allocation and throughput, including how the TLS private key type and size affect these numbers.
Specify the minimum memory to allocate for each controller node for the internal load balancer. Note that Runtime Fabric on VMs / Bare Metal can use up to the maximum amount shown if it is available.

In the Protocol section, select one of the following protocol options:

Accept HTTPS requests…

Select one of the following options:
- and redirect HTTP to HTTPS
  
  After inbound traffic is enabled, HTTP requests sent to Runtime Fabric on VMs / Bare Metal receive a 301 response to redirect to HTTPS on port 443.
- and also accept HTTP requests
  
  HTTP requests sent to Runtime Fabric on VMs / Bare Metal are directed to port 80.
- and drop HTTP requests
  
  HTTP requests sent to Runtime Fabric on VMs / Bare Metal are ignored.

Accept HTTP requests only

Because certificates are not used in this case, use the “Add Domain” button to manually enter domains for routing requests to applications and skip the next step for configuring a TLS context.

Accept HTTP requests only is not recommended as a secure option for external connections. Use this less secure option for network connections for which an external load balancer has previously terminated a secure TLS connection and is forwarding traffic via HTTP to Runtime Fabric on VMs / Bare Metal.

Configure a TLS context for HTTPS.

If you selected Accept HTTPS requests in the Protocol section, all inbound traffic entering Anypoint Runtime Fabric is encrypted using TLS. In this case, when enabling inbound traffic, you can provide up to 10 certificates, but at least one valid TLS certificate is required. See CPU Requirements for Keys and Certificates for more information.
1. Select Add certificate.
  
  You can configure a private key and public certificate for a TLS-enabled server in one of the following ways:
  - Option 1: Upload PEM
    
    Use this option to upload a public certificate and private key in Privacy-Enhanced Mail (PEM) format and use TLS default values. A PEM file is a Base64-encoded ASCII file with a .cer, .crt, or .pem extension.
    
    With this option, you cannot change the default values for TLS versions, ciphers, and other TLS configuration options.
    
    This is the default option when no TLS context exists for Runtime Fabric on VMs / Bare Metal.
    
    For Public certificate, specify a public certificate for the inbound traffic server in PEM format. The Domains field lists domains that are selectable for Application url, which is used for routing requests to an application. By default, the first domain listed is used. Other values can be selected via the Applications→Ingress page.
    
    The certificate must be set with a passphrase and a common name (CN) that specifies the domain for each application deployed to Runtime Fabric.
    
    Select Details to view certificate details, including domains, expiration, and the other configuration settings.
    
    If the CN contains a wildcard, the endpoint for each deployed application takes the form {app-name}.{common-name}.
    
    If the CN does not contain a wildcard, the endpoint takes the form {common-name}/{app-name}.
    
    Specify a value for Private key. This is the PEM formatted file that contains the private key for the certificate.
    
    Optionally, leave the Key password field empty if your key is unencrypted (not recommended).
    
    Specify a value for CA path certificate (optional). The CA path contains the intermediary and root certificates that provide the path (chain of trust) from the certificate to the root. The file is downloadable from the CA that signed your certificate. When you provide an entry in this field, the Runtime Fabric on VMs / Bare Metal load balancer at every connection sends both the certificate and the path to the client. Many clients require the server to send the CA path so that the certificate can be validated.
    
    Select Add certificate after specifying any needed security policies or advanced options. The Key password field is blanked out for security reasons. You can still review public certificate details. If you upload a new key file, this field is again enabled.
    
    The public certificate, private key, and key passcode are saved in the secrets manager.
  - Option 2: Upload JKS
    
    Use this option to upload a Java Keystore (JKS) file and use TLS default values. With this option, you cannot change the default values for TLS versions, ciphers, and other TLS configuration options. A JKS file is a repository for authorization or public key certificates and does not store secret keys.
    
    Specify a value for Keystore File. At a minimum, the keystore file contains the public certificate and private key, also known as a key pair.
    
    Specify a value for Keystore Password, the word or phrase that protects the keystore.
    
    Specify a value for Alias. The alias is used to select a specific key pair.
    
    In the Select alias from keystore window, specify an alias. The alias is used to select a specific key pair.
    
    The following information is displayed: ** The URL format to be used for your apps, based on the certificate’s CN.
    
    + The certificate must be set with a passphrase and a CN that specifies the domain for each application deployed to Runtime Fabric. The domain is used for routing requests to an application. Other values can be selected via the Applications→Ingress page.
    
    +
    
    The Select alias from keystore window is enabled after you specify a keystore file and keystore passcode.
    
    + Select Details to view the information you entered for Keystore File, Keystore Password, and Alias before selecting other options or deploying.
    
    If the CN contains a wildcard, the endpoint for each deployed application takes the form {app-name}.{common-name}.
    
    If the CN does not contain a wildcard, the endpoint takes the form {common-name}/{app-name}.
    
    The expiration date of the secret.
    
    Specify a value for Key Password, the word or phrase that protects the private key.
    
    The Details field is enabled after you specify an alias.
    
    Select Add certificate. The Keystore Passcode and Key Passcode fields are blanked out for security reasons.
    
    If you select a different Alias value, the Key Passcode field is again enabled.
    
    If you upload a new keystore file, the Alias and Keystore Passcode fields are again enabled and the Alias field contents are cleared.
    
    The JKS file information is saved in the global secrets group for your organization.
  - Option 3: Import from Secrets Manager (For Advanced Users)
    
    This option imports a TLS context from the secrets manager, and supports advanced configuration such as creating a TLS context, mutual authentication, selecting ciphers, and selecting TLS versions.
    
    Refer to the instructions in Import a TLS Context from Secrets Manager (Advanced).

(Optional) Select Security Policies

A security policy must be defined in Anypoint Security to be displayed as an option in the HTTP Limits, Web Application Firewall (WAF), IP Whitelist, or Denial of Service (DoS) dropdown lists.

To access a Runtime Fabric on VMs / Bare Metal instance using more than one DNS, add additional DNS entries in the subject alternative names (SAN) certificate property. If a certificate has multiple DNS entries specified in the SAN property, the available URLs are displayed in the Applications→Ingress page when you deploy an application.

To define a security policy in Anypoint Security, you must have the Anypoint Security - Edge entitlement for your Anypoint Platform account. If you do not see Security listed in Management Center, contact your customer success manager to enable Anypoint Security for your account.

Refer to Anypoint Security Policies for Edge for additional information.

(Optional) Select Advanced Options

The following table describes additional configuration options you might need to set for your environment. In this case, Source IP refers to the client making the request.

Table 1. Advanced Configuration Options
Value	Description
Max Connections	The maximum number of simultaneous connections to allow. Default value: 512 connections
Max Requests per Connection	The maximum number of requests per connections to allow. This value ranges from 1 to 4194304. Because this value determines how much reuse a connection allows, consider the amount of CPU required to terminate and reestablish a TLS-encrypted connection when lowering this value. Maximum allowed: 1000 requests per connection Default value: 1000. This value balances security and performance. Refer to Resource Allocation and Performance on Anypoint Runtime Fabric for additional information.
Connection Idle Time-out	The maximum amount of time that allowed for an idle connection. This value helps you terminate idle connections and free resources. This value should always be higher than your Read Request Time-out. Default value: 15 seconds
Read Request Time-out	The maximum amount of time spent to read a request before it is terminated. This value enables requests with large payloads or slow clients to be terminated to keep resources available.v+ This helps guard against connection pool exhaustion from slow requests or from clients who don’t close connections after a response is sent. For example, if a Mule application takes longer than this value to respond, the connection is automatically closed. This value should always be lower than the Connection Idle Time-out value previously configured. Default value: 10 seconds
Read Response Time-out	The maximum amount of time spent to initiate a response before the connection is terminated. This value enables requests with large payloads be terminated to keep resources available. Default value: 300 seconds
Write Response Time-out	The maximum amount of time spent from the end of the request header read to the end of the response write before the request is terminated. Default value: 10 seconds
Max Pipeline Depth	The maximum number of requests to allow from the same client. This value defines how many simultaneous requests a client can send. If a client exceeds this number, the exceeding requests are not read until the requests in the queue receive a response. Default value: 10 requests per client
Source IP header name and enable proxy protocol	Configure the following values based on the applicable scenario: Runtime Fabric on VMs / Bare Metal is not deployed behind a load balancer. These values should not be configured. Source IP header name: Blank Enable proxy protocol: Unchecked Runtime Fabric on VMs / Bare Metal is deployed behind an AWS load balancer with a proxy protocol configured. You must select the enable proxy protocol option. Source IP header name: Blank Enable proxy protocol: Checked Runtime Fabric on VMs / Bare Metal is behind a non-AWS load balancer. If Runtime Fabric on VMs / Bare Metal is deployed behind another type of load balancer, such as F5 or nginx, the source IP address can be provided in an HTTP Header field. In this case, enter the HTTP header name that contains the source IP header. HTTP messages not containing this header field will be rejected. Two common HTTP header names that are used for source IP addresses are: Forwarded: An RFC7239 compliant IP header. X-Forwarded-For: Non-standard pre-2014 header containing one or more IPs from a load balancer (For example: “192.16.23.34, 172.16.21.36") Source IP header name: Non-blank Enable proxy protocol: Unchecked Default value: Blank and unchecked.

If you are using WebSockets:

Provide the correct request headers to upgrade the HTTP connection to WebSockets.
Configure Mule runtime engine with a WebSockets Listener.
Increase the Connection Idle Time-out value to 900 seconds (15 minutes) to ensure consistency with the WebSockets Mule application default value.

(Optional) Configure Internal Load Balancer Logs

You can define the log levels for the internal load balancer. Runtime Fabric supports the following log levels, listed in descending order of verbosity:
- FATAL
- ERROR
- WARNING
- INFO
- VERBOSE
- DEBUG
- TRACE
  
  The more verbose log levels, which include WARNING, INFO, VERBOSE, DEBUG, and TRACE, consume more CPU resources for each request. Consider this when adjusting the log level and allocating resources for the internal load balancer.
  
  By default, the activity across all IPs addresses behind your endpoint is logged. To help reduce CPU consumption when using more verbose log levels, add IP filters to only log-specific IP addresses. This feature also reduces the quantity of logs when debugging a connection for a specific or limited number of IP addresses.
  1. From Anypoint Platform select Runtime Manager.
  2. Select Runtime Fabric.
  3. Select the Inbound Traffic tab, and then select Logs.
  4. Select Add Filter.
  5. In the IP field, enter a single IP address or subset of addresses using CIDR notation.
  6. Select the log level to apply to this IP filter.
  7. Select OK.
Select Save and Deploy to deploy the internal load balancer.

The deployment can take up to a minute to complete.

If there are validation errors, an error message is returned. If the validation is successful, a message in green text is displayed at the bottom-right of the page indicating that the deployment request is accepted. You can view the deployment status at the beginning of the page.

Step 2: Verify That Inbound Traffic Is Enabled

To test inbound traffic for deployed applications, you can send a request using the controller IP address along with a host header set to the domain. The host header depends on the structure of the application URL.

Determine which endpoint exposes the application. The Application url field on the Manage application page in Runtime Manager contains this information.
Run the following cURL command for verification:
```
curl -Lvk -XGET {application-path-from-runtime-manager} --resolve {hostname}:443:{ip-address-of-controller}
```
In the following example, {application-path-from-runtime-manager} is set to https://newapp.example-rtf.dev, and 192.168.64.14 is the IP address of a controller machine in your cluster.
```
curl -Lvk https://newapp.example-rtf.dev/ --resolve newapp.example-rtf.dev:443:192.168.64.14
```

Step 3: Configure an External Load Balancer

After you enable inbound traffic, you must configure Runtime Fabric on VMs / Bare Metal to route incoming traffic to each enabled application for clients to send requests to deployed applications.

For HTTPS requests, you must configure an external load balancer to load balance HTTPS traffic between each controller VM on Runtime Fabric on VMs / Bare Metal. Controller VMs are virtual machines dedicated to run the components that power Anypoint Runtime Fabric. Each controller VM runs a replica of the internal load balancer and is configured to listen on port 443.

Provision the external TCP load balancer to route traffic to the Runtime Fabric on VMs / Bare Metal controller or dedicated nodes with the IPs identified during installation.

External Load Balancer Requirements

When running multiple controller VMs, you must have an external load balancer outside Runtime Fabric on VMs / Bare Metal to front each of the controller VMs.

The external load balancer must support TCP load balancing and must be configured with a server pool containing the IP addresses of each controller VM. A health check must also be configured on the external load balancer, listening on port 443.

This configuration of the external load balancer provides the following benefits:

Maintains high availability.
Protects against failures.
Gracefully handles automated failover if a replica of the internal load balancer restarts or is evicted and rescheduled on another controller VM.

To configure an external load balancer:

Review the information described in Advanced Options when adding an external load balancer.
Configure DNS before using the CN obtained from the TLS certificate. DNS is required to send requests to applications or API gateways deployed to Runtime Fabric. Add an "A record" to your DNS provider to map the CN to the IP address of the external load balancer or controller VMs.

Step 4: Deploy Applications

When you are ready to deploy an application:

Follow the instructions in Deploy a Mule Application to Runtime Fabric.
Verify the application URL.

The Ingress tab allows you to update the configuration for application requests. When you enable inbound traffic, the default behavior is changed to allow for new application deployments. If there are applications deployed to Runtime Fabric before you enable inbound traffic, they do not receive inbound requests until this setting is enabled.

The application’s URL contains the routing path for the application. If the default domain is not the desired domain to be served by the application, select the desired domain from the Domain drop-down list.

View TLS Certificates

To view TLS certificate information for an existing deployment:

Select the Inbound Traffic tab for a Runtime Fabric instance.
Scroll to the Domains section.
Select the … button.
Select View details.

Update or Delete TLS Certificates

To update or delete TLS certificate information:

Select the Inbound Traffic tab in Runtime Manager.
Scroll to the Domains section.
Select the … button.
Select Delete.
To add updated certificate information, select Add certificate and follow the instructions provided in Step 1 to configure a new certificate.

Upgrade Changes

For Runtime Fabric versions 1.5.0 or later, the internal load balancer is upgraded during the Runtime Fabric component upgrade process.

Generate a TLS Certificate for Testing

For testing purposes, you can use the following steps to generate a certificate-key pair:

Run the following command on your machine to generate a certificate-key pair:
```
openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -days 365
```
Type a passphrase for your key.
Complete the requested information. When asked for a common name, supply the domain to be used in your Runtime Fabric.

If you use a wildcard, for example, *.example.com in your common name, your application URLs use the following format: {app-name}.example.com. Otherwise, your application URLs use the format example.com/{app-name}.

TLS Certificate Expiration

Certificates (both self-signed and CA-signed) always have an expiration date. By default, certificates expire one year after they are created.

The following warnings are displayed for certificates that will expire within 30 days to remind you to upload a new certificate-key pair before a certificate expires:

On the Runtime Fabrics page, if a TLS certificate will expire within the next 30 days,TLS Expiring is displayed in the Inbound traffic column.
On the Runtime Fabrics page, when a TLS certificate has expired, a warning is displayed in the Inbound traffic column for that Runtime Fabric instance.
On the Inbound Traffic tab, if a TLS certificate will expire within the next 30 days, a warning is displayed. When a TLS certificate has expired, the expiration date information includes a red warning in the Certificate File field.