Enable Inbound Traffic on Anypoint Runtime Fabric (On VMs / Bare Metal)

Runtime Fabric on VMs / Bare Metal provides load balancing to manage inbound traffic to Mule applications and API proxies. Network traffic is balanced across multiple replicas of an application, which, by default, are deployed across different worker VMs.

You can share Runtime Fabric instances across multiple environments. After installation, configure one or more environments for each Runtime Fabric instance to enable application deployments. Associate at least one environment with each Runtime Fabric instance to deploy Mule applications or API gateways.

Applications deployed to a production environment must be deployed to an instance of Runtime Fabric on VMs / Bare Metal that is separate from nonproduction applications.

By default, inbound traffic is disabled to prevent consuming unnecessary resources. To allow Mule applications or API gateways to listen on inbound connections, you must enable inbound traffic.

Prepare to Enable Inbound Traffic

Before enabling inbound traffic for Runtime Fabric on VMs / Bare Metal, be sure you’ve addressed the following:

Complete the prerequisite network-related tasks in Infrastructure Requirements, including confirguring an external TCP load balancer to manage traffic to applications running in Runtime Fabric on VMs / Bare Metal.
Gather all IP addresses of controller or dedicated nodes (depending on the configuration you choose during installation) that can be reached by the external TCP load balancer. You might not know these IP addresses until you complete the Runtime Fabric on VMs / Bare Metal installation.
Determine if you’ll use shared or dedicated mode for your internal load balancer.By default, the inbound load balancer runs on each controller node configured in Runtime Fabric on VMs / Bare Metal.

For smaller traffic loads, you can share controller nodes for cluster administration and inbound traffic TLS termination.

For larger traffic loads, use dedicated nodes. Refer to Internal Load Balancer for additional information.

Set the number of replicas for inbound traffic to match either the number of controller nodes (for shared mode) or the number of dedicated internal load balancer nodes (for dedicated mode). Include all of the IP addresses for these nodes in your external load balancer or DNS resolution; otherwise, traffic cannot be routed from the load balancer to Runtime Fabric.
Know the domain or domains to use for routing traffic to your applications.
Configure the DNS settings to associate domains that are signed in the TLS certificate to the IP address or addresses served by the external load balancer.

Enable Inbound Traffic

To enable inbound traffic on Runtime Fabric on VMs / Bare Metal, complete the following tasks:

Configure inbound traffic using Anypoint Runtime Manager.
Configure a Transport Layer Security (TLS) context for HTTPS.
Optionally, select security policies.
Optionally, select advanced options.
Optionally, configure internal load balancer logs.
Verify that inbound traffic is enabled.
Configure an external load balancer.
Deploy applications.

Configure Inbound Traffic Using Anypoint Runtime Manager

To configure inbound traffic:

Navigate to Runtime Manager and select Runtime Fabric.
Select the name of your Runtime Fabric instance to open the management page.
In the Associated Environments tab, ensure that your Runtime Fabric instance is associated with the environments in which applications are to be deployed.
Select Inbound Traffic and then select Enable inbound traffic.
In the Resource Allocation section, select Shared Mode or Dedicated Mode.
Specify the minimum number of cores to allocate for each controller node for the internal load balancer.
Specify the minimum memory to allocate for each controller node for the internal load balancer.

In the Protocol section, select one of the following protocol options:

Accept HTTPS requests…

Select one of the following options:
- and redirect HTTP to HTTPS
  
  After inbound traffic is enabled, HTTP requests sent to Runtime Fabric on VMs / Bare Metal receive a 301 response to redirect to HTTPS on port 443.
- and also accept HTTP requests
  
  HTTP requests sent to Runtime Fabric on VMs / Bare Metal are directed to port 80.
- and drop HTTP requests
  
  HTTP requests sent to Runtime Fabric on VMs / Bare Metal are ignored.

Accept HTTP requests only

Because certificates are not used in this case, click Add Domain and manually enter domains for routing requests to applications.

Accept HTTP requests only is not a secure option for external connections. Use this option for network connections for which an external load balancer has previously terminated a secure TLS connection and is forwarding traffic via HTTP to Runtime Fabric on VMs / Bare Metal.

Configure a TLS Context for HTTPS

If you selected Accept HTTPS requests… in the Protocol section, all inbound traffic entering Anypoint Runtime Fabric is encrypted using TLS. In this case, when enabling inbound traffic, you can provide up to ten certificates but at least one valid TLS certificate is required. See CPU Requirements for Keys and Certificates for more information.

When you set up TLS, you have the option to use one of the following:

A public certificate, private key, and CA path

Public certificates and private keys must be in Privacy-Enhanced Mail (PEM) format and use TLS default values. A PEM file is a Base64-encoded ASCII file with a .cer, .crt, or .pem extension. The CA path contains the intermediary and root certificates that provide the path from the certificate to the root. The file is downloadable from the CA that signed your certificate. When you provide an entry in this field, the Runtime Fabric on VMs / Bare Metal load balancer at every connection sends both the certificate and the path to the client. Many clients require the server to send the CA path so that the certificate can be validated.

With this option, you cannot change the default values for TLS versions, ciphers, and other TLS configuration options. This is the default option when no TLS context exists for Runtime Fabric on VMs / Bare Metal.
A Java Keystore (JKS) file

A JKS file is a repository for authorization or public key certificates and does not store secret keys. With this option, you cannot change the default values for TLS versions, ciphers, and other TLS configuration options.
A TLS context imported from secrets manager

This option is for advanced users only. It imports a TLS context from the secrets manager and supports advanced configurations such as creating a TLS context, mutual authentication, selecting ciphers, and selecting TLS versions.

You can change only the TLS configuration and policies for Runtime Fabric instances that you own. For inherited Runtime Fabric instances, the TLS configuration is read-only.

TLS termination is computationally expensive, so allocate enough CPU to increase throughput and decrease latency. Refer to Internal Load Balancer Memory Allocation for more information about CPU allocation and throughput, including how the TLS private key type and size affect these numbers.

In Runtime Manager, select Add certificate.

To configure a TLS context for HTTPs, choose the appropriate option:

Option 1: Upload PEM
1. For Public certificate, specify a public certificate for the inbound traffic server. The Domains field lists domains that can be used for the Application url value, with the first domain listed as the default. You can select other values via the Applications→Ingress page.
  
  The certificate must be set with a passphrase and a common name (CN) that specifies the domain for each application deployed to Runtime Fabric.
  - If the CN contains a wildcard, the endpoint for each deployed application takes the form {app-name}.{common-name}.
  - If the CN does not contain a wildcard, the endpoint takes the form {common-name}/{app-name}.
2. Specify a value for Private key. This is the PEM formatted file that contains the private key for the certificate.
  
  Optionally, leave the Key password field empty if your key is unencrypted (not recommended).
3. Specify a value for CA path certificate (optional).
4. Select Add certificate. The Key password field is blanked out for security reasons.
  
  The public certificate, private key, and key passcode are saved in the secrets manager.
Option 2: Upload JKS
1. Specify a value for Keystore File. At a minimum, the keystore file contains the public certificate and private key.
2. Specify a value for Keystore Password.
3. In the Select alias from keystore window, specify a value for Alias. The alias is used to select a specific key pair.
4. Select Add certificate. The Keystore Passcode and Key Passcode fields are blanked out for security reasons.
  
  The JKS file information is saved in the global secrets group for your organization.

Option 3: Import from Secrets Manager

Refer to the instructions in Importing a TLS Context from Secrets Manager (Advanced).

If you make any changes to the TLS context in Secrets Manager, they are not automatically applied to the inbound traffic configuration. If you make any changes, navigate to the Inbound Traffic tab, and click Save and Deploy.

Optionally Select Security Policies

If you want to add security policies, you must first define them in Anypoint Security before they are displayed as options in the HTTP Limits, Web Application Firewall (WAF), IP Allowlist, or Denial of Service (DoS) dropdown lists.

To access a Runtime Fabric on VMs / Bare Metal instance using more than one FQDN, add FQDN entries in the subject alternative names (SAN) certificate property. If a certificate has multiple FQDN entries specified in the SAN property, the available URLs are displayed in the Applications→Ingress page when you deploy an application. The URLs are not listed in priority, nor is the first item in the list the default.

To define a security policy in Anypoint Security, you must have the Anypoint Security - Edge entitlement for your Anypoint Platform account. If you do not see Security listed in Management Center, contact your customer success manager to enable Anypoint Security for your account. Refer to Anypoint Security Policies for Edge for additional information.

Optionally Select Advanced Options

The following table describes additional configuration options you might set for your environment.

In this table, Source IP refers to the client making the request.

Value	Description
Max Connections	The maximum number of simultaneous connections to allow. Default value: 512 connections
Max Requests per Connection	The maximum number of requests per connections to allow. This value ranges from 1 to 4194304. Because this value determines how much reuse a connection allows, consider the amount of CPU required to terminate and reestablish a TLS-encrypted connection when lowering this value. Maximum allowed: 1000 requests per connection Default value: 1000. This value balances security and performance. Refer to Resource Allocation and Performance on Anypoint Runtime Fabric for additional information.
Connection Idle Time-out	The maximum amount of time that allowed for an idle connection. This value helps you terminate idle connections and free resources. This value should always be higher than your Read Request Time-out. Default value: 15 seconds
Read Request Time-out	The maximum amount of time spent to read a request before it is terminated. This value enables requests with large payloads or slow clients to be terminated to keep resources available. This helps guard against connection pool exhaustion from slow requests or from clients who don’t close connections after a response is sent. For example, if a Mule application takes longer than this value to respond, the connection is automatically closed. This value should always be lower than the Connection Idle Time-out value previously configured. Default value: 10 seconds
Read Response Time-out	The maximum amount of time spent to initiate a response before the connection is terminated. This value enables requests with large payloads be terminated to keep resources available. Default value: 300 seconds
Write Response Time-out	The maximum amount of time spent from the end of the request header read to the end of the response write before the request is terminated. Default value: 10 seconds
Max Pipeline Depth	The maximum number of requests to allow from the same client. This value defines how many simultaneous requests a client can send. If a client exceeds this number, the exceeding requests are not read until the requests in the queue receive a response. Default value: 10 requests per client
Source IP header name and enable proxy protocol	Configure the following values based on the applicable scenario: Runtime Fabric on VMs / Bare Metal is not deployed behind a load balancer. These values should not be configured. Source IP header name: Blank Enable proxy protocol: Unchecked Runtime Fabric on VMs / Bare Metal is deployed behind an AWS load balancer with a proxy protocol configured. You must select the enable proxy protocol option. Source IP header name: Blank Enable proxy protocol: Checked Runtime Fabric on VMs / Bare Metal is behind a non-AWS load balancer. If Runtime Fabric on VMs / Bare Metal is deployed behind another type of load balancer, such as F5 or NGINX, the source IP address can be provided in an HTTP Header field. In this case, enter the HTTP header name that contains the source IP header. HTTP messages not containing this header field will be rejected. Two common HTTP header names that are used for source IP addresses are: Forwarded: An RFC7239 compliant IP header. X-Forwarded-For: Non-standard pre-2014 header containing one or more IPs from a load balancer (For example: “192.16.23.34, 172.16.21.36") Source IP header name: Non-blank Enable proxy protocol: Unchecked Default value: Blank and unchecked.

Value

Description

Max Connections

The maximum number of simultaneous connections to allow.

Default value: 512 connections

Max Requests per Connection

The maximum number of requests per connections to allow.
This value ranges from 1 to 4194304.
Because this value determines how much reuse a connection allows, consider the amount of CPU required to terminate and reestablish a TLS-encrypted connection when lowering this value.

Maximum allowed: 1000 requests per connection

Default value: 1000. This value balances security and performance. Refer to Resource Allocation and Performance on Anypoint Runtime Fabric for additional information.

Connection Idle Time-out

The maximum amount of time that allowed for an idle connection.
This value helps you terminate idle connections and free resources.
This value should always be higher than your Read Request Time-out.

Default value: 15 seconds

Read Request Time-out

The maximum amount of time spent to read a request before it is terminated.
This value enables requests with large payloads or slow clients to be terminated to keep resources available. This helps guard against connection pool exhaustion from slow requests or from clients who don’t close connections after a response is sent.

For example, if a Mule application takes longer than this value to respond, the connection is automatically closed.
This value should always be lower than the Connection Idle Time-out value previously configured.

Default value: 10 seconds

Read Response Time-out

The maximum amount of time spent to initiate a response before the connection is terminated.
This value enables requests with large payloads be terminated to keep resources available.

Default value: 300 seconds

Write Response Time-out

The maximum amount of time spent from the end of the request header read to the end of the response write before the request is terminated.

Default value: 10 seconds

Max Pipeline Depth

The maximum number of requests to allow from the same client.
This value defines how many simultaneous requests a client can send.
If a client exceeds this number, the exceeding requests are not read until the requests in the queue receive a response.

Default value: 10 requests per client

Source IP header name and enable proxy protocol

Configure the following values based on the applicable scenario:

Runtime Fabric on VMs / Bare Metal is not deployed behind a load balancer.
These values should not be configured.

Source IP header name: Blank
Enable proxy protocol: Unchecked
Runtime Fabric on VMs / Bare Metal is deployed behind an AWS load balancer with a proxy protocol configured.
You must select the enable proxy protocol option.

Source IP header name: Blank
Enable proxy protocol: Checked
Runtime Fabric on VMs / Bare Metal is behind a non-AWS load balancer.
If Runtime Fabric on VMs / Bare Metal is deployed behind another type of load balancer, such as F5 or NGINX, the source IP address can be provided in an HTTP Header field. In this case, enter the HTTP header name that contains the source IP header.

HTTP messages not containing this header field will be rejected. Two common HTTP header names that are used for source IP addresses are:
- Forwarded: An RFC7239 compliant IP header.
- X-Forwarded-For: Non-standard pre-2014 header containing one or more IPs from a load balancer (For example: “192.16.23.34, 172.16.21.36")
  
  Source IP header name: Non-blank
  Enable proxy protocol: Unchecked

Default value: Blank and unchecked.

If you are using WebSockets:

Provide the correct request headers to upgrade the HTTP connection to WebSockets.
Configure Mule runtime engine with a WebSockets Listener.
Increase the Connection Idle Time-out value to 900 seconds (15 minutes) to ensure consistency with the WebSockets Mule application default value.

Optionally Configure Internal Load Balancer Logs

You can define the log levels for the internal load balancer. Runtime Fabric supports the following log levels, listed in descending order of verbosity:

FATAL
ERROR
WARNING
INFO
VERBOSE
DEBUG
TRACE

The more verbose log levels, which include WARNING, INFO, VERBOSE, DEBUG, and TRACE, consume more CPU resources for each request. Consider this when adjusting the log level and allocating resources for the internal load balancer.

By default, the activity across all IP addresses behind your endpoint is logged. To help reduce CPU consumption when using more verbose log levels, add IP filters to only log specific IP addresses. This also reduces the number of logs when debugging a connection for a specific or limited number of IP addresses.

From the Inbound Traffic tab, select Logs.
Select Add Filter.
In the IP field, enter a single IP address or subset of addresses using CIDR notation.
Select the log level to apply to this IP filter.
Select OK.
Select Save and Deploy to deploy the internal load balancer.

The deployment can take up to a minute to complete.

If there are validation errors, an error message is returned. If the validation is successful, a message in green text is displayed at the bottom-right of the page indicating that the deployment request is accepted. You can view the deployment status at the beginning of the page.

Verify That Inbound Traffic Is Enabled

To test inbound traffic for deployed applications, you can send a request using the controller IP address along with a host header set to the domain. The host header depends on the structure of the application URL.

Determine which endpoint exposes the application. The Application url field on the Manage application page in Runtime Manager contains this information.
Run the following cURL command for verification:
```
curl -Lvk -XGET {application-path-from-runtime-manager} --resolve {hostname}:443:{ip-address-of-controller}
```
In the following example, {application-path-from-runtime-manager} is set to https://newapp.example-rtf.dev, and 192.168.64.14 is the IP address of a controller machine in your cluster.
```
curl -Lvk https://newapp.example-rtf.dev/ --resolve newapp.example-rtf.dev:443:192.168.64.14
```

Configure an External Load Balancer

After you enable inbound traffic, you must configure Runtime Fabric on VMs / Bare Metal to route incoming traffic to each enabled application for clients to send requests to deployed applications.

For HTTPS requests, you must configure an external load balancer to load balance HTTPS traffic between each controller VM on Runtime Fabric on VMs / Bare Metal. Controller VMs are virtual machines dedicated to run the components that power Anypoint Runtime Fabric. Each controller VM runs a replica of the internal load balancer and is configured to listen on port 443.

Provision the external TCP load balancer to route traffic to the Runtime Fabric on VMs / Bare Metal controller or dedicated nodes with the IPs identified during installation.

External Load Balancer Requirements

When running multiple controller VMs, you must have an external load balancer outside Runtime Fabric on VMs / Bare Metal to front each of the controller VMs.

The external load balancer must support TCP load balancing and must be configured with a server pool containing the IP addresses of each controller VM. A health check must also be configured on the external load balancer, listening on port 443.

This configuration of the external load balancer provides:

High availability
Protection against failure
Automatic failover if a replica of the internal load balancer restarts or is evicted and rescheduled on another controller VM

To configure an external load balancer:

Review the information described in Advanced Options when adding an external load balancer.
Configure DNS before using the CN obtained from the TLS certificate. DNS is required to send requests to applications or API gateways deployed to Runtime Fabric. Add an "A record" to your DNS provider to map the CN to the IP address of the external load balancer or controller VMs.

Deploy Applications

When you are ready to deploy an application:

Follow the instructions in Deploy a Mule Application to Runtime Fabric.
Verify the application URL.

The Ingress tab allows you to update the configuration for application requests. When you enable inbound traffic, the default behavior is changed to allow for new application deployments. If there are applications deployed to Runtime Fabric before you enable inbound traffic, they do not receive inbound requests until this setting is enabled.

The application’s URL contains the routing path for the application. If the default domain is not the desired domain to be served by the application, select the desired domain from the Domain drop-down list.

View TLS Certificates

To view TLS certificate information for an existing deployment:

Select the Inbound Traffic tab for a Runtime Fabric instance.
Scroll to the Domains section.
Select ….
Select View details.

Update or Delete TLS Certificates

To update or delete TLS certificate information:

Select the Inbound Traffic tab in Runtime Manager.
Scroll to the Domains section.
Select ….
Select Delete.
To add updated certificate information, select Add certificate and follow the instructions provided in Step 1 to configure a new certificate.

Switching Between Shared and Dedicated Modes

If you need to switch between shared and dedicated modes, review the following:

All inbound traffic is lost if the shared or dedicated IP addresses are not included in the load balancer or DNS resolution before making the change.
If you change from shared mode to dedicated mode, you must include the dedicated internal load balancer node IP addresses in place of the controller node IP addresses in your external load balancer. Temporarily include both controller and dedicated internal load balancer nodes until you route inbound requests.
If you change from dedicated mode to shared mode, configure the CPU Cores and Memory fields appropriately for your deployment. The amount of available resources can significantly change between dedicated mode, in which all node resources can be consumed, and shared mode.

Upgrade Changes

For Runtime Fabric versions 1.5.0 or later, the internal load balancer is upgraded during the Runtime Fabric component upgrade process.

Generate a TLS Certificate for Testing

For testing purposes, you can use the following steps to generate a certificate-key pair:

Run the following command on your machine to generate a certificate-key pair:
```
openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -days 365
```
Type a passphrase for your key.
Complete the requested information. When asked for a common name, supply the domain to be used in your Runtime Fabric.

If you use a wildcard, for example, *.example.com in your common name, your application URLs use the following format: {app-name}.example.com. Otherwise, your application URLs use the format example.com/{app-name}.

TLS Certificate Expiration

Certificates (both self-signed and CA-signed) always have an expiration date. By default, certificates expire one year after they are created.

The following warnings are displayed for certificates that will expire within 30 days to remind you to upload a new certificate-key pair before a certificate expires:

On the Runtime Fabrics page, if a TLS certificate will expire within the next 30 days,TLS Expiring is displayed in the Inbound traffic column.
On the Runtime Fabrics page, when a TLS certificate has expired, a warning is displayed in the Inbound traffic column for that Runtime Fabric instance.
On the Inbound Traffic tab, if a TLS certificate will expire within the next 30 days, a warning is displayed. When a TLS certificate has expired, the expiration date information includes a red warning in the Certificate File field.