Monitoring Applications Deployed to Runtime Fabric

Applications and API gateways deployed to Runtime Fabric include native support for Anypoint Monitoring to allow viewing metrics within Anypoint Platform.

Runtime Fabric does not provide support for integrating third-party monitoring solutions.

Anypoint Monitoring

Anypoint Monitoring includes tools that provide feedback from Mule flows and components in your application network. Anypoint Monitoring runs as a sidecar in application deployments and gathers metrics for applications and API gateways deployed to Runtime Fabric.

Anypoint Monitoring provides access to log data for applications deployed to Runtime Fabric. To access logs with Anypoint Monitoring, you need the Anypoint Integration Advanced package or a Titanium subscription to Anypoint Platform.

Enable Anypoint Monitoring

By default, Runtime Fabric is configured to send application metrics to Anypoint Monitoring. Data is securely transmitted using mutual TLS encryption. For metrics and logs to be sent to Anypoint Monitoring, outbound network connectivity must be available.

If you need an outbound proxy, configure Anypoint Runtime Fabric to use a SOCKS5 proxy by following the instructions in Manage Proxies Used by Runtime Fabric.

To verify outbound network connectivity on Anypoint Runtime Fabric version 1.8.50 or later, ensure that each node in Runtime Fabric can send outbound TCP (Lumberjack) communication using port 443 to the following domains:

US control plane: dias-ingestor-router.us-east-1.prod.cloudhub.io
EU control plane: dias-ingestor-router.eu-central-1.prod-eu.msap.io

If you are using a version of Anypoint Runtime Fabric older than 1.8.50, use port 5044 instead of port 443. Use the following domains:

US control plane: dias-ingestor-nginx.prod.cloudhub.io
EU control plane: dias-ingestor-nginx.prod-eu.msap.io

Disable Anypoint Monitoring for an Application

When you deploy an application to Runtime Fabric, Anypoint Monitoring is enabled by default. To disable Anypoint Monitoring, pass the following custom property when deploying your application:

anypoint.platform.config.analytics.agent.enabled=false

In some cases, logs with multiple lines might not be fully transmitted to Anypoint Monitoring. This can occur if multiple lines of log output are split across two files due to log rotation inside Runtime Fabric.

Metrics

Each Mule application and API gateway replica runs a metrics agent that collects and transmits metrics data to Anypoint Monitoring via secure mutual TLS encryption. Each replica can cache up to 100 megabytes of metrics, after which the metrics data is rotated to provide storage for more recent data. In the event that metrics cannot be sent to Anypoint Monitoring, this typically provides at least 4 days worth of storage per replica. Note that custom metrics might consume more storage.

Application monitoring metrics are not restored after:

A backup and restore of Runtime Fabric
An upgrade of Runtime Fabric
OS patching of Runtime Fabric nodes

Logs (Titanium)

Each Mule application and API gateway replica can store up to 450 megabytes of log data on disk, after which log data is rotated to provide storage for more recent log data.

Logs generated by each replica of a Mule application and API gateway are collected and written to disk on the worker node on which they are running. Each worker node runs a single log forwarding agent, which monitors this location and transmits the log data to Anypoint Monitoring.

The log forwarding agent can transmit up to 300 kilobytes of log data per second. You should not deploy more than 50 replicas per worker node.
You can toggle application log forwarding at the application level to Anypoint Monitoring from Runtime Fabric v1.5.25 version and later.

Anypoint Monitoring Log Forwarding Prerequisites

The daemon-set that forwards logs to Anypoint Monitoring expects application containers to write logs to the following locations:

/var/log/pods
/var/log/containers
/var/lib/docker/containers

The daemon-set also creates and writes on the host path /var/lib/filebeat-data. For log forwarding to work as expected, ensure that:

The Docker daemon config includes the json-file logging driver.
The Anypoint Monitoring log forwarder daemon-set container has write access to the host on the host path /var/lib/filebeat-data.

Anypoint Visualizer

Anypoint Visualizer displays views of different aspects of an application network graph. You can use the graph to explore your application network, identify problems, and make decisions.

By default, Anypoint Visualizer is disabled when deploying an application to Mule runtime 4.x. See Enable Header Injection for more information.

Due to the impact on performance, enabling Anypoint Visualizer for application deployments with less than 0.2 vCPU is not supported.