Contact Us 1-800-596-4880
  1. Homepage
  2. Runtime Fabric (2.8)
  3. Managing Mule Apps
  4. Monitoring Mule Apps Deployed to Runtime Fabric

  5. Using Anypoint Monitoring

Using Anypoint Monitoring

Mule apps and API gateways deployed to Runtime Fabric include native support for Anypoint Monitoring that allows viewing metrics from within Anypoint Platform.

Anypoint Monitoring

Anypoint Monitoring includes tools that provide feedback from Mule flows and components in your Mule app network. Anypoint Monitoring runs as a sidecar in application deployments and gathers metrics for Mule apps and API gateways deployed to Runtime Fabric. Titanium-plan customers have access to additional capabilities, including application log retrieval and search.

Enable Anypoint Monitoring

By default, Runtime Fabric is configured to send application metrics to Anypoint Monitoring. Data is securely transmitted using mutual TLS encryption. For metrics and logs to be sent to Anypoint Monitoring, outbound network connectivity must be available.

If you need an outbound proxy, configure Anypoint Runtime Fabric to use a SOCKS5 proxy by following the instructions in Configuring Proxies on Runtime Fabric.

To verify outbound network connectivity on Anypoint Runtime Fabric version 1.8.50 or later, ensure that each node in Runtime Fabric can send outbound TCP (Lumberjack) communication using port 443 to the following domains:

  • US control plane:
    dias-ingestor-router.us-east-1.prod.cloudhub.io
    us1.ingest.mulesoft.com

  • EU control plane:
    dias-ingestor-router.eu-central-1.prod-eu.msap.io
    eu1.ingest.mulesoft.com

If you are using a version of Anypoint Runtime Fabric older than 1.8.50, use port 5044 instead of port 443. Use the following domains:

  • US control plane: dias-ingestor-nginx.prod.cloudhub.io

  • EU control plane: dias-ingestor-nginx.prod-eu.msap.io

For further details on Anypoint Monitoring endpoints, refer to configuring your network to support Runtime Fabric.

Metrics

Each Mule app and API gateway replica runs a metrics agent that collects and transmits metrics data to Anypoint Monitoring via secure mutual TLS encryption. Each replica can cache up to 100 megabytes of metrics, after which the metrics data is rotated to provide storage for more recent data. If metrics can’t be sent to Anypoint Monitoring, this typically provides at least four days worth of storage per replica. Custom metrics might consume more storage.

The following Anypoint Runtime Fabric updates do not impact or restore monitoring metrics of applications deployed on Runtime Fabric:

  • A backup and restore of Runtime Fabric

  • An upgrade of Runtime Fabric

  • OS patching of Runtime Fabric nodes

Logs

Anypoint Monitoring provides access to log data for applications deployed to Rutime Fabric. To access logs with Anypoint Monitoring, you need a Titanium subscription.

Each Mule app and API gateway replica can store up to 450 megabytes of log data on disk, after which log data is rotated to provide storage for more recent log data.

Logs generated by each replica of a Mule app and API gateway are collected and written to disk on the worker node on which they are running. Each worker node runs a single log forwarding agent, which monitors this location and transmits the log data to Anypoint Monitoring.

The log forwarding agent can transmit up to 300 kilobytes of log data per second. You should not deploy more than 50 replicas per worker node.

In some cases, logs with multiple lines might not be fully transmitted to Anypoint Monitoring. This can occur if multiple lines of log output are split across two files due to log rotation inside Runtime Fabric.

You can enable log forwarding and set log levels when you deploy applications in Runtime Manager.

Resource and Permission Requirements for Anypoint Monitoring Agent

MuleSoft runs Anypoint Monitoring agents alongside Mule apps to send telemetry to the Anypoint control plane.

For Mule apps deployed or managed using Runtime Fabric agent version 2.5.x or lower, two different agents are used as follows:

  • One DaemonSet per underlying node to transport logs to Anypoint monitoring control plane.

  • One sidecar per application replica to transporting metrics to Anypoint monitoring control plane.

For optimal daemonset functionality, follow these prerequisites:

  1. Check that the application containers write logs to the following locations:

    • /var/log/pods

    • /var/log/containers

    • /var/lib/docker/containers

  2. Check that the DaemonSet creates and writes on the host path /var/lib/filebeat-data.

  3. Ensure that the Docker daemon config includes the json-file logging driver.

  4. Ensure that the Anypoint Monitoring log forwarder DaemonSet container has write access to the host on the host path /var/lib/filebeat-data.

For applications deployed or managed using Runtime Fabric versions 2.6.x and above, MuleSoft consolidates telemetry transport to a single sidecar per application replica.

Dependent on the features you use, the sidecar agent consumes corresponding resources for optimal performance:

  • To enable default Anypoint metrics, the sidecar agent (per application replica) requires 0.05vCPU or Core of CPU allocation and 90MiB memory allocation.

  • If you opt into forwarding application logs to Anypoint Monitoring, the sidecar agent (per application replica) requires up to 30% of CPU limit configured for application replica and up to 150MiB memory allocation

  • If you opt into trace data generation for Mule apps, the sidecar agent (per application replica) requires up to 30% of CPU limit configured for application replica and up to 240MB memory allocation.

Allocate appropriate capacity to the underlying cluster to avoid application deployment failure.

(Optional) Disabling Log Forwarder DeamonSet in Runtime Fabric Versions 2.6.x and Higher

After you upgrade Runtime Fabric agent to version 2.6.0 or higher, all deployments use only a single sidecar for metrics and log forwarding. If you had any Mule apps running under a previous agent, redeploy them to start using the new log forwarding mechanism. Any update you perform in the control plane triggers the redeployment.

After all your deployments use the new sidecar container, the am-log-forwarder DaemonSet just sits idle.

To remove the DeamonSet completely, disable Anypoint Monitoring log forwarding using the corresponding Anypoint Platform API call sending a payload like this: [source,json

{
    "anypointMonitoring": false
}

This disabling does not impact the log forwarding to Anypoint Monitoring via the sidecar.

Anypoint Visualizer

Anypoint Visualizer displays views of different aspects of an application network graph. You can use the graph to explore your application network, identify problems, and make decisions.

By default, Anypoint Visualizer is disabled when deploying an application to Mule runtime 4.x. See Enable Header Injection for more information. Due to the impact on performance, enabling Anypoint Visualizer for application deployments with less than 0.2 vCPU is not supported.

(Optional) Disable Anypoint Monitoring for a Mule App

When you deploy a Mule app to Runtime Fabric, Anypoint Monitoring is enabled by default. To disable Anypoint Monitoring, pass the following custom property when deploying your Mule app:

anypoint.platform.config.analytics.agent.enabled=false
For compliance or geospecific requirements if your team or organization cannot send metrics to Anypoint control plane, you can disable Anypoint metrics if you are not contractually obligated to send these metrics for Anypoint usage and billing purposes.

Enable Trace Data Collection

This feature is available only if your organization has the Anypoint Integration Advanced package or a Titanium subscription to Anypoint Platform. For more information, see the Pricing and Packaging documentation.

Traces enable you to understand the path of a transaction in the MuleSoft ecosystem by providing full visibility into the execution path of API calls or application transactions. When you enable trace data collection for an app, Anypoint Platform collects and exposes tracing information from Mule runtime engine in the OpenTelemetry standard, enabling you to export it using the Anypoint Monitoring Telemetry Exporter.

  • You can enable trace data collection for your Runtime Fabric applications running on Mule version 4.6 or later, and the minimum Runtime Fabric agent version required is 2.6.31 or above.

  • You can enable trace data collection using only the Runtime Manager UI and API, but you can’t enable trace data collection using the Mule Maven plugin, Anypoint CLI, or Anypoint Studio.

To enable trace data collection for an app:

  1. Click the Monitoring tab.

  2. Select the Enable tracing checkbox.

  3. Click Deploy Application if you are enabling the feature for a new deployment, or Apply Changes if you are editing an existing application.

Once you enable tracing, and redeploy your application, the Runtime Manager backend starts instrumenting traces for your application.