Contact Us 1-800-596-4880
  1. Homepage
  2. Runtime Fabric (1.13)
  3. On VMs / Bare Metal

  4. Viewing and Configuring Logs

Viewing and Configuring Logs in Runtime Fabric

Runtime Fabric generates log files for the following:

  • Deployed Mule applications

  • Deployed API proxies

  • Runtime Fabric services

  • Kubernetes services

Log Levels

Runtime Fabric enables you to specify the severity level of messages that are written to the log file.

Runtime Fabric does not support custom logging categories.

Table 1. Runtime Fabric Log Levels
Value Description Command

All Priorities

List all messages.

N/A

ERROR

List only error messages, such when an exception occurs.

priority:ERROR

FATAL

List only fatal messages when an application fails.

priority:FATAL

INFO

List informative messages.

priority:INFO

SYSTEM

List messages about application and worker startup.

priority:SYSTEM

CONSOLE

List messages about console events such as setting the object store.

priority:CONSOLE

WARN

List warning messages.

priority:WARN

DEBUG

List debugging messages.

priority:DEBUG

Log levels are specified per Mule application or API proxy during deployment.

Change the Application Log Level

To change the log level for an application, use Anypoint Runtime Manager to configure the logging.level.<PACKAGE>=LEVEL property.

  1. In Runtime Manager, navigate to an application’s Settings page.

  2. Click the Properties tab.

  3. Select Table View.

  4. In the New Key field, add the logging.level.<PACKAGE> property.

  5. In the New Value field, add the desired log level.

    For example, if you add the key logging.level.org.mule.service.http.impl.service.HttpMessageLogger and a value of DEBUG, DEBUG level log information is emitted from HttpMessageLogger.

View Logs from an Application

Ops Center shows a stream of logs generated by applications and services running on Runtime Fabric on VMs / Bare Metal. This is useful when log forwarding is not configured.

  1. From Ops Center, navigate to Logging.

  2. Select Kubernetes.

  3. Select the Pods tab.

  4. In the drop-down list adjacent to the search input area, select the ID of the environment where the application is deployed.

  5. Locate the pod name that starts with the name of your application.

  6. Select that pod name and then select Logs.

The Logs tab is displayed with a filter applied to your application.

To view the latest logs, select Refresh.

Filters

There are two levels of filters for drilling down in the logs:

  • The Containers filter filters by container name.

  • The Pods filter filters by pod name.

    This is useful for specifying application names that are followed by a wildcard (%).

Forward Logs to External Services

Runtime Fabric on VMs / Bare Metal supports log forwarding to:

  • Anypoint Monitoring

  • Third-party logging solutions using log forwarding output plugins:

    • Azure Log Analytics

    • Elasticsearch

    • Graylog Extended Log Format (GELF)

    • Splunk

    • CloudWatch

    • Syslog

    • Datadog

  • External services, using the rsyslog client service

Limitations for External Log Forwarding

  • Using a Log4j appender and external log forwarding at the same time might lead to duplicate log collection. Use one or the other.

  • Only one external log forwarding configuration is supported at any one time. However, you can enable Anypoint Platform log forwarding and an external log forwarding service at the same time.

  • External log forwarding does not currently support proxies that are configured for a Runtime Fabric cluster. You must explicitly configure firewall rules to allow the outbound IP address and port of the external log forwarding service.

  • External log forwarding does not currently join multiline log entries.

  • Runtime Fabric supports only one-way TLS (client to server) verification.

  • External log forwarding follows Syslog Protocol RFC5424, and therefore hostnames, application names, and process IDs are not automatically mapped to log entries. These fields are mapped when you forward logs to other external services.

Enable Log Forwarding

  1. Install and configure the applicable third-party logging solution. Verify that you have the required resources allocated. Refer to the documentation for your third-party logging solution for details.

  2. From Anypoint Platform, select Runtime Manager.

  3. Select Runtime Fabrics in the sidebar navigation.

  4. Select the applicable Runtime Fabric based on the name used during installation.

  5. Select the Logs tab.

  6. Select the Forward application logs to Anypoint Monitoring (Requires Titanium) checkbox if you have a Titanium subscription and you want to forward logs to Anypoint Monitoring.

  7. Select the Forward application logs to an external service checkbox to forward logs using a log forwarding output plugin.

    1. In the Connects to drop-down list, select the applicable third-party log forwarding solution.

    2. Enter configuration information for your third-party logging solution, as shown in the following sections:

  8. Select the log level:

    • INFO: Informative messages

    • DEBUG: Debugging messages

    • WARNING: Warning messages

    • ERROR: Error messages, such as when an exception occurs

    • FATAL: Fatal messages for when an application fails

  9. Optionally, set the log level for specific a Java package class by selecting the log level, entering the package.name, and pressing the Enter key.

  10. Select Apply changes.

  11. To verify successful log forwarding, manually check the logs using the external log forwarding service.

Azure Log Analytics Configuration Parameters

Table 2. Azure Log Analytics Configuration Parameters
Key Description Required Default Value

Customer ID

Specifies the customer ID or workspace ID string.

Yes

Shared Key

Specifies the primary or secondary connected sources client authentication key.

Yes

Log Type

Specifies the event type name.

No

runtime_fabric

Example output plugin configuration:

[OUTPUT]
    Name        azure
    Customer_ID id
    Shared_Key  key

Elasticsearch Configuration Parameters

Runtime Fabric 1.10.26 and later uses Fluent Bit 1.8.3 for external log forwarding. If you forward logs to Elasticsearch, you must set the configuration parameter Generate_ID to On. By default, Fluent Bit sets this parameter to Off.
Table 3. Elasticsearch Configuration Parameters
Key Description Required Default Value

Host

Specifies the IP address or hostname of the target Elasticsearch instance.

Yes

127.0.0.1

Port

Specifies the TCP port of the target Elasticsearch instance.

Yes

9200

Index

Specifies the index name.

Yes

runtime_fabric

Path

Elasticsearch accepts new data on HTTP query path /_bulk. But it is also possible to serve Elasticsearch behind a reverse proxy on a subpath. This option defines such a path on the output plugin side. It adds a path prefix in the indexing HTTP POST URI.

No

Empty string

Buffer Size

Specifies the buffer size used to read the response from the Elasticsearch HTTP service. This option is useful for debugging purposes where reading the full response is needed. Note that the size of the response grows depending on the number of records inserted. To specify an unlimited amount of memory, set this value to False. Otherwise, set the value according to the Unit Size specification.

No

4KB

HTTP User

Specifies an optional username credential for Elastic X-Pack access.

No

HTTP Passwd

Specifies a password for the user defined in HTTP User.

No

Type

Specifies the type name.

No

runtime_fabric

Logstash Format

Enables Logstash format compatibility. This option takes a Boolean value: True, False, On, or Off.

No

Logstash Prefix

When Logstash Format is enabled, the Index name is composed of a prefix and the date. For example, if Logstash Prefix is set to mydata, your index becomes mydata-YYYY.MM.DD. The last string that is appended belongs to the date when the data is generated.

No

logstash

Logstash DateFormat

Specifies the time format (based on strftime) that is used to generate the second part of the Index name.

No

%Y.%m.%d

Time Key

When Logstash Format is enabled, each record is assigned a new timestamp field. The Time Key property defines the name of that field.

No

@timestamp

Time Key Format

When Logstash Format is enabled, this property defines the format of the timestamp.

No

%Y-%m-%dT%H:%M:%S

Include Tag Key

When enabled, the tag name is appended to the record.

No

Off

Tag Key

When Include Tag Key is enabled, this property defines the key name for the tag.

No

Generate ID

When enabled, generates the _id value for outgoing records. This prevents duplicate records when retrying ES.

No

Off

Replace Dots

When enabled, replaces field name dots ('.') with an underscore ('_'), which is required by Elasticsearch 2.0 through 2.3.

No

Off

Trace Output

When enabled, prints the Elasticsearch API calls to stdout (for diagnosis).

No

Off

Trace Error

When enabled, prints the Elasticsearch API calls to stdout when Elasticsearch returns an error.

No

Off

Current Time Index

Specifies using the current time for index generation instead of message record information.

No

Off

Logstash Prefix Key

Prefixes keys with this string.

No

TLS

Enables or disables TLS support.

No

Off

CA Path Certificate File

Specifies the CA certificate file to be uploaded. This option is available only when TLS is enabled.

No

Example output plugin configuration:

[OUTPUT]
    Name  es
    Host  192.168.2.3
    Port  9200
    Index my_index
    Type  my_type

GELF Configuration Parameters

Table 4. GELF Configuration Parameters
Key Description Required Default Value

Match

Specifies the pattern to match for log tags to be output by this plugin.

Yes

Host

Specifies the IP address or hostname of the target Graylog server.

Yes

127.0.0.1

Port

Specifies the port on which your Graylog GELF input is listening.

Yes

12201

Mode

Specifies the protocol to use (tls, tcp, or udp).

Yes

udp

Gelf Short_Message Key

Specifies a short descriptive message (This must be set in GELF).

Yes

log

Gelf Timestamp Key

Specifies your log timestamp (This should be set in GELF).

No

timestamp

Gelf Host Key

Specifies the key for the value that is used as the name of the host, source, or application that sent the message. (This must be set in GELF).

No

host

Gelf Full Message Key

Specifies the key to use as the long message, which can contain a backtrace. (This is optional in GELF.)

No

full_message

Gelf Level Key

Specifies the key to be used as the log level. Its value must be a standard syslog level (between 0 and 7).

No

level

Packet Size

Specifies the size of packets to be sent if the transport protocol is set to udp.

No

1420

Compress

Specifies compression of your UDP packets if the transport protocol is set to udp.

No

true

TLS

Enables or disables TLS support.

No

Off

CA Path Certificate File

Specifies the CA certificate file to be uploaded. This option is available only when TLS is enabled.

No

Example output plugin configuration:

[OUTPUT]
    Name                    gelf
    Match                   kube.*
    Host                    <your-graylog-server>
    Port                    12201
    Mode                    tcp
    Gelf_Short_Message_Key  log

Splunk Configuration Parameters

Table 5. Splunk Configuration Parameters
Key Description Required Default Value

Host

Specifies the IP address or hostname of the target Splunk service.

Yes

127.0.0.1

Port

Specifies the TCP port of the target Splunk service.

Yes

8088

Splunk Token

Specifies the Authentication Token for the HTTP Event Collector interface.

Yes

HTTP User

Specifies an optional username for basic authentication on the HTTP Event Collector.

No

HTTP Passwd

Specifies a password for the user defined in HTTP User.

No

TLS

Enables or disables TLS support.

No

Off

CA Path Certificate File

Specifies the CA certificate file to be uploaded. This option is available only when TLS is enabled.

No

Example output plugin configuration:

[OUTPUT]
    Name         splunk
    Host         127.0.0.1
    Port         8088
    Splunk_Token xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx
    Tls          On

CloudWatch Configuration Parameters

Table 6. CloudWatch Configuration Parameters
Key Description Required Default Value

Region

Specifies the AWS region

Yes

Aws Access Key Id

Specifies the access key ID for the IAM user

Yes

Aws Secret Access Key

Specifies the AWS secret access key for the IAM user

Yes

Log Group Name

Specifies the name of the CloudWatch log group to which you want log records sent

Yes

runtime_fabric_cloudwatch

Log Stream Name

Specifies the name of the CloudWatch log stream to which you want log records sent

Yes

Log Stream Prefix

Specifies the prefix for the log stream name. The tag is appended to the prefix to construct the full log stream name. Not compatible with the Log Stream Name option.

No

Role Arn

Specifies the ARN of an IAM role to assume (for cross account access).

No

Auto Create Group

Automatically creates the log group. Valid values are "On" or "Off" (case sensitive).

No

Off

Note: You must specify either Log Stream Name or Log Stream Prefix, but not both.

Example output plugin configuration:

[OUTPUT]
    Name                   cloudwatch_logs
    region                 us-east-1
    aws_access_key_id      <your_AWS_access_key_ID>
    aws_secret_access_key  <your_AWS_secret_access_key>
    log_group_name         runtime_fabric_cloudwatch
    log_stream_prefix      my_stream
    auto_create_group      On

Syslog Configuration Parameters

Table 7. Syslog Configuration Parameters
Key Description Required Default Value

Host

Host name or IP address of the remote Syslog server

Yes

Port

TCP or UDP port of the remote Syslog server

Yes

514

Mode

The mode of the transport type

Yes

udp

Syslog format

Syslog protocol format

No

rfc5424

Syslog Maxsize

Integer value that sets the maximum number of bytes per message

No

1024 for rfc3164, 2048 for rfc5424

Example output plugin configuration:

[OUTPUT]
    Name           syslog
    Host           127.0.0.1
    Port           514
    Mode           udp
    Syslog_Format  rfc5424
    Syslog_Maxsize 2048

Datadog Configuration Parameters

Table 8. Datadog Configuration Parameters
Key Description Required Default Value

Host

The host name of the remote Datadog server.

Yes

Api Key

The Datadog API key.

Yes

Compress

Compresses the payload in GZIP format. Datadog recommends setting this to gzip.

No

gzip

Datadog Service

The name of the service generating the logs. For example, runtime_fabric

No

Datadog Source

The type of service. For example, postgres or nginx

No

Datadog Tags

The tags assigned to the logs in Datadog.

No

TLS

End-to-end secure communications protocol support. DataDog does not support CA certificate files.

No

Off

Example output plugin configuration:

[OUTPUT]
    Name        datadog
    Host        http-intake.logs.datadoghq.com
    apikey      <your-datadog-api-key>
    TLS         on
    compress    gzip
    dd_service  <your-app-service>
    dd_source   <your-app-source>
    dd_tags     team:logs,foo:bar

Configure Log Forwarding

The following configuration options are available:

  • Mule Applications: Forward logs from Mule applications.

  • Internal load balancer and components: Forward logs from Runtime Fabric components and the internal load balancer.

  • Runtime Fabric appliance: Forward logs from monitoring and appliance services.

  • All options are enabled by default.

  • Disabling log forwarding for Mule applications is not supported. Log forwarding can be disabled per application during deployment.

Kubernetes Metadata

When forwarding logs to external services, additional metadata might be available in some providers. When performing searches, the user can leverage these metadata labels to help narrow down search results and focus on the logs only from containers that matters.

Table 9. Kubernetes Metadata
Key Description Sample Value

container_name

Specifies the name of the container.

app

container_image

Specifies the full name of the container image.

000.dkr.ecr.us-east-1.amazonaws.com/mulesoft/poseidon-runtime-4.3.0:v1.0.0

container_hash

Specifies the name of the container image and includes the immutable identifier (SHA256 value).

000.dkr.ecr.us-east-1.amazonaws.com/mulesoft/poseidon-runtime-4.3.0@sha256:866daf14a55da1db21c562ddc09487bbb8ffae1dd4cc80b4dc5c64e0ac9c124b

docker_id

Specifies the ID of the container in Docker.

a54a0bf80400b6cd44cae53b9705f5b6330ce4e8efb85c92b3f9c1709829c6fa

host

Specifies the IP of the node on which the pod is running.

10.0.178.87

namespace_name

Specifies the Kubernetes namespace of the pod.

rtf

pod_id

Specifies the ID of the pod.

118cff63-a5ac-11ea-8414-0e281fcb2861

pod_name

Specifies the name of the pod.

testapp-5789f9774-6lxv7

labels_app

Specifies the app label for the application the pod is supporting.

testapp

labels_environment

Specifies the Anypoint environment ID of the environment in which the application is deployed.

24166311-4e5e-4091-b417-42307747267e

labels_organization

Specifies the Anypoint organization ID of the organization in which the application is deployed.

24166311-4e5e-4091-b417-42307747267e

labels_rtf_mulesoft_com_id

Specifies the ID of the application deployment on Anypoint Platform.

194a32bf-bf7e-4c91-aa4e-44132e5aa76d

labels_rtf_mulesoft_com_version

Specifies the spec version ID of the application deployment on Anypoint Platform.

194a32bf-bf7e-4c91-aa4e-44132e5aa76d

labels_type

Specifies the type of deployment that the pod is running.

MuleApplication

Send a Test Message

You can send a test message to verify connectivity between Runtime Fabric on VMs / Bare Metal and the external log forwarding solution:

  1. In Runtime Manager, select Runtime Fabrics in the sidebar navigation.

  2. Select the applicable Runtime Fabric based on the name used during installation.

  3. Select the Log Forwarding tab.

  4. Select Forward application logs to an external service.

  5. Select a service and configure the fields as needed for your log forwarding solution.

  6. Save the changes and apply the configuration to the Runtime Fabric cluster.

  7. To test the configuration and reachability of the log forwarding solution:

    1. Trigger a test message for the selected service.

    2. Verify the text that is forwarded to log forwarding solution.

Disable Anypoint Monitoring or External Log Forwarding per Application

  1. Navigate to Runtime Manager and select Applications.

  2. Select the application for which you want to disable log forwarding.

  3. Select the Logs tab.

  4. Deselect the applicable log forwarding options:

    • Forward logs to Anypoint Monitoring

    • Forward logs to <third-party service>

  5. Select Deploy to redeploy the application.

Forward Logs to Other External Services

Anypoint Runtime Fabric on VMs / Bare Metal also enables you to forward application and cluster logs to external logging solutions other than the third-party solutions supported using log forwarding output plugins. The log forwarder (rsyslog client service) built into Runtime Fabric enables you to send log data to an rsyslog server over TCP or UDP for viewing, retention, and receiving alerts in a centralized destination. Logging solutions such as Logstash provide methods to receive log data from rsyslog clients.

To enable multiline logging, your log entries must begin with an open bracket, [.

  1. Using a terminal, open a shell/SSH connection to a controller VM.

  2. Create a file named log-forwarder.yaml.

  3. Add the following content to this file after customizing based on the table below:

    kind: logforwarder
version: v2
metadata:
   name: log-forwarder
spec:
   address: 192.168.100.1:514
   protocol: udp

    Use the following values as applicable to your environment:

    Key Description

    name

    Specifies the name of the log forwarding rule.

    address

    Specifies the endpoint and port to forward the log data.

    protocol

    Specifies the protocol to send the data to. Supported protocols are TCP or UDP.

  4. Run the following command on the controller VM, referencing the file created earlier.

gravity resource create log-forwarder.yaml

Your logs are forwarded to your external logging solution.

Troubleshoot Log Forwarding Issues

If you experience issues with log forwarding:

  • Verify that the log forwarding configuration specifies the correct host and port information.

  • Verify that the server side configuration, such as networking, is correct and that the server is accessible.

  • Verify log forwarding status:

    1. Navigate to Runtime Manager and select Runtime Fabrics.

    2. Select the name of your Runtime Fabric to open the management page.

    3. Select the Health Details tab.

    4. Verify the following information as applicable:

      • The status of Forwarding Logs To External Provider is Healthy.

      • The status Forwarding Logs to Anypoint Monitoring is Healthy.

  • Run the following commands inside the RTF cluster:

    • kubectl -nrtf get pods -lapp=external-log-forwarder for a list of all external log forwarding pods. Verify that these pods are in Running state.

    • kubectl -nrtf logs -l app=external-log-forwarder to view the logs of external log forwarding pods.

See Also