View and Configure Logging in Runtime Fabric

Runtime Fabric generates log files for the following entities:

Deployed Mule applications
Deployed API proxies
Runtime Fabric services
Kubernetes services

Log Levels

Runtime Fabric enables you to specify the severity level of messages that are written to the log file.

Table 1. Runtime Fabric Log Levels
Value	Description	Command
All Priorities	List all messages.	N/A
ERROR	List only error messages, such when an exception occurs.	priority:ERROR
FATAL	List only fatal messages when an application fails.	priority:FATAL
INFO	List informative messages.	priority:INFO
SYSTEM	List messages about application and worker startup.	priority:SYSTEM
CONSOLE	List messages about console events such as setting the object store.	priority:CONSOLE
WARN	List warning messages.	priority:WARN
DEBUG	List debugging messages.	priority:DEBUG

Log levels are specified per Mule application or API proxy during deployment.

You can use Anypoint Runtime Manager to change the application log level by configuring the following property:

logging.level.<PACKAGE>=LEVEL

For example, if you use Runtime Manager to add the logging.level.org.mule.service.http.impl.service.HttpMessageLogger=DEBUG property, DEBUG level log information is emitted from HttpMessageLogger.

View Logs from an Application

Ops Center shows a stream of logs generated by applications and services running on Runtime Fabric on VMs / Bare Metal. This is useful when log forwarding is not configured.

From Ops Center, navigate to Logging.
Select Kubernetes.
Select the Pods tab.
In the drop-down list adjacent to the search input area, select the ID of the environment where the application is deployed.
Locate the pod name that starts with the name of your application.
Select that pod name and then select Logs.

The Logs tab is displayed with a filter applied to your application.

To view the latest logs, select Refresh.

Filters

There are two levels of filters for drilling down in the logs:

The Containers filter filters by container name.
The Pods filter filters by pod name.

This is useful for specifying application names that are followed by a wildcard (%).

Forward Logs to External Services

Runtime Fabric on VMs / Bare Metal supports log forwarding to:

Anypoint Monitoring
Third-party logging solutions using log forwarding output plugins:
- Azure Log Analytics
- Elasticsearch
- Graylog Extended Log Format (GELF)
- Splunk
- CloudWatch

Only one external log forwarding configuration is supported at any one time. However, you can enable Anypoint log forwarding and an external log forwarding service at the same time.
External log forwarding does not currently support proxies that are configured for a Runtime Fabric cluster. You must explicitly configure firewall rules to allow the outbound IP address and port of the external log forwarding service.

Enable Log Forwarding

Install and configure the applicable third-party logging solution. Verify that you have the required resources allocated. Refer to the documentation for your third-party logging solution for details.
From Anypoint Platform, select Runtime Manager.
Select Runtime Fabrics in the sidebar navigation.
Select the applicable Runtime Fabric based on the name used during installation.
Select the Log Forwarding tab.
Select the Forward to Anypoint Monitoring (Requires Titanium) checkbox if you have a Titanium subscription and you want to forward logs to Anypoint Monitoring.
Select the Forward application logs to an external service checkbox to forward logs using a log forwarding output plugin.
1. In the Connects to drop-down list, select the applicable third-party log forwarding solution.
2. Enter configuration information for your third-party logging solution, as shown in the following sections:
3. Select Apply changes.
4. To verify successful log forwarding, manually check the logs using the external log forwarding service.

Azure Log Analytics Configuration Parameters

Table 2. Azure Log Analytics Configuration Parameters
Key	Description	Required	Default Value
Customer ID	Specifies the customer ID or workspace ID string.	Yes
Shared Key	Specifies the primary or secondary connected sources client authentication key.	Yes
Log Type	Specifies the event type name.	No	`runtime_fabric`

Example output plugin configuration:

[OUTPUT]
    Name        azure
    Customer_ID id
    Shared_Key  key

Elasticsearch Configuration Parameters

Table 3. Elasticsearch Configuration Parameters
Key	Description	Required	Default Value
Host	Specifies the IP address or hostname of the target Elasticsearch instance.	Yes	127.0.0.1
Port	Specifies the TCP port of the target Elasticsearch instance.	Yes	9200
Index	Specifies the index name.	Yes	runtime_fabric
Path	Elasticsearch accepts new data on HTTP query path `/_bulk`. But it is also possible to serve Elasticsearch behind a reverse proxy on a subpath. This option defines such a path on the output plugin side. It adds a path prefix in the indexing HTTP POST URI.	No	Empty string
Buffer Size	Specifies the buffer size used to read the response from the Elasticsearch HTTP service. This option is useful for debugging purposes where reading the full response is needed. Note that the size of the response grows depending on the number of records inserted. To specify an unlimited amount of memory, set this value to `False`. Otherwise, set the value according to the Unit Size specification.	No	4KB
HTTP User	Specifies an optional username credential for Elastic X-Pack access.	No
HTTP Passwd	Specifies a password for the user defined in `HTTP User`.	No
Type	Specifies the type name.	No	runtime_fabric
Logstash Format	Enables Logstash format compatibility. This option takes a Boolean value: `True`, `False`, `On`, or `Off`.	No
Logstash Prefix	When `Logstash Format` is enabled, the `Index` name is composed of a prefix and the date. For example, if `Logstash Prefix` is set to `mydata`, your index becomes `mydata-YYYY.MM.DD`. The last string that is appended belongs to the date when the data is generated.	No	logstash
Logstash DateFormat	Specifies the time format (based on `strftime`) that is used to generate the second part of the `Index` name.	No	%Y.%m.%d
Time Key	When `Logstash Format` is enabled, each record is assigned a new timestamp field. The `Time Key` property defines the name of that field.	No	@timestamp
Time Key Format	When `Logstash Format` is enabled, this property defines the format of the timestamp.	No	%Y-%m-%dT%H:%M:%S
Include Tag Key	When enabled, the tag name is appended to the record.	No	Off
Tag Key	When `Include Tag Key` is enabled, this property defines the key name for the tag.	No
Generate ID	When enabled, generates the `_id` value for outgoing records. This prevents duplicate records when retrying ES.	No	Off
Replace Dots	When enabled, replaces field name dots ('.') with an underscore ('_'), which is required by Elasticsearch 2.0 through 2.3.	No	Off
Trace Output	When enabled, prints the Elasticsearch API calls to stdout (for diagnosis).	No	Off
Trace Error	When enabled, prints the Elasticsearch API calls to stdout when Elasticsearch returns an error.	No	Off
Current Time Index	Specifies using the current time for index generation instead of message record information.	No	Off
Logstash Prefix Key	Prefixes keys with this string.	No
TLS	Enables or disables TLS support.	No	Off
CA Path Certificate File	Specifies the CA certificate file to be uploaded. This option is available only when TLS is enabled.	No

Example output plugin configuration:

[OUTPUT]
    Name  es
    Host  192.168.2.3
    Port  9200
    Index my_index
    Type  my_type

GELF Configuration Parameters

Table 4. GELF Configuration Parameters
Key	Description	Required	Default Value
Match	Specifies the pattern to match for log tags to be output by this plugin.	Yes
Host	Specifies the IP address or hostname of the target Graylog server.	Yes	127.0.0.1
Port	Specifies the port on which your Graylog GELF input is listening.	Yes	12201
Mode	Specifies the protocol to use (`tls`, `tcp`, or `udp`).	Yes	udp
Gelf Short_Message Key	Specifies a short descriptive message (This must be set in GELF).	Yes	log
Gelf Timestamp Key	Specifies your log timestamp (This should be set in GELF).	No	timestamp
Gelf Host Key	Specifies the key for the value that is used as the name of the host, source, or application that sent the message. (This must be set in GELF).	No	host
Gelf Full Message Key	Specifies the key to use as the long message, which can contain a backtrace. (This is optional in GELF.)	No	full_message
Gelf Level Key	Specifies the key to be used as the log level. Its value must be a standard syslog level (between 0 and 7).	No	level
Packet Size	Specifies the size of packets to be sent if the transport protocol is set to `udp`.	No	1420
Compress	Specifies compression of your UDP packets if the transport protocol is set to `udp`.	No	true
TLS	Enables or disables TLS support.	No	Off
CA Path Certificate File	Specifies the CA certificate file to be uploaded. This option is available only when TLS is enabled.	No

Example output plugin configuration:

[OUTPUT]
    Name                    gelf
    Match                   kube.*
    Host                    <your-graylog-server>
    Port                    12201
    Mode                    tcp
    Gelf_Short_Message_Key  log

Splunk Configuration Parameters

Table 5. Splunk Configuration Parameters
Key	Description	Required	Default Value
Host	Specifies the IP address or hostname of the target Splunk service.	Yes	127.0.0.1
Port	Specifies the TCP port of the target Splunk service.	Yes	8088
Splunk Token	Specifies the Authentication Token for the HTTP Event Collector interface.	Yes
HTTP User	Specifies an optional username for basic authentication on the HTTP Event Collector.	No
HTTP Passwd	Specifies a password for the user defined in `HTTP User`.	No
TLS	Enables or disables TLS support.	No	Off
CA Path Certificate File	Specifies the CA certificate file to be uploaded. This option is available only when TLS is enabled.	No

Example output plugin configuration:

[OUTPUT]
    Name         splunk
    Host         127.0.0.1
    Port         8088
    Splunk_Token xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx
    Tls          On

CloudWatch Configuration Parameters

Table 6. CloudWatch Configuration Parameters
Key	Description	Required	Default Value
Region	Specifies the AWS region	Yes
Aws Access Key Id	Specifies the access key ID for the IAM user	Yes
Aws Secret Access Key	Specifies the AWS secret access key for the IAM user	Yes
Log Group Name	Specifies the name of the CloudWatch log group to which you want log records sent	Yes	runtime_fabric_cloudwatch
Log Stream Name	Specifies the name of the CloudWatch log stream to which you want log records sent	Yes
Log Stream Prefix	Specifies the prefix for the log stream name. The tag is appended to the prefix to construct the full log stream name. Not compatible with the `Log Stream Name` option.	No
Role Arn	Specifies the ARN of an IAM role to assume (for cross account access).	No
Auto Create Group	Automatically creates the log group. Valid values are "On" or "Off" (case sensitive).	No	Off

Note: You must specify either Log Stream Name or Log Stream Prefix, but not both.

Example output plugin configuration:

[OUTPUT]
    Name                   cloudwatch_logs
    region                 us-east-1
    aws_access_key_id      <your_AWS_access_key_ID>
    aws_secret_access_key  <your_AWS_secret_access_key>
    log_group_name         runtime_fabric_cloudwatch
    log_stream_prefix      my_stream
    auto_create_group      On

Syslog Configuration Parameters

Table 7. Syslog Configuration Parameters
Key	Description	Required	Default Value
Host	Host name or IP address of the remote Syslog server	Yes
Port	TCP or UDP port of the remote Syslog server	Yes	514
Mode	The mode of the transport type	Yes	udp
Syslog format	Syslog protocol format	No	rfc5424
Syslog Maxsize	Integer value that sets the maximum number of bytes per message	No	1024 for rfc3164, 2048 for rfc5424

Example output plugin configuration:

[OUTPUT]
    Name           syslog
    Host           127.0.0.1
    Port           514
    Mode           udp
    Syslog_Format  rfc5424
    Syslog_Maxsize 2048

Configure Log Forwarding

The following configuration options are available:

Mule Applications: Forward logs from Mule applications.
Internal load balancer and components: Forward logs from Runtime Fabric components and the internal load balancer.
Runtime Fabric appliance: Forward logs from monitoring and appliance services.

All options are enabled by default.
Disabling log forwarding for Mule applications is not supported. Log forwarding can be disabled per application during deployment.

Kubernetes Metadata

When forwarding logs to external services, additional metadata might be available in some providers. When performing searches, the user can leverage these metadata labels to help narrow down search results and focus on the logs only from containers that matters.

Table 8. Kubernetes Metadata
Key	Description	Sample Value
container_name	Specifies the name of the container.	app
container_image	Specifies the full name of the container image.	000.dkr.ecr.us-east-1.amazonaws.com/mulesoft/poseidon-runtime-4.3.0:v1.0.0
container_hash	Specifies the name of the container image and includes the immutable identifier (SHA256 value).	000.dkr.ecr.us-east-1.amazonaws.com/mulesoft/poseidon-runtime-4.3.0@sha256:866daf14a55da1db21c562ddc09487bbb8ffae1dd4cc80b4dc5c64e0ac9c124b
docker_id	Specifies the ID of the container in Docker.	a54a0bf80400b6cd44cae53b9705f5b6330ce4e8efb85c92b3f9c1709829c6fa
host	Specifies the IP of the node on which the pod is running.	10.0.178.87
namespace_name	Specifies the Kubernetes namespace of the pod.	rtf
pod_id	Specifies the ID of the pod.	118cff63-a5ac-11ea-8414-0e281fcb2861
pod_name	Specifies the name of the pod.	testapp-5789f9774-6lxv7
labels_app	Specifies the app label for the application the pod is supporting.	testapp
labels_environment	Specifies the Anypoint environment ID of the environment in which the application is deployed.	24166311-4e5e-4091-b417-42307747267e
labels_organization	Specifies the Anypoint organization ID of the organization in which the application is deployed.	24166311-4e5e-4091-b417-42307747267e
labels_rtf_mulesoft_com_id	Specifies the ID of the application deployment on Anypoint Platform.	194a32bf-bf7e-4c91-aa4e-44132e5aa76d
labels_rtf_mulesoft_com_version	Specifies the spec version ID of the application deployment on Anypoint Platform.	194a32bf-bf7e-4c91-aa4e-44132e5aa76d
labels_type	Specifies the type of deployment that the pod is running.	MuleApplication

Send a Test Message

You can send a test message to verify connectivity between Runtime Fabric on VMs / Bare Metal and the external log forwarding solution:

In Runtime Manager, select Runtime Fabrics in the sidebar navigation.
Select the applicable Runtime Fabric based on the name used during installation.
Select the Log Forwarding tab.
Select Forward application logs to an external service.
Select a service and configure the fields as needed for your log forwarding solution.
Save the changes and apply the configuration to the Runtime Fabric cluster.
To test the configuration and reachability of the log forwarding solution:
1. Trigger a test message for the selected service.
2. Verify the text that is forwarded to log forwarding solution.

Disable Anypoint Monitoring or External Log Forwarding per Application

Navigate to Runtime Manager and select Applications.
Select the application for which you want to disable log forwarding.
Select the Logs tab.
Deselect the applicable log forwarding options:
- Forward logs to Anypoint Monitoring
- Forward logs to <third-party service>
Select Deploy to redeploy the application.

Forward Logs to Other External Services

Anypoint Runtime Fabric on VMs / Bare Metal also enables you to forward application and cluster logs to external logging solutions other than the third-party solutions supported using log forwarding output plugins. The log forwarder (rsyslog client service) built into Runtime Fabric enables you to send log data to an rsyslog server over TCP or UDP for viewing, retention, and receiving alerts in a centralized destination. Logging solutions such as Logstash provide methods to receive log data from rsyslog clients.

Anypoint Runtime Fabric on VMs / Bare Metal provides dashboards and alerts for critical metrics when performance or availability is compromised. An SMTP server is required to receive alerts. To view and configure Anypoint Runtime Fabric dashboards and alerts using Ops Center:

Using a terminal, open a shell/SSH connection to a controller VM.
Create a file named log-forwarder.yaml.

Add the following content to this file after customizing based on the table below:

kind: logforwarder
version: v2
metadata:
   name: log-forwarder
spec:
   address: 192.168.100.1:514
   protocol: udp

Use the following values as applicable to your environment:

Table 9. Log Forwarding Configuration Parameters
Key	Description
`name`	Specifies the name of the log forwarding rule.
`address`	Specifies the endpoint and port to forward the log data.
`protocol`	Specifies the protocol to send the data to. Supported protocols are TCP or UDP.

Run the following command on the controller VM, referencing the file created earlier.

gravity resource create log-forwarder.yaml

Your logs are forwarded to your external logging solution.

Troubleshoot Log Forwarding Issues

If you experience issues with log forwarding:

Verify that the log forwarding configuration specifies the correct host and port information.
Verify that the server side configuration, such as networking, is correct and that the server is accessible.
Verify log forwarding status:
1. Navigate to Runtime Manager and select Runtime Fabrics.
2. Select the name of your Runtime Fabric to open the management page.
3. Select the Health Details tab.
4. Verify the following information as applicable:
  - The status of Forwarding Logs To External Provider is Healthy.
  - The status Forwarding Logs to Anypoint Monitoring is Healthy.
Run the following commands inside the RTF cluster:
- kubectl -nrtf get pods -lapp=external-log-forwarder for a list of all external log forwarding pods. Verify that these pods are in Running state.
- kubectl -nrtf logs -l app=external-log-forwarder to view the logs of external log forwarding pods.

See Also