All Priorities
View and Configure Logging in Runtime Fabric
Runtime Fabric generates log files for the following:
-
Deployed Mule applications
-
Deployed API proxies
-
Runtime Fabric services
-
Kubernetes services
Log Levels
Runtime Fabric enables you to specify the severity level of messages that are written to the log file.
Runtime Fabric does not support custom logging categories.
Value | Description | Command |
---|---|---|
List all messages. |
N/A |
|
ERROR |
List only error messages, such when an exception occurs. |
priority:ERROR |
FATAL |
List only fatal messages when an application fails. |
priority:FATAL |
INFO |
List informative messages. |
priority:INFO |
SYSTEM |
List messages about application and worker startup. |
priority:SYSTEM |
CONSOLE |
List messages about console events such as setting the object store. |
priority:CONSOLE |
WARN |
List warning messages. |
priority:WARN |
DEBUG |
List debugging messages. |
priority:DEBUG |
Log levels are specified per Mule application or API proxy during deployment.
Change the Application Log Level
To change the log level for an application, use Anypoint Runtime Manager to configure the logging.level.<PACKAGE>=LEVEL
property.
-
In Runtime Manager, navigate to an application’s Settings page.
-
Click the Properties tab.
-
Select Table View.
-
In the New Key field, add the
logging.level.<PACKAGE>
property. -
In the New Value field, add the desired log level.
For example, if you add the key
logging.level.org.mule.service.http.impl.service.HttpMessageLogger
and a value ofDEBUG
,DEBUG
level log information is emitted fromHttpMessageLogger
.
View Logs from an Application
Ops Center shows a stream of logs generated by applications and services running on Runtime Fabric on VMs / Bare Metal. This is useful when log forwarding is not configured.
-
From Ops Center, navigate to Logging.
-
Select Kubernetes.
-
Select the Pods tab.
-
In the drop-down list adjacent to the search input area, select the ID of the environment where the application is deployed.
-
Locate the pod name that starts with the name of your application.
-
Select that pod name and then select Logs.
The Logs tab is displayed with a filter applied to your application.
To view the latest logs, select Refresh. |
Forward Logs to External Services
Runtime Fabric on VMs / Bare Metal supports log forwarding to:
-
Anypoint Monitoring
-
Third-party logging solutions using log forwarding output plugins:
-
Azure Log Analytics
-
Elasticsearch
-
Graylog Extended Log Format (GELF)
-
Splunk
-
CloudWatch
-
Syslog
-
Datadog
-
-
External services, using the
rsyslog
client service
Limitations for External Log Forwarding
-
Only one external log forwarding configuration is supported at any one time. However, you can enable Anypoint Platform log forwarding and an external log forwarding service at the same time.
-
External log forwarding does not currently support proxies that are configured for a Runtime Fabric cluster. You must explicitly configure firewall rules to allow the outbound IP address and port of the external log forwarding service.
-
External log forwarding does not currently join multiline log entries.
-
Runtime Fabric supports only one-way TLS (client to server) verification.
-
You cannot programmatically change the log-patterns for external log forwarders. Additionally, Runtime Fabric does not support custom log appenders in
log4j2.xml
. -
External log forwarding follows Syslog Protocol RFC5424, and therefore hostnames, application names, and process IDs are not automatically mapped to log entries. These fields are mapped when you forward logs to other external services.
Enable Log Forwarding
-
Install and configure the applicable third-party logging solution. Verify that you have the required resources allocated. Refer to the documentation for your third-party logging solution for details.
-
From Anypoint Platform, select Runtime Manager.
-
Select Runtime Fabrics in the sidebar navigation.
-
Select the applicable Runtime Fabric based on the name used during installation.
-
Select the Logs tab.
-
Select the Forward application logs to Anypoint Monitoring (Requires Titanium) checkbox if you have a Titanium subscription and you want to forward logs to Anypoint Monitoring.
-
Select the Forward application logs to an external service checkbox to forward logs using a log forwarding output plugin.
-
In the Connects to drop-down list, select the applicable third-party log forwarding solution.
-
Enter configuration information for your third-party logging solution, as shown in the following sections:
-
Select Apply changes.
-
To verify successful log forwarding, manually check the logs using the external log forwarding service.
-
Azure Log Analytics Configuration Parameters
Key | Description | Required | Default Value |
---|---|---|---|
Customer ID |
Specifies the customer ID or workspace ID string. |
Yes |
|
Shared Key |
Specifies the primary or secondary connected sources client authentication key. |
Yes |
|
Log Type |
Specifies the event type name. |
No |
|
Example output plugin configuration:
[OUTPUT]
Name azure
Customer_ID id
Shared_Key key
Elasticsearch Configuration Parameters
Runtime Fabric 1.10.26 and later uses Fluent Bit 1.8.3 for external log forwarding. If you forward logs to Elasticsearch, you must set the configuration parameter |
Key | Description | Required | Default Value |
---|---|---|---|
Host |
Specifies the IP address or hostname of the target Elasticsearch instance. |
Yes |
127.0.0.1 |
Port |
Specifies the TCP port of the target Elasticsearch instance. |
Yes |
9200 |
Index |
Specifies the index name. |
Yes |
runtime_fabric |
Path |
Elasticsearch accepts new data on HTTP query path |
No |
Empty string |
Buffer Size |
Specifies the buffer size used to read the response from the Elasticsearch HTTP service. This option is useful for debugging purposes where reading the full response is needed. Note that the size of the response grows depending on the number of records inserted. To specify an unlimited amount of memory, set this value to |
No |
4KB |
HTTP User |
Specifies an optional username credential for Elastic X-Pack access. |
No |
|
HTTP Passwd |
Specifies a password for the user defined in |
No |
|
Type |
Specifies the type name. |
No |
runtime_fabric |
Logstash Format |
Enables Logstash format compatibility. This option takes a Boolean value: |
No |
|
Logstash Prefix |
When |
No |
logstash |
Logstash DateFormat |
Specifies the time format (based on |
No |
%Y.%m.%d |
Time Key |
When |
No |
@timestamp |
Time Key Format |
When |
No |
%Y-%m-%dT%H:%M:%S |
Include Tag Key |
When enabled, the tag name is appended to the record. |
No |
Off |
Tag Key |
When |
No |
|
Generate ID |
When enabled, generates the |
No |
Off |
Replace Dots |
When enabled, replaces field name dots ('.') with an underscore ('_'), which is required by Elasticsearch 2.0 through 2.3. |
No |
Off |
Trace Output |
When enabled, prints the Elasticsearch API calls to stdout (for diagnosis). |
No |
Off |
Trace Error |
When enabled, prints the Elasticsearch API calls to stdout when Elasticsearch returns an error. |
No |
Off |
Current Time Index |
Specifies using the current time for index generation instead of message record information. |
No |
Off |
Logstash Prefix Key |
Prefixes keys with this string. |
No |
|
TLS |
Enables or disables TLS support. |
No |
Off |
CA Path Certificate File |
Specifies the CA certificate file to be uploaded. This option is available only when TLS is enabled. |
No |
Example output plugin configuration:
[OUTPUT]
Name es
Host 192.168.2.3
Port 9200
Index my_index
Type my_type
GELF Configuration Parameters
Key | Description | Required | Default Value |
---|---|---|---|
Match |
Specifies the pattern to match for log tags to be output by this plugin. |
Yes |
|
Host |
Specifies the IP address or hostname of the target Graylog server. |
Yes |
127.0.0.1 |
Port |
Specifies the port on which your Graylog GELF input is listening. |
Yes |
12201 |
Mode |
Specifies the protocol to use ( |
Yes |
udp |
Gelf Short_Message Key |
Specifies a short descriptive message (This must be set in GELF). |
Yes |
log |
Gelf Timestamp Key |
Specifies your log timestamp (This should be set in GELF). |
No |
timestamp |
Gelf Host Key |
Specifies the key for the value that is used as the name of the host, source, or application that sent the message. (This must be set in GELF). |
No |
host |
Gelf Full Message Key |
Specifies the key to use as the long message, which can contain a backtrace. (This is optional in GELF.) |
No |
full_message |
Gelf Level Key |
Specifies the key to be used as the log level. Its value must be a standard syslog level (between 0 and 7). |
No |
level |
Packet Size |
Specifies the size of packets to be sent if the transport protocol is set to |
No |
1420 |
Compress |
Specifies compression of your UDP packets if the transport protocol is set to |
No |
true |
TLS |
Enables or disables TLS support. |
No |
Off |
CA Path Certificate File |
Specifies the CA certificate file to be uploaded. This option is available only when TLS is enabled. |
No |
Example output plugin configuration:
[OUTPUT]
Name gelf
Match kube.*
Host <your-graylog-server>
Port 12201
Mode tcp
Gelf_Short_Message_Key log
Splunk Configuration Parameters
Key | Description | Required | Default Value |
---|---|---|---|
Host |
Specifies the IP address or hostname of the target Splunk service. |
Yes |
127.0.0.1 |
Port |
Specifies the TCP port of the target Splunk service. |
Yes |
8088 |
Splunk Token |
Specifies the Authentication Token for the HTTP Event Collector interface. |
Yes |
|
HTTP User |
Specifies an optional username for basic authentication on the HTTP Event Collector. |
No |
|
HTTP Passwd |
Specifies a password for the user defined in |
No |
|
TLS |
Enables or disables TLS support. |
No |
Off |
CA Path Certificate File |
Specifies the CA certificate file to be uploaded. This option is available only when TLS is enabled. |
No |
Example output plugin configuration:
[OUTPUT]
Name splunk
Host 127.0.0.1
Port 8088
Splunk_Token xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx
Tls On
CloudWatch Configuration Parameters
Key | Description | Required | Default Value |
---|---|---|---|
Region |
Specifies the AWS region |
Yes |
|
Aws Access Key Id |
Specifies the access key ID for the IAM user |
Yes |
|
Aws Secret Access Key |
Specifies the AWS secret access key for the IAM user |
Yes |
|
Log Group Name |
Specifies the name of the CloudWatch log group to which you want log records sent |
Yes |
runtime_fabric_cloudwatch |
Log Stream Name |
Specifies the name of the CloudWatch log stream to which you want log records sent |
Yes |
|
Log Stream Prefix |
Specifies the prefix for the log stream name. The tag is appended to the prefix to construct the full log stream name. Not compatible with the |
No |
|
Role Arn |
Specifies the ARN of an IAM role to assume (for cross account access). |
No |
|
Auto Create Group |
Automatically creates the log group. Valid values are "On" or "Off" (case sensitive). |
No |
Off |
Note: You must specify either Log Stream Name
or Log Stream Prefix
, but not both.
Example output plugin configuration:
[OUTPUT]
Name cloudwatch_logs
region us-east-1
aws_access_key_id <your_AWS_access_key_ID>
aws_secret_access_key <your_AWS_secret_access_key>
log_group_name runtime_fabric_cloudwatch
log_stream_prefix my_stream
auto_create_group On
Syslog Configuration Parameters
Key | Description | Required | Default Value |
---|---|---|---|
Host |
Host name or IP address of the remote Syslog server |
Yes |
|
Port |
TCP or UDP port of the remote Syslog server |
Yes |
514 |
Mode |
The mode of the transport type |
Yes |
udp |
Syslog format |
Syslog protocol format |
No |
rfc5424 |
Syslog Maxsize |
Integer value that sets the maximum number of bytes per message |
No |
1024 for rfc3164, 2048 for rfc5424 |
Example output plugin configuration:
[OUTPUT]
Name syslog
Host 127.0.0.1
Port 514
Mode udp
Syslog_Format rfc5424
Syslog_Maxsize 2048
Datadog Configuration Parameters
Key | Description | Required | Default Value |
---|---|---|---|
Host |
The host name of the remote Datadog server. |
Yes |
|
Api Key |
The Datadog API key. |
Yes |
|
Compress |
Compresses the payload in GZIP format. Datadog recommends setting this to |
No |
|
Datadog Service |
The name of the service generating the logs. For example, |
No |
|
Datadog Source |
The type of service. For example, |
No |
|
Datadog Tags |
The tags assigned to the logs in Datadog. |
No |
|
TLS |
End-to-end secure communications protocol support. DataDog does not support CA certificate files. |
No |
Off |
Example output plugin configuration:
[OUTPUT]
Name datadog
Host http-intake.logs.datadoghq.com
apikey <your-datadog-api-key>
TLS on
compress gzip
dd_service <your-app-service>
dd_source <your-app-source>
dd_tags team:logs,foo:bar
Configure Log Forwarding
The following configuration options are available:
-
Mule Applications: Forward logs from Mule applications.
-
Internal load balancer and components: Forward logs from Runtime Fabric components and the internal load balancer.
-
Runtime Fabric appliance: Forward logs from monitoring and appliance services.
|
Kubernetes Metadata
When forwarding logs to external services, additional metadata might be available in some providers. When performing searches, the user can leverage these metadata labels to help narrow down search results and focus on the logs only from containers that matters.
Key | Description | Sample Value |
---|---|---|
container_name |
Specifies the name of the container. |
app |
container_image |
Specifies the full name of the container image. |
000.dkr.ecr.us-east-1.amazonaws.com/mulesoft/poseidon-runtime-4.3.0:v1.0.0 |
container_hash |
Specifies the name of the container image and includes the immutable identifier (SHA256 value). |
000.dkr.ecr.us-east-1.amazonaws.com/mulesoft/poseidon-runtime-4.3.0@sha256:866daf14a55da1db21c562ddc09487bbb8ffae1dd4cc80b4dc5c64e0ac9c124b |
docker_id |
Specifies the ID of the container in Docker. |
a54a0bf80400b6cd44cae53b9705f5b6330ce4e8efb85c92b3f9c1709829c6fa |
host |
Specifies the IP of the node on which the pod is running. |
10.0.178.87 |
namespace_name |
Specifies the Kubernetes namespace of the pod. |
rtf |
pod_id |
Specifies the ID of the pod. |
118cff63-a5ac-11ea-8414-0e281fcb2861 |
pod_name |
Specifies the name of the pod. |
testapp-5789f9774-6lxv7 |
labels_app |
Specifies the app label for the application the pod is supporting. |
testapp |
labels_environment |
Specifies the Anypoint environment ID of the environment in which the application is deployed. |
24166311-4e5e-4091-b417-42307747267e |
labels_organization |
Specifies the Anypoint organization ID of the organization in which the application is deployed. |
24166311-4e5e-4091-b417-42307747267e |
labels_rtf_mulesoft_com_id |
Specifies the ID of the application deployment on Anypoint Platform. |
194a32bf-bf7e-4c91-aa4e-44132e5aa76d |
labels_rtf_mulesoft_com_version |
Specifies the spec version ID of the application deployment on Anypoint Platform. |
194a32bf-bf7e-4c91-aa4e-44132e5aa76d |
labels_type |
Specifies the type of deployment that the pod is running. |
MuleApplication |
Send a Test Message
You can send a test message to verify connectivity between Runtime Fabric on VMs / Bare Metal and the external log forwarding solution:
-
In Runtime Manager, select Runtime Fabrics in the sidebar navigation.
-
Select the applicable Runtime Fabric based on the name used during installation.
-
Select the Log Forwarding tab.
-
Select Forward application logs to an external service.
-
Select a service and configure the fields as needed for your log forwarding solution.
-
Save the changes and apply the configuration to the Runtime Fabric cluster.
-
To test the configuration and reachability of the log forwarding solution:
-
Trigger a test message for the selected service.
-
Verify the text that is forwarded to log forwarding solution.
-
Disable Anypoint Monitoring or External Log Forwarding per Application
-
Navigate to Runtime Manager and select Applications.
-
Select the application for which you want to disable log forwarding.
-
Select the Logs tab.
-
Deselect the applicable log forwarding options:
-
Forward logs to Anypoint Monitoring
-
Forward logs to <third-party service>
-
-
Select Deploy to redeploy the application.
Forward Logs to Other External Services
Anypoint Runtime Fabric on VMs / Bare Metal also enables you to forward application and cluster logs to external logging solutions other than the third-party solutions supported using log forwarding output plugins. The log forwarder (rsyslog
client service) built into Runtime Fabric enables you to send log data to an rsyslog
server over TCP or UDP for viewing, retention, and receiving alerts in a centralized destination. Logging solutions such as Logstash provide methods to receive log data from rsyslog
clients.
To enable multiline logging, your log entries must begin with an open bracket, |
-
Using a terminal, open a shell/SSH connection to a controller VM.
-
Create a file named
log-forwarder.yaml
. -
Add the following content to this file after customizing based on the table below:
kind: logforwarder version: v2 metadata: name: log-forwarder spec: address: 192.168.100.1:514 protocol: udp
Use the following values as applicable to your environment:
Key Description name
Specifies the name of the log forwarding rule.
address
Specifies the endpoint and port to forward the log data.
protocol
Specifies the protocol to send the data to. Supported protocols are TCP or UDP.
-
Run the following command on the controller VM, referencing the file created earlier.
gravity resource create log-forwarder.yaml
Your logs are forwarded to your external logging solution.
Troubleshoot Log Forwarding Issues
If you experience issues with log forwarding:
-
Verify that the log forwarding configuration specifies the correct host and port information.
-
Verify that the server side configuration, such as networking, is correct and that the server is accessible.
-
Verify log forwarding status:
-
Navigate to Runtime Manager and select Runtime Fabrics.
-
Select the name of your Runtime Fabric to open the management page.
-
Select the Health Details tab.
-
Verify the following information as applicable:
-
The status of Forwarding Logs To External Provider is
Healthy
. -
The status Forwarding Logs to Anypoint Monitoring is
Healthy
.
-
-
-
Run the following commands inside the RTF cluster:
-
kubectl -nrtf get pods -lapp=external-log-forwarder
for a list of all external log forwarding pods. Verify that these pods are inRunning
state. -
kubectl -nrtf logs -l app=external-log-forwarder
to view the logs of external log forwarding pods.
-