Contact Us 1-800-596-4880
  1. Homepage
  2. Flex Gateway (1.8)
  3. Monitoring in Connected Mode

  4. Configure Logs for Third-Party Services

Configuring Flex Gateway Log Output for Third-Party Services

Anypoint Flex Gateway enables you to deliver runtime and access logs to any supported Fluent Bit v3.0 output type. You configure log output via a custom YAML file. Additionally, to view access logs, you must also apply a Message Logging policy.

The following configuration file examples demonstrate how to direct Flex Gateway logs to output types: File, HTTP, and Splunk. However, you can use any Fluent Bit v3.0 output type, such as Azure Log Analytics, Amazon S3, Kafka, and more. For a complete list of Fluent Bit output types, see the Fluent Bit documentation for your Fluent Bit version.

As well as being able to configure additional Fluent Bit log output types, Flex Gateway sends runtime logs to runtime manager and access logs to API Manager. To learn more, see:

20%

25%

20%

Configure Log Output for Flex Gateway as a Linux Service

Configure Log Output for Flex Gateway in a Docker Container

Configure Log Output for Flex Gateway in a Kubernetes Cluster

Before You Begin

Before configuring Flex Gateway log output, complete the following tasks:

Additionally, depending on your output, ensure that the service where you direct logs to is operational and that you have the necessary information:

  • Splunk:

    • A running Splunk service. You can test Splunk with a free Splunk Cloud trial.

    • Added an HTTP Event Collector (HEC) data input to the Cloud or Enterprise Splunk service. See the Splunk documentation about how to add a data input, and how to create an HEC token.

  • Dynatrace:

    • A running Dynatrace service. You can test Dynatrace with a free Dynatrace trial.

    • Environment ID. To find your Dynatrace environment ID, see Environment ID

    • Access token. To generate a Dynatrace access token, see Access Tokens. Ensure that the token has the Ingest logs token scope.

Runtime Logs and Access Logs

Runtime and access logs both appear in the same log output. However, you can choose whether to include both logs in your log output. To use access logs, you must also apply a Message Logging policy.

Runtime Logs

Runtime logs include the following information:

  • The startup and shutdown of Flex Gateway, Fluent Bit, and Envoy

  • Deployed APIs and policies

  • Possible misconfigurations of Flex Gateway or other errors

Access Logs

Access logs describe incoming interaction with Envoy over a period of time and incoming requests and responses for specific APIs.

To enable access logs, you must apply a Message Logging policy to the API instances you want to monitor. To apply a Message Logging policy, see Configuring Access Logs.

Logging Configuration File

The logging object configures the delivery of runtime and access logs enabled via the message logging policy. Logs are delivered to any supported Fluent Bit v3.0 output.

apiVersion: gateway.mulesoft.com/v1alpha1
kind: Configuration
metadata:
  name: <value>
  namespace: <namespace name>
spec:
  logging:
    outputs:
    - name: <output-name>
      type: <output-type>
      parameters:
        <param-name>: <param-value>
    runtimeLogs:
      logLevel: <value>
      outputs: <value>
    accessLogs:
      outputs: <value>
Parameter Required or Optional Default Value Description

logging.outputs[#].name

Required

N/A

The name of this output to later refer to in runtime and access logs configurations.

logging.outputs[#].type

Required

N/A

An output type supported by Fluent Bit. For Fluent Bit output types, see Fluent Bit documentation for your Fluent Bit version.

logging.outputs[#].parameters

Required

N/A

A map of parameters for the specific Fluent Bit output type. For Fluent Bit output type parameters, see Fluent Bit documentation for your Fluent Bit version.

logging.accessLogs.outputs

Optional

Empty

A list of output names to redirect access logs to.

logging.runtimeLogs.logLevel

Optional

info

A parameter specifying log detail. The supported logLevel types listed in decreasing verbosity are debug, info, warn, error, and fatal.

logging.runtimeLogs.outputs

Optional

Empty

A list of output names to redirect runtime logs to.

Leaving a value blank applies the default value to your configuration.

In addition to parameters above, Flex Gateway offers variables for logging output. When configured, the variables render as one of their respective outputs in the logs:

Variable Description Outputs

date

Date and time of the logged event

A specific time, for example, 17/11/2022-09:48:27AM

logger

Flex Gateway service where the logged event occurred

flex-gateway-agent, flex-gateway-envoy, and flex-gateway-fluent

level

Loglevel of the logged event

debug, info, warn, error, or fatal

kind

Log type

runtimeLog or accessLog

To view configuration files with applied values, see the following configuration file examples:

Configure Log Output for Flex Gateway as a Linux Service

  1. Create a YAML configuration file in the Flex Gateway configuration directory:

    sudo touch /usr/local/share/mulesoft/flex-gateway/conf.d/logs-config.yaml

  2. Update the file with your desired output configuration details as shown in the following examples:

  3. If you are configuring access logs, see Configure Access Logs to apply a Message Logging policy.

Configure Log Output for Flex Gateway in a Docker Container

If you have already added a volume for a folder with your Flex Gateway configuration files, skip to the create and save a YAML file step.

  1. Press Ctrl+C to stop your Flex Gateway and any replicas.

  2. Create a folder named app in the directory with your Flex Gateway configuration files.

  3. Restart your Flex Gateway with an additional volume for the new app directory:

    docker run --rm \
-v "$(pwd)":/usr/local/share/mulesoft/flex-gateway/conf.d \
-p 8080:8080 \
mulesoft/flex-gateway
    Specify an optional name you want to assign to your Flex Replica by including the following: -e FLEX_NAME=<name-for-flex-replica> \.

  4. Create and save a YAML file with your desired output configuration details as shown in the following examples:

  5. If you are configuring access logs, see Configure Access Logs to apply a Message Logging policy.

Configure Log Output for Flex Gateway in a Kubernetes Cluster

To configure logging for Flex Gateway:

  1. Create a new resource using a YAML configuration file with your desired output configuration details as shown in the following examples:

  2. If you are configuring access logs, see Configure Access Logs to apply a Message Logging policy.

File Configuration Example

Configure Flex Gateway to send formatted runtime and access logs to a file called log.txt (Fluent Bit File output), in the /var/log directory. As an example, the following definition specifies a spec.logging.outputs.type value of file:

apiVersion: gateway.mulesoft.com/v1alpha1
kind: Configuration
metadata:
  name: logging-config
spec:
  logging:
    outputs:
    - name: default
      type: file
      parameters:
        file: /var/log/log.txt # Docker: replace `/var/log` with the absolute path to the mounted configuration directory
        format: template
        template: |
          [{date}][{logger}][{level}][{kind}] {message}
    runtimeLogs:
      logLevel: info
      outputs:
      - default
    accessLogs:
      outputs:
      - default
For Docker, in spec.logging.output.parameters.file, replace /var/log with the absolute path to the mounted configuration directory.

HTTP Configuration Example

You can configure Flex Gateway to send formatted runtime and access logs to a HTTP output. As an example, the following definition specifies a spec.logging.outputs.type value of HTTP:

apiVersion: gateway.mulesoft.com/v1alpha1
kind: Configuration
metadata:
  name: logging-config
spec:
  logging:
    outputs:
    - name: default
      type: HTTP
      parameters:
        host: collectors.au.sumologic.com
        port: "443"
        URI: /receiver/v1/http/[PrivateKey]
        format: json_lines
        json_date_key: timestamp
        json_date_format: iso8601
    runtimeLogs:
      logLevel: info
      outputs:
      - default
    accessLogs:
      outputs:
      - default

Splunk Configuration Example

Splunk begins ingesting logs soon after you apply this resource.

apiVersion: gateway.mulesoft.com/v1alpha1
kind: Configuration
metadata:
  name: splunk-config
spec:
  logging:
    outputs:
    - name: splunk
      type: splunk
      parameters:
        host: <instance-name>.splunkcloud.com
        port: "8088"
        splunk_token: <Splunk's HEC token>
        tls: "on"
        tls.verify: "off"
    runtimeLogs:
      logLevel: info
      outputs: [splunk]
    accessLogs:
      outputs: [splunk]

Dynatrace Configuration Example

You can configure Flex Gateway to send formatted runtime and access logs to a Dynatrace environment HTTP output. As an example, the following file contains the necessary configurations:

apiVersion: gateway.mulesoft.com/v1alpha1
kind: Configuration
metadata:
  name: dynatrace-config
spec:
  logging:
    outputs:
    - name: dynatrace
      type: http
      parameters:
        host: <environmentID>.live.dynatrace.com
        Port: "443"
        allow_duplicated_headers: "false"
        header:
          - Content-Type application/json; charset=utf-8
          - Authorization Api-Token <accessToken>
        URI: /api/v2/logs/ingest
        Format: json
        json_date_key: timestamp
        json_date_format: iso8601
        tls: "on"
        tls.verify: "off"
    runtimeLogs:
      logLevel: info
      outputs:
        - dynatrace
    accessLogs:
      outputs:
        - dynatrace

For more information about configuration options, see the Fluent Bit documentation for your Fluent Bit version.

Configure Access Logs

To configure access logs in your log output, you must apply a Message Logging policy to the APIs whose request and response data you want to monitor. If you have previously applied a Message Logging policy to an API instance, you do not need to configure another.

To configure a Message Logging policy, see Configuring Message Logging for Flex Gateway in Connected Mode.

Configuring a Message Logging policy makes access logs visible in both API Manager Logs and your third-party log services.

If your access logs are not visible, it is possible that a policy with a higher execution order is causing an error before the Message Logging policy can execute. To reorder policy execution, see Ordering Policies.

Logging Event Examples

The following log output examples demonstrate possible outputs that may occur for the following events:

Graceful Shutdowns

[flex-gateway-agent][info] SignalHandler: Signal terminated received
[flex-gateway-agent][info] FilesystemWatcher(/usr/local/share/mulesoft/flex-gateway/conf.d): Stopping with context done
[flex-gateway-agent][info] XdsService: Stopping with context done
[flex-gateway-agent][error] flex-gateway-envoy: Stopped with error context canceled
[flex-gateway-agent][info] FilesystemWatcher(/etc/mulesoft/flex-gateway/conf.d): Stopping with context done
[flex-gateway-agent][error] flex-gateway-fluent: Stopped with error context canceled
[flex-gateway-agent][error] flex-gateway-fluent: Stopped with error context canceled
[flex-gateway-agent][info] Server dropped connection
[flex-gateway-agent][info] Agent communication closed

Service Disconnections

[flex-gateway-agent][error] WebSocket receiver got error: read tcp 172.17.0.2:60780->100.64.1.37:443: i/o timeout
[flex-gateway-agent][info] Server dropped connection
[flex-gateway-agent][warn] Dropped connection due to error read tcp 172.17.0.2:60780->100.64.1.37:443: i/o timeout
[flex-gateway-agent][info] Reconnecting after connection was dropped
[flex-gateway-agent][error] Error connecting to Websocket Server: dial tcp: lookup arm-mcm2-service.kstg.msap.io on 192.168.65.7:53: dial udp 192.168.65.7:53: connect: network is unreachable

Envoy Crash

[flex-gateway-envoy][critical] Caught Segmentation fault, suspect faulting address 0xffff87e8fb89485b
[flex-gateway-envoy][critical] Backtrace (use tools/stack_decode.py to get line numbers):
[flex-gateway-envoy][critical] Envoy version: 36cbae1aeedd8c3b4eb88cd28268d21fe0905ce5/1.23.0/Clean/RELEASE/BoringSSL
[flex-gateway-envoy][critical] #0: __restore_rt [0x7ff79db2f140]
[flex-gateway-envoy][critical] #1: [0x557dcfad249b]
[flex-gateway-envoy][critical] #2: [0x557dcdf35237]
[flex-gateway-envoy][critical] #3: [0x557dce16d49f]
[flex-gateway-envoy][critical] #4: [0x557dce1a8ff5]
[flex-gateway-envoy][critical] #5: [0x557dce1a8e47]
[flex-gateway-envoy][critical] #6: ...
[flex-gateway-agent][error] flex-gateway-envoy: Stopped with error flex-gateway-envoy: command error: signal: segmentation fault
[flex-gateway-agent][error] flex-gateway-fluent: Stopped with error context canceled
[flex-gateway-agent][error] flex-gateway-fluent: Stopped with error context canceled
[flex-gateway-agent][info] FilesystemWatcher(/etc/mulesoft/flex-gateway/conf.d): Stopping with context done
[flex-gateway-agent][info] FilesystemWatcher(/usr/local/share/mulesoft/flex-gateway/conf.d): Stopping with context done
[flex-gateway-agent][info] XdsService: Stopping with context done
[flex-gateway-agent][info] Server dropped connection
[flex-gateway-agent][info] Agent communication closed

See Also