Configuring Flex Gateway Distributed Tracing in Connected Mode

OpenTelemetry provides a unified standard for instrumenting, generating, collecting, and exporting telemetry data across distributed systems. Flex Gateway supports OpenTelemetry, enabling you to configure the generation of distributed tracing data for all API traffic managed by the gateway. This integration delivers deep visibility into API request flows, making it easier to analyze system behavior, troubleshoot issues, and understand interactions between Flex Gateway APIs and external services.

Tracing enables you to track Flex Gateway API interactions. When multiple systems and services are involved, distributed tracing tracks API requests as they flow through distributed environments, providing a comprehensive view of the API’s execution.

With the Anypoint Monitoring Telemetry Exporter, you can send Flex Gateway trace data to third-party observability and analytics platforms, including Azure Monitor, Splunk HEC, or any OpenTelemetry-compliant tool. This capability allows you to leverage end-to-end observability, correlating API traces with data from other systems to build a comprehensive view of your environment.

Before You Begin

To use the OpenTelemetry with Flex Gateway, you must be familiar with:

Distributed tracing concepts, including traces and spans.

If you’re new to distributed tracing and OpenTelemetry, we recommend starting with What is OpenTelemetry and Understanding distributed tracing.

Export Traces from the Anypoint Monitoring Telemetry Exporter

To export traces to third party providers using the Anypoint Monitoring Telemetry Exporter, see How to export telemetry data to third-party monitoring systems.

To export traces using the Anypoint Monitoring Telemetry Exporter, first Export Traces to Anypoint Monitoring.

Export Traces to Anypoint Monitoring

Edit and then apply this configuration resource to enable distributed tracing. This configuration resource applies to all API instances. This example sets sampling to 0% for all traces, and sampling is configured on a per-API basis using policies.

apiVersion: gateway.mulesoft.com/v1beta1
kind: Configuration
metadata:
  name: tracing
  namespace: test-ns
spec:
  tracing:
    provider:
      type: anypoint
      properties:
        timeout: 10s #
    sampling: # Set all sampling to 0% - this example shows enabling samping for the gateway, but only generating trace data for sampling on a per-API basis using the Tracing policy.
      client: 0
      random: 0
      overall: 0
      #
      # Add labels such as these example labels, if you want to configure labels globally. You can also configure labels on a per-API basis by using the Tracing policy.
      labels:
      - name: c-env-tag
        type: environment
        keyName: FLEX_NAME
        defaultValue: DEFAULT_VAL
      - name: c-header-tag
        type: requestHeader
        keyName: :method
        defaultValue: DEFAULT_METHOD
      - name: c-literal-tag
        type: literal
        defaultValue: the-tag

Export Traces to a gRPC Open Telemetry Endpoint

To configure Open Telemetry tracing with a gRPC endpoint in Flex, use the following YAML configuration as an example:

apiVersion: gateway.mulesoft.com/v1beta1
kind: Configuration
metadata:
  name: tracing
  namespace: test-ns
spec:
  tracing:
    provider:
      type: opentelemetry
      properties:
        # otel collector service in this example it is installed in default namespace
        address: h2://splunk-otel-collector-agent.default.svc.cluster.local:4317
        serviceType: grpc
        serviceName: my-flex-name
        timeout: 10s
        resourceAttributes:
          any-key: any-value
  sampling:
    client: 100
    random: 100
    overall: 100
  labels:
    - name: c-env-tag
      type: environment
      keyName: FLEX_NAME
      defaultValue: DEFAULT_VAL
    - name: c-header-tag
      type: requestHeader
      keyName: :method
      defaultValue: DEFAULT_METHOD
    - name: c-literal-tag
      type: literal
      defaultValue: the-tag

Export Traces to a HTTP Open Telemetry Endpoint

To configure Open Telemetry tracing with a HTTP endpoint, use the following YAML configuration as an example:

apiVersion: gateway.mulesoft.com/v1beta1
kind: Configuration
metadata:
  name: tracing
  namespace: test-ns
spec:
  tracing:
    provider:
      type: opentelemetry
      properties:
        address: http://opentelemetry:4318
        tracesUri: http://opentelemetry:4318/v1/traces
        serviceType: http
        serviceName: the-name
        timeout:     10s
        # headers to be added in the http request to otel
        httpHeaders:
          auth: samething
        resourceAttributes:
          nay-key: nay-value
  sampling:
    client: 100
    random: 100
    overall: 100
  labels:
    - name: c-env-tag
      type: environment
      keyName: FLEX_NAME
      defaultValue: DEFAULT_VAL
    - name: c-header-tag
      type: requestHeader
      keyName: :method
      defaultValue: DEFAULT_METHOD
    - name: c-literal-tag
      type: literal
      defaultValue: the-tag

Configure Distributed Tracing For Specific APIs

You can override the globally configured distributed tracing settings by applying the Tracing Policy to specific APIs. You must first enable and configure distributed tracing for the gateway. You can then apply this policy to the APIs you want to instrument:

Tracing Policy