Configuring Flex Gateway Distributed Tracing in Connected Mode

OpenTelemetry provides a unified standard for instrumenting, generating, collecting, and exporting telemetry data across distributed systems. Flex Gateway supports OpenTelemetry, enabling you to configure the generation of distributed tracing data for all API traffic managed by the gateway. This integration delivers deep visibility into API request flows, making it easier to analyze system behavior, troubleshoot issues, and understand interactions between Flex Gateway APIs and external services.

Tracing enables you to track Flex Gateway API interactions. When multiple systems and services are involved, distributed tracing tracks API requests as they flow through distributed environments, providing a comprehensive view of the API’s execution.

With the Anypoint Monitoring Telemetry Exporter, you can send Flex Gateway trace data to third-party observability and analytics platforms, including Azure Monitor, Splunk HEC, or any OpenTelemetry-compliant tool. This capability allows you to leverage end-to-end observability, correlating API traces with data from other systems to build a comprehensive view of your environment.

Before You Begin

To use the OpenTelemetry with Flex Gateway, you must be familiar with:

Distributed tracing concepts, including traces and spans.

If you’re new to distributed tracing and OpenTelemetry, we recommend starting with What is OpenTelemetry and Understanding distributed tracing.

Export traces from the Anypoint Monitoring Telemetry Exporter

To export traces to third party providers using the Anypoint Monitoring Telemetry Exporter, see How to export telemetry data to third-party monitoring systems.

To export traces using the Anypoint Monitoring Telemetry Exporter, first configure the Telemetry Exporter in Anypoint Monitoring.

Export Traces to Anypoint Monitoring

Edit and then apply this configuration resource to enable distributed tracing. This configuration resource applies to all API instances. This example sets sampling to 0% for all traces, and sampling is configured on a per-API basis using policies.

apiVersion: gateway.mulesoft.com/v1beta1
kind: Configuration
metadata:
  name: tracing
  namespace: test-ns
spec:
  tracing:
    provider:
      type: anypoint
      properties:
        timeout: 10s
    sampling:
      client: 0
      random: 0
      overall: 0
    labels:
    - name: c-env-tag
      type: environment
      keyName: FLEX_NAME
      defaultValue: DEFAULT_VAL
    - name: c-header-tag
      type: requestHeader
      keyName: :method
      defaultValue: DEFAULT_METHOD
    - name: c-literal-tag
      type: literal
      defaultValue: the-tag

Export Traces to a gRPC OpenTelemetry Endpoint

To configure OpenTelemetry tracing with a gRPC endpoint in Flex, use the following YAML configuration as an example:

apiVersion: gateway.mulesoft.com/v1beta1
kind: Configuration
metadata:
  name: tracing
  namespace: test-ns
spec:
  tracing:
    provider:
      type: opentelemetry
      properties:
        address: h2://splunk-otel-collector-agent.default.svc.cluster.local:4317
        serviceType: grpc
        serviceName: my-flex-name
        timeout: 10s
        resourceAttributes:
          any-key: any-value
    sampling:
      client: 100
      random: 100
      overall: 100
    labels:
    - name: c-env-tag
      type: environment
      keyName: FLEX_NAME
      defaultValue: DEFAULT_VAL
    - name: c-header-tag
      type: requestHeader
      keyName: :method
      defaultValue: DEFAULT_METHOD
    - name: c-literal-tag
      type: literal
      defaultValue: the-tag

Export Traces to a HTTP OpenTelemetry Endpoint

To configure OpenTelemetry tracing with a HTTP endpoint, use the following YAML configuration as an example:

apiVersion: gateway.mulesoft.com/v1beta1
kind: Configuration
metadata:
  name: tracing
  namespace: test-ns
spec:
  tracing:
    provider:
      type: opentelemetry
      properties:
        address: http://opentelemetry:4318
        tracesUri: http://opentelemetry:4318/v1/traces
        serviceType: http
        serviceName: the-name
        timeout: 10s
        httpHeaders:
          auth: samething
        resourceAttributes:
          nay-key: nay-value
    sampling:
      client: 100
      random: 100
      overall: 100
    labels:
    - name: c-env-tag
      type: environment
      keyName: FLEX_NAME
      defaultValue: DEFAULT_VAL
    - name: c-header-tag
      type: requestHeader
      keyName: :method
      defaultValue: DEFAULT_METHOD
    - name: c-literal-tag
      type: literal
      defaultValue: the-tag

Configure Distributed Tracing For Specific APIs

You can override the globally configured distributed tracing settings by applying the Tracing Policy to specific APIs. You must first enable and configure distributed tracing for the gateway. You can then apply this policy to the APIs you want to instrument:

Tracing Policy