Configuring Flex Gateway Distributed Tracing in Local Mode

OpenTelemetry provides a unified standard for instrumenting, generating, collecting, and exporting telemetry data across distributed systems. Flex Gateway supports OpenTelemetry, enabling you to configure the generation of distributed tracing data for all API traffic managed by the gateway. This integration delivers deep visibility into API request flows, making it easier to analyze system behavior, troubleshoot issues, and understand interactions between Flex Gateway APIs and external services.

Tracing enables you to track Flex Gateway API interactions. When multiple systems and services are involved, distributed tracing tracks API requests as they flow through distributed environments, providing a comprehensive view of the API’s execution.

With the Anypoint Monitoring Telemetry Exporter, you can send Flex Gateway trace data to third-party observability and analytics platforms, including Azure Monitor, Splunk HEC, or any OpenTelemetry-compliant tool. This capability allows you to leverage end-to-end observability, correlating API traces with data from other systems to build a comprehensive view of your environment.

Before You Begin

To use the OpenTelemetry with Flex Gateway, you must be familiar with:

Distributed tracing concepts, including traces and spans.

If you’re new to distributed tracing and OpenTelemetry, we recommend starting with What is OpenTelemetry and Understanding distributed tracing.

Export Traces to a gRPC Open Telemetry Endpoint

To configure Open Telemetry tracing with a gRPC endpoint in Flex, use the following YAML configuration as an example:

apiVersion: gateway.mulesoft.com/v1beta1
kind: Configuration
metadata:
  name: tracing
  namespace: test-ns
spec:
  tracing:
    provider:
      type: opentelemetry
      properties:
        # otel collector service in this example it is installed in default namespace
        address: h2://splunk-otel-collector-agent.default.svc.cluster.local:4317
        serviceType: grpc
        serviceName: my-flex-name
        timeout: 10s
        resourceAttributes:
          any-key: any-value
  sampling:
    client: 100
    random: 100
    overall: 100
  labels:
    - name: c-env-tag
      type: environment
      keyName: FLEX_NAME
      defaultValue: DEFAULT_VAL
    - name: c-header-tag
      type: requestHeader
      keyName: :method
      defaultValue: DEFAULT_METHOD
    - name: c-literal-tag
      type: literal
      defaultValue: the-tag

Export Traces to a HTTP Open Telemetry Endpoint

To configure Open Telemetry tracing with a HTTP endpoint, use the following YAML configuration as an example:

apiVersion: gateway.mulesoft.com/v1beta1
kind: Configuration
metadata:
  name: tracing
  namespace: test-ns
spec:
  tracing:
    provider:
      type: opentelemetry
      properties:
        address: http://opentelemetry:4318
        tracesUri: http://opentelemetry:4318/v1/traces
        serviceType: http
        serviceName: the-name
        timeout:     10s
        # headers to be added in the http request to otel
        httpHeaders:
          auth: samething
        resourceAttributes:
          nay-key: nay-value
  sampling:
    client: 100
    random: 100
    overall: 100
  labels:
    - name: c-env-tag
      type: environment
      keyName: FLEX_NAME
      defaultValue: DEFAULT_VAL
    - name: c-header-tag
      type: requestHeader
      keyName: :method
      defaultValue: DEFAULT_METHOD
    - name: c-literal-tag
      type: literal
      defaultValue: the-tag

Configure Distributed Tracing For Specific APIs

You can override the globally configured distributed tracing settings by applying the Tracing Policy to specific APIs. You must first enable and configure distributed tracing for the gateway. You can then apply this policy to the APIs you want to instrument:

Tracing Policy