Configuring Omni Gateway Distributed Tracing in Local Mode

OpenTelemetry provides a unified standard for instrumenting, generating, collecting, and exporting telemetry data across distributed systems. Omni Gateway supports OpenTelemetry, enabling you to configure the generation of distributed tracing data for all API traffic managed by the gateway. This integration delivers deep visibility into API request flows, making it easier to analyze system behavior, troubleshoot issues, and understand interactions between Omni Gateway APIs and external services.

Tracing enables you to track Omni Gateway API interactions. When multiple systems and services are involved, distributed tracing tracks API requests as they flow through distributed environments, providing a comprehensive view of the API’s execution.

With the Anypoint Monitoring Telemetry Exporter, you can send Omni Gateway trace data to third-party observability and analytics platforms, including Azure Monitor, Splunk HEC, or any OpenTelemetry-compliant tool. This capability allows you to leverage end-to-end observability, correlating API traces with data from other systems to build a comprehensive view of your environment.

Configure distributed tracing in Local Mode using one of these methods:

Configuration YAML resource: Applies to all API instances.
Policy: Applies to one API instance and provides granular control.

Before You Begin

To use the OpenTelemetry with Omni Gateway, you must be familiar with:

Distributed tracing concepts, including traces and spans.

If you’re new to distributed tracing and OpenTelemetry, we recommend starting with What is OpenTelemetry and Understanding distributed tracing.

Configure with a YAML Resource

Edit and apply a YAML configuration resource to configure distributed tracing for all API instances in the gateway. Local Mode supports exporting traces to a gRPC or HTTP OpenTelemetry endpoint.

Export Traces to a gRPC OpenTelemetry Endpoint

To configure OpenTelemetry tracing with a gRPC endpoint in Omni, apply this YAML configuration:

apiVersion: gateway.mulesoft.com/v1beta1
kind: Configuration
metadata:
  name: tracing
  namespace: test-ns
spec:
  tracing:
    provider:
      type: opentelemetry
      properties:
        address: h2://splunk-otel-collector-agent.default.svc.cluster.local:4317
        serviceType: grpc
        serviceName: my-flex-name
        timeout: 10s
        resourceAttributes:
          any-key: any-value
    sampling:
      client: 100
      random: 100
      overall: 100
    labels:
    - name: c-env-tag
      type: environment
      keyName: FLEX_NAME
      defaultValue: DEFAULT_VAL
    - name: c-header-tag
      type: requestHeader
      keyName: :method
      defaultValue: DEFAULT_METHOD
    - name: c-literal-tag
      type: literal
      defaultValue: the-tag

Export Traces to an HTTP OpenTelemetry Endpoint

To configure OpenTelemetry tracing with an HTTP endpoint, apply this YAML configuration:

apiVersion: gateway.mulesoft.com/v1beta1
kind: Configuration
metadata:
  name: tracing
  namespace: test-ns
spec:
  tracing:
    provider:
      type: opentelemetry
      properties:
        address: http://opentelemetry:4318
        tracesUri: http://opentelemetry:4318/v1/traces
        serviceType: http
        serviceName: the-name
        timeout: 10s
        httpHeaders:
          auth: samething
        resourceAttributes:
          nay-key: nay-value
    sampling:
      client: 100
      random: 100
      overall: 100
    labels:
    - name: c-env-tag
      type: environment
      keyName: FLEX_NAME
      defaultValue: DEFAULT_VAL
    - name: c-header-tag
      type: requestHeader
      keyName: :method
      defaultValue: DEFAULT_METHOD
    - name: c-literal-tag
      type: literal
      defaultValue: the-tag

Configure Distributed Tracing for Specific APIs

To override the globally configured distributed tracing settings for specific APIs, apply the Tracing policy. First enable and configure distributed tracing for the gateway, then apply this policy to the APIs to instrument:

Tracing Policy