Configuring Omni Gateway Distributed Tracing in Connected Mode

OpenTelemetry provides a unified standard for instrumenting, generating, collecting, and exporting telemetry data across distributed systems. Omni Gateway supports OpenTelemetry, enabling you to configure the generation of distributed tracing data for all API traffic managed by the gateway. This integration delivers deep visibility into API request flows, making it easier to analyze system behavior, troubleshoot issues, and understand interactions between Omni Gateway APIs and external services.

Tracing enables you to track Omni Gateway API interactions. When multiple systems and services are involved, distributed tracing tracks API requests as they flow through distributed environments, providing a comprehensive view of the API’s execution.

You can export Omni Gateway trace data to third-party observability and analytics platforms, such as New Relic, Azure Monitor, Splunk, or any OpenTelemetry-compliant tool. Omni Gateway can send traces directly to an OpenTelemetry endpoint, or, for gateways connected to Anypoint Platform, send traces to Anypoint Monitoring, where the Anypoint Monitoring Telemetry Exporter forwards them to third-party providers. Exporting traces enables end-to-end observability, so you can correlate API traces with data from other systems to build a comprehensive view of your environment.

Configure distributed tracing in Connected Mode using one of these methods:

Runtime Manager runtime configurations: Applies to all API instances in the gateway.
Configuration YAML resource: Applies to all API instances.
Policy: Applies to one API instance and provides granular control.

Configuration Precedence

When you configure a setting in more than one place, Omni Gateway uses the first value that is set, in this order of precedence:

Policy: A policy applied to the API instance.
Runtime Manager runtime configuration or Configuration resource: Used if no policy value is set.
Built-in default: Used if none of the other values is set. A policy value overrides the gateway-level value for the API instance it’s applied to. API instances without the policy use the gateway-level value.

Before You Begin

To use the OpenTelemetry with Omni Gateway, you must be familiar with:

Distributed tracing concepts, including traces and spans.

If you’re new to distributed tracing and OpenTelemetry, we recommend starting with What is OpenTelemetry and Understanding distributed tracing.

Configuring Omni Gateway Distributed Tracing in Runtime Manager

In Anypoint Platform, select Runtime Manager.
Select Omni Gateways.
Select your Omni Gateway.
Open Runtime Configurations.
Open Distributed Tracing.
Enable Distributed tracing.
Set the Sampling percentage to the percentage of transactions to trace.

To configure sampling only on a per-API basis with the Tracing policy, set Sampling percentage to 0.
Apply the changes.

The Sampling percentage corresponds to the overall sampling percentage. The client and random sampling percentages default to 100 when Sampling percentage is set to a non-zero value.

Configure with a YAML Resource

Edit and apply a YAML configuration resource to configure distributed tracing for all API instances in the gateway. Connected Mode supports exporting traces to Anypoint Monitoring, a gRPC OpenTelemetry endpoint, or an HTTP OpenTelemetry endpoint.

Export Traces to Anypoint Monitoring

Edit and apply this configuration resource to enable distributed tracing. This configuration resource applies to all API instances. This example sets global sampling to 0% and policies configure sampling per API.

apiVersion: gateway.mulesoft.com/v1beta1
kind: Configuration
metadata:
  name: tracing
  namespace: test-ns
spec:
  tracing:
    provider:
      type: anypoint
      properties:
        timeout: 10s
    sampling:
      client: 0
      random: 0
      overall: 0
    labels:
    - name: c-env-tag
      type: environment
      keyName: FLEX_NAME
      defaultValue: DEFAULT_VAL
    - name: c-header-tag
      type: requestHeader
      keyName: :method
      defaultValue: DEFAULT_METHOD
    - name: c-literal-tag
      type: literal
      defaultValue: the-tag

Export Traces to a gRPC OpenTelemetry Endpoint

To configure OpenTelemetry tracing with a gRPC endpoint in Omni, apply this YAML configuration:

apiVersion: gateway.mulesoft.com/v1beta1
kind: Configuration
metadata:
  name: tracing
  namespace: test-ns
spec:
  tracing:
    provider:
      type: opentelemetry
      properties:
        address: h2://splunk-otel-collector-agent.default.svc.cluster.local:4317
        serviceType: grpc
        serviceName: my-flex-name
        timeout: 10s
        resourceAttributes:
          any-key: any-value
    sampling:
      client: 100
      random: 100
      overall: 100
    labels:
    - name: c-env-tag
      type: environment
      keyName: FLEX_NAME
      defaultValue: DEFAULT_VAL
    - name: c-header-tag
      type: requestHeader
      keyName: :method
      defaultValue: DEFAULT_METHOD
    - name: c-literal-tag
      type: literal
      defaultValue: the-tag

Export Traces to an HTTP OpenTelemetry Endpoint

To configure OpenTelemetry tracing with an HTTP endpoint, apply this YAML configuration:

apiVersion: gateway.mulesoft.com/v1beta1
kind: Configuration
metadata:
  name: tracing
  namespace: test-ns
spec:
  tracing:
    provider:
      type: opentelemetry
      properties:
        address: http://opentelemetry:4318
        tracesUri: http://opentelemetry:4318/v1/traces
        serviceType: http
        serviceName: the-name
        timeout: 10s
        httpHeaders:
          auth: samething
        resourceAttributes:
          nay-key: nay-value
    sampling:
      client: 100
      random: 100
      overall: 100
    labels:
    - name: c-env-tag
      type: environment
      keyName: FLEX_NAME
      defaultValue: DEFAULT_VAL
    - name: c-header-tag
      type: requestHeader
      keyName: :method
      defaultValue: DEFAULT_METHOD
    - name: c-literal-tag
      type: literal
      defaultValue: the-tag

Configure Distributed Tracing for Specific APIs

To override the globally configured distributed tracing settings for specific APIs, apply the Tracing policy. First enable and configure distributed tracing for the gateway, then apply this policy to the APIs to instrument:

Tracing Policy

Export Traces from Anypoint Monitoring to Third Parties

To export traces to third-party providers using the Anypoint Monitoring Telemetry Exporter, see Exporting Telemetry Data to Third-Party Monitoring Systems.

To export traces using the Anypoint Monitoring Telemetry Exporter, first configure the Telemetry Exporter in Anypoint Monitoring.