Contact Us 1-800-596-4880
  1. Homepage
  2. Omni Gateway (1.13)
  3. Securing Omni Gateway Instances with Policies
  4. Inbound Policies Directory

  5. A2A Quality Evaluation

A2A Quality Evaluation Policy

Policy Name

A2A Quality Evaluation

Summary

Evaluates A2A agent task responses with an LLM to score quality and completeness

Category

A2A

First Omni Gateway version available

v1.13.0

Release Notes

A2A Quality Evaluation Policy

Returned Status Codes

No return codes exist for this policy.
This policy supports Agent2Agent Protocol (A2A) version v0.3.0. To learn more about A2A versions, see A2A Releases.

Summary

The A2A Quality Evaluation policy evaluates completed A2A agent task responses using an LLM to score completeness against the original user intent. The policy only adds its verdict as response headers and emits metrics for observability. The policy doesn’t block or modify the agent’s response.

The policy evaluates quality when the final agent response is available:

  • Synchronous responses - When message/send returns a 200 status with the final response, the quality LLM is invoked inline before the response is returned.

  • Asynchronous responses - When message/send returns a 202 status with a task ID, the policy stores the task metadata. Later, when tasks/get returns a completed state, the policy invokes the quality LLM and caches the verdict for subsequent polls.

This policy supports A2A message/send and tasks/get requests across all transport protocols.

Configuring Policy Parameters

Omni Gateway Local Mode

The A2A Quality Evaluation policy isn’t supported in Local Mode.

Managed Omni Gateway and Omni Gateway Connected Mode

When you apply the policy from the UI, the following parameters are displayed:

Element Description Required Default

Quality LLM URL

OpenAI-compatible chat completions endpoint for quality evaluation. The policy POSTs request to <qualityLlmUrl>/v1/chat/completions.

For example: https://api.openai.com

Yes

None

Quality LLM API Key

Bearer token for authenticating to the quality LLM endpoint. This value is encrypted at rest.

Yes

None

Quality LLM Model

Model name to use for quality evaluation. The name must be a valid model name supported by the configured endpoint.

For example: gpt-4o or gpt-4o-mini

No

gpt-4o

Request Timeout (ms)

Timeout in milliseconds for the quality LLM call. If the call exceeds this timeout, the verdict isn’t attached and the policy logs a warning.

No

10000

How This Policy Works

The A2A Quality Evaluation policy intercepts A2A requests and evaluates completed agent responses using a separate LLM. The policy works differently for synchronous and asynchronous agent responses.

Request Filtering

The policy only processes requests that meet all of the following criteria:

  • HTTP method is POST

  • Content-Type header matches a JSON MIME type

  • Request body parses as a valid JSON-RPC request

  • JSON-RPC method is either message/send or tasks/get

All other requests pass through without evaluation.

Synchronous Quality Flow (message/send with 200)

When an agent returns a synchronous 200 response to message/send:

  1. The policy captures the original user message and the agent’s final response.

  2. The policy invokes the quality LLM with a prompt containing both the user message and agent response.

  3. The quality LLM returns a verdict with three fields:

    • evaluation: "yes" or "no" (whether the agent fully addressed the user’s request)

    • value: "1", "2", or "3" (quality score: 1=poor, 2=adequate, 3=excellent)

    • explanation: Free-text explanation of the verdict

  4. The policy adds quality verdict headers to the response.

    For header details, see Quality Verdict Headers.

  5. The policy emits a mulesoft_agent_calls counter metric with the verdict tags (see Observability Metrics).

  6. The response is returned to the client with the original agent response body and added quality headers.

The quality LLM call happens inline, so it adds latency (up to the configured timeout) to the synchronous response path.

Asynchronous Quality Flow (message/send with 202, then tasks/get)

When an agent returns an asynchronous 202 response to message/send:

  1. The policy captures the original user message and the returned task ID.

  2. The policy stores the task metadata.

  3. The response is returned immediately to the client with no quality headers and no LLM call.

Later, when a client polls with tasks/get and the upstream returns state: "completed":

  1. The policy invokes the quality LLM, stores the verdict, adds quality verdict headers, and emits the observability metric.

  2. The response is returned with the quality headers.

Subsequent polls of the same completed task return the stored verdict without re-invoking the LLM.

Quality Verdict Headers

When a quality verdict is available, the policy adds these headers to the response:

Header Description

x-quality-evaluation

"yes" or "no" indicating whether the agent fully addressed the user’s request.

x-quality-score

Quality score: "1" (poor), "2" (adequate), or "3" (excellent).

x-quality-explanation

Free-text explanation of the verdict provided by the quality LLM.

Observability Metrics

Every time the quality LLM is invoked (synchronous 200 responses and the first tasks/get request that evaluates a completed task), the policy emits a single increment of the mulesoft_agent_calls counter metric.

The metric includes these tags:

Tag Values

mulesoft.a2a.task_resolution

"yes", "no" (verdict from the quality LLM), or "unknown" if the LLM call failed or the response couldn’t be parsed.

mulesoft.a2a.quality_score

"1", "2", or "3" (score from the quality LLM), or "unknown" if the LLM call failed.

mulesoft.a2a.method

"message/send" or "tasks/get" (the A2A method that triggered the evaluation).

mulesoft.a2a.latency_bucket

One of 20 bounded log-scale labels representing the quality LLM call latency in milliseconds: "le_10", "le_15", "le_25", "le_40", "le_60", "le_90", "le_130", "le_200", "le_300", "le_450", "le_700", "le_1000", "le_1500", "le_2200", "le_3300", "le_5000", "le_7500", "le_11000", "le_16000", or "gt_16000". Only emitted on successful LLM calls.

mulesoft.a2a.failure_reason

Only emitted when task_resolution="unknown". Possible values: "transport_error", "http_<code>", "parse_error", "no_choices", or "invalid_evaluation".
The metric counts distinct quality LLM calls, not distinct tasks. A task answered synchronously via 200 produces one metric increment. A task accepted as 202 and later polled to completed also produces one metric increment (on the first tasks/get poll that evaluates the completed task). Subsequent polls against an already-evaluated task don’t emit additional metrics.

A sustained non-zero rate of task_resolution="unknown" indicates that the quality LLM is unreachable or returning errors. Operators should alert on this condition.

Failure Modes

The policy is designed to fail open — quality evaluation failures do not block agent responses. Instead, failures are logged and reflected in observability metrics.

Failure Impact

Quality LLM unreachable, non-200 response, or parse error

No quality verdict headers are attached. The metric is emitted with task_resolution="unknown", quality_score="unknown", and an appropriate failure_reason tag. A warning is logged.

Verdict storage fails

The storage error is ignored. The policy may re-invoke the quality LLM on a subsequent tasks/get request. Both LLM calls are counted in the metric.

Request body is not valid JSON-RPC

The request passes through unchanged. No quality evaluation is performed.

Quality LLM call exceeds the configured timeout

Treated as a transport error. No quality verdict headers are attached. The metric is emitted with task_resolution="unknown" and failure_reason="transport_error".

Constraints and Limitations

  • Synchronous latency - For synchronous 200 responses, the quality LLM call happens inline and can add up to the configured timeout in latency.

  • Asynchronous poll latency - For asynchronous tasks, the first tasks/get poll that returns a completed state incurs the quality LLM call latency.

Configuration Examples

The following examples demonstrate common configurations for the A2A Quality Evaluation policy using different LLM models, timeouts, and endpoint configurations.

Basic Configuration

This example configures the policy to evaluate agent responses using the gpt-4o model with a 10-second timeout:

apiVersion: gateway.mulesoft.com/v1alpha1
kind: PolicyBinding
metadata:
  name: a2a-quality-evaluation
spec:
  targetRef:
    kind: ApiInstance
    name: my-a2a-agent
  policyRef:
    name: a-two-a-quality-evaluation-flex
  config:
    qualityLlmUrl: "https://api.openai.com"
    qualityLlmApiKey: "sk-proj-..."
    qualityLlmModel: "gpt-4o"
    timeout: 10000

Custom Model and Timeout

This example uses a faster model (gpt-4o-mini) with a shorter timeout for lower latency:

apiVersion: gateway.mulesoft.com/v1alpha1
kind: PolicyBinding
metadata:
  name: a2a-quality-evaluation-fast
spec:
  targetRef:
    kind: ApiInstance
    name: my-a2a-agent
  policyRef:
    name: a-two-a-quality-evaluation-flex
  config:
    qualityLlmUrl: "https://api.openai.com"
    qualityLlmApiKey: "sk-proj-..."
    qualityLlmModel: "gpt-4o-mini"
    timeout: 5000

Using a Custom OpenAI-Compatible Endpoint

This example uses a self-hosted or alternative OpenAI-compatible endpoint:

apiVersion: gateway.mulesoft.com/v1alpha1
kind: PolicyBinding
metadata:
  name: a2a-quality-evaluation-custom
spec:
  targetRef:
    kind: ApiInstance
    name: my-a2a-agent
  policyRef:
    name: a-two-a-quality-evaluation-flex
  config:
    qualityLlmUrl: "https://custom-llm.example.com"
    qualityLlmApiKey: "custom-api-key"
    qualityLlmModel: "custom-model-v1"
    timeout: 15000

Sample Response Headers

The following examples demonstrate the quality evaluation headers and metrics added by the policy for different response scenarios.

Synchronous Quality Flow

When an agent returns a synchronous 200 response, the policy evaluates the response inline and adds quality headers.

  1. The initial request is sent:

    {
  "jsonrpc": "2.0",
  "id": 1,
  "method": "message/send",
  "params": {
    "message": {
      "parts": [
        {
          "kind": "text",
          "text": "Cancel my order #12345"
        }
      ]
    }
  }
}

  2. The upstream agent returns 200 with the agent’s reply. The policy invokes the quality LLM, which returns:

    {
  "evaluation": "yes",
  "value": "3",
  "explanation": "Order cancellation confirmed. The agent successfully canceled order #12345."
}

  3. The policy adds these headers and emits these metrics:

    • Response headers:

      x-quality-evaluation: yes
x-quality-score: 3
x-quality-explanation: Order cancellation confirmed. The agent successfully canceled order #12345.

    • Metric emitted:

      mulesoft_agent_calls{
  task_resolution="yes",
  quality_score="3",
  method="message/send",
  latency_bucket="le_300"
} 1

Asynchronous Quality Flow

When an agent returns a 202 response, the policy stores the task metadata and defers evaluation until the task is polled.

  1. Initial request is sent:

    {
  "jsonrpc": "2.0",
  "id": 1,
  "method": "message/send",
  "params": {
    "message": {
      "parts": [
        {
          "kind": "text",
          "text": "Cancel my order #12345"
        }
      ]
    }
  }
}

  2. The upstream agent returns 202:

    {
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "id": "task-abc-123"
  }
}

    The policy stores the task metadata. No quality headers are added. No metric is emitted.

  3. The client polls the task:

    {
  "jsonrpc": "2.0",
  "id": 2,
  "method": "tasks/get",
  "params": {
    "id": "task-abc-123"
  }
}

  4. The upstream agent returns 200 with completed status:

    {
  "jsonrpc": "2.0",
  "id": 2,
  "result": {
    "status": "completed",
    "message": {
      "parts": [
        {
          "kind": "text",
          "text": "Your order #12345 has been canceled."
        }
      ]
    }
  }
}

  5. The policy invokes the quality LLM, stores the verdict, and adds these headers and emits these metrics:

    • Response headers:

      x-quality-evaluation: yes
x-quality-score: 3
x-quality-explanation: Order cancellation confirmed.

    • Metric emitted:

      mulesoft_agent_calls{
  task_resolution="yes",
  quality_score="3",
  method="tasks/get",
  latency_bucket="le_450"
} 1

Subsequent polls of the same task return the stored verdict and do not invoke the LLM or emit additional metrics.

See Also