Troubleshoot Policy Enforcement Issues

Even though you applied policies to your services, the Mule adapter might not seem to be enforcing these policies, responding with the following error messages:

Service port is not named
Policy not disassociated after deleting bindings
Binding service name does not match actual service name
Application pod does not have an Envoy sidecar
Internal error with 500 error code

Service Port Is Not Named

When you create a service, you must provide specifications, such as ports and selectors. Each of these has its own set of parameters that must be defined. When you assign a port to a service, you must also specify a name for that port, such as "http," to avoid an error. Istio requires you to name every port.

Causes

A name for the service port is not specified.

Diagnose

To diagnose this issue, verify the definition of the service that exposes the application:

$ kubectl -n namespace get services -o yaml

For example, the following service does not have a named port:

apiVersion: v1
items:
- apiVersion: v1
  kind: Service
  metadata:
    creationTimestamp: "2019-10-09T19:08:28Z"
    name: hello-kubernetes
    namespace: default
    resourceVersion: "7075"
    selfLink: /api/v1/namespaces/default/services/hello-kubernetes
    uid: 2d0719a5-eac8-11e9-93c9-06b247319cf6
  spec:
    clusterIP: 172.20.177.73
    externalTrafficPolicy: Cluster
    ports:
    - nodePort: 32231
      port: 80
      protocol: TCP
      targetPort: 8080
    selector:
      app: hello-kubernetes
    sessionAffinity: None
    type: LoadBalancer

Solution

Istio requires you to name every port. To resolve this error, re-create the service with a named port:

apiVersion: v1
kind: Service
metadata:
  name: hello-kubernetes
spec:
  type: LoadBalancer
  ports:
  - port: 80
    name: http
    targetPort: 8080
  selector:
    app: hello-kubernetes

Policy Not Disassociated after Deleting Bindings

After you delete the bindings, the policies applied to the service might not be successfully removed.

Causes

When a binding was deleted, an associated traffic rule was not deleted. Because of this issue, traffic is still being sent to the Mule adapter.

Diagnose

To diagnose this issue, verify that the service rules have been deleted:

$ kubectl get rules -l serviceName=service name

If you determine that the rules are still applied, manually delete those rules.

Solution

Delete the incorrect rules applied to the service:

$ kubectl delete rules -l serviceName=service_name

Binding Service Name Does Not Match Actual Service Name

When you create the binding, you must provide a service name. If the service name provided does not match the actual service name, policies are not enforced.

Causes

Binding was created pointing to an nonexistent service.

Diagnose

To diagnose this issue:

Review the service bindings using either of the following commands:
- $ asmctl api binding list
- $ asmctl adapter binding list
Compare the service with the bindings:

$ kubectl -n <namespace> get services

In the above example, service names must be, for example customer-service, and NOT customers.nto-payment.svc.cluster.nto-cluster

Solution

To resolve this issue:

Delete the existing bindings by running any of the commands:
- $ asmctl api binding delete --name=binding_name --namespace=binding_namespace
- $ asmctl adapter binding delete --name=binding_name --namespace=binding_namespace
Re-create the bindings to point to the actual services:

Binding service does not match actual service resolution

Application Pod Does Not Have an Envoy Sidecar

Policies are not enforced due to an Envoy error in the application pod.

Causes

Applications were not restarted in the namespace that was marked for injecting Envoy sidecar for every pod.

Diagnose

To diagnose this issue, verify that the sidecar has an envoy proxy associated to it:

$ asmctl management check sidecar

Solution

If the terminal does not display the status as Injected, you must restart the pods:

$ kubectl -n <namespace> delete pods <pod_name>

Internal Error with 500 Error Code

The applications managed by Anypoint Service Mesh generate a 500 error code with the following error description:

INTERNAL:grpcmule-nto-payment.handler.service-mesh.INTERNAL:grpcmule-nto-payment.handler.service-mesh

Causes

This error occurs if the license expires and, as a result, the adapter restarts.

Diagnose

To diagnose this issue:

Display the adapter pod:

$ kubectl get pods -n service-mesh
Retrieve the adapter logs:

$ kubectl -n service-mesh logs $(kubectl -n service-mesh get pod -l namespace=replace_with_namespace -oname) -c mule

If the license has expired, an error similar to the following is displayed:

Solution

To resolve this issue:

Replace the existing license with a valid one:

$ asmctl management config license --license=/path/to/license.lic
Redeploy the adapter:

$ kubectl -n service-mesh patch $(kubectl -n service-mesh get deploy -l namespace=replace_with_namespace -oname) --type=json -p="[{\"op\": \"replace\", \"path\": \"/spec/template/metadata/annotations/kubectl.kubernetes.io~1restartedAt\",\"value\":\"$(date -u +%FT%TZ)\"}]"